Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Function to extract fink classification from the alert doesn't need pyspark #116

Open
Theodlz opened this issue Apr 29, 2022 · 1 comment

Comments

@Theodlz
Copy link
Contributor

Theodlz commented Apr 29, 2022

In
/fink_filters/classification.py pyspark is imported on top of the file. However, for most functions defined there (2 out of 3), pyspark is not needed nor used. Only extract_fink_classification uses it, but extract_fink_classification_from_pdf(pdf) doesn't.

What's the expected behavior :

We would expect pyspark not to be needed when a user imports extract_fink_classification_from_pdf(pdf) to extract the classification from a simple panda dataframe, or even when using extract_fink_classification_() that just needs the right fields from the alert (this function is used by the one mentionned before, and the one mentionned next). But only when using extract_fink_classification that is meant to be used with Apache Spark.

What can be done :

The easiest solution would be to have the basic classification function extract_fink_classification_ (used by both panda and spark functions later on) in its own module. And then 2 seperate modules, one that imports pandas as needed, and one that imports spark, where both import the module that contains extract_fink_classification_.

Otherwise, maybe just finding a way to import the right libraries used by each function, rather than all of them when importing the whole module (what i mean is that, if i import only ``extract_fink_classification_from_pdf, it could maybe only import the libraries used by it, and the same would happen when importing only extract_fink_classification` where this time it would import pyspark).

@JulienPeloton
Copy link
Member

Priority when I return from holidays ;-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants