Global cache_dir variable for exact, fuzzy, and semantic deduplication
#384
Conversation
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
sarahyurick
changed the title
cache_dir variable for exact, fuzzy, and semantic deduplication
| You also need to set a global variable representing the cache directory where the outputs are written: | ||
|
|
||
| .. code-block:: python | ||
|
|
||
| from nemo_curator.cache import initialize_cache_directory | ||
|
|
||
| initialize_cache_directory("cache_dir") |
There was a problem hiding this comment.
Could also make more sense to call this something else, like deduplication_outputs.
| id_column=id_column, | ||
| id_column_type=id_column_type, | ||
| which_to_keep=config.which_to_keep, | ||
| output_dir=os.path.join(cache_dir, config.clustering_save_loc), |
There was a problem hiding this comment.
Might want to re-add output_dir to SemanticClusterLevelDedup and/or add it as a parameter for SemDedup
| self.sorted_clusters_dir = os.path.join( | ||
| get_cache_directory(), "clustering", "sorted" | ||
| ) | ||
| self.output_dir = os.path.join(get_cache_directory(), "clustering") |
There was a problem hiding this comment.
| self.output_dir = os.path.join(get_cache_directory(), "clustering") | |
| self.output_dir = os.path.join(get_cache_directory(), "duplicates") |
?
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
| from nemo_curator.utils.file_utils import expand_outdir_and_mkdir | ||
|
|
||
| # Global variable to store the cache directory | ||
| _global_cache_dir = None |
There was a problem hiding this comment.
Instead of a global variable, we should use the singleton design pattern here, where there is a Cache class that creates the static singleton instance (if it doesn't exist), or returns an existing one (if it was already created).
Subsequently, the init and get methods can be nicely wrapped inside that class.
There was a problem hiding this comment.
@sarahyurick please let me know what you think and whether my comment makes sense to you
There was a problem hiding this comment.
Thank you! Yes, that makes sense to me. Sorry I haven't gotten back to this earlier, but I expect to have more changes ready next week.
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
Signed-off-by: Sarah Yurick <[email protected]>
sarahyurick
added
gpuci
TODO: