Activity

1. pjbull commented on Feb 25, 2025

   

   pjbull

   on Feb 25, 2025

   Member

   More actions

   Thanks @Haydnspass, did you see these docs on pickling/unpickling?
   https://cloudpathlib.drivendata.org/stable/authentication/#pickling-cloudpath-objects

   Another option is to set the env var for the local cache dir:
   https://cloudpathlib.drivendata.org/stable/caching/#keeping-the-cache-around

   Let me know if either of these options address your scenario.
2. Haydnspass commented on Feb 27, 2025

   

   Haydnspass

   on Feb 27, 2025

   Last edited by Haydnspass

   Author

   More actions

   @pjbull thanks for the docs, indeed I missed that.
   Trying setting the client as default did not work for me, however I don't know if thats some extra magic going on in multiprocessing (via joblib) or if I did something wrong.

   For the moment I use this workaround by subclassing S3Path:

   class S3Path(cloudpathlib.S3Path):
       """Temporary child for fixing cache dir"""
   
       def __getstate__(self) -> dict[str, Any]:
           state = super().__getstate__()
           state["_client_cache_dir"] = self.client._local_cache_dir if self.client is not None else None
           return state
   
       def __setstate__(self, state: dict[str, Any]) -> None:
           self.__dict__ = state
           if (p := state["_client_cache_dir"]) is not None:
               self.client._local_cache_dir = p

   I'll follow up
3. pjbull commented on Feb 27, 2025

   

   pjbull

   on Feb 27, 2025

   Member

   More actions

   @Haydnspass Do you have a minimally reproducible example we could work with to see what's happening with the multiprocessing? There might be potential workarounds or enhancements we want to do.
4. Haydnspass commented on Mar 28, 2025

   

   Haydnspass

   on Mar 28, 2025

   Author

   More actions

   @pjbull sorry for getting back to you late.

   Hmm, indeed i thought my parallel processing using joblib is the problem, but that appears not to be the case in my minimal example here:

   from pathlib import Path
   
   import cloudpathlib as cl
   from joblib import Parallel, delayed
   
   
   def local_cache_dir(path_s3: cl.S3Path) -> Path:
       return Path(path_s3.fspath)
   
   
   def test_cache_dir(s3_client: cl.S3Client):
       s3_client.set_as_default_client()
       path_s3 = cl.S3Path("[s3 path]")
   
       assert str(s3_client._local_cache_dir) in str(local_cache_dir(path_s3).parent)
   
       out = Parallel(n_jobs=1)(delayed(local_cache_dir)(path_s3) for _ in range(1))[0]
       assert str(s3_client._local_cache_dir) in str(out)

   So I figure it could be a combination of joblib and pytorch multiprocessing but I will have to further investigate.
5. pjbull

   added

   cachingIssues that would be fixed or improved by a separate, customizable caching layer.Issues that would be fixed or improved by a separate, customizable caching layer.

   on Aug 4, 2025
6. sumamazaeem

   added 5 commits that reference this issue on Oct 17, 2026

   fix: resolve issue drivendataorg#504 - preserve explicit persistent l…

   ...

   68474f3

   docs: add HISTORY.md UNRELEASED entry for issue drivendataorg#504 fix

   c361e9c

   docs: restore full HISTORY.md (keep drivendataorg#504 UNRELEASED entry)

   63fcda8

   docs: restore complete HISTORY.md with only the issue drivendataorg#504…

   ...

   1cd816b

   fix: resolve issue drivendataorg#504 - preserve explicit persistent l…

   ...

   deb85c3