null cache

[weidezhang]

Hi,
I wonder if I can use alluxio to replace null cache as petastorm cache. do you see performance gain by using alluxio ?

[weidezhang]

also i wonder if the cache is shared across all trainer processes and across all epoches ? will the cache need to be reset after each training epoch ?

[selitvin]

I am not familiar with alluxio. Implementing a new cache version and plugging it in should be pretty easy. The cache is not automatically shared across processes. However, if you are using a shareabe storage (e.g. local disk for processes running on the same machine), then the cache will be shared (as long as the storage/cache-class implementation) properly handles race conditions.

[selitvin]

If you'd like to take a shot implementing an alluxio based cache, I would be happy to help with a review.

[weidezhang]

thanks for the info. it's helpful. we will do some investigation and contribute back to community if possible.