2.3.0rc1

What's Changed

• Fix BufferedReader error caused by opening an empty file in S3 in "rb" mode by @belltailjp in #270
• Sphinx 5 warns as language is None by @kuenishi in #274
• Control buffering for underlying file object for Zip(FS) by @kuenishi in #273
• Sparse file local cache by @kuenishi in #275
• Thread safety by @kuenishi in #277
• Multiprocessing-safe sparse file cache by @kuenishi in #278
• Limit the sparse cache size by @kuenishi in #279

Sparse file cache example with multiprocessing would be like:

        import os
        from pfio.v2 import from_url

        with from_url("s3://bucket/your.zip",
                      local_cache=True, reset_on_fork=True) as fs:

          pid = os.fork()
          if pid:
            os.wait()
          else:
            with fs.open("file-in-zip.jpg", 'rb') as fp:
              data = rp.read()

$ sha1sum dist/pfio-2.3.0rc1*
782f5ef08f08e4de64dac5175b300bce3dd46a38  dist/pfio-2.3.0rc1-py3-none-any.whl
4d21ede0d89f059830d681b74b8c464febd567e4  dist/pfio-2.3.0rc1.tar.gz

Full Changelog: 2.2.0...2.3.0rc1

2.2.0 Release

Major updates:

• Python 3.6 dropped
• Default cache directory moved from ~/.pfio to ~/.cache/pfio in favor of XDG standard.
• New option reset_on_fork introduced to each FSes.

All:

• #260 by kuenishi was closed 14 minutes ago
• #266 by kuenishi was merged 18 minutes ago
• #267 by knshnb was merged 14 days ago
• #261 by kuenishi was closed 22 days ago
• #264 by kuenishi was merged 22 days ago
• #226 by kuenishi was closed 22 days ago
• #265 by kuenishi was merged 22 days ago
• #262 by kuenishi was closed 22 days ago
• #263 by kuenishi was merged 22 days ago
• #259 by kuenishi was merged on Jan 12

453ad4a231b87f64499ec70abcc02b80ac96b861d8e7ccea831760e6398fb26c  pfio-2.2.0-py3-none-any.whl
b7aff43978b9fbbd685c586d79ea7c1679d6dc8bef9052132c353a0945c21b6f  pfio-2.2.0.tar.gz

2.1.2 Release

• Prevent race condition for from_url(create=True) in case of parallel processing cat:bug
  #254 by belltailjp was merged 20 minutes ago

2.1.1 Release

• Fix import degradation cat:bug
  #251 by belltailjp was merged 1 hour ago

2.1.0 Release

Includes incompatible changes.

• Release/2.1 - remove v1 API
  #250 by kuenishi was merged 1 minute ago
• Provide 'create' option in each filesystem (v2 API)
  #245 by belltailjp was merged 2 hours ago
• Update Sphinx RTD theme cat:document
  #249 by kuenishi was merged 7 days ago
• Buffered S3 reader cat:enhancement
  #247 by belltailjp was merged 7 days ago
• File-like object returned from open_url is extremely slow with S3 cat:performance
  #241 by belltailjp was closed 7 days ago
• Fix S3.rename didn't correctly remove the old file cat:bug
  #248 by belltailjp was merged 8 days ago
• Fix pfio.v2.Hdfs.isdir returns True to non-existent path cat:bug
  #246 by belltailjp was merged 8 days ago

2.0.1 Release

• Opening a giant (exceeding 4GB~?) zip in S3 using pfio.v2.from_url raises "BadZipFile: Bad magic number for central directory" cat:bug
  #239 by belltailjp was closed 3 hours ago
• Add readline method to _ObjectReader so S3 file object can be fed to pickle.load
  #243 by belltailjp was merged 3 hours ago
• [Ready] Fix _ObjectReader.seek(0, os.SEEK_END) case cat:bug
  #242 by belltailjp was merged 3 hours ago
• Path.glob has different behavior from standard pathlib.Path cat:bug
  #235 by disktnk was closed 2 days ago
• Fix pathlib return value cat:bug
  #240 by kuenishi was merged 2 days ago
• xml.etree.ElementTree.Element.getchildren is abolished in python 3.9 cat:bug
  #238 by belltailjp was merged 2 days ago
• Fix forkederror cat:bug
  #236 by kuenishi was merged 2 days ago
• Extend CI timeout
  #237 by kuenishi was merged 2 days ago

2.0.0 Release

https://pypi.org/project/pfio/2.0.0/

Incompatible Deprecation

• V1 API #195 (will be removed in 2.1.0)

Addition and Fixes

• Support Python 3.10 #227
• Support zipfile.is_zipfile itself or replacement feature in PFIO cat:feature #208
• Add zipfile.is_zipfile support #230
• Loader bench #229
• Bump and fix pyenv as 2.1.0, update Python versions #228
• Read-only mmaped file cache #224
• Set 0 to pyarrow.fs.HadoopFileSystem port in v2 API to make it consistent to the v1 API behavior cat:bug #225 by @belltailjp
• Support "zip" over S3 #220
• Treat prefix ending with "/" as directory #219 by @HiroakiMikami
• Add force_type=None option to pfio.v2.from_url() cat:enhancement #211
• Use relative path as an argument of boto3 cat:code-fix #212 by @HiroakiMikami
• Add kwargs to open_url #210 by @HiroakiMikami
• Make kwargs available in open_url too #217
• Fix bug in S3.list cat:bug #213 by @HiroakiMikami
• pathlib-compat API cat:feature #192
• Fix typo #207 by @superbrothers
• Add .parent, .name, .suffix #201
• Add pathlib-like API supporting glob and path concatination cat:feature #199
• Fix listing root directory #198

2.0.0a3 Release

• Add zipfile.is_zipfile support (#230)
• Add loader-bench #229
• Bump and fix pyenv as 2.1.0, update Python versions (#228)
• Read-only mmaped file cache (Experimental, #224)
• Set 0 to HDFS port (#225), Thanks to @belltailjp
• Support ZIP on S3 (#223)

2.0.0a2 Release

• list() now does not prints path prefix #213
• pfio.v2.open_url() supports keyword arguments as well as from_url() #217
• Omit first /on using S3 #212
• Add force_type keyword argument to open_url() #218
• Imitate directory on path with trailing / #219

This pre-release has been realized by huge contribution from and discussion with @HiroakiMikami . Thanks!

2.0.0a1 Release

Alpha release

• New feature: pfio.v2.pathlib
• Depreaction: toplevel v1 API e.g. pfio.open()

Release: https://pypi.org/project/pfio/2.0.0a1/