Compress shapefiles

[kdeloach]

Investigate enabling compression on NHD shapefiles to reduce filesize. This could potentially improve the responsiveness of RWD after a new worker has been provisioned.

Ref: #46 (comment)

[mmcfarland]

I don't believe there is an OGR driver for compressing/reading compressed shapefiles, but we should do this for the rasters via gdal drivers.

[emiliom]

In case it helps ... OGR can read zipped (or tarballed) shape files using the "virtual file system" (vfs/vsi) drivers or approach. I couldn't find a good, focused documentation on this at the official gdal/ogr web site, other than this terse and unhelpful ogr "memory" driver page. But you can google for "vsi" or "vsizip" and find references easily. I found this old but good blog post, plus a somewhat old GDAL/OGR wiki page.

There are probably important performance issues that you should read up on before pursuing that route, though. I've used vsi stuff sporadically, and not enough to know much about performance issues.

[mmcfarland]

That's a good tip, @emiliom. I've use vsicurl with gdal, we'll check out vsizip to see if the compression on the shapefiles buys us much and at what cost.

[emiliom]

Glad it was helpful, @mmcfarland. I'd be interested in hearing (say, via this github issue) what you learn about cost tradeoffs, once you've tested it -- regardless of whether that's in the next 24 hours or 24 weeks ...