Abstract backend for pluggable OP

[lovato]

GAE does not accept any python package that needs to be compiled.
And Flask-Images meets that requirement.

But F-I uses File I/O, which is not allowed inside GAE.

I can help with this one, but need to know first if you are already working in something like this... Or what would be the best approach. Get tied to some GAE offering (like https://developers.google.com/appengine/docs/java/googlecloudstorageclient/) or make it all-in-memory.

Best
Marco

[iurisilvio]

I don't have much experience with GAE, but looks like googlecloudstorage is the way to go. The all-in-memory is possible with GAE? It will use a lot of memory and Google has some resource limits, right?

The GAE PIL works with the current code?

AFAIK, nobody is working on a custom backend and I'm +1 to that. Not only for GAE, also for S3 and other backends in a pluggable way.

[mikeboers]

I'd also lean towards the googlecloudstorage as that would be s shit-ton of memory usage (in the naive case).

I was also thinking of making a storage backend. I tend to use the very simple mapping API for this sort of thing, and then only rely upon __getitem__ and __setitem__. Then you can quickly do a file adapter (which we would use by default), memory (via a dict), Redis, memcached, etc., and wrap that API in other tools to add automatic expiry times, etc..

[lovato]

My idea of using RAM only is to not store anything. Resize & deliver mode only. I can then use a cache server (varnish?) approach if I dont want to re-do all the work. It would make it simpler to anyone who does not have a plain /tmp available.

In my particular case, in GAE, I can enable its own frontend cache, like suggested in this article: http://www.smallte.ch/blog-read_en_34003.html

Does it make sense?
Lovato

[iurisilvio]

Yes, makes sense even outside GAE.

A backend abstraction is the first thing to be done. 👍

[mikeboers]

Looking at how we already do the cacheing, my idea about just using the __get/setitem__ protocol is not terribly efficient as it will reduce the potential effectiveness of using sendfile, among other things.

We also need to abstract the remote image fetching as well.

So, a more complete list of things that would require abstracting are:

• getting cached remote images;
• setting cached remote images;
• setting cached resized images;
• serving the final image.

[lovato]

Yes, we dont need to fetch the same origin all the time. But what if a proxy can serve images? This article is old, but perhaps illustrate what a url.fetch can be. Of couse this is valid on GAE, cant talk about other approaches.

So the wrappers can have their own sets of customization, depending the infrastructure they will be deployed to.