Develop Archiver for Azure Blob Store

[uromahn]

Author: Ulrich Romahn (@uromahn)

Summary of the feature being proposed

Temporal supports continuous archiving of records of the history and visibility store to limit the size of the primary persistence store.
Currently the default implementation supports three archival stores:

• Filesystem
• AWS S3, and
• Google Cloud Blob Storage

However, there is currently no archiver for Azure Blob Store.

For organizations wanting to use Azure and Blob Store for archiving, the only viable option today is to deploy an S3 Proxy and configure s3 as the archival store in Temporal. While this is working, it introduces another necessary component and an additional potential point of failure making the overall system slightly less reliable.

What value does this feature bring to Temporal?

Organizations who currently use MS Azure to host their Temporal workloads can use Azure Blob Store natively for archiving. This will remove the requirement to use a third-party component proxying the S3 API to the Azure Blob Store API.

Are you willing to implement this feature yourself?

Yes, although progress might be slow due to my day-to-day work commitments.

[bergundy]

Hi @uromahn,

We don't have plans to add this to the server codebase, we don't have the resources to maintain and test this regularly.
We can add an injection point where you can add your own archival provider though. That would require that you fork the server, implement your own archival provider, and inject it where the server is constructed in the CLI entry point.

[uromahn]

@bergundy that is a bit disappointing, although I kind of understand.

The challenge will be that I need this primarily for our own deployment (we are a heavy user of MS Azure) and always building our own custom release of Temporal will not fly here at my employer!
So, I need a way to make this more pluggable where I can build an archiver independently from the main server and then use that for archiving.
Any guidance you could provide to potentially do that (although this would increase the scope of this "project" significantly)?

By the way, you may want to talk to your product team. I see that the archiving feature is still marked as "experimental". I would consider this a critical feature in a large enterprise setting and having native support for Azure Blob Store may be as important for those enterprise customers. From an engineering perspective, I hear and understand you but from an overall product perspective, your priorities just don't fully line up with real customers. (Not everybody wants to deploy on AWS or GCP!)