Flexibility to specify channel logs platform (other than DynamoDB)

[noorahealthtech]

I noticed that the more recent versions of Rapidpro come with tight DynamoDB integration, which means that all customers need to use AWS if they want to see any sort of channel logs. For people hosted on GCP or any other platform this is hugely restrictive because the logs contain valuable information that we use to debug the system.

At Noora Health, we use Rapidpro extensively for our campaigns and CTAs via WhatsApp and are thinking of implementing a common interface for channel logs and lets the user pick the respective service (DynamoDB, Firestore, BigTable etc.) for their particular installation. We can even potentially let users provide this via a dashboard in the future, depending on where they want their logs.

[ericnewcomer]

Have a look at ScyllaDB. It is OSS and is API compliant with DynamoDB. Theoretically you could make this all work on GCP without implementing an entirely different backend. Haven't tried it myself, so curious how you get on if you decide to go this route.

[rowanseymour]

Just to add to that: we're in the process of moving more things into DynamoDB, and yes in theory that should all work with ScyllaDB that provides a DynamoDB adapter. But... we run RapidPro and TextIt in AWS so we won't be able to test that. We can't even promise that the platform can work against S3 compatible object stores because again, we don't have a deployment that does that.

[noorahealthtech]

Got it, we'll try to be ready with Firestore / ScyllaDB adapters as these new things come along 😁

[noorahealthtech]

Yeah, at the moment we're noodling with the idea of offering Firestore or ScyllaDB as a potential replacement for channel logs (with DynamoDB as default, but with options to use other log stores) - will keep you folks posted on what we learn!

ScyllaDB is interesting because it seems like the actual code requires minimal changes, but I'm not sure if we want to take it upon ourselves to host it. 🤔

[rowanseymour]

Don't forget also that all metrics are Cloudwatch these days (we were previously using Librato but they don't really exist anymore). Supporting non-AWS hosting environments has long been something we've left to the people who really want to go that way to figure out (e.g. using Minio instead of S3), and we've been happy to make small changes to configuration code to let them do that. But I don't think we would accept significant code changes into the official repos to create abstraction layers because that ends up being more maintenance work for us.