DB collection exports

[matr1x-hackmud]

A problem as old as time in Hackmud is that each user has their own MongoDB collection, containing data. Sometimes, this is vast amounts of data that piles up, as we are one to do as video-game datahoarders. This in aggregate can lead to a lot of strain on the DB server; both to store the data and perform lookups against it (particularly if unindexed).

It would be nice to have an API endpoint of some kind (likely similar in scope to the chat API) that would allow us to perform a read-only export of all of the data in our users' collections, for storage in an external location like another MongoDB server, or just as flat file tarballs or something. This would give datahoarders a really strong incentive to vacuum up their DBs to be more performant in-game, without jumping through the hoops of our current database export solutions.

As a workaround, players are able to set up - with extreme effort (and a large amount of cron_bots and user slots) - a database backup system that makes use of chat messages that are then picked up with the chat API for out-of-game data exports. This requires effort to ensure the data comes out clean and is secure, as well as provides additional strain on the server for the duration of the backup process (which may be ongoing if you consume a lot of data).

A standardised and easily-accessible means of data custody would be a lot more equitable for the playerbase and reduce DB load :D

[hmdunce]

Some personal experience:

I use base32768 to get more efficiency in chat exports, since the chat character limit applies to the UTF-16 JS string -- even though it is presumably stored as UTF-8 on the disk...

I only do these exports as one-offs so far. I haven't built a robust process yet! It was still a good amount of work to get these one-offs working.

For a long time I had loads of garbage, but I was afraid I might #db.r something important, so I only deleted obvious garbage I happened to find. I guess I always hoped to build a robust export process and then I never did. 😄