fix download counter

[mreid-tt]

Description

This PR implements the download counter function and includes:

• rewrite catalog download links using md5 hashes for id
• allow downloads of noarch builds to pass arch checks
• rewrite nas tests for new url structure

fixes #22

Notes

While this implementation is functional, it lacks efficiency as it generates a database entry for every download request from a client. As mentioned in #112 (comment), the platform currently manages approximately 3.5 million requests per day with a 97% hit ratio on Fastly as CDN. Implementing this would lead to a significant surge in server traffic, potentially causing platform outages for users due to approximately a 33x increase in load.

Given the above, help is requested to find a more efficient implementation of this function.

[hgy59]

@mreid-tt If you find a solution for the download counters, it would be nice to display the counters somewhere in the admin UI.

[mreid-tt]

hey @Diaoul @publicarray, would either of you be able to work with me to get the logs from the Fastly CDN to begin developing a parsing process for them?

[publicarray]

@mreid-tt Fastly can send logs to another platform but doesn't store logs itself. At least not with our current plan
https://docs.fastly.com/en/guides/integrations#_streaming-logs

FYI we average about 38 requests per second.

I also remember that I did for a brief period had logs go to sumologic so that I could debug an issue and after the issue was fixed the consensus at the time was not to send logs to it anymore

@Diaoul

[publicarray]

The last 24h:

[mreid-tt]

@publicarray Thanks for your response! Regarding the logging options, we don’t need anything too complex. A daily log dump via FTP or SFTP to our server would suffice. From there, the log could be parsed, ingested into our database, and then deleted.

Based on our traffic of 38 requests/second (approximately 3.3 million requests/day), the uncompressed log file might be around 3.1 GB. To manage this, we could allocate a dedicated 5 GB partition for uploads, ensuring it only holds one day of logs. This way, if a cron job fails, uploads will be blocked rather than risking the server running out of space.

Would love to hear your thoughts on this approach.

[publicarray]

Good Idea, Fastly can push logs via SFTP and auto rotate logs: https://docs.fastly.com/en/guides/log-streaming-sftp. But aren't we usually low on storage on that server?

[mreid-tt]

Good Idea, Fastly can push logs via SFTP and auto rotate logs: https://docs.fastly.com/en/guides/log-streaming-sftp. But aren't we usually low on storage on that server?

You are correct, the last daily storage report looks like this:

Filesystem      Size  Used Avail Use% Mounted on
 /dev/sda1        79G   62G   14G  82% /
53G     /etc/synocommunity/data

We may not need as much space as a whole day since the documentation for setting this up has this line:

By default, logs are placed in your root directory every hour using the file naming format YYYY-mm-ddThh:mm:ss-.

Thus if we process the logs hourly as they are received we should have much less of a storage risk.

[Diaoul]

It some point I would love to move hosting because currently I don't think I'm paying a fair price for this VPS at 15 EUR per months for 2 vcores, 4GB of RAM and 80 GB storage.

If you have some hosting recommendation I'm all ears!

[mreid-tt]

It some point I would love to move hosting because currently I don't think I'm paying a fair price for this VPS at 15 EUR per months for 2 vcores, 4GB of RAM and 80 GB storage.

If you have some hosting recommendation I'm all ears!

Interesting. Are we also hosting content with Fastly? If so, this would involve not only moving the VPS service but also the CDN. Alternatively, will Fastly continue to provide CDN services if we choose to migrate our VPS to one of their designated cloud partners?

EDIT: After a quick search, I found the existing pricing to be surprisingly reasonable in comparison. Here are a few options worth exploring:

1. Hetzner Cloud
2. Contabo VPS
3. IONOS VPS
4. A2 Hosting (Unmanaged VPS)
5. Hostinger VPS Hosting

Additionally, this video highlights some other factors to consider.

[Diaoul]

Hetzner wanted to verify my ID and all plus it comes with restriction on egress traffic. Given we upload a lot of data, I don't want to be limited by that so I gave up.

I've taken a VPS Comfort at OVH. It's twice everything we have today but at a cheaper price. Now the challenge will be to move everything there 😅

[mreid-tt]

I've taken a VPS Comfort at OVH. It's twice everything we have today but at a cheaper price. Now the challenge will be to move everything there 😅

From the website, I noticed that the Comfort plan is currently offered as part of a Black Friday promotion. Are you certain it will renew at this discounted price after the initial period? Additionally, I saw this note on the webpage:

For any VPS solutions hosted in Sydney and Singapore: 3 TB/month for the Comfort range. Bandwidth is reduced to 10 Mbps once the monthly quota has been exceeded.

Regarding migration, I assume the process would involve cloning our current configuration, conducting some tests, and piloting with a few users using a new URL for the Package Sources configuration. Once everything is confirmed to be working as expected, we could proceed to update the DNS records to point to the new server. During this process, it would be a good opportunity to explore logging options and potentially move this PR forward.

[mikespub]

If you're mostly interested in statistics, you might simply use their historical stats API instead of streaming logs from Fastly.

[publicarray]

@mreid-tt @Diaoul how hard is it to program the api server to manage files via sftp/s3 protocols rather than the local file system, so we can use the server as a database only and move the storage to a different platform (would increase cost however)

[mreid-tt]

Hey @publicarray, off the top of my head, changing the download path via the API should be fairly straightforward. Uploads, on the other hand, would be more involved since the API currently ingests the file to generate the signature and checksum. That process would need to be adjusted to perform those steps and then push the signed file to an alternate storage platform. We'd also need to account for changes to file management — like handling deletions and other lifecycle operations.

What are you thinking in terms of direction here?

[publicarray]

Thanks yes the upload is the most difficult process but I think it's not overly so as in we can process the file on the server as before and once done upload it to a new location if the api keys or credentials are set for sftp/s3

Last time I checked (few years ago) BunnyCDN was fairly cheap to host files on but my info might be out of date.

I think there have been a few cases where the storage on the server is limiting us and something like this was proposed as a solution before a while a go.

I'm not good with python and i assume during the actual migration if the server is updated, the files need to moved to the correct new location so the server will find them again with limited impact. FYI the json api from Synology allows us to point the download files to a different server so we can also modify the responses. So the actual downloads can be handled directly.

[publicarray]

I might not the best person to ask regarding direction but there are solutions to these problems if the problem becomes severe enough. Maybe @hgy59 can say more he has been more involved than i have been

If it helps a few years ago when we needed a CDN i asked a few providers including BunnyCDN and at that time where happy to include us in their open source program. For a limited time we where also part of the Cloudflare open source program. At that time Cloudflare did not have a file storage service like it has now. I haven't looked yet if fastly has a file storage API