BUG | Download are super slow

[spirillen]

What is the problem you are experiencing?

When I'm going to find current records on a PD FP report, I'm using fresh copies of the all-links to find what is listed.
So downloading this huge list with 500kb/s are taking about 3 - 3,5 minutes of extra time to handle FP requests, just wasting time, per case

Any change we could speed this up?

How can we reproduce the problem?

To reproduce the problem:

1. curl -Lo- 'https://phish.co.za/latest/ALL-phishing-links.lst' | grep -i '$record'
2. or wget -qO- 'https://phish.co.za/latest/ALL-phishing-links.lst' | grep -i '$record'

Do you have a screenshot?

Screenshot

What did you expect to happen?

I expected that...

Is there a workaround?

This would takes a few seconds as I run this from a data center with TB connections

Additional context

I would like to add...

Log information

Click to expand

Paste your log file here between the back tics.

[spirillen]

Or maybe an API we can connect to with tokens?