Blocklist import script for IPTools/IPSet
https://github.com/AubsUK/aubs-blocklist-importer
Top of Page
Information
Quick Start
Configurable Options
Files Used and Created
Planned changes
Example outputs
Removal
Notes
|Back to top|
This is a simple blocklist import script that works with single IPv4 addresses (no ranges or IPv6 support yet).
- Runs automatically (via Cron)
- Imports a list of IPs to block from a URL text file
- Strips out anything non-IPv4 related
- Removes duplicates
- Custom lists to override the importing blocklist
- Allow list - Never block anything on this list even if it is in the download (e.g. your own or customer IPs)
- Block list - Always block anything on this list even if it isn't in the download (e.g. IPs of frequent attackers or spammers)
- Checks for an existing chain and that everything is already set up, or creates them
- Compares the new import blocklist against the existing blocked list
- only import new IPs
- remove old IPs not on the new list
- Checks the new live list matches the filtered import list
- If it doesn't, it clears the configuration and tries to re-import the previous (known-good) list
- It then checks if the re-import of the known-good list is successful
- Full logging
- Email notifications
- Email Success/Failure switches, these allow set days when 'success' and 'failure' emails are sent.
- Notifications can also be set so if a 'failure' occurs, a the next 'success' is also alerted, even if it's not a 'success' alert day.
|Back to top|
Switch to a secure location to hold the script
cd /usr/local/sbin/
Clone the repository as root so permissions are set appropriately
sudo git clone https://github.com/AubsUK/aubs-blocklist-importer
or just create the three files manually, and copy their contents
Make the script executable
cd aubs-blocklist-importer
sudo chmod 700 aubs-blocklist-importer.sh
Edit override-allowlist.txt and override-blocklist.txt to include IPs to never block (e.g. your own servers) and always block (servers that frequently attack you)
sudo nano override-allowlist.txt
sudo nano override-blocklist.txt
Add an entry into Cron
sudo nano /etc/crontab
Add in:
# Blocklist Importer
0 * * * * root /usr/local/sbin/aubs-blocklist-importer/aubs-blocklist-importer.sh # Run on the hour, every hour
Note: Make it as frequent as required (within reason, check the blocklist's website to confirm the maximum);
Following the Quick Start instructions and not modifying any variables, the following files are used:
| File Name | Purpose |
|---|---|
| aubs-blocklist-importer.sh | The script file |
| override-allowlist.txt | Override allow-list containing IPs one on each line to always allow even if they *are* in the blocklist |
| override-blocklist.txt | Override block-list containing IPs one on each line to always block even if they *are not* in the blocklist |
| Last_Run_Status.txt | Stores the status of the last run and the day |
| File Name | Purpose |
|---|---|
| aubs-blocklist-importer.log | Stores the logs from each run |
|
.git/ (folder and all sub files)
images/ (folder and all sub files) LICENSE README.md |
Git files, not used by the script |
If DELETE_ALL_FILES_ON_COMPLETION is set to false the following files will remain in the main folder, otherwise they will be deleted after each run
| File Name | Purpose |
|---|---|
| ip-blocklist.download | Main file that the download list is imported into and processed |
| ip-blocklist.download.compare.add | Items processed that aren't in the existing (to be added) |
| ip-blocklist.download.compare.rem | Existing items that aren't in the processed (to be removed) |
| ip-blocklist.download.Dedupe | Downloaded file processed with duplicates removed |
| ip-blocklist.download.IPv4 | Downloaded file processed with only IPv4 addresses |
| ip-blocklist.download.Original | Copy of the original download file |
| ip-blocklist.download.OverrideAllow | Downloaded file processed with override allow-list addresses removed |
| ip-blocklist.download.OverrideAllowTEMP | Temporary override allow-list files sorted and deduped |
| ip-blocklist.download.OverrideBlock | Downloaded file processed with override block-list addresses added |
| ip-blocklist.download.OverrideBlockTEMP | Temporary override block-list files sorted and deduped |
| ip-blocklist.existing | List of existing IPs from the current IP chain |
| ip-blocklist.existing.check1 | List of IPs to confirm successful import |
| ip-blocklist.existing.check2 | List of IPs to confirm successful import> |
| ip-blocklist.existing.validate1 | List of IPs remaining after checking |
| ip-blocklist.existing.validate2 | List of IPs remaining after checking |
| Variable | Description | Default |
|---|---|---|
|
|
Clear out the IPTable and IPSet for the $CHAINNAME on each run | false |
|
|
Delete the temporary files on completion, useful for debugging if something is wrong with the block lists | true |
|
|
URL of the text file which contains all the IPs to use | http://lists.blocklist.de/lists/all.txt |
|
|
Name of the chain to import the IPs in to | blocklist-de |
|
|
The action script should apply to the firewall rule Either ALLOW (for known IPs), BLOCK, or REJECT for this Chain |
REJECT |
|
|
Minimum IPs in the download file to consider it a legitimate download | 100 |
|
|
Path of the script file |
"$(dirname "$(realpath "$0")")/" |
|
|
/path/to/ temp files location |
Uses |
|
|
Base filename for the temporary files created | ip-blocklist |
|
|
Display name of the sender | Notifications |
|
|
Senders email address |
Automatically configured to |
|
|
Recipient email address (multuple recipients separated by commas) |
Automatically configured to |
|
|
Start of the subject for success and failure emails |
|
|
|
Days SUCCESS emails should be sent - Leave blank to disable [1=Monday, 7=Sunday] (1,4=Mon,Thu) |
1,4 |
|
|
When to send success emails (if run multiple times a day) [NONE, FIRST, ALL] (only on the days in SUCCESS_DAYS) |
FIRST |
|
|
Days FAILURE emails should be sent - Leave blank to disable [1=Monday, 7=Sunday] (1,2,3,4,5,6,7=Mon,Tue,Wed,Thu,Fri,Sat,Sun) |
1,2,3,4,5,6,7 |
|
|
When to send failure emails (if run multiple times a day) [NONE, FIRST, ALL] |
FIRST |
|
|
For multi-day runs, as long as the FAILURE_DAYS is 1-7 and FAILURE_TYPE isn't NONE, when a FAILURE is received after a SUCCESS, an email will be sent (last run=success, this run=failure).
The same will happen for SUCCESS if SUCCESS_DAYS is 1-7 and SUCCESS_TYPE isn't NONE, that after a FAILURE, a SUCCESS email will be received. On the other hand, as FAILUREs will be sent, a SUCCESS might not to confirm it has been restored until the next SUCCESS_DAY when a SUCCESS can be received. Set this to true and a FAILURE/SUCCESS email will be sent the first time the new status changes, but no other times unless scheduled. |
true |
|
|
Location of the override allow-list |
The same as |
|
|
Filename of the override allow-list | override-allowlist.txt |
|
|
Location of the override block-list |
The same as |
|
|
Filename of the override allow-list | override-blocklist.txt |
|
|
Location of the log file |
A new directory in the /var/log/ path called |
|
|
Filename of the log file | aubs-blocklist.log |
|
|
Location of the last run file | $BASE_PATH |
|
|
Filename of the last run information (this includes the day number for use in email allowed days) | Last_Run_Status.txt |
|
|
Location of the main packages used. These should normally be installed, but if not, it'll report in the log and stop running |
$(which iptables)
|
|Back to top|
The script contains two useful test lines when the script goes through the validation checks.
The first pretends the filtered download list $Blocklist_File has 5 lines less than it should, so when the imported list doesn't match the filtered download list, it'll try and restore the last known good list (line 449):
#sed -i '1,5d' $BLOCKLIST_FILE #TESTING1 == REMOVE THE FIRST FIVE LINES FROM THE FILTERED ORIGINAL FILE
And the second is used after the first validation check fails, which then pretends the second validation check of the last known good list $BLOCKLIST_EXISTING has 5 lines less than it should, so when the last known good list doesn't match the last known good import validation, it'll output that it's all failed (line 495):
# sed -i '1,5d' $BLOCKLIST_EXISTING #TESTING2 == REMOVE THE FIRST FIVE LINES FROM THE ORIGINAL EXISTING FILE
- Allow cron to take the download file URL and chain name as variables, so multiple can be run from one script
- Using the same chain with multiple blocklists (perhaps download all at once, then filter through before adding - Size limitations?).
- Incorporate IPv6 IP addresses
- If a firewall rule exists in the chain, check if the ACTION is the same each time and change if it's different e.g. DROP to REJECT
- Check if the path is a path or a file/path for all variables
BASE_PATH_CheckPath=${BASE_PATH%/*}
BASE_PATH_CheckFile=${BASE_PATH##*/}
echo "PATH [ $BASE_PATH_CheckPath ]"
echo "FILE [ $BASE_PATH_CheckFile ]"
- Check if BASE_PATH is a valid and/or a 'bad' path like in /proc/ or something
- --DONE--
Change logging to give the option to enter additional test (e.g. 'done' at the end of the previous logged line) - Consider removing the variables for the programs being used, I don't really think these are necessary because the ones being used are mostly 'standard' - Check if they are POSIX, or alternatives. Most others being used are: date, touch, echo, if, exit, rm, mv, cp, wc, sed, comm, cat. [ipset was not on some of my servers]
- Work with subnets, expand them to individual IPs or if IPSet allows them.
- --DONE--
Enable/Disable email notifications, or set them to only send every X days (and list the days in email). - --DONE--
Check import was successful - Warn if any 'override allow' exist in the blocklist
- Allow use of list from a local file (e.g. manual syncing)
- --DONE--
Don't import if downloaded file contains less than a defined number of rows - If a run results in a 'success' but errors or critical, it should send a FAILURE email.
- Correct spelling mistake on L638
- Remove debugging messages from L201 to L217 and L222 and L233 and L248
- Add automatic retry count/delay to reduce the number of failure emails.
- The original download contained [20127] rows; filtering out 76 rows not IPv4 [20051]; no duplicate IPs found (still [20051]).
- 3 unique Override Allow IPs weren't in the list, so none to remove (still [20051]); 1 unique Override Block IP present to add (took it up to [20052]).
- All the IPtables / IPsets configs exist, nothing new to do.
- Exported the existing list and compared it; only added [2211] IPs and rmeoved [2866].
- Exported the new existing list [20052]; compared it to the expected filtered download list [20052]; confirmed both match
- Finished in 6 seconds.
me@server:/usr/local/sbin/aubs-blocklist-importer$ sudo ./aubs-blocklist-importer.sh
Tue 26 Jul 22:39:42 BST 2022: ================================================================================
Tue 26 Jul 22:39:42 BST 2022:
Tue 26 Jul 22:39:42 BST 2022: Using Base Path [ /usr/local/sbin/aubs-blocklist-importer/ ]
Tue 26 Jul 22:39:42 BST 2022: Deleting any existing blocklist files. (/usr/local/sbin/aubs-blocklist-importer/ip-blocklist.*)
Tue 26 Jul 22:39:42 BST 2022: Downloading the most recent IP list from http://lists.blocklist.de/lists/all.txt... Successful [20127]
Tue 26 Jul 22:39:43 BST 2022:
Tue 26 Jul 22:39:43 BST 2022: Filter out anything not an IPv4 address [20051]
Tue 26 Jul 22:39:43 BST 2022: Removing duplicate IPs. [20051]
Tue 26 Jul 22:39:43 BST 2022: Removing Override allow-list IPs (3 unique) [20051]
Tue 26 Jul 22:39:43 BST 2022: Adding Override block-list IPs... (1 unique) [20052]
Tue 26 Jul 22:39:43 BST 2022:
Tue 26 Jul 22:39:43 BST 2022: Checking the configuration for 'blocklist-de'...
Tue 26 Jul 22:39:43 BST 2022: IP set already exists
Tue 26 Jul 22:39:43 BST 2022: Chain already exists
Tue 26 Jul 22:39:43 BST 2022: Chain already in INPUT
Tue 26 Jul 22:39:43 BST 2022: Firewall rule already exists in the chain
Tue 26 Jul 22:39:43 BST 2022:
Tue 26 Jul 22:39:43 BST 2022: Getting the existing list for the 'blocklist-de' IP set
Tue 26 Jul 22:39:43 BST 2022:
Tue 26 Jul 22:39:43 BST 2022: Comparing the New and Existing lists...
Tue 26 Jul 22:39:43 BST 2022: Adding [2211] new IPs into the IP set... Done
Tue 26 Jul 22:39:45 BST 2022: Removing [2866] old IPs from the IP set... Done
Tue 26 Jul 22:39:48 BST 2022:
Tue 26 Jul 22:39:48 BST 2022: Checking imported 'blocklist-de' matches downloaded list... Filtered Download [20052] - Filtered Existing [20052]... Validated
Tue 26 Jul 22:39:48 BST 2022:
Tue 26 Jul 22:39:48 BST 2022: Process finished in 0 Minutes and 6 Seconds.
Tue 26 Jul 22:39:48 BST 2022: Writing last status of [SUCCESS6] to /usr/local/sbin/aubs-blocklist-importer/Last_Run_Status.txt
Tue 26 Jul 22:39:48 BST 2022: NOT sending SUCCESS email
Tue 26 Jul 22:39:48 BST 2022: Deleting any existing blocklist files. (/usr/local/sbin/aubs-blocklist-importer/ip-blocklist.*)
Tue 26 Jul 22:39:48 BST 2022:
Tue 26 Jul 22:39:48 BST 2022: ================================================================================
- Pretty similar to the successful run.
- Testing removed 5 IPs from the filtered download [20068] --> [20063]; doesn't match the live list [20063].
- Rebuilds IPtable and IPset config for the chain.
- Imports the list and confirms it matches the last known good.
me@server:/usr/local/sbin/aubs-blocklist-importer$ sudo ./aubs-blocklist-importer.sh
Tue 26 Jul 23:25:25 BST 2022: ================================================================================
Tue 26 Jul 23:25:25 BST 2022:
Tue 26 Jul 23:25:25 BST 2022: Using Base Path [ /usr/local/sbin/aubs-blocklist-importer/ ]
Tue 26 Jul 23:25:25 BST 2022: Deleting any existing blocklist files. (/usr/local/sbin/aubs-blocklist-importer/ip-blocklist.*)
Tue 26 Jul 23:25:25 BST 2022: Downloading the most recent IP list from http://lists.blocklist.de/lists/all.txt... Successful [20141]
Tue 26 Jul 23:25:26 BST 2022:
Tue 26 Jul 23:25:26 BST 2022: Filter out anything not an IPv4 address [20067]
Tue 26 Jul 23:25:26 BST 2022: Removing duplicate IPs. [20067]
Tue 26 Jul 23:25:26 BST 2022: Removing Override allow-list IPs (3 unique) [20067]
Tue 26 Jul 23:25:26 BST 2022: Adding Override block-list IPs... (1 unique) [20068]
Tue 26 Jul 23:25:26 BST 2022:
Tue 26 Jul 23:25:26 BST 2022: Checking the configuration for 'blocklist-de'...
Tue 26 Jul 23:25:26 BST 2022: IP set already exists
Tue 26 Jul 23:25:26 BST 2022: Chain already exists
Tue 26 Jul 23:25:26 BST 2022: Chain already in INPUT
Tue 26 Jul 23:25:26 BST 2022: Firewall rule already exists in the chain
Tue 26 Jul 23:25:26 BST 2022:
Tue 26 Jul 23:25:26 BST 2022: Getting the existing list for the 'blocklist-de' IP set
Tue 26 Jul 23:25:26 BST 2022:
Tue 26 Jul 23:25:26 BST 2022: Comparing the New and Existing lists...
Tue 26 Jul 23:25:26 BST 2022: Adding [68] new IPs into the IP set... Done
Tue 26 Jul 23:25:26 BST 2022: Removing [52] old IPs from the IP set... Done
Tue 26 Jul 23:25:26 BST 2022:
Tue 26 Jul 23:25:26 BST 2022: Checking imported 'blocklist-de' matches downloaded list... Filtered Download [20063] - Filtered Existing [20068]... ERROR !!! - They don't match
Tue 26 Jul 23:25:26 BST 2022: An error occurred with importing the download
Tue 26 Jul 23:25:26 BST 2022:
Tue 26 Jul 23:25:26 BST 2022: Resetting the chain
Tue 26 Jul 23:25:26 BST 2022: Flushed IPTable Chain
Tue 26 Jul 23:25:26 BST 2022: Flushed IPSet Chain
Tue 26 Jul 23:25:26 BST 2022: Destroyed IPSet Chain
Tue 26 Jul 23:25:26 BST 2022: Deleted IPTable INPUT Join
Tue 26 Jul 23:25:26 BST 2022: Deleted IPTable Chain
Tue 26 Jul 23:25:26 BST 2022: Creating a new chain
Tue 26 Jul 23:25:26 BST 2022: New IP set created
Tue 26 Jul 23:25:26 BST 2022: New chain created
Tue 26 Jul 23:25:26 BST 2022: Chain added to INPUT
Tue 26 Jul 23:25:26 BST 2022: Firewall rule created
Tue 26 Jul 23:25:26 BST 2022:
Tue 26 Jul 23:25:26 BST 2022: Importing the previous existing list... Done
Tue 26 Jul 23:25:48 BST 2022: Re-checking restored 'blocklist-de' version matches original existing... Original [20052] - Current [20052]... Validated
Tue 26 Jul 23:25:48 BST 2022:
Tue 26 Jul 23:25:48 BST 2022: Process finished in 0 Minutes and 23 Seconds.
Tue 26 Jul 23:25:48 BST 2022:
Tue 26 Jul 23:25:48 BST 2022: ================================================================================
- As with the unsuccessful run, the download import failed. This time, the restore also fails.
- Testing removed 5 IPs from the filtered download [20068] --> [20063]; doesn't match the live list [20063].
- Rebuilds IPtable and IPset config for the chain.
- Imports the last known good list
- Testing removes 5 IPs from the last known good after importing [20052] --> [20047]; fails to match the live list [20052].
- Not much can be done now, the IPset will contain what it has, but may need manual intervention.
- The next automatic run may correct this.
me@server:/usr/local/sbin/aubs-blocklist-importer$ sudo ./aubs-blocklist-importer.sh
Tue 26 Jul 23:28:16 BST 2022: ================================================================================
Tue 26 Jul 23:28:16 BST 2022:
Tue 26 Jul 23:28:16 BST 2022: Using Base Path [ /usr/local/sbin/aubs-blocklist-importer/ ]
Tue 26 Jul 23:28:16 BST 2022: Deleting any existing blocklist files. (/usr/local/sbin/aubs-blocklist-importer/ip-blocklist.*)
Tue 26 Jul 23:28:16 BST 2022: Downloading the most recent IP list from http://lists.blocklist.de/lists/all.txt... Successful [20141]
Tue 26 Jul 23:28:16 BST 2022:
Tue 26 Jul 23:28:16 BST 2022: Filter out anything not an IPv4 address [20067]
Tue 26 Jul 23:28:16 BST 2022: Removing duplicate IPs. [20067]
Tue 26 Jul 23:28:16 BST 2022: Removing Override allow-list IPs (3 unique) [20067]
Tue 26 Jul 23:28:16 BST 2022: Adding Override block-list IPs... (1 unique) [20068]
Tue 26 Jul 23:28:17 BST 2022:
Tue 26 Jul 23:28:17 BST 2022: Checking the configuration for 'blocklist-de'...
Tue 26 Jul 23:28:17 BST 2022: IP set already exists
Tue 26 Jul 23:28:17 BST 2022: Chain already exists
Tue 26 Jul 23:28:17 BST 2022: Chain already in INPUT
Tue 26 Jul 23:28:17 BST 2022: Firewall rule already exists in the chain
Tue 26 Jul 23:28:17 BST 2022:
Tue 26 Jul 23:28:17 BST 2022: Getting the existing list for the 'blocklist-de' IP set
Tue 26 Jul 23:28:17 BST 2022:
Tue 26 Jul 23:28:17 BST 2022: Comparing the New and Existing lists...
Tue 26 Jul 23:28:17 BST 2022: Adding [68] new IPs into the IP set... Done
Tue 26 Jul 23:28:17 BST 2022: Removing [52] old IPs from the IP set... Done
Tue 26 Jul 23:28:17 BST 2022:
Tue 26 Jul 23:28:17 BST 2022: Checking imported 'blocklist-de' matches downloaded list... Filtered Download [20063] - Filtered Existing [20068]... ERROR !!! - They don't match
Tue 26 Jul 23:28:17 BST 2022: An error occurred with importing the download
Tue 26 Jul 23:28:17 BST 2022:
Tue 26 Jul 23:28:17 BST 2022: Resetting the chain
Tue 26 Jul 23:28:17 BST 2022: Flushed IPTable Chain
Tue 26 Jul 23:28:17 BST 2022: Flushed IPSet Chain
Tue 26 Jul 23:28:17 BST 2022: Destroyed IPSet Chain
Tue 26 Jul 23:28:17 BST 2022: Deleted IPTable INPUT Join
Tue 26 Jul 23:28:17 BST 2022: Deleted IPTable Chain
Tue 26 Jul 23:28:17 BST 2022: Creating a new chain
Tue 26 Jul 23:28:17 BST 2022: New IP set created
Tue 26 Jul 23:28:17 BST 2022: New chain created
Tue 26 Jul 23:28:17 BST 2022: Chain added to INPUT
Tue 26 Jul 23:28:17 BST 2022: Firewall rule created
Tue 26 Jul 23:28:17 BST 2022:
Tue 26 Jul 23:28:17 BST 2022: Importing the previous existing list... Done
Tue 26 Jul 23:28:39 BST 2022: Re-checking restored 'blocklist-de' version matches original existing... Original [20047] - Current [20052]... ERROR !!! - Still an issue
Tue 26 Jul 23:28:39 BST 2022:
Tue 26 Jul 23:28:39 BST 2022: Process finished in 0 Minutes and 23 Seconds.
Tue 26 Jul 23:28:39 BST 2022:
Tue 26 Jul 23:28:39 BST 2022: ================================================================================
- As with the unsuccessful run, the download import failed. This time, the restore also fails.
- Testing removed 5 IPs from the filtered download [20068] --> [20063]; doesn't match the live list [20063].
- Rebuilds IPtable and IPset config for the chain.
- Imports the last known good list
- Testing removes 5 IPs from the last known good after importing [20052] --> [20047]; fails to match the live list [20052].
- Not much can be done now, the IPset will contain what it has, but may need manual intervention.
- The next automatic run may correct this.
me@server:/usr/local/sbin/aubs-blocklist-importer$ sudo ./aubs-blocklist-importer.shSat 10 Dec 22:21:14 GMT 2022: ================================================================================
Sat 10 Dec 22:21:14 GMT 2022:
Sat 10 Dec 22:21:14 GMT 2022: Using Base Path [ /usr/local/sbin/aubs-blocklist-importer/ ]
Sat 10 Dec 22:21:14 GMT 2022: Deleting any existing blocklist files. (/usr/local/sbin/aubs-blocklist-importer/ip-blocklist.*)
Sat 10 Dec 22:21:14 GMT 2022: Downloading the most recent IP list from http://lists.blocklist.de/lists/all.txt... IP blocklist could not be downloaded from 'http://lists.blocklist.de/lists/all.txt' [ Downloaded 0, below minimum of 100]
Sat 10 Dec 22:21:14 GMT 2022: Writing last status of [FAILURE7] to /usr/local/sbin/aubs-blocklist-importer/Last_Run_Status.txt
Sat 10 Dec 22:21:14 GMT 2022: Sending FAILURE email
Sat 10 Dec 22:21:14 GMT 2022: ================================================================================
If you installed the script using the quick start guide, it's pretty easy to remove.
Edit crontab and remove the # Blocklist Importer line and the line below it:
sudo nano /etc/crontab
Move into the sbin folder and delete the aubs-blocklist-importer folder:
cd /usr/local/sbin/
sudo rm -r aubs-blocklist-importer
Move into the logs folder and delete the aubs-blocklist-importer folder:
cd /var/log/
sudo rm -r aubs-blocklist-importer
That's it, everything has been removed.
|Back to top|
This script was born through the need for a script to do exactly what I wanted. I took a lot of inspiration from Lexo.ch, and lots of support from Stack Overflow and related sites, along with may other sites.