[MERGE] release/v1.0.0 into main

Release SummerFIM v1.0.0

.gitignore

@@ -0,0 +1 @@
+**/__pycache__/

README.md

@@ -1,2 +1,121 @@
+![](https://img.shields.io/badge/SummerFIM-v1.0.0-orange.svg)
+
 # SummerFIM
-SummerFIM is a simple File Integrity Monitor (FIM) coded in Python.
+SummerFIM is a simple File Integrity Monitor (FIM) coded in Python. Basically, this program works from SHA2 checksum comparison.
+
+## Motivation:
+I'm a datahoarder and therefore I have many HDDs and it has happened to me that some files on one of these HDDs have been corrupted and I only found out much later in a while that the backup of that HDD had been damaged, that is, I lost all those files forever. SummerFIM comes to solve this problem!
+
+## SummerFIM usage suggestion:
+Have 2 HDDs with the exact same files and every 6 months run SummerFIM on each one of them and compare the old status with the new status of each one HDD and even between the two HDD status, since they are the same, if in any of these comparisons any hash is different, certainly your HDD has corrupted files. It is very unlikely that both HDDs will corrupt at the same time, so you should replace the corrupted HDD with a new one and copy all files from the healthy HDD to the HDD that replaced the corrupted one.
+And that way you will have very good accuracy in the integrity of your files.
+
+## How to install SummerFIM:
+Run the following commands in your Linux terminal:
+```bash
+$ cd ~/
+$ git clone https://github.com/davidsf026/SummerFIM -b v1.0.0 ~/.SummerFIM
+$ alias summer='cd ~/.SummerFIM ; python3 ~/.SummerFIM/main.py ; cd ~/'
+$ echo "alias summer='cd ~/.SummerFIM ; python3 ~/.SummerFIM/main.py ; cd ~/'" >> ~/.bashrc
+```
+
+Set the path you want to monitore editing pathToScan key value in `~/.SummerFIM/app/settings/settings.yaml` file. Example:
+```yaml
+pathToScan: "/home/david/Documents/Test"
+filesListWithHashesTextFilePath: "app/data/hash_file.sha2"
+```
+
+SummerFIM is now installed!
+
+## How to use SummerFIM:
+
+### 0. Showing all avaible commands:
+Open a Linux terminal and start SummerFIM CLI with `summer` command and run `help` instruction in SummerFIM CLI:
+```bash
+$ summer
+summer> help
+help     Display this help.
+init     First checksum of selected path in settings.yaml file.
+update   Update checksum of selected path in settings.yaml file.
+exit     Exit SummerFIM.
+summer> exit
+Bye bye!
+```
+
+### 1. Initializing defined path monitoring:
+After installing SummerFIM correctly, the first thing you do is initialize the monitoring of defined path in `settings.yaml`:
+Open a Linux terminal and start SummerFIM CLI with `summer` command and run `init` instruction in SummerFIM CLI:
+```bash
+$ summer
+summer> init
+[ 1/3 ] 917f134de9e411465a28d076e3b395ec9a9faf58dbf7e62d4bd4ee90c838c6ca */home/david/Documents/Test/1.txt
+[ 2/3 ] ff5d016f12764b10f99041c02575c73517115d67bda134aeff3dd9b53ede2592 */home/david/Documents/Test/2.txt
+[ 3/3 ] e6476c095dae12456964423db181068538bf6d67dc80ba8b2db80fdbddd59508 */home/david/Documents/Test/3.txt
+summer> exit
+Bye bye!
+```
+
+Now SummerFIM has recursively scanned the defined path and checksummed all files present, this data is saved in `~/.SummerFIM/app/data/hash_file.sha2` as demonstrated below:
+```bash
+$ cat ~/.SummerFIM/app/data/hash_file.sha2
+917f134de9e411465a28d076e3b395ec9a9faf58dbf7e62d4bd4ee90c838c6ca */home/david/Documents/Test/1.txt
+ff5d016f12764b10f99041c02575c73517115d67bda134aeff3dd9b53ede2592 */home/david/Documents/Test/2.txt
+e6476c095dae12456964423db181068538bf6d67dc80ba8b2db80fdbddd59508 */home/david/Documents/Test/3.txt
+```
+
+### 2. Updating monitoring of an ALREADY INITIALIZED path:
+After initializing the monitoring of the defined path and it is expected that you spend some time changing the defined path, whether deleting files or creating new ones and whenever you wish, you can update the monitoring of the defined path where the checksum of the new files will be done and added to `hash_file.sha2` and files that have been deleted in the meantime will have their data deleted from `hash_file.sha2`.
+To update the defined path, open a Linux terminal and start SummerFIM CLI with `summer` command and run `update` instruction in SummerFIM CLI:
+```bash
+$ summer
+summer> update
+[ INFO ] 2 files deletions were detected:
+[ 1/2 ] /home/david/Documents/Test/1.txt
+[ 2/2 ] /home/david/Documents/Test/2.txt
+
+[ INFO ] 3 new files were detected:
+[ 1/3 ] 2ge5gg78f87bf23807b2398h23f08n23f9m23f80ba8b2db808h23f08n23fm2fd */home/david/Documents/Test/4.txt
+[ 2/3 ] h7hdf8721386fbv12bdae12456964423db181068538bf6d67dc80ba8b2db8008 */home/david/Documents/Test/5.txt
+[ 3/3 ] 308h23f08n2378f87bf23807b239086964423db1810685384423db1810685384 */home/david/Documents/Test/6.txt
+summer> exit
+Bye bye!
+```
+
+### 3. Checking path integrity:
+Open a Linux terminal and start SummerFIM CLI with `summer` command and run TWICE `update` instruction in SummerFIM CLI:
+```bash
+$ summer
+summer> update
+[ INFO ] 2 files deletions were detected:
+[ 1/2 ] /home/david/Documents/Test/1.txt
+[ 2/2 ] /home/david/Documents/Test/2.txt
+
+[ INFO ] 3 new files were detected:
+[ 1/3 ] 2ge5gg78f87bf23807b2398h23f08n23f9m23f80ba8b2db808h23f08n23fm2fd */home/david/Documents/Test/4.txt
+[ 2/3 ] h7hdf8721386fbv12bdae12456964423db181068538bf6d67dc80ba8b2db8008 */home/david/Documents/Test/5.txt
+[ 3/3 ] 308h23f08n2378f87bf23807b239086964423db1810685384423db1810685384 */home/david/Documents/Test/6.txt
+summer> update
+[ INFO ] 0 files deletions were detected:
+
+[ INFO ] 0 new files were detected:
+summer> exit
+Bye bye!
+```
+The update statement is run twice to ensure the recorded path state in `hash_file.sha2` matches the current path state.
+Now run the init instruction so that a new `hash_file.sha2` is generated and thus all checksum hashes are generated again.
+
+```bash
+$ summer
+summer> init
+[ 1/4 ] e6476c095dae12456964423db181068538bf6d67dc80ba8b2db80fdbddd59508 */home/david/Documents/Test/3.txt
+[ 2/4 ] e6476c095dae12456964423db181068538bf6d67dc80ba8b2db80fdbddd59508 */home/david/Documents/Test/4.txt
+[ 3/4 ] e6476c095dae12456964423db181068538bf6d67dc80ba8b2db80fdbddd59508 */home/david/Documents/Test/5.txt
+[ 4/4 ] e6476c095dae12456964423db181068538bf6d67dc80ba8b2db80fdbddd59508 */home/david/Documents/Test/6.txt
+[ WARNING ] Can't save, file already exists.
+Do you want to replace app/data/hash_file.sha2? yes
+[ INFO ] Replacing file...
+summer> exit
+Bye bye!
+```
+
+Notice that in SummerFIM's data folder (`~/.SummerFIM/app/data`) we will have two files, `hash_file.sha2` and `hash_file_old.sha2`, both have data from the same files the difference is that `hash_file_old.sha2` had the hashes recalculated recently and the `hash_file_old.sha2` has hashes recorded from the init prior to the init recently run and all the update instructions prior to the init recently run, that is, if a comparison is made between these two files and some hash is different, certainly the path has corrupted files. Currently SummerFIM does not have the ability to compare `hash_file.sha2` and `hash_file_old.sha2` files, but it can easily be done with any diff checker, I recommend the following: https://www.diffchecker.com/

app/checksum.py

@@ -0,0 +1,74 @@
+import hashlib
+
+# CUSTOM TEXT
+INFO = '[ \033[94mINFO\033[0m ]'
+ERROR = '[ \033[91mERROR\033[0m ]'
+WARNING = '[ \033[93mWARNING\033[0m ]'
+OKGREEN = '\033[92m'
+OKYELLOW = '\033[93m'
+BOLD = '\033[1m'
+ENDC = '\033[0m'
+
+class Checksum:
+    def sha2(filesList):
+        temp = []
+        filesListLenght = len(filesList)
+        currentIndexfilesList = 0
+        for i in filesList:
+            currentIndexfilesList +=1
+            sha256_hash = hashlib.sha256()
+            try:
+                with open(i,"rb") as f:
+                    for byte_block in iter(lambda: f.read(4096),b""):
+                        sha256_hash.update(byte_block)
+                    outputSha256 = sha256_hash.hexdigest()
+                    permissionErrorException = False
+            except PermissionError:
+                print(ERROR + ' Checksum permission error on file: ' + i)
+                errorCounter +=1
+                permissionErrorException = True
+            if permissionErrorException == False:
+                temp.append(outputSha256 + ' *' + i)
+                print(BOLD + "[ " + OKGREEN + str(currentIndexfilesList) + '/' + str(filesListLenght) + ENDC + " ] " + ENDC + outputSha256 + ' *' + i)
+        filesList = temp
+        temp = []
+
+        return filesList
+
+    def sha2UpdateList(oldFilesListWithHashes, newFilesListWithoutHashes):
+
+        oldFilesListWithoutHashes = [i[66:] for i in oldFilesListWithHashes]
+
+        # COMPARING ARRAYS oldFilesListWithoutHashes AND newFilesListWithoutHashes TO FIND OUT WHICH FILES WERE DELETED
+        filesDeletionsArray = list(set(oldFilesListWithoutHashes) - set(newFilesListWithoutHashes))
+        filesDeletionsArray = sorted(filesDeletionsArray)
+
+        print(INFO + " " + str(len(filesDeletionsArray)) + " files deletions were detected:")
+        currentIndexDeletionsArray = 0
+        for i in filesDeletionsArray:
+            currentIndexDeletionsArray +=1
+            print(BOLD + "[ " + OKYELLOW + str(currentIndexDeletionsArray) + '/' + str(len(filesDeletionsArray)) + ENDC + " ] " + ENDC + i)
+        print("")
+
+        # REMOVING THE ITEMS PRESENT IN THE filesDeletionsArray INTO oldFilesListWithHashes
+        temp = []
+        for item in oldFilesListWithHashes:
+            if item[66:] not in filesDeletionsArray:
+                temp.append(item)
+        newFilesListWithHashes = temp
+        temp = []
+
+        # COMPARING ARRAYS oldFilesListWithoutHashes and newFilesListWithoutHashes TO FIND OUT WHICH FILES WERE ADDED
+        filesAdditionsArray = list(set(newFilesListWithoutHashes) - set(oldFilesListWithoutHashes))
+        filesAdditionsArray = sorted(filesAdditionsArray)
+
+        print(INFO + " " + str(len(filesAdditionsArray)) + " new files were detected:")
+
+        # CHECKSUMING HASHES OF filesAdditionsArray ITEMS
+        filesAdditionsArray = Checksum.sha2(filesAdditionsArray)
+
+        # JOINING newFilesListWithHashes AND filesAdditionsArray
+        newFilesListWithHashes = newFilesListWithHashes + filesAdditionsArray
+        newFilesListWithHashes = sorted(newFilesListWithHashes, key=lambda x:x[66:])
+
+        return newFilesListWithHashes

app/files_operations.py

@@ -0,0 +1,35 @@
+import os
+
+# CUSTOM TEXT
+INFO = '[ \033[94mINFO\033[0m ]'
+ERROR = '[ \033[91mERROR\033[0m ]'
+WARNING = '[ \033[93mWARNING\033[0m ]'
+
+class FilesOperations:
+    def saveArrayInTextFile(array, textFilePath):
+        alreadyExists = os.path.exists(textFilePath)
+        if alreadyExists:
+            print(WARNING + " Can't save, file already exists.")
+            while True:
+                CHOICE = input("Do you want to replace " + textFilePath + "? ")
+                if CHOICE == "yes":
+                    print(INFO + " Replacing file...")
+                    os.remove(textFilePath)
+                    alreadyExists = False
+                    break
+                elif CHOICE == "no":
+                    print(INFO + " Canceling operation...")
+                    break
+                else:
+                    print(WARNING + " Unknown option, only yes or no.")
+        if alreadyExists == False:
+            textFileObject = open(textFilePath, 'a', encoding="utf-8")
+            for i in array:
+                textFileObject.write(i + '\n')
+            textFileObject.close()
+
+    def backup(fileToBackupPath):
+        fileToBackupPath_str = str(fileToBackupPath)
+        index = fileToBackupPath_str.rfind(".")
+        backupFilePath = fileToBackupPath_str[:index] + "_old" + fileToBackupPath_str[index:]
+        os.rename(fileToBackupPath, backupFilePath)

app/main_operations.py

@@ -0,0 +1,33 @@
+from app.scan import *
+from app.checksum import *
+from app.files_operations import *
+from app.settings_yaml_vars import *
+import os
+
+class MainOperations:
+    def init(pathToScan, filesListWithHashesTextFilePath):
+        summerDataPath = str(SettingsYamlVars.filesListWithHashesTextFilePath)
+        lastSlashOccurenceIndexInSummerDataPath = summerDataPath.rfind('/')
+        summerDataPath = summerDataPath[:lastSlashOccurenceIndexInSummerDataPath]
+
+        alreadyExists = os.path.exists(summerDataPath)
+
+        if alreadyExists == False:
+            os.makedirs(summerDataPath)
+
+        filesList = Scan.path(pathToScan)
+        filesListWithHashes = Checksum.sha2(filesList)
+        FilesOperations.saveArrayInTextFile(filesListWithHashes, filesListWithHashesTextFilePath)
+
+    def update(pathToScan, filesListWithHashesTextFilePath):
+        currentFilesListWithoutHashes = Scan.path(pathToScan)
+        oldFilesListWithHashes = Scan.textFile(filesListWithHashesTextFilePath)
+        currentFilesListWithHashes = Checksum.sha2UpdateList(oldFilesListWithHashes, currentFilesListWithoutHashes)
+        FilesOperations.backup(filesListWithHashesTextFilePath)
+        FilesOperations.saveArrayInTextFile(currentFilesListWithHashes, filesListWithHashesTextFilePath)
+
+    def help():
+        print('help \t Display this help.')
+        print('init \t First checksum of selected path in settings.yaml file.')
+        print('update \t Update checksum of selected path in settings.yaml file.')
+        print('exit \t Exit SummerFIM.')

app/scan.py

@@ -0,0 +1,32 @@
+import os, fnmatch
+import yaml
+
+class Scan:
+    def yaml(yamlFilepath):
+        with open(yamlFilepath) as yamlFile: 
+            yamlArray = yaml.safe_load(yamlFile)
+
+        return yamlArray
+
+    def path(pathToScan):
+        path = str(pathToScan)
+        pattern = '*'
+        filesListArray = []
+
+        for dName, sdName, fList in os.walk(path):
+            for fileName in fList:
+                if fnmatch.fnmatch(fileName, pattern):
+                    filesListArray.append(os.path.join(dName, fileName))
+        filesListArray = sorted(filesListArray)
+
+        return filesListArray
+
+    def textFile(path):
+        textFileObject = open(path, encoding="utf-8")
+        textFileArray = [i[:-1] for i in textFileObject.readlines()]
+
+        textFileObject.close()
+
+        textFileArray = sorted(textFileArray)
+
+        return textFileArray

app/settings/settings.yaml

@@ -0,0 +1,2 @@
+pathToScan: "PUT HERE THE PATH YOU WANT TO MONITOR"
+filesListWithHashesTextFilePath: "app/data/hash_file.sha2"

app/settings_yaml_vars.py

@@ -0,0 +1,7 @@
+from app.scan import *
+
+class SettingsYamlVars:
+    settingsYamlArray = Scan.yaml('app/settings/settings.yaml')
+
+    pathToScan = settingsYamlArray['pathToScan']
+    filesListWithHashesTextFilePath = settingsYamlArray['filesListWithHashesTextFilePath']