diff --git a/.gitignore b/.gitignore index d990d02..1a048d9 100644 --- a/.gitignore +++ b/.gitignore @@ -49,4 +49,7 @@ ipban.ini apps/uploads/* # Synology Drive Client -.sync-exclude.lst \ No newline at end of file +.sync-exclude.lst + +# IPBan Log Directory +log/ \ No newline at end of file diff --git a/README.md b/README.md index 8fab391..f4b2cf1 100644 --- a/README.md +++ b/README.md @@ -15,14 +15,15 @@ It was designed with an emphasis on security to meet organization/business needs # ❗ This project is still in pre-release. # 🐛 Known Bugs -- Worker management needs reimplementation - - Currently working on this +- ~~Worker management needs reimplementation~~ + - ~~Currently working on this~~ - Manually requeuing jobs fail - ~~Searching plates will only work if pagination position is on page 1~~ - ~~This is a grid.js issue [#1314](https://github.com/grid-js/gridjs/issues/1314) [#1344](https://github.com/grid-js/gridjs/pull/1334) [#1311](https://github.com/grid-js/gridjs/issues/1311).~~ # ✨ Upcoming Features - Integrate [Apprise](https://github.com/caronc/apprise) +- Add something similar to [DahuaSunriseSunset](https://github.com/bp2008/DahuaSunriseSunset) - Enhance search functionality - Add - Direction @@ -42,6 +43,20 @@ It was designed with an emphasis on security to meet organization/business needs - Add audit logs for each action - Add support for 2FA/MFA +# Development +- Run/Debug Configuration +Install Redis server and start it.
+Additionally, you can set Redis server to start automatically. See the Bare Server section + - OpenALPR-Webhook + - Set the Script path to `/app.py` + - Parameters should be set to `--host=0.0.0.0 --port=8080` + - Add `DEBUG=True` into Environment variables + - Set the Working directory to `` + - Redis Worker Server + - Add a new Run/Debug Configuration named "Worker Manager Server" + - Set the Script path to `/apps/workers.py` + - Set the Working directory to `/apps` + # Installation @@ -53,73 +68,171 @@ TBD 2. git https://github.com/mibs510/OpenALPR-Webhook 3. cd OpenALPR-Webhook 4. pip3 install -r requirements.txt -5. ./venv/Scripts/activate +5. ./venv/bin/activate 6. ./app.py --host=0.0.0.0 --port=8080 +#### Linux systemd service +You will want to create a service file to automatically start OpenALPR-Webhook upon each reboot. + +`sudo nano /etc/systemd/system/oalpr-wh.service` + + +` +[Unit] +Description=OpenALPR-Webhook +After=network.target + +[Service] +User=user +WorkingDirectory=/home/user/OpenALPR-Webhook +ExecStart=/home/user/OpenALPR-Webhook/app.py --host=0.0.0.0 --port=8080 +Restart=always + +[Install] +WantedBy=multi-user.target +` + +Be sure to modify `User`, `WorkingDirectory`, and `ExecStart` +
+Then execute: +
+`sudo systemctl daemon-reload` +`sudo systemctl enable oalpr-wh` +`sudo systemctl start oalpr-wh` + ### New Instance -Head over to the URL of your server. You will be required to login. Click on 'register' to create a super admin account. +Head over to the URL of your server. You will be required to login. Click 'register' to create a super admin account.
-After creating a super admin account, the register link will disappear as a protective measure against unauthorized account creation. +After creating a super admin account, the register link will throw an 'Access Denied' as a protective measure against unauthorized account creation.
-Accounts will need to be created manually by an administrator under Settings/Users +Accounts will need to be created manually by an administrator under Settings/Users. # Documentation ### Dashboard ___ - -### Alerts/Blacklist +The dashboard displays some simple statistics, recent alerts, and license plate captures. +### Alerts/Custom Alerts ___ -### Alerts/Search +Under Alerts/Custom Alerts, users can view alerts handled by OpenALPR-Webhook. +Each user, including administrators and the super administrator, can only view their own custom alerts. +Administrators and the super administrator can add other users as additional contacts to be notified when a match occurs. + +#### Print +Users can print a report by clicking on the print icon on the upper right hand corner. +Printing also allows the report to be saved as a PDF. + +#### Add a custom alert +To add a custom alert, go to Search->License Plates, click on the record, and then click on bell icon + +#### Region Match +Users have the ability to enable Match Region while adding a custom alert in Search->License Plates or
+Enabling this will tell OpenALPR-Webhook to require a region match of the +license plate for it to send a notification. The non-matched record will still appear in the Past History section under +Alerts/Custom Alerts or Search/License Plates. +### Alerts/Rekor™ Scout ___ +Rekor™ Scout alerts arrive from Rekor as alerts. You cannot modify the alert in OpenALPR-Webhook, to modify these alerts, ### Search ___ +Note: The Vehicle Information section for each report contains specifications of the type of vehicle that includes make, model, year, and body type. +OpenALPR-Webhook does not generate this data. This data is generated by Rekor Watchman Agent. +#### License Plates +View and search license plates grouped with vehicle details. +###### API_KEY +This field displays the last four characters of the API key that was used to submit this record. +This is useful for administrators to perform a reverse search of the user that is responsible for Rekor POSTing data. + + +#### Vehicles +View and search vehicles that did not have a license plate detected. +___ ### Settings/Agents ___ > Available to administrators only. > +Edit agent connection details here. These settings allow OpenALPR-Webhook to download high resolution images directly from the agent. +Users are not allowed to delete or add agents manually. Agents are registered as new agents are discovered by OpenALPR-Webhook. +Administrators can enable them after being registered for OpenALPR-Webhook to utilize. ### Settings/Cameras ___ > Available to administrators only. > +Similar to Settings/Agents. This section allows to specify connection details for each camera. +These settings are used to forcefully focus and zoom a camera at a specified interval. ### Settings/General ___ > Available to administrators only. > -Disable uuid_img download (pulls from agent if available real-time when viewing/printing reports) +#### Report Settings +These settings are used to rebrand generated reports using the print function. +#### IPBan Settings +An extended addon for +#### POST Auth Settings +###### Disable POST +Suspend all POSTing to OpenALPR-Webhook. +###### No Authorization Required +Highly unrecommended. This allows anyone (or thing) to POST data into OpenALPR-Webhook. This is a security issue as it +allows untrusted data into OpenALPR-Webhook. +###### Users & Admin API Tokens +The second-best option. This allows every user to POST data into OpenALPR-Webhook. +###### Admin API Tokens +The default option. Only data from Rekor that contains an admin's `API_KEY` is allowed to POST. +#### General Settings +###### Public URL +Specify the public URL used to access OpenALPR-Webhook. +Although not used by OpenALPR-Webhook at the moment, certain features that are yet to be implemented will require a valid URL. ### Settings/Maintenance/App ___ -Reinitiate cache -
-Redownload all missing uuid_imgs to db/locally -
-Trim database to keep X months of plates -
-Trim database to keep x months of high-res/uuid_imgs -
-Remove all high-res/uuid_imgs from db > Available to administrators only. > +#### Worker Manager Server +The Worker Manager Server is responsible for spawning and terminating Redis workers as needed. One worker is spawned for every agent and camera that is enabled. This allows OpenALPR-Webhook to scale as needed without interruptions. +Because Redis forks workers on the process level, the Worker Manager Server only runs on *nix systems. + + +##### Restart +Restart the server when experiencing issues with worker allocation. This will not restart the webserver. + + +##### Shutdown +Shutting down the server is essential when performing a soft restart on the webserver. This makes sure that no worker turns to a zombie. + + +These actions are not needed when performing a system reboot. ### Settings/Maintenance/Redis ___ > Available to administrators only. > +A front end to Redis server(s). This was made possible with [rq-dashboard](https://github.com/Parallels/rq-dashboard). +For each agent enabled, a worker is spawned and listens on the default queue. +Unlike for agents, a worker is spawned for each enabled camera and listens to a queue named the ID of the camera. +#### Queues +View a list of queues with job status +#### Jobs +View a list of jobs. +Note: It is advisable that no job, other than those of type `download_plate_image()`, be re-queued. +#### Workers +View a list of workers, the current job, and its associated queues ### Settings/Profile ___ Users can edit basic information about themselves such as name, website, email address, phone number, time zone, etc. -
+

Each user has a unique `API_KEY`. The `API_KEY` key used to authorize Rekor Scout to POST data onto the webhook endpoint. -
+

Administrators can set a global setting to limit which `API_KEY`'s can POST data. Refer to Settings/General. +

To begin receiving data into OpenALPR-Webhook, copy your `API_KEY` into [Rekor Scout](cloud.openalpr.com) > Configuration > WebHooks Configuration > Add New Webhook > Custom Data
`API_KEY: vvvvvvvv-wwww-xxxx-yyyy-zzzzzzzzzzzz`
-Be sure to fill in all other fields such as Destination URL, Description, check Send All Plate Reads, check Send Matching Alerts, and check Send Reads missing plate. +Be sure to fill in all other fields such as Destination URL, Description, check Send All Plate Reads, Send Matching Alerts, and Send Reads missing plate. +![Custom Data field](media/API_KEY.png) ### Settings/Notifications ___ > Available to administrators only. > +Specify notification settings for Twilio and SMTP. Valid SMTP settings are required to reset user passwords in Settings/Users. ### Settings/Users ___ > Available to administrators only. @@ -127,4 +240,17 @@ ___ Administrators can create users, edit users, change user roles, and suspend user accounts.
Once an account has been created, it cannot be deleted. This is to preserve accounts and their API tokens for audit -purposes. \ No newline at end of file +purposes (a feature yet to be implemented). + +#### Edit User +###### Status +The super administrator account cannot be suspended. + +###### Administrator +The super administrator account cannot be demoted. + +###### Reset Password +A valid SMTP server is required to reset passwords. A new generated password will be emailed to the user. + +#### Create/Add User +Click on the 'User+' icon located in upper right-hand corner to add a user. diff --git a/app.py b/app.py index 7aeca84..3a371fb 100644 --- a/app.py +++ b/app.py @@ -27,11 +27,12 @@ if DEBUG: - app.logger.info('DEBUG = ' + str(DEBUG)) - app.logger.info('Page Compression = ' + 'FALSE' if DEBUG else 'TRUE') - app.logger.info('DBMS = ' + app_config.SQLALCHEMY_DATABASE_URI) + app.logger.info("DEBUG = {}".format(str(DEBUG))) + app.logger.info("Page Compression = {}".format('FALSE' if DEBUG else 'TRUE')) + app.logger.info("DBMS = {}".format(app_config.SQLALCHEMY_DATABASE_URI)) if __name__ == "__main__": with app.app_context(): - app.run(host="0.0.0.0", port=8080) + # app.run(host="0.0.0.0", port=8080, debug=True, passthrough_errors=True, use_debugger=False, use_reloader=False) + app.run(host="0.0.0.0", port=8080) \ No newline at end of file diff --git a/apps/__init__.py b/apps/__init__.py index c85aea6..fc195c4 100644 --- a/apps/__init__.py +++ b/apps/__init__.py @@ -1,8 +1,11 @@ -import os.path +import logging +import os import platform import subprocess +import time from datetime import datetime +import setproctitle from flask_ipban import IpBan from flask_migrate import Migrate from redis import Redis @@ -13,15 +16,17 @@ from importlib import import_module from flask_mail import Mail +import log import version from apps.alpr.ipban_config import IPBanConfig -from apps.alpr.enums import WorkerType +from worker_manager import WorkerManager +from worker_manager_enums import WorkerType, WMSCommand +setproctitle.setproctitle("OpenALPR-Webhook") mail = Mail() db = SQLAlchemy() migrate = Migrate() default_q = Queue(WorkerType.General.value, connection=Redis()) -camera_q = Queue(WorkerType.Camera.value, connection=Redis()) Base = declarative_base() login_manager = LoginManager() ip_ban_config = IPBanConfig() @@ -41,7 +46,7 @@ def register_blueprints(app): 'alpr.routes.settings.cameras', 'alpr.routes.settings.general', 'alpr.routes.settings.maintenance', 'alpr.routes.settings.maintenance.rq_dashboard', 'alpr.routes.settings.notifications', 'alpr.routes.settings.profile', - 'alpr.routes.settings.users', 'home'): + 'alpr.routes.settings.users', 'alpr.routes.vehicle', 'home'): module = import_module('apps.{}.routes'.format(module_name)) app.register_blueprint(module.blueprint) @@ -57,7 +62,7 @@ def initialize_databases(): pass @app.before_first_request - def initialize_settings(): + def initialize_cache(): # Initiate cache when needed from apps.alpr.models.cache import Cache now = datetime.now() @@ -69,6 +74,8 @@ def initialize_settings(): Cache.filter_by_year(this_year) Cache.filter_by_year(next_year) + @app.before_first_request + def initialize_settings(): # Create default settings when needed from apps.alpr.models.settings import GeneralSettings settings = GeneralSettings.get_settings() @@ -77,23 +84,56 @@ def initialize_settings(): settings = GeneralSettings() settings.save() - # Start redis workers on Linux only - if platform.system() == "Linux": + @app.before_first_request + def start_workers(): + start_redis_workers() + + +def start_redis_workers(): + # Start redis workers on Linux only + if platform.system() == "Linuxx": + # Worker Manager Server + worker_manager_server = WorkerManager(WMSCommand.ACK) + worker_manager_server.debug = True + try: + subprocess.Popen(['python3', os.path.abspath(os.path.dirname(__file__) + "/..") + + '/worker_manager_server.py']) + time.sleep(3) + worker_manager_server.send() + except Exception as ex: + logging.error(ex) + + if worker_manager_server.last_connection(): + # General workers to download UUID images from agents + from apps.alpr.models.settings import AgentSettings + enabled_agents = AgentSettings.get_all_enabled() + for agent in enabled_agents: + worker_manager_server.command = WMSCommand.START_WORKER + worker_manager_server.worker_type = WorkerType.General + worker_manager_server.worker_id = agent.agent_uid + worker_manager_server.send() + # Cameras from apps.alpr.models.settings import CameraSettings - camera_workers = CameraSettings.get_all_enabled_count() - workers_cmd = subprocess.Popen(["python3", os.path.dirname(os.path.realpath(__file__)) + - "/workers.py", "-c", str(camera_workers), "-g", str(camera_workers)], - stderr=subprocess.PIPE, stdout=subprocess.PIPE) - output, error = workers_cmd.communicate() - if workers_cmd.returncode != 0: - app.logger.info("workers_cmd.returncode = {}".format(workers_cmd.returncode)) - app.logger.info("workers_cmd.errorcode = {}".format(str(error, 'utf-8'))) - - app.logger.info("workers_cmd.stdout = {}".format(workers_cmd.stdout)) - app.logger.info("workers_cmd.stderr = {}".format(workers_cmd.stderr)) + from apps.alpr import queue + enabled_cameras = CameraSettings.get_all_enabled() + try: + for camera in enabled_cameras: + worker_manager_server.command = WMSCommand.START_WORKER + worker_manager_server.worker_type = WorkerType.Camera + worker_manager_server.worker_id = camera.camera_id + worker_manager_server.send() + time.sleep(1) + # Add the function to the queue + q = Queue(camera.camera_id, connection=Redis()) + q.enqueue(queue.focus_camera, args=(camera.camera_id,), job_timeout=-1) + except Exception as ex: + logging.exception(ex) + else: + logging.error("Last connection to Worker Manager Server failed. Could not spin up redis workers!") def create_app(config) -> Flask: + log.init("OpenALPR-Webhook.log") app = Flask(__name__) app.config.from_object(config) mail.init_app(app) diff --git a/apps/alpr/beautify.py b/apps/alpr/beautify.py index 9fd2916..9d3a650 100644 --- a/apps/alpr/beautify.py +++ b/apps/alpr/beautify.py @@ -1,4 +1,5 @@ import json +import logging import time import pycountry @@ -68,11 +69,12 @@ def human_format(num: int) -> str: return '{}{}'.format('{:f}'.format(num).rstrip('0').rstrip('.'), ['', 'K', 'M', 'B', 'T'][magnitude]) -# Thanks to @hostingutilities.com at stackoverflow.com -# https://stackoverflow.com/a/43750422 -def human_size(bytes: int, units=[' bytes','KB','MB','GB','TB', 'PB', 'EB']) -> str: - """ Returns a human readable string representation of bytes """ - return str(bytes) + units[0] if bytes < 1024 else human_size(bytes >> 10, units[1:]) +def human_size(num, suffix="B"): + for unit in ["", "Ki", "Mi", "Gi", "Ti", "Pi", "Ei", "Zi"]: + if abs(num) < 1024.0: + return f"{num:3.1f}{unit}{suffix}" + num /= 1024.0 + return f"{num:.1f}Yi{suffix}" def get_negative_mtm_svg_html_arrow() -> Markup: @@ -103,4 +105,4 @@ def name(ugly_name: str) -> str: def print_json(json_obj: str) -> None: - print(json.dumps(json_obj, ensure_ascii=False, indent=4)) + logging.debug(json.dumps(json_obj, ensure_ascii=False, indent=4)) diff --git a/apps/alpr/enums.py b/apps/alpr/enums.py index 8997b91..141d9fe 100644 --- a/apps/alpr/enums.py +++ b/apps/alpr/enums.py @@ -13,8 +13,9 @@ class AccountVerified(enum.Enum): class ChartType(enum.Enum): ALERT_CHART = "alert-chart" + CUSTOM_ALERT = "custom-alert" PLATES_CAPTURED_CHART = "plates-captured-chart" - TOP_REGION_CHART = "top-region-chart" + TOP_SECOND_REGION_CHART = "top-second-region-chart" class DataType(enum.Enum): @@ -23,18 +24,9 @@ class DataType(enum.Enum): VEHICLE = "vehicle" -class MultiProcessCommand(enum.Enum): - START_WORKER = "start-worker" - STOP_WORKER = "stop-worker" - CLOSE_CONNECTION = "close" - - class UserRole(enum.Enum): ADMIN = "ADMIN" REGULAR = "NONADMIN" -class WorkerType(enum.Enum): - # worker type = queue name - Camera = 'cameras' - General = 'default' + diff --git a/apps/alpr/models/alpr_alert.py b/apps/alpr/models/alpr_alert.py index c56bb68..78bf6e6 100644 --- a/apps/alpr/models/alpr_alert.py +++ b/apps/alpr/models/alpr_alert.py @@ -98,7 +98,7 @@ def filter_by_id_and_beautify(cls, _id: int) -> {}: else: return None - def filter_epoch_time(self) -> "ALPRAlert": + def filter_epoch_time(self) -> ["ALPRAlert"]: self.collection = self.query.filter((ALPRAlert.epoch_time / 1000) >= self.start_of_month_timestamp).filter( (ALPRAlert.epoch_time / 1000) <= self.end_of_month_timestamp).all() return self.collection diff --git a/apps/alpr/models/alpr_group.py b/apps/alpr/models/alpr_group.py index ceed530..6976eb6 100644 --- a/apps/alpr/models/alpr_group.py +++ b/apps/alpr/models/alpr_group.py @@ -71,6 +71,10 @@ def __init__(self, **kwargs): def filter_by_id(cls, _id: int) -> "ALPRGroup": return cls.query.filter_by(id=_id).first() + @classmethod + def filter_by_best_plate_number(cls, best_plate_number: str) -> "[ALPRGroup]": + return cls.query.filter_by(best_plate_number=best_plate_number).order_by(ALPRGroup.id.desc()).all() + @classmethod def filter_by_id_and_beautify(cls, _id: int) -> {}: record = cls.filter_by_id(_id) @@ -96,7 +100,7 @@ def filter_by_id_and_beautify(cls, _id: int) -> {}: 'epoch_start': record.epoch_start, 'epoch_start_datetime': dt.astimezone(record.epoch_start), 'epoch_end': record.epoch_end, - 'epoch_end_datetime': dt.astimezone(record.epoch_start), + 'epoch_end_datetime': dt.astimezone(record.epoch_end), 'best_confidence_percent': record.best_confidence_percent, 'best_region': beautify.country(record.best_region), 'travel_direction_class_tag': beautify.direction(record.travel_direction), @@ -117,17 +121,17 @@ def filter_by_id_and_beautify(cls, _id: int) -> {}: return None @classmethod - def get_latest_agent_label(cls, _agent_uid: int) -> str: + def get_latest_agent_label(cls, _agent_uid: str) -> str: record = cls.query.filter_by(agent_uid=_agent_uid).order_by(ALPRGroup.id.desc()).first() return record.web_server_config['agent_label'] @classmethod - def get_latest_agent_type(cls, _agent_uid: int) -> str: + def get_latest_agent_type(cls, _agent_uid: str) -> str: record = cls.query.filter_by(agent_uid=_agent_uid).order_by(ALPRGroup.id.desc()).first() return record.agent_type @classmethod - def get_latest_agent_version(cls, _agent_uid: int) -> str: + def get_latest_agent_version(cls, _agent_uid: str) -> str: record = cls.query.filter_by(agent_uid=_agent_uid).order_by(ALPRGroup.id.desc()).first() return record.agent_version @@ -151,6 +155,10 @@ def get_latest_camera_gps_longitude(cls, _camera_id: int) -> str: record = cls.query.filter_by(camera_id=_camera_id).order_by(ALPRGroup.id.desc()).first() return record.gps_longitude + @classmethod + def get_latest_by_best_plate_number(cls, best_plate_number: str, limit=2) -> "[ALPRGroup]": + return cls.query.filter_by(best_plate_number=best_plate_number).order_by(ALPRGroup.id.desc()).limit(limit) + @classmethod def get_oldest_agent_epoch_start(cls, _agent_uid: int) -> int: record = cls.query.filter_by(agent_uid=_agent_uid).order_by(ALPRGroup.id.asc()).first() diff --git a/apps/alpr/models/cache.py b/apps/alpr/models/cache.py index 86e2429..6682bf6 100644 --- a/apps/alpr/models/cache.py +++ b/apps/alpr/models/cache.py @@ -1,5 +1,6 @@ import logging import os +import pathlib import platform from datetime import datetime, timedelta @@ -26,20 +27,33 @@ class Cache(db.Model): year = db.Column(db.Integer) all_time_plates_captured = db.Column(db.Integer, default=0) all_time_alerts = db.Column(db.Integer, default=0) + all_time_custom_alerts = db.Column(db.Integer, default=0) all_time_vehicles = db.Column(db.Integer, default=0) month = db.Column(NestedMutableJson, - default=[{"license_plates_captured": 0, "alerts": 0, "vehicles": 0, "cameras": {}, "regions": {}}, - {"license_plates_captured": 0, "alerts": 0, "vehicles": 0, "cameras": {}, "regions": {}}, - {"license_plates_captured": 0, "alerts": 0, "vehicles": 0, "cameras": {}, "regions": {}}, - {"license_plates_captured": 0, "alerts": 0, "vehicles": 0, "cameras": {}, "regions": {}}, - {"license_plates_captured": 0, "alerts": 0, "vehicles": 0, "cameras": {}, "regions": {}}, - {"license_plates_captured": 0, "alerts": 0, "vehicles": 0, "cameras": {}, "regions": {}}, - {"license_plates_captured": 0, "alerts": 0, "vehicles": 0, "cameras": {}, "regions": {}}, - {"license_plates_captured": 0, "alerts": 0, "vehicles": 0, "cameras": {}, "regions": {}}, - {"license_plates_captured": 0, "alerts": 0, "vehicles": 0, "cameras": {}, "regions": {}}, - {"license_plates_captured": 0, "alerts": 0, "vehicles": 0, "cameras": {}, "regions": {}}, - {"license_plates_captured": 0, "alerts": 0, "vehicles": 0, "cameras": {}, "regions": {}}, - {"license_plates_captured": 0, "alerts": 0, "vehicles": 0, "cameras": {}, "regions": {}}] + default=[{"license_plates_captured": 0, "alerts": 0, "custom_alerts": 0, "vehicles": 0, + "cameras": {}, "regions": {}}, + {"license_plates_captured": 0, "alerts": 0, "custom_alerts": 0, "vehicles": 0, + "cameras": {}, "regions": {}}, + {"license_plates_captured": 0, "alerts": 0, "custom_alerts": 0, "vehicles": 0, + "cameras": {}, "regions": {}}, + {"license_plates_captured": 0, "alerts": 0, "custom_alerts": 0, "vehicles": 0, + "cameras": {}, "regions": {}}, + {"license_plates_captured": 0, "alerts": 0, "custom_alerts": 0, "vehicles": 0, + "cameras": {}, "regions": {}}, + {"license_plates_captured": 0, "alerts": 0, "custom_alerts": 0, "vehicles": 0, + "cameras": {}, "regions": {}}, + {"license_plates_captured": 0, "alerts": 0, "custom_alerts": 0, "vehicles": 0, + "cameras": {}, "regions": {}}, + {"license_plates_captured": 0, "alerts": 0, "custom_alerts": 0, "vehicles": 0, + "cameras": {}, "regions": {}}, + {"license_plates_captured": 0, "alerts": 0, "custom_alerts": 0, "vehicles": 0, + "cameras": {}, "regions": {}}, + {"license_plates_captured": 0, "alerts": 0, "custom_alerts": 0, "vehicles": 0, + "cameras": {}, "regions": {}}, + {"license_plates_captured": 0, "alerts": 0, "custom_alerts": 0, "vehicles": 0, + "cameras": {}, "regions": {}}, + {"license_plates_captured": 0, "alerts": 0, "custom_alerts": 0, "vehicles": 0, + "cameras": {}, "regions": {}}] ) def __init__(self, year=datetime.now().year, month=datetime.now().month): @@ -67,10 +81,24 @@ def get_alert_count(self, year: int, month: int) -> int: return cache.month[month - 1]['alerts'] + def get_combined_alert_count(self, year: int, month: int) -> int: + cache = self.filter_by_year(year) + if cache is None: + return 0 + + return cache.month[month - 1]['alerts'] + cache.month[month - 1]['custom_alerts'] + + def get_custom_alert_count(self, year: int, month: int) -> int: + cache = self.filter_by_year(year) + if cache is None: + return 0 + + return cache.month[month - 1]['custom_alerts'] + def get_all_db_file_sizes(self, raw=False) -> str: if raw: - return str(os.path.getsize("apps/db/alpr_alert.sqlite") + \ - os.path.getsize("apps/db/alpr_group.sqlite") + \ + return str(os.path.getsize("apps/db/alpr_alert.sqlite") + + os.path.getsize("apps/db/alpr_group.sqlite") + os.path.getsize("apps/db/vehicle.sqlite")) return beautify.human_size(os.path.getsize("apps/db/alpr_alert.sqlite") + @@ -116,21 +144,24 @@ def get_chart_series(self, chart: ChartType) -> []: last_month = 12 for i in range(12): - if chart == ChartType.PLATES_CAPTURED_CHART or chart == ChartType.TOP_REGION_CHART: + if chart == ChartType.PLATES_CAPTURED_CHART or chart == ChartType.TOP_SECOND_REGION_CHART: # License Plates Captured this_month_license_plate_count = self.get_license_plate_count(year, last_month) if chart == ChartType.PLATES_CAPTURED_CHART: series.append(this_month_license_plate_count) - elif chart == ChartType.TOP_REGION_CHART: + elif chart == ChartType.TOP_SECOND_REGION_CHART: # Top Region - this_month_top_region_count = self.get_top_region_count(year, last_month) - series.append(this_month_top_region_count) + this_month_top_second_region_count = self.get_top_second_region_count(year, last_month) + series.append(this_month_top_second_region_count) elif chart == ChartType.ALERT_CHART: # Alerts this_month_alert_count = self.get_alert_count(year, last_month) series.append(this_month_alert_count) + elif chart == ChartType.CUSTOM_ALERT: + this_month_custom_alert_count = self.get_custom_alert_count(year, last_month) + series.append(this_month_custom_alert_count) # Move to previous month last_month -= 1 @@ -152,8 +183,8 @@ def get_chart_labels(self, chart=None) -> []: for i in range(12): start_of_month = datetime(year=year, month=month, day=1) - if chart == ChartType.TOP_REGION_CHART: - pretty_region = beautify.country(self.get_top_region(year, month)) + if chart == ChartType.TOP_SECOND_REGION_CHART: + pretty_region = beautify.country(self.get_top_second_region(year, month)) series.append("{} - {}".format(start_of_month.strftime("%b %Y"), pretty_region)) else: series.append(start_of_month.strftime("%b %Y")) @@ -215,31 +246,33 @@ def get_quick_stats(self) -> {}: year -= 1 last_month = 12 - this_month_alert_count = self.get_alert_count(year, month) + this_month_alert_count = self.get_combined_alert_count(year, month) this_month_license_plate_count = self.get_license_plate_count(year, month) # print("this_month_license_plate_count = {}".format(this_month_license_plate_count)) - this_month_top_region = self.get_top_region(year, month) - # print("this_month_top_region = {}".format(this_month_top_region)) - this_month_top_region_count = self.get_top_region_count(year, month) - # print("this_month_top_region_count = {}".format(this_month_top_region_count)) + this_month_top_second_region = self.get_top_second_region(year, month) + # print("this_month_top_second_region = {}".format(this_month_top_second_region)) + this_month_top_second_region_count = self.get_top_second_region_count(year, month) + # print("this_month_top_second_region_count = {}".format(this_month_top_second_region_count)) response['this_month_alert_count'] = this_month_alert_count + response['this_month_rekor_alert_count'] = self.get_alert_count(year, month) + response['this_month_custom_alert_count'] = self.get_custom_alert_count(year, month) response['this_month_license_plate_count'] = this_month_license_plate_count - response['this_month_top_region'] = beautify.country(this_month_top_region) - response['this_month_top_region_count'] = this_month_top_region_count + response['this_month_top_second_region'] = beautify.country(this_month_top_second_region) + response['this_month_top_second_region_count'] = this_month_top_second_region_count last_month_alert_count = self.get_alert_count(year, last_month) last_month_license_plate_count = self.get_license_plate_count(year, last_month) # print("last_month_license_plate_count = {}".format(last_month_license_plate_count)) - last_month_top_region = self.get_top_region(year, last_month) - # print("last_month_top_region = {}".format(last_month_top_region)) - last_month_top_region_count = self.get_region_count(year, last_month, this_month_top_region) - # print("last_month_top_region_count = {}".format(last_month_top_region_count)) + last_month_top_second_region = self.get_top_second_region(year, last_month) + # print("last_month_top_second_region = {}".format(last_month_top_second_region)) + last_month_top_second_region_count = self.get_region_count(year, last_month, this_month_top_second_region) + # print("last_month_top_second_region_count = {}".format(last_month_top_second_region_count)) response['last_month_alert_count'] = last_month_alert_count response['last_month_license_plate_count'] = last_month_license_plate_count - response['last_month_top_region'] = beautify.country(last_month_top_region) - response['last_month_top_region_count'] = last_month_top_region_count + response['last_month_top_second_region'] = beautify.country(last_month_top_second_region) + response['last_month_top_second_region_count'] = last_month_top_second_region_count # Calculate Month-To-Month for the number of plates captured if last_month_license_plate_count != 0: @@ -289,29 +322,37 @@ def get_quick_stats(self) -> {}: response["this_month_alerts_mtm_percent"] = mtm_alerts_percent - # Calculate Month-To-Month for the top region - if last_month_top_region != "N/A" and this_month_top_region != "N/A": - mtm_region_percent = ((this_month_top_region_count / last_month_top_region_count) - 1) * 100 - mtm_region_percent = round(mtm_region_percent, 1) - # Determine the color/class of the percentage - if mtm_region_percent < 0: - response['this_month_top_region_mtm_class'] = "text-danger" - response['this_month_top_region_mtm_svg_html_arrow'] = beautify.get_negative_mtm_svg_html_arrow() - elif mtm_region_percent == 0: - response['this_month_top_region_mtm_class'] = "" - elif mtm_region_percent > 0: - response['this_month_top_region_mtm_class'] = "text-success" - response['this_month_top_region_mtm_svg_html_arrow'] = beautify.get_positive_mtm_svg_html_arrow() - elif last_month_top_region_count == 0 and this_month_top_region_count != 0: + # Calculate Month-To-Month for the top second region + if last_month_top_second_region != "N/A" and this_month_top_second_region != "N/A": + if last_month_top_second_region == this_month_top_second_region: + mtm_region_percent = \ + ((this_month_top_second_region_count / last_month_top_second_region_count) - 1) * 100 + mtm_region_percent = round(mtm_region_percent, 1) + # Determine the color/class of the percentage + if mtm_region_percent < 0: + response['this_month_top_second_region_mtm_class'] = "text-danger" + response['this_month_top_second_region_mtm_svg_html_arrow'] = \ + beautify.get_negative_mtm_svg_html_arrow() + elif mtm_region_percent == 0: + response['this_month_top_second_region_mtm_class'] = "" + elif mtm_region_percent > 0: + response['this_month_top_second_region_mtm_class'] = "text-success" + response['this_month_top_second_region_mtm_svg_html_arrow'] = \ + beautify.get_positive_mtm_svg_html_arrow() + else: + mtm_region_percent = 100 + response['this_month_top_second_region_mtm_class'] = "text-success" + response['this_month_top_second_region_mtm_svg_html_arrow'] = beautify.get_positive_mtm_svg_html_arrow() + elif last_month_top_second_region_count == 0 and this_month_top_second_region_count != 0: mtm_region_percent = 100 - response['this_month_top_region_mtm_class'] = "text-success" - response['this_month_top_region_mtm_svg_html_arrow'] = beautify.get_positive_mtm_svg_html_arrow() + response['this_month_top_second_region_mtm_class'] = "text-success" + response['this_month_top_second_region_mtm_svg_html_arrow'] = beautify.get_positive_mtm_svg_html_arrow() else: mtm_region_percent = 0 - response['this_month_top_region_mtm_class'] = "" - response['this_month_top_region_mtm_svg_html_arrow'] = "" + response['this_month_top_second_region_mtm_class'] = "" + response['this_month_top_second_region_mtm_svg_html_arrow'] = "" - response["this_month_top_region_mtm_percent"] = mtm_region_percent + response["this_month_top_second_region_mtm_percent"] = mtm_region_percent return response @@ -403,7 +444,15 @@ def get_top_region(self, year, month) -> str: return "N/A" length = len(list(cache.month[month - 1]['regions'])) - return str(list(cache.month[month - 1]['regions'].keys())[1]) if length != 0 else "N/A" + return str(list(cache.month[month - 1]['regions'].keys())[0]) if length != 0 else "N/A" + + def get_top_second_region(self, year, month) -> str: + cache = self.filter_by_year(year) + if cache is None: + return "N/A" + + length = len(list(cache.month[month - 1]['regions'])) + return str(list(cache.month[month - 1]['regions'].keys())[1]) if length >= 1 else "N/A" def get_top_region_count(self, year: int, month: int) -> int: cache = self.filter_by_year(year) @@ -411,7 +460,15 @@ def get_top_region_count(self, year: int, month: int) -> int: return 0 length = len(list(cache.month[month - 1]['regions'])) - return int(list(cache.month[month - 1]['regions'].values())[1]) if length != 0 else 0 + return int(list(cache.month[month - 1]['regions'].values())[0]) if length != 0 else 0 + + def get_top_second_region_count(self, year: int, month: int) -> int: + cache = self.filter_by_year(year) + if cache is None: + return 0 + + length = len(list(cache.month[month - 1]['regions'])) + return int(list(cache.month[month - 1]['regions'].values())[1]) if length >= 1 else 0 def init(self): try: @@ -504,7 +561,8 @@ def init(self): agent_cache = AgentCache.filter_by_agent_uid(agent_uid) if agent_settings is None: agent_settings = AgentSettings(agent_uid, alpr_group.get_latest_agent_label(agent_uid)) - agent_settings.created = datetime.fromtimestamp(ALPRGroup.get_oldest_agent_epoch_start(agent_uid) / 1000) + agent_settings.created = \ + datetime.fromtimestamp(ALPRGroup.get_oldest_agent_epoch_start(agent_uid) / 1000) agent_settings.save() if agent_cache is None: agent_cache = AgentCache(agent_uid, alpr_group.get_latest_agent_label(agent_uid)) @@ -514,7 +572,6 @@ def init(self): agent_cache.agent_type = alpr_group.get_latest_agent_type(agent_uid) agent_cache.save() - # Populate cache cache.all_time_plates_captured += this_month_license_plate_count cache.all_time_alerts += this_month_alert_count @@ -651,35 +708,3 @@ def save(self) -> None: db.session.close() error = str(e.__dict__['orig']) raise InvalidUsage(error, 422) - - -class Counter(db.Model): - __bind_key__ = 'cache' - __tablename__ = 'Counter' - - id = db.Column(db.Integer, primary_key=True) - key = db.Column(db.String) - count = db.Column(db.Integer) - - def __init__(self, key: str): - self.key = key - self.count = 0 - - @classmethod - def filter_by_key(cls, key: str) -> "Counter": - return cls.query.filter_by(key=key).first() - - def one_down(self): - self.count -= 1 - - def one_up(self): - self.count += 1 - - def save(self) -> None: - try: - db.session.add(self) - db.session.commit() - except SQLAlchemyError as e: - db.session.rollback() - db.session.close() - logging.exception(e) diff --git a/apps/alpr/models/custom_alert.py b/apps/alpr/models/custom_alert.py index 2d75611..6686a92 100644 --- a/apps/alpr/models/custom_alert.py +++ b/apps/alpr/models/custom_alert.py @@ -1,4 +1,4 @@ -from datetime import datetime +import logging from flask_login import current_user @@ -95,25 +95,38 @@ def filter_by_license_plate(cls, _license_plate: str) -> "CustomAlert": def filter_by_submitted_user_id(cls, _submitted_by_user_id: int) -> ["CustomAlert"]: return cls.query.filter_by(submitted_by_user_id=_submitted_by_user_id).all() - def get_dashboard_records(self, current_user, n=3) -> []: - custom_alerts = self.query.filter_by(submitted_by_user_id=current_user.id).order_by(CustomAlert.id.desc()).limit(n) + def get_dashboard_records(self, n=3) -> []: + custom_alerts = \ + self.query.filter_by(submitted_by_user_id=current_user.id).order_by(CustomAlert.id.desc()).limit(n) data = [] dt = helper.Timezone(current_user) - for record in custom_alerts: - alpr_group = ALPRGroup.filter_by_id(record.alpr_group_id) - data.append({ - 'id': record.id, - 'month': dt.month(alpr_group.epoch_start), - 'day': dt.day(alpr_group.epoch_start), - 'plate_number': record.license_plate, - 'epoch_time_datetime': dt.astimezone(alpr_group.epoch_start), - 'site_name': alpr_group.web_server_config['agent_label'], - 'camera_name': alpr_group.web_server_config['camera_label'], - 'description': helper.shorten_description(record.description) - }) + for alert in custom_alerts: + alpr_group = ALPRGroup.get_latest_by_best_plate_number(alert.license_plate) + for record in alpr_group: + data.append({ + 'id': record.id, + 'month': dt.month(record.epoch_start), + 'day': dt.day(record.epoch_start), + 'plate_number': record.best_plate_number, + 'epoch_time_datetime': dt.astimezone(record.epoch_start), + 'site_name': record.web_server_config['agent_label'], + 'camera_name': record.web_server_config['camera_label'], + 'description': helper.shorten_description(alert.description) + }) return data + def delete(self) -> None: + try: + db.session.delete(self) + db.session.commit() + except SQLAlchemyError as e: + db.session.rollback() + db.session.close() + error = str(e.__dict__['orig']) + raise InvalidUsage(error, 422) + return + def save(self) -> None: try: db.session.add(self) diff --git a/apps/alpr/models/settings.py b/apps/alpr/models/settings.py index 8f0d509..5330c79 100644 --- a/apps/alpr/models/settings.py +++ b/apps/alpr/models/settings.py @@ -51,9 +51,13 @@ def filter_by_id(cls, id: str) -> "AgentSettings": return cls.query.filter_by(id=id).first() @classmethod - def get_all(cls, _agent_uid: str) -> "AgentSettings": + def get_all(cls, _agent_uid: str) -> ["AgentSettings"]: return cls.query.all() + @classmethod + def get_all_enabled(cls) -> ["AgentSettings"]: + return cls.query.filter_by(enabled=True) + def save(self) -> None: try: db.session.add(self) @@ -98,13 +102,9 @@ def filter_by_id(cls, _id: int) -> "CameraSettings": return cls.query.filter_by(id=_id).first() @classmethod - def get_all_enabled(cls) -> []: + def get_all_enabled(cls) -> ["CameraSettings"]: return cls.query.filter_by(enable=True) - @classmethod - def get_all_enabled_count(cls) -> int: - return cls.query.filter_by(enable=True).count() - @classmethod def get_camera_label(cls, _camera_id: int) -> "CameraSettings": camera = cls.query.filter_by(camera_id=_camera_id).first() @@ -142,7 +142,7 @@ class EmailNotificationSettings(db.Model): recipients = db.Column(db.String) @classmethod - def get_recipients(cls) -> []: + def get_recipients(cls) -> [str]: settings = cls.query.filter_by(id=id).first() recipients = settings.recipients @@ -176,7 +176,7 @@ class GeneralSettings(db.Model): logo = db.Column(db.String, default=default_org_logo) org_name = db.Column(db.String, default="OpenALPR-Webhook") post_auth = db.Column(db.Enum(PostAuth), default=PostAuth.ADMINS_ONLY) - public_url = db.Column(db.String) + public_url = db.Column(db.String, default="https://openalpr-webhook") @classmethod def get_settings(cls) -> "GeneralSettings": diff --git a/apps/alpr/models/vehicle.py b/apps/alpr/models/vehicle.py index 8eee21d..d6aef01 100644 --- a/apps/alpr/models/vehicle.py +++ b/apps/alpr/models/vehicle.py @@ -4,6 +4,9 @@ from apps import db, helpers from sqlalchemy.ext.mutable import MutableDict + +from apps.alpr import beautify +from apps.alpr.models.alpr_group import ALPRGroup from apps.exceptions.exception import InvalidUsage @@ -70,7 +73,50 @@ def filter_by_id(cls, _id: int) -> "Vehicle": return cls.query.filter_by(id=_id).first() @classmethod - def filter_epoch_start(self) -> "Vehicle": + def filter_by_id_and_beautify(cls, _id: int) -> {}: + record = cls.filter_by_id(_id) + + if record: + dt = helpers.Timezone(current_user, msecs=True) + return { + 'uuid_jpg': record.uuid_jpg, + 'overview_jpeg': record.overview_jpeg, + 'vehicle_crop_jpeg': record.vehicle_crop_jpeg, + 'agent_label': ALPRGroup.get_latest_agent_label(record.agent_uid), + 'agent_uid': record.agent_uid, + 'agent_version': record.agent_version, + 'agent_type': record.agent_type, + 'camera_label': ALPRGroup.get_latest_camera_label(record.camera_id), + 'camera_id': record.camera_id, + 'gps_latitude': record.gps_latitude, + 'gps_longitude': record.gps_longitude, + 'country': beautify.name(record.country), + 'id': record.id, + 'epoch_start': record.epoch_start, + 'epoch_start_datetime': dt.astimezone(record.epoch_start), + 'epoch_end': record.epoch_end, + 'epoch_end_datetime': dt.astimezone(record.epoch_end), + 'is_vehicle_confidence_percent': round(float(record.vehicle['is_vehicle'][0]['confidence']), 0), + 'travel_direction_class_tag': beautify.direction(record.travel_direction), + 'travel_direction': round(float(record.travel_direction), 0), + 'vehicle_color_name': record.vehicle_color_name, + 'vehicle_color_confidence': record.vehicle_color_confidence, + 'vehicle_year_name': record.vehicle_year_name, + 'vehicle_year_confidence': record.vehicle_year_confidence, + 'vehicle_make_name': record.vehicle_make_name, + 'vehicle_make_confidence': record.vehicle_make_confidence, + 'vehicle_make_model_name': record.vehicle_make_model_name, + 'vehicle_make_model_confidence': record.vehicle_make_model_confidence, + 'vehicle_body_type_name': record.vehicle_body_type_name, + 'vehicle_body_type_confidence': record.vehicle_body_type_confidence, + 'vehicle_signature': record.vehicle_signature + } + + else: + return None + + @classmethod + def filter_epoch_start(self) -> ["Vehicle"]: self.collection = self.query.filter((Vehicle.epoch_start / 1000) >= self.start_of_month_timestamp).filter( (Vehicle.epoch_start / 1000) <= self.end_of_month_timestamp).all() return self.collection @@ -98,6 +144,5 @@ def save(self) -> None: except Exception as e: db.session.rollback() db.session.close() - print(e) error = str(e.__dict__['orig']) raise InvalidUsage(error, 422) diff --git a/apps/alpr/notify.py b/apps/alpr/notify.py index a1e0e4b..4608fe4 100644 --- a/apps/alpr/notify.py +++ b/apps/alpr/notify.py @@ -1,4 +1,5 @@ import enum +import logging import smtplib import ssl from email.mime.text import MIMEText @@ -29,11 +30,11 @@ def __init__(self): self._settings = EmailNotificationSettings.get_settings() self.recipients = self._settings.recipients - def send(self) -> None: + def send(self) -> bool: # Stop if Email is disabled if not self._settings.enabled: - return + return False try: # Split the string to create a list: user1@example.com,user2@example.com -> @@ -47,15 +48,17 @@ def send(self) -> None: msg = MIMEText(self.body) msg['To'] = recipient msg['From'] = self._settings.username_email - msg['Subject'] = "[{}] OpenALPR-Webhook: {}".format(self.tag, self.subject) + msg['Subject'] = "[{}] {} - OpenALPR-Webhook".format(self.tag, self.subject) server.login(self._settings.username_email, self._settings.password) server.send_message(msg) except Exception as ex: - raise ex + logging.exception(ex) + return False + return True def send_test(self) -> None: try: - self.tag = Tag.TEST + self.tag = Tag.TEST.value self.subject = "SMTP Test" self.body = "This is a test 🧪 message from OpenALPR-Web🪝!" self.send() diff --git a/apps/alpr/queue.py b/apps/alpr/queue.py index a045f18..b9f16f7 100644 --- a/apps/alpr/queue.py +++ b/apps/alpr/queue.py @@ -40,27 +40,32 @@ def download_plate_image(agent_uid: str, img_uuid: str, data_type: str, foreign_ # Get agent IP/hostname & port agent = AgentSettings.filter_by_agent_uid(agent_uid) + + download_dir = os.path.abspath(os.path.dirname(__file__) + "../../../") + "/" + download_folder_name + + # Create directory when needed + if not os.path.exists(download_dir): + os.makedirs(download_dir) + + # Email object email = Email() - email.tag = "Agent" + email.tag = Tag.AGENT.value + email.subject = "Plate Image Save Failed" + email.recipients = EmailNotificationSettings.recipients if agent is None: - logging.info("Agent does not exist.") - email.subject = "Unknown Agent" - email.body = "Failed to download high resolution image. Agent does not exist." \ - "\n\nAgent UID: {}\nIMG UID: {}".format(agent_uid, img_uuid) - email.send() - raise Exception("Failed to download a plate image. Agent does not exist.") + logging.info("Agent (ID: {}) does not exist.".format(agent_uid)) + raise Exception("Failed to download a plate image. Agent (ID: {}) does not exist.".format(agent_uid)) elif agent.enabled: + url = "http://{}:{}/img/{}.jpg".format(agent.ip_hostname, agent.port, img_uuid) try: - url = "http://{}:{}/img/{}.jpg".format(agent.ip_hostname, agent.port, img_uuid) logging.info("Downloading: {}".format(url)) # Download it req = requests.get(url, stream=True) if req.status_code == 200: # Create a path to save it to - full_file_path = Path(os.path.abspath(os.path.dirname(__file__) + "../../../") + "/" + - download_folder_name + img_uuid + ".jpg").absolute() + full_file_path = Path(download_dir + img_uuid + ".jpg").absolute() # Write to disk with open(full_file_path, 'wb') as jpg: shutil.copyfileobj(req.raw, jpg) @@ -68,63 +73,74 @@ def download_plate_image(agent_uid: str, img_uuid: str, data_type: str, foreign_ else: logging.info("Failed to download high resolution plate image. HTTP status_code={}".format( req.status_code)) - email.subject = "High Resolution Image Download Failed" - email.body = "Failed to download high resolution image. HTTP status code from agent was not 200.\n\n" \ - "Agent UID: {}\nIMG UID: {}\nHTTP Status: {}".format(agent_uid, img_uuid, - req.status_code) + + email.body = "⚠️ Failed to download high resolution image. " \ + "HTTP status code from agent was not 200.\n\nLabel: {}\nUID: {}\n" \ + "IMG UID: {}\nURL: {}\nHTTP Status: {}".format( + agent.agent_label, agent_uid, img_uuid, url, req.status_code) email.send() - raise Exception("Failed to download the plate image. HTTP status_code={}".format(req.status_code)) + + raise Exception("Failed to download high resolution plate image. HTTP status_code={}".format( + req.status_code)) except Exception as ex: logging.info("Failed to download the plate image") - email.subject = "High Resolution Image Download Failed" - email.body = "Failed to download high resolution image. Incorrect IP/hostname & port? Make sure" \ - "OpenALPR-Webhook can access the agent.\n\nAgent UID: {}\nIMG UID: {}\n" \ - "Exception: {}".format(agent_uid, img_uuid, ex) + + email.body = "⚠️ Failed to download high resolution image. Incorrect IP/hostname & port? Make sure" \ + "OpenALPR-Webhook can access the agent.\n\nLabel: {}\nUID: {}\nIMG UID: {}\n" \ + "URL: {}\nException: {}".format(agent.agent_label, agent_uid, img_uuid, url, ex) email.send() raise Exception(ex) # Find the original record record = None + dt = DataType(data_type) - if data_type == DataType.GROUP: + if dt == DataType.GROUP: record = ALPRGroup.filter_by_id(foreign_id) - elif data_type == DataType.ALERT: + elif dt == DataType.ALERT: record = ALPRAlert.filter_by_id(foreign_id) - elif data_type == DataType.VEHICLE: + elif dt == DataType.VEHICLE: record = Vehicle.filter_by_id(foreign_id) + elif dt is None: + raise Exception("Unknown data_type = {}".format(data_type)) - # Insert the image in the db - try: - # Read it while encoding it into base64 - with open(full_file_path, "rb") as image_file: - encoded_string = base64.b64encode(image_file.read()) - uuid_jpg = encoded_string.decode("utf-8") + if record is not None: + # Insert the image in the db + try: + # Read it while encoding it into base64 + with open(full_file_path, "rb") as image_file: + encoded_string = base64.b64encode(image_file.read()) + uuid_jpg = encoded_string.decode("utf-8") - # Insert into `uuid_jpg column` - record.uuid_jpg = uuid_jpg + # Insert into `uuid_jpg column` + record.uuid_jpg = uuid_jpg - # Save/update the db - record.save() + # Save/update the db + record.save() - # Delete the .jpg afterwards - if os.path.isfile(full_file_path): - os.remove(full_file_path) - return True - except Exception as ex: - logging.exception(ex) - logging.info("Failed to save the plate image") - EmailNotificationSettings.send("App", "Plate Image Save Failed", - "Failed to save plate image into the database." - "\n\nAgent UID: {}\nIMG UID: {}\nException: {}".format(agent_uid, img_uuid, - ex)) - # Delete the jpg if something happens above - # if os.path.isfile(full_file_path): - # os.remove(full_file_path) + # Delete the .jpg afterwards + if os.path.isfile(full_file_path): + os.remove(full_file_path) + return True + except Exception as ex: + logging.exception(ex) + logging.info("Failed to save the plate image") + + email.body = "⚠️ Failed to download high resolution image from agent.\n\n" \ + "Label: {}\nUID: {}\nImage UID: {}\nException: {}\n".format(agent.agent_label, + agent.agent_uid, img_uuid, ex) + email.send() + + # Delete the jpg if something happens above + if os.path.isfile(full_file_path): + os.remove(full_file_path) + raise Exception(ex) + else: + logging.error("Record #{} of type {} not found. dt = {}".format(foreign_id, data_type, dt)) - raise Exception(ex) elif not agent.enabled: - logging.info("Agent is disabled.") + logging.info("Agent (Label: {}) is disabled.".format(agent.agent_label)) def focus_camera(camera_id: int): @@ -133,6 +149,12 @@ def focus_camera(camera_id: int): app = create_app(app_config) app.app_context().push() + # Email + email = Email() + email.tag = Tag.CAMERA.value + email.subject = "Force Focus & Zoom Failed" + email.recipients = EmailNotificationSettings.recipients + # Run in an infinite loop unless if an administrator disables forced focus & zoom checks while True: # Get camera IP/hostname, port, username, password, etc. @@ -154,79 +176,86 @@ def focus_camera(camera_id: int): try: dahua_if.set_focus_and_zoom() except Exception as ex: - logging.error("Could not set camera focus and zoom.\nException: {}".format(ex)) + logging.error("Could not set camera focus and zoom.") + logging.exception(ex) + + # Send to each email address specified in the Notifications settings page + if camera.notify_on_failed_interval_check: + email.body = "⚠️ Failed to force focus and zoom camera.\n\n" \ + "Label: {}\nUID: {}\nException: {}\n".format(camera.camera_label, camera.camera_id, ex) + + email.send() # Sleep - for second in range(camera.focus_zoom_interval_check * 60): - time.sleep(1) + time.sleep(camera.focus_zoom_interval_check) else: raise Exception("Unsupported camera manufacturer '{}' for camera ID# {}.".format(camera.manufacturer, camera.camera_id)) -def send_alert(custom_alert_id: int): +def send_alert(custom_alert_id: int, alpr_group_id: int): # Make the application context available here. This function is forked into a separate process and the database # connections needs to be reintroduced. app = create_app(app_config) app.app_context().push() + # Email + email = Email() + email.tag = Tag.ALERT.value + try: custom_alert = CustomAlert.filter_by_id(custom_alert_id) - alpr_group = ALPRGroup.filter_by_id(custom_alert.alpr_group_id) + custom_alert_alpr_group = ALPRGroup.filter_by_id(custom_alert.alpr_group_id) + alpr_group = ALPRGroup.filter_by_id(alpr_group_id) if custom_alert: - if alpr_group: - report_settings = GeneralSettings.get_settings() - submitted_user = User.find_by_id(custom_alert.submitted_by_user_id) - submitted_user_profile = UserProfile.find_by_user_id(custom_alert.submitted_by_user_id) - - email = Email() - email.tag = Tag.ALERT - email.subject = "{} Match! - OpenALPR-Webhook".format(custom_alert.license_plate) - # Add a publicly accessible URL - email.body = "🚨 Custom Alert: {}\n\n{}\n\nLocation/Agent: {}\n\nCamera: {}\n\nOrganization: {}\n\n" \ - "OpenALPR-Webhook".\ - format(custom_alert.license_plate, custom_alert.description, - alpr_group.web_server_config['agent_label'], - alpr_group.web_server_config['camera_label'], report_settings.org_name) - - # Send email alert to receipt first - if helper.are_valid_email_recipients(submitted_user.email): - email.recipients = submitted_user.email - email.send() + email.subject = "{} Match!".format(custom_alert.license_plate) + + report_settings = GeneralSettings.get_settings() + submitted_user = User.find_by_id(custom_alert.submitted_by_user_id) + submitted_user_profile = UserProfile.find_by_user_id(custom_alert.submitted_by_user_id) + + # Add a publicly accessible URL + email.body = "🚨 Custom Alert: {}\n\n{}\nLocation/Agent: {}\nCamera: {}\nOrganization: {}\n".format( + custom_alert.license_plate, custom_alert.description, custom_alert_alpr_group.web_server_config['agent_label'], + custom_alert_alpr_group.web_server_config['camera_label'], report_settings.org_name) + + # Send email alert to submitter first + if helper.are_valid_email_recipients(submitted_user.email): + email.recipients = submitted_user.email + email.send() - sms = SMS() - sms.msg = "🚨 Custom Alert: {}\n\n{}\n\nLocation/Agent: {}\n\nCamera: {}\n\nOrganization: {}\n\n" \ - "OpenALPR-Webhook".format(custom_alert.license_plate, custom_alert.description, - alpr_group.web_server_config['agent_label'], - alpr_group.web_server_config['camera_label'], - report_settings.org_name) - - # Send email alert to receipt first - if helper.are_valid_sms_recipients(submitted_user_profile.phone): - sms.recipients = submitted_user_profile.phone - sms.send() - - # Send to additional recipients - if custom_alert.notify_user_ids is not None: - for id in custom_alert.notify_user_ids: - user = User.find_by_id(id) - user_profile = UserProfile.find_by_user_id(id) + sms = SMS() + sms.msg = "🚨 Custom Alert: {}\n\n{}\n\nLocation/Agent: {}\n\nCamera: {}\n\nOrganization: {}\n\n" \ + "OpenALPR-Webhook".format(custom_alert.license_plate, custom_alert.description, + custom_alert_alpr_group.web_server_config['agent_label'], + custom_alert_alpr_group.web_server_config['camera_label'], + report_settings.org_name) + + # Send sms alert to submitter first + if helper.are_valid_sms_recipients(submitted_user_profile.phone): + sms.recipients = submitted_user_profile.phone + sms.send() + + # Send to additional recipients + if custom_alert.notify_user_ids is not None: + for _id in custom_alert.notify_user_ids: + user = User.find_by_id(_id) + user_profile = UserProfile.find_by_user_id(_id) + if user is not None: if helper.are_valid_email_recipients(user.email): email.recipients = user.email email.send() + if user_profile is not None: if helper.are_valid_sms_recipients(user_profile.phone): sms.recipients = user_profile.phone sms.send() else: - print("ALPR Group #{} was not found in the database.".format(custom_alert.alpr_group_id)) logging.exception("ALPR Group #{} was not found in the database.".format(custom_alert.alpr_group_id)) raise Exception("ALPR Group #{} was not found in the database.".format(custom_alert.alpr_group_id)) else: - print("Custom alert #{} was not found in the database.".format(custom_alert_id)) logging.exception("Custom alert #{} was not found in the database.".format(custom_alert_id)) raise Exception("Custom alert #{} was not found in the database.".format(custom_alert_id)) except Exception as ex: - print(ex) logging.exception(ex) diff --git a/apps/alpr/routes/alert/routes.py b/apps/alpr/routes/alert/routes.py index 4ed4ef9..e234928 100644 --- a/apps/alpr/routes/alert/routes.py +++ b/apps/alpr/routes/alert/routes.py @@ -15,6 +15,9 @@ @blueprint.route('/custom/', methods=["GET"]) @login_required def custom_alert(id): + if id is None: + return render_template('home/page-404.html') + alert = CustomAlert.filter_by_id_and_beautify(id) dt = helpers.Timezone(current_user) @@ -33,6 +36,9 @@ def custom_alert(id): @blueprint.route('/custom/print/', methods=["GET"]) @login_required def print_custom_alert(id): + if id is None: + return render_template('home/page-404.html') + alert = ALPRAlert.filter_by_id_and_beautify(id) dt = helpers.Timezone(current_user) @@ -50,6 +56,9 @@ def print_custom_alert(id): @blueprint.route('/rekor/', methods=["GET"]) @login_required def alpr_alert(id): + if id is None: + return render_template('home/page-404.html') + alert = ALPRAlert.filter_by_id_and_beautify(id) dt = helpers.Timezone(current_user) @@ -67,6 +76,9 @@ def alpr_alert(id): @blueprint.route('/rekor/print/', methods=["GET"]) @login_required def print_alpr_alert(id): + if id is None: + return render_template('home/page-404.html') + alert = ALPRAlert.filter_by_id_and_beautify(id) dt = helpers.Timezone(current_user) diff --git a/apps/alpr/routes/alerts/custom/routes.py b/apps/alpr/routes/alerts/custom/routes.py index 25e7bfd..5f33662 100644 --- a/apps/alpr/routes/alerts/custom/routes.py +++ b/apps/alpr/routes/alerts/custom/routes.py @@ -1,10 +1,10 @@ import logging -from flask import request, jsonify +from flask import request, jsonify, render_template from flask_login import current_user, login_required import apps.helpers as helper -from apps import db +from apps import db, helpers from apps.alpr.models.alpr_group import ALPRGroup from apps.alpr.models.custom_alert import CustomAlert from apps.alpr.routes.alerts.custom import blueprint @@ -22,7 +22,8 @@ def add(): region_match = bool(data.get('region_match')) description = data.get('description') username = current_user.username - notify_user_ids = str(data.get('notify_user_ids')).split(',') + notify_user_ids = [] if data.get('notify_user_ids') == 'null' else str(data.get('notify_user_ids')).split(',') + user = User.find_by_username(username) @@ -48,23 +49,49 @@ def add(): return jsonify({'error': message['duplicate_custom_alert']}), 404 +@blueprint.route('/delete/', methods=["PUT"]) +@login_required +def delete(id): + + alert_record = CustomAlert.filter_by_id(id) + user = User.find_by_username(current_user.username) + + if alert_record: + if user: + if alert_record.submitted_by_user_id == user.id: + alert_record.delete() + else: + return jsonify({'error': message['illegal_access']}), 404 + else: + return jsonify({'error': message['user_not_found']}), 404 + else: + return jsonify({'message': message['custom_alert_not_found']}), 200 + + return jsonify({'message': message['custom_alert_added_successfully']}), 200 + + @blueprint.route('/edit', methods=["PUT"]) @login_required def edit(): data = request.form id = int(data.get('id')) + custom_alert = CustomAlert.filter_by_id(id) region_match = bool(data.get('region_match')) description = data.get('description') username = current_user.username - notify_user_ids = data.get('notify_user_ids') if data.get('notify_user_ids') == 'null' else str(data.get('notify_user_ids')).split(',') + notify_user_ids = [] if data.get('notify_user_ids') == 'null' else str(data.get('notify_user_ids')).split(',') user = User.find_by_username(username) - if notify_user_ids != "null" and user.status != RoleType['ADMIN']: + if notify_user_ids != "null" and user.role != RoleType['ADMIN']: return jsonify({'error': message['illegal_access']}), 404 - custom_alert = CustomAlert.filter_by_id(id) + # Each user can only edit their own records + if custom_alert: + if custom_alert.submitted_by_user_id != user.id: + return jsonify({'error': message['illegal_access']}), 404 + if custom_alert: try: custom_alert.region_match = region_match @@ -107,7 +134,7 @@ def query(): 'id': record.id, 'site': alpr_group.web_server_config['agent_label'], 'camera': alpr_group.web_server_config['camera_label'], - 'plate_number': record.license_plate, + 'plate_number': alpr_group.best_plate_number, 'plate_crop_jpeg': alpr_group.best_plate['plate_crop_jpeg'], 'direction': alpr_group.travel_direction_class_tag, 'confidence': alpr_group.best_confidence_percent, @@ -124,6 +151,9 @@ def query(): @blueprint.route('//choices.js', methods=["GET"]) @login_required def setChoices(id): + if id is None: + return render_template('home/page-404.html') + custom_alert = CustomAlert.filter_by_id(int(id)) if custom_alert: return jsonify(helper.setChoices(current_user, custom_alert.notify_user_ids)) diff --git a/apps/alpr/routes/alerts/routes.py b/apps/alpr/routes/alerts/routes.py index 9c4ab0a..7c2d9c9 100644 --- a/apps/alpr/routes/alerts/routes.py +++ b/apps/alpr/routes/alerts/routes.py @@ -1,6 +1,7 @@ from flask import render_template from flask_login import login_required + from apps.alpr.routes.alerts import blueprint diff --git a/apps/alpr/routes/capture/routes.py b/apps/alpr/routes/capture/routes.py index 155b4bc..d706275 100644 --- a/apps/alpr/routes/capture/routes.py +++ b/apps/alpr/routes/capture/routes.py @@ -14,6 +14,9 @@ @blueprint.route('/', methods=["GET"]) @login_required def plate(id): + if id is None: + return render_template('home/page-404.html') + license_plate = ALPRGroup.filter_by_id_and_beautify(id) dt = helpers.Timezone(current_user) user_profile = UserProfile.find_by_user_id(current_user.id) @@ -31,6 +34,9 @@ def plate(id): @blueprint.route('/print/', methods=["GET"]) @login_required def print_plate(id): + if id is None: + return render_template('home/page-404.html') + license_plate = ALPRGroup.filter_by_id_and_beautify(id) dt = helpers.Timezone(current_user) diff --git a/apps/alpr/routes/search/routes.py b/apps/alpr/routes/search/routes.py index ba7a1d4..94923f1 100644 --- a/apps/alpr/routes/search/routes.py +++ b/apps/alpr/routes/search/routes.py @@ -1,9 +1,11 @@ from flask import render_template, request from flask_login import login_required, current_user -from sqlalchemy import func from apps import db, helpers +from apps.alpr import beautify +from apps.alpr.models.alpr_alert import ALPRAlert from apps.alpr.models.alpr_group import ALPRGroup +from apps.alpr.models.vehicle import Vehicle from apps.alpr.routes.search import blueprint @@ -13,9 +15,45 @@ def search(): return render_template('home/search.html', segment='search') -@blueprint.route('/query', methods=["GET"]) +@blueprint.route('/query/alert/', methods=["GET"]) @login_required -def query(): +def query_alert_plate(plate): + if plate is None: + return render_template('home/page-404.html') + + query = ALPRAlert.query.filter_by(plate_number=plate).order_by(ALPRAlert.id.desc()) + total = query.count() + + # pagination + start = request.args.get('start', type=int, default=-1) + length = request.args.get('length', type=int, default=-1) + if start != -1 and length != -1: + query = query.offset(start).limit(length) + + dt = helpers.Timezone(current_user) + data = [] + for record in query: + data.append({ + 'id': record.id, + 'site': ALPRGroup.get_latest_agent_label(record.agent_uid), + 'camera': ALPRGroup.get_latest_camera_label(record.group['camera_id']), + 'plate_number': record.plate_number, + 'plate_crop_jpeg': record.group['best_plate']['plate_crop_jpeg'], + 'direction': record.travel_direction_class_tag, + 'confidence': record.best_confidence_percent, + 'time': dt.astimezone(record.epoch_time) + }) + + # response + return { + 'data': data, + 'total': total, + } + + +@blueprint.route('/query/group', methods=["GET"]) +@login_required +def query_group(): query = ALPRGroup.query.order_by(ALPRGroup.id.desc()) # search filter @@ -52,7 +90,115 @@ def query(): } -def get_count(q): - count_q = q.statement.with_only_columns([func.count()]).order_by(None) - count = q.session.execute(count_q).scalar() - return count +@blueprint.route('/query/', methods=["GET"]) +@login_required +def query_plate(plate): + if plate is None: + return render_template('home/page-404.html') + + query = ALPRGroup.query.filter_by(best_plate_number=plate).order_by(ALPRGroup.id.desc()) + total = query.count() + + # pagination + start = request.args.get('start', type=int, default=-1) + length = request.args.get('length', type=int, default=-1) + if start != -1 and length != -1: + query = query.offset(start).limit(length) + + dt = helpers.Timezone(current_user) + data = [] + for record in query: + data.append({ + 'id': record.id, + 'site': record.web_server_config['agent_label'], + 'camera': record.web_server_config['camera_label'], + 'plate_number': record.best_plate_number, + 'plate_crop_jpeg': record.best_plate['plate_crop_jpeg'], + 'direction': record.travel_direction_class_tag, + 'confidence': record.best_confidence_percent, + 'time': dt.astimezone(record.epoch_start) + }) + + # response + return { + 'data': data, + 'total': total, + } + + +@blueprint.route('/query/vehicle', methods=["GET"]) +@login_required +def query_vehicle(): + query = Vehicle.query.order_by(Vehicle.id.desc()) + + # search filter + search = request.args.get('search') + if search: + query = query.filter(db.or_(Vehicle.vehicle_color_name.like(f'%{search}%'), + Vehicle.vehicle_make_name.like(f'%{search}%'), + Vehicle.vehicle_make_model_name.like(f'%{search}%'))) + + total = query.count() + + # pagination + start = request.args.get('start', type=int, default=-1) + length = request.args.get('length', type=int, default=-1) + if start != -1 and length != -1: + query = query.offset(start).limit(length) + + dt = helpers.Timezone(current_user) + data = [] + for record in query: + data.append({ + 'id': record.id, + 'site': ALPRGroup.get_latest_agent_label(record.agent_uid), + 'camera': ALPRGroup.get_latest_camera_label(record.camera_id), + 'color': beautify.name(record.vehicle_color_name), + 'ym': record.vehicle_year_name + " " + beautify.name(record.vehicle_make_model_name), + 'vehicle_crop_jpeg': record.vehicle_crop_jpeg, + 'direction': record.travel_direction_class_tag, + 'time': dt.astimezone(record.epoch_start) + }) + + # response + return { + 'data': data, + 'total': total, + } + + +@blueprint.route('/query/vehicle/signature/', methods=["GET"]) +@login_required +def query_vehicle_signature(signature): + if signature is None: + return render_template('home/page-404.html') + + query = Vehicle.query.filter_by(vehicle_signature=signature).order_by(Vehicle.id.desc()) + + total = query.count() + + # pagination + start = request.args.get('start', type=int, default=-1) + length = request.args.get('length', type=int, default=-1) + if start != -1 and length != -1: + query = query.offset(start).limit(length) + + dt = helpers.Timezone(current_user) + data = [] + for record in query: + data.append({ + 'id': record.id, + 'site': ALPRGroup.get_latest_agent_label(record.agent_uid), + 'camera': ALPRGroup.get_latest_camera_label(record.camera_id), + 'color': beautify.name(record.vehicle_color_name), + 'ym': record.vehicle_year_name + " " + beautify.name(record.vehicle_make_model_name), + 'vehicle_crop_jpeg': record.vehicle_crop_jpeg, + 'direction': record.travel_direction_class_tag, + 'time': dt.astimezone(record.epoch_start) + }) + + # response + return { + 'data': data, + 'total': total, + } diff --git a/apps/alpr/routes/settings/agents/routes.py b/apps/alpr/routes/settings/agents/routes.py index 0b1313c..e6880cd 100644 --- a/apps/alpr/routes/settings/agents/routes.py +++ b/apps/alpr/routes/settings/agents/routes.py @@ -1,12 +1,18 @@ import logging +import time + from flask import render_template, request, jsonify from flask_login import current_user, login_required +from rq import Queue + +import worker_manager from apps import db from apps.alpr.models.settings import AgentSettings from apps.alpr.routes.settings.agents import blueprint from apps.authentication.routes import ROLE_ADMIN import apps.helpers as helper from apps.helpers import message +from worker_manager_enums import WorkerType, WMSCommand @blueprint.route('/search', methods=["GET"]) @@ -98,6 +104,7 @@ def edit(): agent = AgentSettings.filter_by_id(data.get('id')) if agent: + previously_enabled = agent.enabled try: # Update it! agent.ip_hostname = ip_hostname @@ -107,7 +114,22 @@ def edit(): # Reload workers and queues. Someone may have enabled an agent which will require a worker and queue # dedicated to that agent - # reload_wqs() + + try: + if previously_enabled != enabled: + if enabled: + wms = worker_manager.WorkerManager(WMSCommand.START_WORKER) + wms.worker_id = agent.agent_uid + wms.worker_type = WorkerType.General + wms.debug = True + wms.send() + else: + wms = worker_manager.WorkerManager(WMSCommand.STOP_WORKER) + wms.worker_id = agent.agent_uid + wms.debug = True + wms.send() + except Exception as ex: + logging.exception(ex) except Exception as ex: logging.exception(ex) diff --git a/apps/alpr/routes/settings/cameras/manufacturers/Dahua.py b/apps/alpr/routes/settings/cameras/manufacturers/Dahua.py index b88af55..5d806d4 100644 --- a/apps/alpr/routes/settings/cameras/manufacturers/Dahua.py +++ b/apps/alpr/routes/settings/cameras/manufacturers/Dahua.py @@ -108,7 +108,7 @@ def set_focus_and_zoom(self) -> bool: for i in range(5): response = requests.get(url, auth=HTTPDigestAuth(self.username, self.password)) time.sleep(1) - logging.debug("Camera {}/{} HTTP response status code: {}".format(self.label, self.id, + logging.debug("Camera {} (ID: {}) HTTP response status code: {}".format(self.label, self.id, response.status_code)) except Exception as ex: logging.exception(ex) @@ -122,5 +122,4 @@ def set_focus_and_zoom(self) -> bool: format(self.focus, self.zoom, values['status.Focus'], values['status.Zoom'])) return False except Exception as ex: - logging.exception(ex) - raise ex + raise Exception(ex) diff --git a/apps/alpr/routes/settings/cameras/routes.py b/apps/alpr/routes/settings/cameras/routes.py index abee873..b716f45 100644 --- a/apps/alpr/routes/settings/cameras/routes.py +++ b/apps/alpr/routes/settings/cameras/routes.py @@ -1,8 +1,13 @@ import logging +import time + from flask import render_template, request, jsonify from flask_login import current_user, login_required +from redis.client import Redis +from rq import Queue from apps import db +from apps.alpr import queue from apps.alpr.models.cache import CameraCache from apps.alpr.models.settings import CameraSettings from apps.alpr.routes.settings.cameras import blueprint @@ -10,6 +15,8 @@ from apps.authentication.routes import ROLE_ADMIN from apps.helpers import message import apps.helpers as helper +from worker_manager import WorkerManager +from worker_manager_enums import WMSCommand, WorkerType @blueprint.route('/edit', methods=['GET', 'POST']) @@ -111,7 +118,7 @@ def save(): focus_zoom_interval_check = data.get('focus_zoom_interval_check') notify_on_failed_interval_check = bool(data.get('notify_on_failed_interval_check')) manufacturer = data.get('manufacturer') - enable = data.get('enable') + enable = bool(data.get('enable')) # Check to if we can even put the values through the validators if len(hostname) == 0: @@ -145,8 +152,28 @@ def save(): camera.focus_zoom_interval_check = focus_zoom_interval_check camera.notify_on_failed_interval_check = notify_on_failed_interval_check camera.manufacturer = manufacturer - camera.enable = bool(enable) + previously_enabled = camera.enable + camera.enable = enable camera.save() + try: + if previously_enabled != enable: + if enable: + wms = WorkerManager(WMSCommand.START_WORKER) + wms.debug = True + wms.worker_type = WorkerType.Camera + wms.worker_id = camera.camera_id + wms.send() + time.sleep(1) + # Add the function to the queue + q = Queue(camera.camera_id, connection=Redis()) + q.enqueue(queue.focus_camera, args=(camera.camera_id,), job_timeout=-1) + else: + wms = WorkerManager(WMSCommand.STOP_WORKER) + wms.debug = True + wms.worker_id = camera.camera_id + wms.send() + except Exception as ex: + logging.exception(ex) except Exception as ex: logging.exception(ex) return "could_not_process" diff --git a/apps/alpr/routes/settings/general/routes.py b/apps/alpr/routes/settings/general/routes.py index d33b7b4..2db0b3b 100644 --- a/apps/alpr/routes/settings/general/routes.py +++ b/apps/alpr/routes/settings/general/routes.py @@ -113,9 +113,9 @@ def edit_report_settings(): logging.exception(ex) return jsonify({'error': message['error_updating_brand_logo']}), 500 - # Organization name - settings.org_name = data.get('org_name') - settings.save() + # Organization name + settings.org_name = data.get('org_name') + settings.save() return jsonify({'message': message['report_settings_saved']}), 200 else: diff --git a/apps/alpr/routes/settings/maintenance/routes.py b/apps/alpr/routes/settings/maintenance/routes.py index 35d3c0c..22c09ae 100644 --- a/apps/alpr/routes/settings/maintenance/routes.py +++ b/apps/alpr/routes/settings/maintenance/routes.py @@ -1,13 +1,13 @@ import logging +import platform from flask import render_template, jsonify from flask_login import login_required, current_user -from marshmallow import fields import apps.alpr.get as Get from apps.alpr.models.alpr_alert import ALPRAlert from apps.alpr.models.alpr_group import ALPRGroup -from apps.alpr.models.cache import Cache, Counter, CameraCache, AgentCache +from apps.alpr.models.cache import Cache, CameraCache, AgentCache from apps.alpr.models.vehicle import Vehicle from apps.alpr.routes.settings.maintenance import blueprint from apps.api.schemas.alpr_alert_schema import ALPRAlertSchema @@ -18,6 +18,8 @@ from apps.api.service.vehicle_service import VehicleService from apps.authentication.models import User from apps.authentication.routes import ROLE_ADMIN +from worker_manager import WorkerManager +from worker_manager_enums import WMSCommand @blueprint.route('/init/cache', methods=["GET"]) @@ -30,7 +32,6 @@ def init_cache_db(): Cache.query.delete() AgentCache.query.delete() CameraCache.query.delete() - Counter.query.delete() cache = Cache.filter_by_year() if cache is None: @@ -58,58 +59,36 @@ def import_db(): get = Get.Get() group_collection = get.collection(database="group") - print("len(group_collection) = {}".format(len(group_collection))) + logging.info("len(group_collection) = {}".format(len(group_collection))) alert_collection = get.collection(database="alert") - print("len(alert_collection) = {}".format(len(alert_collection))) + logging.info("len(alert_collection) = {}".format(len(alert_collection))) vehicle_collection = get.collection(database="vehicle") - print("len(vehicle_collection) = {}".format(len(vehicle_collection))) - - alpr_group_counter = Counter.filter_by_key("alpr_group") - if alpr_group_counter is None: - alpr_group_counter = Counter("alpr_group") - - alpr_alert_counter = Counter.filter_by_key("alpr_alert") - if alpr_alert_counter is None: - alpr_alert_counter = Counter("alpr_alert") - - vehicle_counter = Counter.filter_by_key("vehicle") - if vehicle_counter is None: - vehicle_counter = Counter("vehicle") + logging.info("len(vehicle_collection) = {}".format(len(vehicle_collection))) for i in range(len(group_collection)): request_data = group_collection[i] - alpr_group_counter.one_up() try: validated_data = alpr_group_schema.load(request_data) alpr_group_service.create(validated_data) except Exception as ex: - alpr_group_counter.one_down() - print("transfer_db: (alpr_group) ex = {}".format(ex)) - print("transfer_db: (alpr_group) request_data = {}".format(request_data)) + logging.debug("transfer_db: (alpr_group) ex = {}".format(ex)) + logging.debug("transfer_db: (alpr_group) request_data = {}".format(request_data)) for i in range(len(alert_collection)): request_data = alert_collection[i] - alpr_alert_counter.one_up() try: validated_data = alpr_alert_schema.load(request_data) alpr_alert_service.create(validated_data) except Exception as ex: - alpr_alert_counter.one_down() - print("transfer_db: (alpr_alert) ex = {}".format(ex)) - print("transfer_db: (alpr_alert) request_data = {}".format(request_data)) + logging.debug("transfer_db: (alpr_alert) ex = {}".format(ex)) + logging.debug("transfer_db: (alpr_alert) request_data = {}".format(request_data)) for i in range(len(vehicle_collection)): request_data = vehicle_collection[i] - vehicle_counter.one_up() try: validated_data = vehicle_schema.load(request_data) vehicle_service.create(validated_data) except Exception as ex: - vehicle_counter.one_down() - print("transfer_db: (vehicle) ex = {}".format(ex)) - print("transfer_db: (vehicle) request_data = {}".format(request_data)) - - alpr_group_counter.save() - alpr_alert_counter.save() - vehicle_counter.save() + logging.debug("transfer_db: (vehicle) ex = {}".format(ex)) + logging.debug("transfer_db: (vehicle) request_data = {}".format(request_data)) # Rewrite the API_TOKEN to that of the super_admin super_admin = User.find_by_id(1) @@ -130,3 +109,54 @@ def import_db(): vehicle.save() return jsonify({'msg': "Records migrated to SQLite!"}), 200 + + +@blueprint.route('/shutdown/wms', methods=["POST"]) +@login_required +def shutdown_wms(): + if current_user.role != ROLE_ADMIN: + return render_template('home/page-403.html') + + if platform.system() != "Linux": + return jsonify({'error': 'Unsupported platform. Redis server is not running.'}), 404 + + try: + wms = WorkerManager(WMSCommand.STOP_ALL) + logging.debug("Sending STOP_ALL command") + wms.send() + wms.command = WMSCommand.STOP_SERVER + logging.debug("Sending STOP_SERVER command") + wms.send() + except Exception or TimeoutError as ex: + logging.exception(ex) + return jsonify({'error': str(ex)}), 404 + + return jsonify({'message': 'Worker Manager Server shutdown successfully!'}), 200 + + +@blueprint.route('/restart/wms', methods=["GET"]) +@login_required +def restart_wms(): + if current_user.role != ROLE_ADMIN: + return render_template('home/page-403.html') + + try: + + wms = WorkerManager(WMSCommand.STOP_ALL) + logging.debug("Sending STOP_ALL command") + wms.send() + wms.command = WMSCommand.STOP_SERVER + logging.debug("Sending STOP_SERVER command") + wms.send() + + from apps import start_redis_workers + start_redis_workers() + + except Exception or TimeoutError as ex: + logging.exception(ex) + if TimeoutError: + return jsonify({'error': 'Connection to Worker Manager Server timed out!'}), 404 + elif Exception: + return jsonify({'error': 'Unknown error occurred!'}), 404 + + return jsonify({'msg': 'Worker Manager Server shutdown successfully!'}), 200 diff --git a/apps/alpr/routes/settings/maintenance/rq_dashboard/test.py b/apps/alpr/routes/settings/maintenance/rq_dashboard/test.py deleted file mode 100644 index 761855c..0000000 --- a/apps/alpr/routes/settings/maintenance/rq_dashboard/test.py +++ /dev/null @@ -1,9 +0,0 @@ -from apps import default_q -from apps.alpr.routes.settings.maintenance.rq_dashboard.queue_functions import count_words_at_url - -if __name__ == "__main__": - for i in range(10): - job = default_q.enqueue(count_words_at_url, 'http://nvie.com') - print(job.result) - - diff --git a/apps/alpr/routes/settings/profile/routes.py b/apps/alpr/routes/settings/profile/routes.py index 8dd5784..93f86ca 100644 --- a/apps/alpr/routes/settings/profile/routes.py +++ b/apps/alpr/routes/settings/profile/routes.py @@ -6,6 +6,7 @@ from flask_login import login_required, current_user from werkzeug.utils import secure_filename +from apps import helpers from apps.alpr.routes.settings.profile import blueprint from apps.authentication.models import User, UserProfile from apps.authentication.routes import upload_folder_name, ROLE_ADMIN, ROLE_USER, STATUS_ACTIVE, STATUS_SUSPENDED @@ -55,6 +56,8 @@ def edit(): # Change avatar if image: + # Create directory if it doesn't exist + helpers.mkdir(upload_folder_name) filename = unique_file_name(secure_filename(image.filename)) full_file_path = os.path.join(upload_folder_name, filename) try: diff --git a/apps/alpr/routes/settings/routes.py b/apps/alpr/routes/settings/routes.py index 8c41138..acc78d5 100644 --- a/apps/alpr/routes/settings/routes.py +++ b/apps/alpr/routes/settings/routes.py @@ -3,12 +3,14 @@ from flask_login import login_required, current_user from flask_paginate import get_page_parameter, Pagination -from apps import IPBanConfig, ip_ban_config +from apps import ip_ban_config from apps.alpr.models.settings import EmailNotificationSettings, TwilioNotificationSettings, GeneralSettings, PostAuth from apps.alpr.routes.settings import blueprint from apps.alpr.routes.settings.cameras.manufacturers import get_camera_manufacturers from apps.authentication.models import User, UserProfile from apps.authentication.routes import ROLE_ADMIN +from worker_manager import WorkerManager +from worker_manager_enums import WMSCommand @blueprint.route('/agents', methods=["GET"]) @@ -39,6 +41,24 @@ def general(): ipban=ip_ban_config.get_settings(), post_auth_levels=PostAuth) +@blueprint.route('/maintenance/app', methods=["GET"]) +@login_required +def maintenance_app(): + if current_user.role != ROLE_ADMIN: + return render_template('home/page-403.html') + + wms_status = False + try: + wms = WorkerManager(WMSCommand.ACK) + wms.send() + wms_status = wms.last_connection() + except Exception: + pass + + return render_template('settings/maintenance-app.html', segment='settings-maintenance-app', + wms_status=wms_status) + + @blueprint.route('/notifications', methods=["GET"]) @login_required def notifications(): @@ -127,4 +147,4 @@ def users(): segment='settings-users' ) - return redirect(url_for('home_blueprint.index')) \ No newline at end of file + return redirect(url_for('home_blueprint.index')) diff --git a/apps/alpr/routes/settings/users/routes.py b/apps/alpr/routes/settings/users/routes.py index e9bdaa7..3045f42 100644 --- a/apps/alpr/routes/settings/users/routes.py +++ b/apps/alpr/routes/settings/users/routes.py @@ -6,7 +6,7 @@ from apps import db, helpers from apps.alpr.models.settings import EmailNotificationSettings -from apps.alpr.notify import Email +from apps.alpr.notify import Email, Tag from apps.alpr.routes.settings.users import blueprint from apps.authentication.models import User, UserProfile from apps.authentication.routes import STATUS_ACTIVE, STATUS_SUSPENDED, ROLE_ADMIN, ROLE_USER @@ -23,7 +23,7 @@ def check_smtp(): settings = EmailNotificationSettings.get_settings() if settings is None: - return jsonify({'error': 'Settings have not been initialized'}), 404 + return jsonify({'error': 'Empty SMTP settings. Valid SMTP settings required to reset user passwords.'}), 404 # Validate SMTP settings is_valid_hostname = helpers.is_valid_hostname(settings.hostname) @@ -106,6 +106,12 @@ def edit(): profile.address = data.get('address') profile.zipcode = data.get('zipcode') profile.phone = data.get('phone') + + # Check phone number validity + if data.get('phone') != "": + if not helpers.are_valid_sms_recipients(profile.phone): + return jsonify({'error': message['invalid_phone_number']}), 404 + profile.email = data.get('email') profile.website = data.get('website') @@ -123,6 +129,12 @@ def edit(): profile.address = data.get('address') profile.zipcode = data.get('zipcode') profile.phone = data.get('phone') + + # Check phone number validity + if data.get('phone') != "": + if not helpers.are_valid_sms_recipients(profile.phone): + return jsonify({'error': message['invalid_phone_number']}), 404 + profile.website = data.get('website') profile.save() @@ -200,14 +212,14 @@ def reset_password(): password = pg.generate() user.password = hash_pass(password) email = Email() - email.tag = "Account" - email.subject = "Account Password Reset" + email.tag = Tag.ACCOUNT.value + email.subject = "Password Reset" email.body = "Hello {},\nYour password has been reset.\nYour new password is: {}".format(user.username, password) email.recipients = [user.email] - email.send() # Don't save the new user password unless an email was sent without an exception - user.save() + if email.send(): + user.save() except Exception as ex: logging.exception(ex) return jsonify({'error': 'Something went wrong! Please make sure email SMTP settings are correct.'}), 404 diff --git a/apps/alpr/routes/vehicle/__init__.py b/apps/alpr/routes/vehicle/__init__.py new file mode 100644 index 0000000..ef10da5 --- /dev/null +++ b/apps/alpr/routes/vehicle/__init__.py @@ -0,0 +1,7 @@ +from flask import Blueprint + +blueprint = Blueprint( + 'vehicle', + __name__, + url_prefix='/vehicle' +) diff --git a/apps/alpr/routes/vehicle/routes.py b/apps/alpr/routes/vehicle/routes.py new file mode 100644 index 0000000..bbc9566 --- /dev/null +++ b/apps/alpr/routes/vehicle/routes.py @@ -0,0 +1,51 @@ +import datetime + +from flask import render_template +from flask_login import login_required, current_user + +from apps import helpers +from apps.alpr.models.cache import CameraCache +from apps.alpr.models.settings import GeneralSettings +from apps.alpr.models.vehicle import Vehicle +from apps.alpr.routes.vehicle import blueprint +from apps.authentication.models import UserProfile, User + + +@blueprint.route('/', methods=["GET"]) +@login_required +def vehicle(id): + if id is None: + return render_template('home/page-404.html') + + vehicle = Vehicle.filter_by_id_and_beautify(id) + dt = helpers.Timezone(current_user) + user_profile = UserProfile.find_by_user_id(current_user.id) + + if vehicle is None: + return render_template('home/page-404.html') + else: + cached_camera = CameraCache.filter_by_id_and_beautify(vehicle['camera_id']) + return render_template('home/vehicle.html', segment='search', vehicle=vehicle, + date=dt.astimezone(datetime.datetime.utcnow()), + user_profile=user_profile, cached_camera=cached_camera, + settings=GeneralSettings.get_settings(), users=User.get_list_of_users_w_user_profiles()) + + +@blueprint.route('/print/', methods=["GET"]) +@login_required +def vehicle_print(id): + if id is None: + return render_template('home/page-404.html') + + vehicle = Vehicle.filter_by_id_and_beautify(id) + dt = helpers.Timezone(current_user) + user_profile = UserProfile.find_by_user_id(current_user.id) + + if vehicle is None: + return render_template('home/page-404.html') + else: + cached_camera = CameraCache.filter_by_id_and_beautify(vehicle['camera_id']) + return render_template('home/vehicle-print.html', segment='search', vehicle=vehicle, + date=dt.astimezone(datetime.datetime.utcnow()), + user_profile=user_profile, cached_camera=cached_camera, + settings=GeneralSettings.get_settings(), users=User.get_list_of_users_w_user_profiles()) diff --git a/apps/api/controller/webhook_controller.py b/apps/api/controller/webhook_controller.py index 03ae3c2..bb28f11 100644 --- a/apps/api/controller/webhook_controller.py +++ b/apps/api/controller/webhook_controller.py @@ -50,16 +50,13 @@ def post(self): return BaseController.errorGeneral("POST has been disabled"), 403 elif auth_level == PostAuth.USERS_ADMINS or auth_level == PostAuth.ADMINS_ONLY: try: - # Re-serialize the object "custom_data": "{\"API_KEY\": \"aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee\"}" -> - # "custom_data": {"API_KEY": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"} - json_obj = json.loads(request.json['custom_data']) - - api_key = json_obj['API_KEY'] - if api_key != "": - token_holder = User.find_by_api_token(api_key) - if token_holder is None: - return BaseController.errorGeneral("Unknown API_KEY holder"), 403 - else: + try: + api_key = request.json['custom_data']['API_KEY'] + if api_key != "": + token_holder = User.find_by_api_token(api_key) + if token_holder is None: + return BaseController.errorGeneral("Unknown API_KEY holder"), 403 + except KeyError: return BaseController.errorGeneral("Missing API_KEY in custom_data"), 403 if auth_level == PostAuth.ADMINS_ONLY: @@ -71,7 +68,7 @@ def post(self): return BaseController.errorGeneral("Could not process custom_data") # Redefine custom_data for the schema validators - request.json['custom_data'] = json_obj + # request.json['custom_data'] = json_obj # Enumerate the 'data_type' key data_type = DataType(request.json['data_type']) diff --git a/apps/api/service/cache_service.py b/apps/api/service/cache_service.py index 038e06a..2387f88 100644 --- a/apps/api/service/cache_service.py +++ b/apps/api/service/cache_service.py @@ -5,6 +5,7 @@ from apps import default_q from apps.alpr.enums import DataType +from apps.alpr.models.alpr_group import ALPRGroup from apps.alpr.models.cache import Cache, CameraCache from apps.alpr.models.custom_alert import CustomAlert from apps.alpr.models.settings import AgentSettings, CameraSettings @@ -15,14 +16,14 @@ class CacheService: - foreign_id = None + alpr_group_id = None now = datetime.now() this_year = now.year # From 0 - 11 not 1 - 12 this_month = now.month - 1 - def __init__(self, request_json: json, foreign_id: int): - self.foreign_id = foreign_id + def __init__(self, request_json: json, alpr_group_id: int): + self.alpr_group_id = alpr_group_id self.request = request_json # Load the cache if it already exists @@ -31,16 +32,17 @@ def __init__(self, request_json: json, foreign_id: int): # Create a row if the year doesn't exist if not self.cache: self.cache = Cache() - + def update(self): try: - if self.request['data_type'] == DataType.GROUP: + data_type = DataType(self.request['data_type']) + if data_type == DataType.GROUP: # Increase counters self.cache.all_time_plates_captured += 1 self.cache.month[self.this_month]['license_plates_captured'] += 1 # Objects self.cache.month[self.this_month]['cameras'] = \ - self.increase_dict_value_count("cameras", self.request['camera_id']) + self.increase_dict_value_count("cameras", str(self.request['camera_id'])) self.cache.month[self.this_month]['regions'] = \ self.increase_dict_value_count("regions", self.request['best_region']) @@ -53,8 +55,11 @@ def update(self): else: # Always overwrite the user definable values, they may have changed camera_id_cache.camera_label = self.request['web_server_config']['camera_label'] - camera_id_cache.gps_latitude = self.request['gps_latitude'] - camera_id_cache.gps_longitude = self.request['gps_longitude'] + # Don't overwrite them back to -1 + if self.request['gps_latitude'] != -1: + camera_id_cache.gps_latitude = self.request['gps_latitude'] + if self.request['gps_longitude'] != -1: + camera_id_cache.gps_longitude = self.request['gps_longitude'] camera_id_cache.country = self.request['country'] camera_id_cache.save() @@ -64,7 +69,8 @@ def update(self): agent_uid = AgentSettings.filter_by_agent_uid(self.request['agent_uid']) if agent_uid is None: - agent_uid = AgentSettings(self.request['agent_uid'], self.request['web_server_config']['agent_label']) + agent_uid = AgentSettings(self.request['agent_uid'], + self.request['web_server_config']['agent_label']) else: # Always overwrite the label, it may have changed agent_uid.agent_label = self.request['web_server_config']['agent_label'] @@ -74,7 +80,8 @@ def update(self): camera_id = CameraSettings.filter_by_camera_id(self.request['camera_id']) if camera_id is None: - camera_id = CameraSettings(self.request['camera_id'], self.request['web_server_config']['camera_label']) + camera_id = CameraSettings(self.request['camera_id'], + self.request['web_server_config']['camera_label']) else: # Always overwrite the label, it may have changed camera_id.camera_label = self.request['web_server_config']['camera_label'] @@ -83,29 +90,51 @@ def update(self): # Check to see if we need to send an alert custom_alert = CustomAlert.filter_by_license_plate(self.request['best_plate_number']) if custom_alert: - # Send the id to the custom alert to the queue to notify recipients - default_q.enqueue(send_alert, custom_alert.id) - - elif self.request['data_type'] == DataType.ALERT: + this_record = ALPRGroup.filter_by_id(self.alpr_group_id) + custom_alert_alpr_group_record = ALPRGroup.filter_by_id(custom_alert.alpr_group_id) + if this_record: + def enqueue(): + # Send the id to the custom alert to the queue to notify recipients + default_q.enqueue(send_alert, custom_alert.id, self.alpr_group_id) + # Increase custom_alerts + self.cache.all_time_custom_alerts += 1 + self.cache.month[self.this_month]['custom_alerts'] += 1 + + if custom_alert.region_match: + if this_record.best_region == custom_alert_alpr_group_record.best_region: + enqueue() + else: + logging.info("Plate region match is enabled but does not match") + else: + enqueue() + + elif data_type == DataType.ALERT: self.cache.all_time_alerts += 1 self.cache.month[self.this_month]['alerts'] += 1 - elif self.request['data_type'] == DataType.VEHICLE: + elif data_type == DataType.VEHICLE: self.cache.all_time_vehicles += 1 self.cache.month[self.this_month]['vehicles'] += 1 - # Update/save the cache - self.cache.save() - # Prevent circular/infinite loop import from apps.alpr.queue import download_plate_image # Send it to the queue to download the uuid.jpg from the origin agent - default_q.enqueue(download_plate_image, self.request['agent_uid'], self.request['best_uuid'], - self.request['data_type'], self.foreign_id, retry=Retry(max=5, interval=60)) + if data_type == DataType.ALERT: + best_uuid = self.request['group']['best_uuid'] + else: + best_uuid = self.request['best_uuid'] + + default_q.enqueue(download_plate_image, self.request['agent_uid'], best_uuid, self.request['data_type'], + self.alpr_group_id) + + # Update/save the cache + self.cache.save() return self.cache except Exception as ex: logging.exception(ex) + logging.debug("data_type = {}".format(self.request['data_type'])) + logging.debug(json.dumps(self.request, ensure_ascii=False, indent=4)) raise Exception(ex) def increase_dict_value_count(self, dict_index: str, key: str) -> {}: diff --git a/apps/config.py b/apps/config.py index 2b3f683..2901b17 100644 --- a/apps/config.py +++ b/apps/config.py @@ -1,5 +1,6 @@ import configparser import os +from logging.config import dictConfig from os.path import exists import secrets diff --git a/apps/helpers.py b/apps/helpers.py index af7b82f..eba6f58 100644 --- a/apps/helpers.py +++ b/apps/helpers.py @@ -1,5 +1,7 @@ import datetime +import logging import os +import subprocess import uuid import re @@ -17,6 +19,30 @@ message = Messages.message +def netstat() -> float: + try: + netstat = subprocess.Popen(['netstat', '-aon'], stdout=subprocess.PIPE) + grep_3565 = subprocess.Popen(['grep', '3565'], stdin=netstat.stdout, stdout=subprocess.PIPE) + grep_time_wait = subprocess.Popen(['grep', 'TIME_WAIT'], stdin=grep_3565.stdout, stdout=subprocess.PIPE) + tail_n_1 = subprocess.Popen(['tail', '-n1'], stdin=grep_time_wait.stdout, stdout=subprocess.PIPE) + awk = subprocess.Popen(['awk', '{print $8}'], stdin=tail_n_1.stdout, stdout=subprocess.PIPE) + + stdout, stderr = awk.communicate() + stdout = stdout.decode('utf-8') + logging.debug("stdout = {}".format(stdout)) + + if awk.returncode == 0: + if stdout != "": + stdout = stdout.split('/') + seconds = stdout[0].strip('(') + logging.debug("seconds = {}".format(seconds)) + return float(seconds) + return 0.00 + return 0.00 + except Exception: + return 0.00 + + def setChoices(current_user, user_ids: []) -> []: users = User.query choices = [] @@ -27,9 +53,13 @@ def setChoices(current_user, user_ids: []) -> []: if user_ids is not None: if str(user.id) in user_ids: selected = True + if user_profile.full_name != "": + label = user_profile.full_name + " (" + user.email + ")" + else: + label = user.username + " (" + user.email + ")" choices.append({ 'value': user.id, - 'label': user_profile.full_name + " (" + user.email + ")", + 'label': label, 'selected': selected }) return choices @@ -176,8 +206,7 @@ def emailValidate(email): return False -def createFolder(folder_name): - """ create folder for save csv """ +def mkdir(folder_name): if not os.path.exists(f'{folder_name}'): os.makedirs(f'{folder_name}') diff --git a/apps/home/routes.py b/apps/home/routes.py index 0e70753..fcc4fc9 100644 --- a/apps/home/routes.py +++ b/apps/home/routes.py @@ -16,7 +16,7 @@ def index(): cache = Cache().filter_by_year() alpr_group_records = ALPRGroup().get_dashboard_records() alpr_alert_records = ALPRAlert().get_dashboard_records() - custom_alerts = CustomAlert().get_dashboard_records(current_user) + custom_alerts = CustomAlert().get_dashboard_records() return render_template('home/index.html', segment='index', alpr_group_records=alpr_group_records, alpr_alert_records=alpr_alert_records, custom_alerts=custom_alerts, @@ -24,9 +24,10 @@ def index(): us_map_regions=cache.get_us_map_series(), plates_captured_chart_series=cache.get_chart_series(ChartType.PLATES_CAPTURED_CHART), alert_chart_series=cache.get_chart_series(ChartType.ALERT_CHART), - top_region_chart_series=cache.get_chart_series(ChartType.TOP_REGION_CHART), + custom_alert_chart_series=cache.get_chart_series(ChartType.CUSTOM_ALERT), + top_region_chart_series=cache.get_chart_series(ChartType.TOP_SECOND_REGION_CHART), plates_captured_alerts_chart_labels=cache.get_chart_labels(), - top_region_chart_labels=cache.get_chart_labels(ChartType.TOP_REGION_CHART), + top_region_chart_labels=cache.get_chart_labels(ChartType.TOP_SECOND_REGION_CHART), number_of_records=cache.get_number_of_records(), size_of_databases=cache.get_all_db_file_sizes(), top_cameras=cache.get_top_cameras(), number_of_records_raw=cache.get_number_of_records(raw=True), diff --git a/apps/messages.py b/apps/messages.py index 492b6e5..151bb04 100644 --- a/apps/messages.py +++ b/apps/messages.py @@ -73,6 +73,7 @@ class Messages: 'general_settings_saved': 'General settings saved!', 'general_settings_not_saved': 'Could not save general settings!', 'custom_alert_added_successfully': 'Custom alert added successfully', + 'custom_alert_not_found': 'Custom alert not found', 'custom_alert_updated_successfully': 'Custom alert updated successfully', 'duplicate_custom_alert': 'A custom alert with this license plate number already exists for you.', 'illegal_access': "Illegal access", diff --git a/apps/static/assets/img/brand/dark.svg b/apps/static/assets/img/brand/dark.svg deleted file mode 100644 index d82922e..0000000 --- a/apps/static/assets/img/brand/dark.svg +++ /dev/null @@ -1,13 +0,0 @@ - - - - - - - diff --git a/apps/static/assets/img/brand/light.svg b/apps/static/assets/img/brand/light.svg deleted file mode 100644 index 2f1ed79..0000000 --- a/apps/static/assets/img/brand/light.svg +++ /dev/null @@ -1 +0,0 @@ - \ No newline at end of file diff --git a/apps/static/assets/img/favicon/android-chrome-192x192.png b/apps/static/assets/img/favicon/android-chrome-192x192.png index 74d876a..ffa762a 100644 Binary files a/apps/static/assets/img/favicon/android-chrome-192x192.png and b/apps/static/assets/img/favicon/android-chrome-192x192.png differ diff --git a/apps/static/assets/img/favicon/android-chrome-512x512.png b/apps/static/assets/img/favicon/android-chrome-512x512.png index 7aa65b4..7402007 100644 Binary files a/apps/static/assets/img/favicon/android-chrome-512x512.png and b/apps/static/assets/img/favicon/android-chrome-512x512.png differ diff --git a/apps/static/assets/img/favicon/apple-touch-icon.png b/apps/static/assets/img/favicon/apple-touch-icon.png index 28fc29b..5a2633c 100644 Binary files a/apps/static/assets/img/favicon/apple-touch-icon.png and b/apps/static/assets/img/favicon/apple-touch-icon.png differ diff --git a/apps/static/assets/img/favicon/browserconfig.xml b/apps/static/assets/img/favicon/browserconfig.xml index cc90142..6f29df1 100644 --- a/apps/static/assets/img/favicon/browserconfig.xml +++ b/apps/static/assets/img/favicon/browserconfig.xml @@ -2,8 +2,8 @@ - - #F2F4F6 + + #1f2937 diff --git a/apps/static/assets/img/favicon/favicon-16x16.png b/apps/static/assets/img/favicon/favicon-16x16.png index c935d8e..544e741 100644 Binary files a/apps/static/assets/img/favicon/favicon-16x16.png and b/apps/static/assets/img/favicon/favicon-16x16.png differ diff --git a/apps/static/assets/img/favicon/favicon-32x32.png b/apps/static/assets/img/favicon/favicon-32x32.png index a987b58..f268be0 100644 Binary files a/apps/static/assets/img/favicon/favicon-32x32.png and b/apps/static/assets/img/favicon/favicon-32x32.png differ diff --git a/apps/static/assets/img/favicon/favicon.ico b/apps/static/assets/img/favicon/favicon.ico index 2d07568..51e4d79 100644 Binary files a/apps/static/assets/img/favicon/favicon.ico and b/apps/static/assets/img/favicon/favicon.ico differ diff --git a/apps/static/assets/img/favicon/favicon.png b/apps/static/assets/img/favicon/favicon.png deleted file mode 100644 index 6a751f2..0000000 Binary files a/apps/static/assets/img/favicon/favicon.png and /dev/null differ diff --git a/apps/static/assets/img/favicon/manifest.json b/apps/static/assets/img/favicon/manifest.json deleted file mode 100644 index 1d22e99..0000000 --- a/apps/static/assets/img/favicon/manifest.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "name": "Bootstrap", - "short_name": "Bootstrap", - "icons": [ - { - "src": "/docs/4.3/assets/img/favicons/android-chrome-192x192.png", - "sizes": "192x192", - "type": "image/png" - }, - { - "src": "/docs/4.3/assets/img/favicons/android-chrome-512x512.png", - "sizes": "512x512", - "type": "image/png" - } - ], - "start_url": "/?utm_source=a2hs", - "theme_color": "#563d7c", - "background_color": "#563d7c", - "display": "standalone" -} diff --git a/apps/static/assets/img/favicon/mstile-144x144.png b/apps/static/assets/img/favicon/mstile-144x144.png new file mode 100644 index 0000000..b67efbf Binary files /dev/null and b/apps/static/assets/img/favicon/mstile-144x144.png differ diff --git a/apps/static/assets/img/favicon/mstile-150x150.png b/apps/static/assets/img/favicon/mstile-150x150.png index 5cef431..14f24cf 100644 Binary files a/apps/static/assets/img/favicon/mstile-150x150.png and b/apps/static/assets/img/favicon/mstile-150x150.png differ diff --git a/apps/static/assets/img/favicon/mstile-310x150.png b/apps/static/assets/img/favicon/mstile-310x150.png new file mode 100644 index 0000000..6ec6099 Binary files /dev/null and b/apps/static/assets/img/favicon/mstile-310x150.png differ diff --git a/apps/static/assets/img/favicon/mstile-310x310.png b/apps/static/assets/img/favicon/mstile-310x310.png new file mode 100644 index 0000000..51f7b07 Binary files /dev/null and b/apps/static/assets/img/favicon/mstile-310x310.png differ diff --git a/apps/static/assets/img/favicon/mstile-70x70.png b/apps/static/assets/img/favicon/mstile-70x70.png new file mode 100644 index 0000000..a943c83 Binary files /dev/null and b/apps/static/assets/img/favicon/mstile-70x70.png differ diff --git a/apps/static/assets/img/favicon/safari-pinned-tab.svg b/apps/static/assets/img/favicon/safari-pinned-tab.svg index 6be6e39..3058a98 100644 --- a/apps/static/assets/img/favicon/safari-pinned-tab.svg +++ b/apps/static/assets/img/favicon/safari-pinned-tab.svg @@ -1,11 +1,29 @@ - - - - - - + + + + +Created by potrace 1.14, written by Peter Selinger 2001-2017 + + + + diff --git a/apps/static/assets/img/favicon/site.webmanifest b/apps/static/assets/img/favicon/site.webmanifest index 2d28b24..cb632d4 100644 --- a/apps/static/assets/img/favicon/site.webmanifest +++ b/apps/static/assets/img/favicon/site.webmanifest @@ -3,17 +3,17 @@ "short_name": "OpenALPR-Webhook", "icons": [ { - "src": "/android-chrome-192x192.png", + "src": "/static/assets/img/favicon/android-chrome-192x192.png", "sizes": "192x192", "type": "image/png" }, { - "src": "/android-chrome-512x512.png", + "src": "/static/assets/img/favicon/android-chrome-512x512.png", "sizes": "512x512", "type": "image/png" } ], - "theme_color": "#1F2937", - "background_color": "#F2F4F6", + "theme_color": "#1f2937", + "background_color": "#1f2937", "display": "standalone" } diff --git a/apps/static/assets/img/marker.svg b/apps/static/assets/img/marker.svg deleted file mode 100644 index ed4d224..0000000 --- a/apps/static/assets/img/marker.svg +++ /dev/null @@ -1,2 +0,0 @@ - - \ No newline at end of file diff --git a/apps/static/assets/img/paypal-logo.svg b/apps/static/assets/img/paypal-logo.svg deleted file mode 100644 index 17535e6..0000000 --- a/apps/static/assets/img/paypal-logo.svg +++ /dev/null @@ -1,69 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/apps/static/assets/img/profile-cover.jpg b/apps/static/assets/img/profile-cover.jpg deleted file mode 100644 index b6bbf9b..0000000 Binary files a/apps/static/assets/img/profile-cover.jpg and /dev/null differ diff --git a/apps/static/assets/img/themesberg-logo-alt.svg b/apps/static/assets/img/themesberg-logo-alt.svg deleted file mode 100644 index 3a6e4a6..0000000 --- a/apps/static/assets/img/themesberg-logo-alt.svg +++ /dev/null @@ -1 +0,0 @@ - \ No newline at end of file diff --git a/apps/static/assets/img/themesberg.svg b/apps/static/assets/img/themesberg.svg deleted file mode 100644 index b28fdc3..0000000 --- a/apps/static/assets/img/themesberg.svg +++ /dev/null @@ -1 +0,0 @@ -logo \ No newline at end of file diff --git a/apps/templates/accounts/register.html b/apps/templates/accounts/register.html index 6f68aee..263e209 100644 --- a/apps/templates/accounts/register.html +++ b/apps/templates/accounts/register.html @@ -101,7 +101,7 @@

Register

- +
Login diff --git a/apps/templates/home/alert.html b/apps/templates/home/alert.html index c632e7d..23b8528 100644 --- a/apps/templates/home/alert.html +++ b/apps/templates/home/alert.html @@ -5,6 +5,7 @@ {% block stylesheets %} + {% endblock stylesheets %} @@ -42,25 +43,6 @@

Rekor™ Scout Alert

-{#
#} -{#
#} -{#
#} -{#
#} -{# #} -{#
#} -{#
#} -{#

{{ settings.org_name }}

#} -{#
#} -{#
#} -{#
#} -{#
#} -{#
#} -{#
#} -{#

Alert Report

#} -{#
#} -{#
#} -{#
#} -{#
#}
{% if alert['uuid_jpg'] %} {{ alert['best_plate_number'] }} @@ -197,6 +179,12 @@
Description
{{ alert['description'] }}
+
+
+ +
+
+
@@ -209,7 +197,7 @@
Description
{% block javascripts %} - + {% endblock javascripts %} diff --git a/apps/templates/home/capture.html b/apps/templates/home/capture.html index 8e87f73..84468af 100644 --- a/apps/templates/home/capture.html +++ b/apps/templates/home/capture.html @@ -5,6 +5,7 @@ {% block stylesheets %} + {% endblock stylesheets %} @@ -21,6 +22,7 @@ + @@ -46,25 +48,6 @@

License Plate Report

-{#
#} -{#
#} -{#
#} -{#
#} -{# #} -{#
#} -{#
#} -{#

{{ settings.org_name }}

#} -{#
#} -{#
#} -{#
#} -{#
#} -{#
#} -{#
#} -{#

License Plate Report

#} -{#
#} -{#
#} -{#
#} -{#
#}
{% if license_plate['uuid_jpg'] %} {{ license_plate['best_plate_number'] }} @@ -196,6 +179,12 @@
Vehicle Information
+
+
+ +
+
+
@@ -207,6 +196,7 @@
Vehicle Information -
+ + + + +
+
+
+
+
+
+
+
+
+
+
+
+ + {% endblock content %} @@ -69,7 +95,7 @@

License Plates

{ id: 'time', name: 'Time', sort: false }, ], server: { - url: '{{ url_for('search.query') }}', + url: '{{ url_for('search.query_group') }}', then: results => results.data, total: results => results.total, }, @@ -89,9 +115,55 @@

License Plates

url: (prev, page, limit) => { return updateUrl(prev, {start: page * limit, length: limit}); }, - }, + } }, }).render(document.getElementById('licensePlates')); + + new gridjs.Grid({ + columns: [ + { id: 'id', name:'ID', hidden: true, sort: false }, + { id: 'site', name: 'Site', sort: false }, + { id: 'camera', name: 'Camera', sort: false }, + { id: 'color', name: 'Color', sort: false }, + { id: 'ym', name: 'Year Make Model', sort: false, + formatter: (cell, row) => { + return gridjs.h('text-default', { + className: '', + }, gridjs.html(`${cell}`)); + } + }, + { id: 'vehicle_crop_jpeg', name: 'Vehicle', sort: false, formatter: (cell) => + gridjs.html(``) + }, + { id: 'direction', name: 'Direction', sort: false, formatter: (cell) => + gridjs.html(``) + }, + { id: 'time', name: 'Time', sort: false }, + ], + server: { + url: '{{ url_for('search.query_vehicle') }}', + then: results => results.data, + total: results => results.total, + }, + search: { + enabled: true, + debounceTimeout: 1000, + server: { + url: (prev, search) => { + return updateUrl(prev, {search}); + }, + }, + }, + pagination: { + enabled: true, + resetPageOnUpdate: true, + server: { + url: (prev, page, limit) => { + return updateUrl(prev, {start: page * limit, length: limit}); + }, + }, + }, + }).render(document.getElementById('Vehicles')); {% endblock javascripts %} diff --git a/apps/templates/home/vehicle-print.html b/apps/templates/home/vehicle-print.html new file mode 100644 index 0000000..13f33c9 --- /dev/null +++ b/apps/templates/home/vehicle-print.html @@ -0,0 +1,140 @@ +{% extends "layouts/base-print.html" %} + +{% block title %} {{ vehicle['vehicle_year_name'] }} {{ vehicle['vehicle_make_model_name'] }} {% endblock %} + + +{% block stylesheets %}{% endblock stylesheets %} + +{% block content %} + +
+
+
+
+
+
+
+ +
+
+

{{ settings.org_name }}

+
+
+
+
+
+
+

Vehicle Report

+
+
+
+
+
+ {% if vehicle['uuid_jpg'] %} + {{ vehicle['vehicle_year_name'] }} {{ vehicle['vehicle_make_model_name'] }} + {% elif vehicle['overview_jpeg'] %} + {{ vehicle['vehicle_year_name'] }} {{ vehicle['vehicle_make_model_name'] }} + {% endif %} +
+
+
+ {% if vehicle['vehicle_crop_jpeg'] %} + {{ vehicle['vehicle_year_name'] }} {{ vehicle['vehicle_make_model_name'] }} + {% endif %} +
+
+
+
+

+ {{ vehicle['vehicle_year_name'] }} {{ vehicle['vehicle_make_model_name'] }} +

+
+
+
+
+
+
+
Agent Information
+
+
Label:
+
{% if cached_agent %}{{ cached_agent.agent_label }}{% else %}{{ vehicle['agent_label'] }}{% endif %}
+
UID:
+
{{ vehicle['agent_uid'] }}
+
Version:
+
{{ vehicle['agent_version'] }}
+
Type:
+
{{ vehicle['agent_type'] }}
+
+
+
+
Camera Information
+
+
Label:
+
{% if cached_camera %}{{ cached_camera['camera_label'] }}{% else %}{{ license_plate['camera_label'] }}{% endif %}
+
ID:
+
{% if cached_camera %}{{ cached_camera['camera_id'] }}{% else %}{{ license_plate['camera_id'] }}{% endif %}
+
GPS Latitude:
+
{% if cached_camera %}{{ cached_camera['gps_latitude'] }}{% else %}{{ license_plate['gps_latitude'] }}{% endif %}
+
GPS Longitude:
+
{% if cached_camera %}{{ cached_camera['gps_longitude'] }}{% else %}{{ license_plate['gps_longitude'] }}{% endif %}
+
Country:
+
{% if cached_camera %}{{ cached_camera['country'] }}{% else %}{{ license_plate['country'] }}{% endif %}
+
+
+
+
+
+
Capture Information
+
+
Capture No.:
+
{{ vehicle['id'] }}
+
Start Timestamp:
+
{{ vehicle['epoch_start_datetime'] }}
+
End Timestamp:
+
{{ vehicle['epoch_end_datetime'] }}
+
Epoch Start:
+
{{ vehicle['epoch_start'] }}
+
Epoch End:
+
{{ vehicle['epoch_end'] }}
+
Vehicle Confidence:
+
{{ vehicle['is_vehicle_confidence_percent'] }}
+
Travel Direction:
+
({{ vehicle['travel_direction'] }}°)
+
+
+
+
Vehicle Information + + + + + + + +
+
+
Color:
+
{{ vehicle['vehicle_color_name'] }} ({{ vehicle['vehicle_color_confidence'] }})
+
Year:
+
{{ vehicle['vehicle_year_name'] }} ({{ vehicle['vehicle_year_confidence'] }})
+
Make:
+
{{ vehicle['vehicle_make_name'] }} ({{ vehicle['vehicle_make_confidence'] }})
+
Model:
+
{{ vehicle['vehicle_make_model_name'] }} ({{ vehicle['vehicle_make_model_confidence'] }})
+
Body Type:
+
{{ vehicle['vehicle_body_type_name'] }} ({{ vehicle['vehicle_body_type_confidence'] }})
+
+
+ +
+ +
+
+
+ +{% endblock content %} + + +{% block javascripts %}{% endblock javascripts %} diff --git a/apps/templates/home/vehicle.html b/apps/templates/home/vehicle.html new file mode 100644 index 0000000..a03c103 --- /dev/null +++ b/apps/templates/home/vehicle.html @@ -0,0 +1,288 @@ +{% extends "layouts/base.html" %} + +{% block title %} {{ vehicle['vehicle_year_name'] }} {{ vehicle['vehicle_make_model_name'] }} {% endblock %} + + +{% block stylesheets %} + + + +{% endblock stylesheets %} + +{% block content %} + +
+ +
+
+

Vehicle Report

+

+
+
+ + + + + + +
+
+
+
+
+
+
+ {% if vehicle['uuid_jpg'] %} + {{ vehicle['vehicle_year_name'] }} {{ vehicle['vehicle_make_model_name'] }} + {% elif vehicle['overview_jpeg'] %} + {{ vehicle['vehicle_year_name'] }} {{ vehicle['vehicle_make_model_name'] }} + {% endif %} +
+
+
+ {% if vehicle['vehicle_crop_jpeg'] %} + {{ vehicle['vehicle_year_name'] }} {{ vehicle['vehicle_make_model_name'] }} + {% endif %} +
+
+
+
+

+ {{ vehicle['vehicle_year_name'] }} {{ vehicle['vehicle_make_model_name'] }} +

+
+
+
+
+
+
+
Agent Information
+
+
Label:
+
{% if cached_agent %}{{ cached_agent.agent_label }}{% else %}{{ vehicle['agent_label'] }}{% endif %}
+
UID:
+
{{ vehicle['agent_uid'] }}
+
Version:
+
{{ vehicle['agent_version'] }}
+
Type:
+
{{ vehicle['agent_type'] }}
+
+
+
+
Camera Information
+
+
Label:
+
{% if cached_camera %}{{ cached_camera.camera_label }}{% else %}{{ vehicle['camera_label'] }}{% endif %}
+
ID:
+
{% if cached_camera %}{{ cached_camera.camera_id }}{% else %}{{ vehicle['camera_id'] }}{% endif %}
+
GPS Latitude:
+
+ {% if cached_camera %} + {% if cached_camera.gps_latitude != -1 and cached_camera.gps_longitude != -1 %} + {{ cached_camera.gps_latitude }} + {% else %} + {{ cached_camera['gps_latitude'] }} + {% endif %} + {% else %} + {% if vehicle['gps_latitude'] != -1 and vehicle['gps_longitude'] != -1 %} + {{ vehicle['gps_latitude'] }} + {% else %} + {{ vehicle['gps_latitude'] }} + {% endif %} + {% endif %} +
+
GPS Longitude:
+
+ {% if cached_camera %} + {% if cached_camera.gps_longitude != -1 and cached_camera.gps_latitude != -1 %} + {{ cached_camera.gps_longitude }} + {% else %} + {{ cached_camera.gps_longitude }} + {% endif %} + {% else %} + {% if vehicle['gps_longitude'] != -1 and vehicle['gps_latitude'] != -1 %} + {{ vehicle['gps_longitude'] }} + {% else %} + {{ vehicle['gps_longitude'] }} + {% endif %} + {% endif %} +
+
Country:
+
{% if cached_camera %}{{ cached_camera.country }}{% else %}{{ vehicle['country'] }}{% endif %}
+
+
+
+
+
+
Capture Information
+
+
Capture No.:
+
{{ vehicle['id'] }}
+
Start Timestamp:
+
{{ vehicle['epoch_start_datetime'] }}
+
End Timestamp:
+
{{ vehicle['epoch_end_datetime'] }}
+
Epoch Start:
+
{{ vehicle['epoch_start'] }}
+
Epoch End:
+
{{ vehicle['epoch_end'] }}
+
Vehicle Confidence:
+
{{ vehicle['is_vehicle_confidence_percent'] }}
+
Travel Direction:
+
({{ vehicle['travel_direction'] }}°)
+
+
+
+
Vehicle Information + + + + + + + +
+
+
Color:
+
{{ vehicle['vehicle_color_name'] }} ({{ vehicle['vehicle_color_confidence'] }})
+
Year:
+
{{ vehicle['vehicle_year_name'] }} ({{ vehicle['vehicle_year_confidence'] }})
+
Make:
+
{{ vehicle['vehicle_make_name'] }} ({{ vehicle['vehicle_make_confidence'] }})
+
Model:
+
{{ vehicle['vehicle_make_model_name'] }} ({{ vehicle['vehicle_make_model_confidence'] }})
+
Body Type:
+
{{ vehicle['vehicle_body_type_name'] }} ({{ vehicle['vehicle_body_type_confidence'] }})
+
+
+ +
+
+
+ +
+
+
+ +
+
+
+ +{% endblock content %} + + +{% block javascripts %} + + + + +{% endblock javascripts %} diff --git a/apps/templates/includes/sidebar.html b/apps/templates/includes/sidebar.html index e762c93..430c111 100644 --- a/apps/templates/includes/sidebar.html +++ b/apps/templates/includes/sidebar.html @@ -151,7 +151,7 @@

Hi, {{ current_user.full_name }}

@@ -186,7 +194,7 @@
Please follow this guide for a strong password:
-
+
User image
@@ -214,6 +222,10 @@