Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create tracking_chain-milestone_4.md #1039

Merged
merged 1 commit into from
Nov 15, 2023
Merged

Create tracking_chain-milestone_4.md #1039

merged 1 commit into from
Nov 15, 2023

Conversation

FedeC87p
Copy link
Contributor

Milestone Delivery Checklist

  • The milestone-delivery-template.md has been copied and updated.
  • This pull request is being made by the same account as the accepted application.
  • I have disclosed any and all sources of reused code in the submitted repositories and have done my due diligence to meet its license requirements.
  • In case of acceptance, an invoice must be submitted and the payment will be transferred to the BTC/ETH/fiat account provided in the application.
  • The delivery is according to the Guidelines for Milestone Deliverables.

Link to the application pull request: w3f/Grants-Program#1847

@keeganquigley
Copy link
Contributor

Thanks for the delivery @FedeC87p someone will look at it shortly.

@takahser takahser self-assigned this Nov 2, 2023
@takahser takahser self-requested a review November 2, 2023 08:59
@takahser
Copy link
Contributor

takahser commented Nov 6, 2023

@FedeC87p sorry for the delay here; we've a bit of a backlog currently. Just FYI, I'm working on the evaluation and I'll be back with more feedback soon.

@FedeC87p
Copy link
Contributor Author

FedeC87p commented Nov 7, 2023

@FedeC87p sorry for the delay here; we've a bit of a backlog currently. Just FYI, I'm working on the evaluation and I'll be back with more feedback soon.

Don't worry, there's no problem.

takahser
takahser requested changes Nov 8, 2023
View reviewed changes
Copy link
Contributor

@takahser takahser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@FedeC87p thanks for your patience, I added my evaluation here. Don't hesitate to ask questions, if anything is unclear.

@FedeC87p
Copy link
Contributor Author

FedeC87p commented Nov 8, 2023

@FedeC87p thanks for your patience, I added my evaluation here. Don't hesitate to ask questions, if anything is unclear.

Hi,
yes, for those who have never used TrackingChain it is probably best to start from the Step by Step guide:
https://github.com/TrackingChains/TrackingChain/wiki/Configuration-Step-By-Step

Furthermore, you can find the generic guide where all the features of the program are explained:
https://github.com/TrackingChains/TrackingChain/wiki

The docker compose allows you to replace many of the Step By Step operations, as it automatically loads all the services with the configurations and a pre SEED of the data in order to provide a ready-to-use application for different chains (Rococo Contract, Shibuya Ink , Shibuya EVM, MoonBase EVM).

@takahser
Copy link
Contributor

takahser commented Nov 8, 2023

@FedeC87p Do I have to go through the whole guide to test the functionality of M4 or is there a shortcut?

The docker compose allows you to replace many of the Step By Step operations, as it automatically loads all the services with the configurations and a pre SEED of the data in order to provide a ready-to-use application for different chains (Rococo Contract, Shibuya Ink , Shibuya EVM, MoonBase EVM).

While I appreciate your explanations I didn't quite get how they're relevant to the errors.

@FedeC87p
Copy link
Contributor Author

FedeC87p commented Nov 8, 2023

DockerCompose, the error is due to the first launch (as the services are running queries while the database is still being loaded), it is described in the link below.

https://github.com/TrackingChains/TrackingChain/wiki/Docker-Compose

I modified the docs to point directly to the project file

@FedeC87p
Copy link
Contributor Author

FedeC87p commented Nov 8, 2023

No, in my opinion you can start docker, which already has everything you need. (The errors that docker gives are temporary, so you can ignore them and restart any services if they were shut down due to the initial error).

In my opinion, once you start docker, just follow this guide:
https://github.com/TrackingChains/TrackingChain/wiki/Demo-Step-By-Step

Here you can upload an onchain data and then verify Milestone 4

If it is useful, I am also available for a video call

@takahser
Copy link
Contributor

takahser commented Nov 8, 2023

@FedeC87p thanks for the offer, I think a video call might make sense and accelerate the evaluation process. Feel free to book a slot here.

In the next versions a checker will be inserted to wait for the complete loading of the sql server.

Regarding this, is there any reason not to do it now already? AFAIK you can make services dependent on each other with docker compose.

@takahser
Copy link
Contributor

takahser commented Nov 9, 2023

@FedeC87p thanks for your time today. As discussed, I've updated the evaluation - the only thing missing is the resolution of the docker-compose startup bug.

@FedeC87p
Copy link
Contributor Author

FedeC87p commented Nov 9, 2023

@takahser thanks for the support.

I solved it by adding a check on each worker to check if the database is ready, this way it will only start the jobs in that case.

This way I can prevent even cases where you are not using Docker.

At most from the logs you will be able to find only one database access error (which is what is done to check if the database is ready).

You can already test the behavior by running docker-compose up

@takahser
Copy link
Contributor

@FedeC87p sorry for the delay here. I've pulled the latest changes (69742af590942439e95adab80cfe458f7637aa8e) and tried again, but I'm still getting the SqlExceptions and InvalidOperationExceptions. Note that I've removed the old containers and images before running docker compose up.

% docker compose up
[+] Building 0.0s (0/0)                                                                                                                                                      docker:desktop-linux
[+] Running 8/5
 ✔ Container trackingchain-trackingdb-1                                                                                                                      Created                         0.0s 
 ! trackingdb The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested                                 0.0s 
 ✔ Container trackingchain-triage-1                                                                                                                          Created                         0.0s 
 ✔ Container trackingchain-webapp-1                                                                                                                          Created                         0.0s 
 ✔ Container trackingchain-aggregator-pool-1                                                                                                                 Created                         0.0s 
 ✔ Container trackingchain-transaction-generator-1                                                                                                           Created                         0.0s 
 ✔ Container trackingchain-transaction-watcher-1                                                                                                             Created                         0.0s 
 ✔ Container trackingchain-monitor-1                                                                                                                         Created                         0.0s 
Attaching to trackingchain-aggregator-pool-1, trackingchain-monitor-1, trackingchain-trackingdb-1, trackingchain-transaction-generator-1, trackingchain-transaction-watcher-1, trackingchain-triage-1, trackingchain-webapp-1
trackingchain-trackingdb-1             | SQL Server 2022 will run as non-root by default.
trackingchain-trackingdb-1             | This container is running as user mssql.
trackingchain-trackingdb-1             | To learn more visit https://go.microsoft.com/fwlink/?linkid=2099216.
trackingchain-webapp-1                 | [15:09:28.135  WRN][]: Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.  Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository 
trackingchain-triage-1                 | [15:09:28.152  INF][]: Migration Db Worker running.  TrackingChain.TriageAPI.HostedService.MigratorDBHostedService 
trackingchain-webapp-1                 | [15:09:28.247  WRN][]: No XML encryptor configured. Key {6f6e6e77-8b98-4203-90c0-3c5a0c7a0b00} may be persisted to storage in unencrypted form.  Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager 
trackingchain-webapp-1                 | [15:09:28.291  WRN][]: Overriding address(es) 'http://+:80'. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead.  Microsoft.AspNetCore.Server.Kestrel 
trackingchain-webapp-1                 | [15:09:28.302  INF][]: Now listening on: http://0.0.0.0:12345  Microsoft.Hosting.Lifetime 
trackingchain-webapp-1                 | [15:09:28.302  INF][]: Application started. Press Ctrl+C to shut down.  Microsoft.Hosting.Lifetime 
trackingchain-webapp-1                 | [15:09:28.302  INF][]: Hosting environment: Production  Microsoft.Hosting.Lifetime 
trackingchain-webapp-1                 | [15:09:28.302  INF][]: Content root path: /app  Microsoft.Hosting.Lifetime 
trackingchain-aggregator-pool-1        | [15:09:28.562  WRN][]: The query uses the 'First'/'FirstOrDefault' operator without 'OrderBy' and filter operators. This may lead to unpredictable results.  Microsoft.EntityFrameworkCore.Query 
trackingchain-transaction-generator-1  | [15:09:28.657  WRN][]: The query uses the 'First'/'FirstOrDefault' operator without 'OrderBy' and filter operators. This may lead to unpredictable results.  Microsoft.EntityFrameworkCore.Query 
trackingchain-transaction-watcher-1    | [15:09:28.775  WRN][]: The query uses the 'First'/'FirstOrDefault' operator without 'OrderBy' and filter operators. This may lead to unpredictable results.  Microsoft.EntityFrameworkCore.Query 
trackingchain-monitor-1                | [15:09:28.920  WRN][]: The query uses the 'First'/'FirstOrDefault' operator without 'OrderBy' and filter operators. This may lead to unpredictable results.  Microsoft.EntityFrameworkCore.Query 
trackingchain-trackingdb-1             | 2023-11-14 15:09:31.69 Server      Setup step is copying system data file 'C:\templatedata\master.mdf' to '/var/opt/mssql/data/master.mdf'.
2023-11-14 15:09:31.72 Server      Did not find an existing master data file /var/opt/mssql/data/master.mdf, copying the missing default master and other system database files. If you have moved the database location, but not moved the database files, startup may fail. To repair: shutdown SQL Server, move the master database to configured location, and restart.
2023-11-14 15:09:31.72 Server      Setup step is copying system data file 'C:\templatedata\mastlog.ldf' to '/var/opt/mssql/data/mastlog.ldf'.
2023-11-14 15:09:31.73 Server      Setup step is copying system data file 'C:\templatedata\model.mdf' to '/var/opt/mssql/data/model.mdf'.
2023-11-14 15:09:31.73 Server      Setup step is copying system data file 'C:\templatedata\modellog.ldf' to '/var/opt/mssql/data/modellog.ldf'.
2023-11-14 15:09:31.74 Server      Setup step is copying system data file 'C:\templatedata\msdbdata.mdf' to '/var/opt/mssql/data/msdbdata.mdf'.
2023-11-14 15:09:31.75 Server      Setup step is copying system data file 'C:\templatedata\msdblog.ldf' to '/var/opt/mssql/data/msdblog.ldf'.
2023-11-14 15:09:31.75 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_replicatedmaster.mdf' to '/var/opt/mssql/data/model_replicatedmaster.mdf'.
2023-11-14 15:09:31.76 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_replicatedmaster.ldf' to '/var/opt/mssql/data/model_replicatedmaster.ldf'.
2023-11-14 15:09:31.76 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_msdbdata.mdf' to '/var/opt/mssql/data/model_msdbdata.mdf'.
2023-11-14 15:09:31.77 Server      Setup step is FORCE copying system data file 'C:\templatedata\model_msdblog.ldf' to '/var/opt/mssql/data/model_msdblog.ldf'.
2023-11-14 15:09:31.80 Server      Microsoft SQL Server 2022 (RTM-CU9) (KB5030731) - 16.0.4085.2 (X64) 
trackingSep 27 2023 12:05:43           | 
trackingCopyright (C) 2022 Microsoft Corporation
trackingDeveloper Edition (64-bit) on Linux (Ubuntu 20.04.6 LTS) <X64>
2023-11-14 15:09:31.80 Server      UTC adjustment: 0:00
2023-11-14 15:09:31.80 Server      (c) Microsoft Corporation.
2023-11-14 15:09:31.80 Server      All rights reserved.
2023-11-14 15:09:31.80 Server      Server process ID is 504.
2023-11-14 15:09:31.80 Server      Logging SQL Server messages in file '/var/opt/mssql/log/errorlog'.
2023-11-14 15:09:31.80 Server      Registry startup parameters: 
tracking -d /var/opt/mssql/data/master.mdf
tracking -l /var/opt/mssql/data/mastlog.ldf
tracking -e /var/opt/mssql/log/errorlog| 
2023-11-14 15:09:31.81 Server      SQL Server detected 1 sockets with 12 cores per socket and 12 logical processors per socket, 12 total logical processors; using 12 logical processors based on SQL Server licensing. This is an informational message; no user action is required.
2023-11-14 15:09:31.81 Server      SQL Server is starting at normal priority base (=7). This is an informational message only. No user action is required.
2023-11-14 15:09:31.81 Server      Detected 6276 MB of RAM. This is an informational message; no user action is required.
2023-11-14 15:09:31.81 Server      Using conventional memory in the memory manager.
2023-11-14 15:09:31.81 Server      Detected pause instruction latency: 0 cycles.
2023-11-14 15:09:31.81 Server      Page exclusion bitmap is enabled.
2023-11-14 15:09:31.85 Server      Buffer pool extension is not supported on Linux platform.
2023-11-14 15:09:31.85 Server      Buffer Pool: Allocating 1048576 bytes for 909398 hashPages.
2023-11-14 15:09:31.90 Server      Automatic soft-NUMA was enabled because SQL Server has detected hardware NUMA nodes with greater than 8 physical cores.
2023-11-14 15:09:31.99 Server      Buffer pool extension is already disabled. No action is necessary.
2023-11-14 15:09:32.38 Server      Installing Client TLS certificates to the store.
2023-11-14 15:09:32.39 Server      Error searching first file in /var/opt/mssql/security/ca-certificates: 3(The system cannot find the path specified.)
2023-11-14 15:09:32.39 Server      CPU vectorization level(s) detected:  SSE SSE2 SSE3 SSSE3 SSE41 SSE42 POPCNT
2023-11-14 15:09:32.45 Server      Successfully initialized the TLS configuration. Allowed TLS protocol versions are ['1.0 1.1 1.2']. Allowed TLS ciphers are ['ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:!DHE-RSA-AES256-GCM-SHA384:!DHE-RSA-AES128-GCM-SHA256:!DHE-RSA-AES256-SHA:!DHE-RSA-AES128-SHA'].
2023-11-14 15:09:32.50 Server      Query Store settings initialized with enabled = 1, 
SetThreadAffinity failed: 87           | 
2023-11-14 15:09:32.52 Server      The maximum number of dedicated administrator connections for this instance is '1'
2023-11-14 15:09:32.52 Server      Node configuration: node 0: CPU mask: 0x000000000000003f:0 Active CPU mask: 0x000000000000003f:0. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.
2023-11-14 15:09:32.52 Server      Node configuration: node 1: CPU mask: 0x0000000000000fc0:0 Active CPU mask: 0x0000000000000fc0:0. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.
2023-11-14 15:09:32.54 Server      Using dynamic lock allocation.  Initial allocation of 2500 Lock blocks and 5000 Lock Owner blocks per node.  This is an informational message only.  No user action is required.
2023-11-14 15:09:32.63 Server      In-Memory OLTP initialized on lowend machine.
2023-11-14 15:09:32.66 Server      CLR version v4.0.30319 loaded.
2023-11-14 15:09:32.69 Server      [INFO] Created Extended Events session 'hkenginexesession'
2023-11-14 15:09:32.69 Server      Database Instant File Initialization: enabled. For security and performance considerations see the topic 'Database Instant File Initialization' in SQL Server Books Online. This is an informational message only. No user action is required.
ForceFlush is enabled for this instance. 
2023-11-14 15:09:32.70 Server      Total Log Writer threads: 3. This is an informational message; no user action is required.
2023-11-14 15:09:32.71 Server      Software Usage Metrics is disabled.
2023-11-14 15:09:32.74 spid36s     [1]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0
2023-11-14 15:09:32.75 spid36s     Starting up database 'master'.
ForceFlush feature is enabled for log durability.
2023-11-14 15:09:32.88 spid36s     The tail of the log for database master is being rewritten to match the new sector size of 4096 bytes.  1024 bytes at offset 396288 in file /var/opt/mssql/data/mastlog.ldf will be written.
2023-11-14 15:09:33.07 spid36s     Converting database 'master' from version 927 to the current version 957.
2023-11-14 15:09:33.07 spid36s     Database 'master' running the upgrade step from version 927 to version 928.
2023-11-14 15:09:33.21 spid36s     Database 'master' running the upgrade step from version 928 to version 929.
2023-11-14 15:09:33.24 spid36s     Database 'master' running the upgrade step from version 929 to version 930.
2023-11-14 15:09:33.25 spid36s     Database 'master' running the upgrade step from version 930 to version 931.
2023-11-14 15:09:33.25 spid36s     Database 'master' running the upgrade step from version 931 to version 932.
2023-11-14 15:09:33.26 spid36s     Database 'master' running the upgrade step from version 932 to version 933.
2023-11-14 15:09:33.26 spid36s     Database 'master' running the upgrade step from version 933 to version 934.
2023-11-14 15:09:33.27 spid36s     Database 'master' running the upgrade step from version 934 to version 935.
2023-11-14 15:09:33.29 spid36s     Database 'master' running the upgrade step from version 935 to version 936.
2023-11-14 15:09:33.29 spid36s     Database 'master' running the upgrade step from version 936 to version 937.
2023-11-14 15:09:33.30 spid36s     Database 'master' running the upgrade step from version 937 to version 938.
2023-11-14 15:09:33.30 spid36s     Database 'master' running the upgrade step from version 938 to version 939.
2023-11-14 15:09:33.30 spid36s     Database 'master' running the upgrade step from version 939 to version 940.
2023-11-14 15:09:33.31 spid36s     Database 'master' running the upgrade step from version 940 to version 941.
2023-11-14 15:09:33.31 spid36s     Database 'master' running the upgrade step from version 941 to version 942.
2023-11-14 15:09:33.31 spid36s     Database 'master' running the upgrade step from version 942 to version 943.
2023-11-14 15:09:33.33 spid36s     Database 'master' running the upgrade step from version 943 to version 944.
2023-11-14 15:09:33.34 spid36s     Database 'master' running the upgrade step from version 944 to version 945.
2023-11-14 15:09:33.36 Server      External governance manager initialized
2023-11-14 15:09:33.37 spid36s     Database 'master' running the upgrade step from version 945 to version 946.
2023-11-14 15:09:33.38 spid36s     Database 'master' running the upgrade step from version 946 to version 947.
2023-11-14 15:09:33.38 spid36s     Database 'master' running the upgrade step from version 947 to version 948.
2023-11-14 15:09:33.39 spid36s     Database 'master' running the upgrade step from version 948 to version 949.
2023-11-14 15:09:33.40 spid36s     Database 'master' running the upgrade step from version 949 to version 950.
2023-11-14 15:09:33.40 spid36s     Database 'master' running the upgrade step from version 950 to version 951.
2023-11-14 15:09:33.41 spid36s     Database 'master' running the upgrade step from version 951 to version 952.
2023-11-14 15:09:33.41 spid36s     Database 'master' running the upgrade step from version 952 to version 953.
2023-11-14 15:09:33.42 spid36s     Database 'master' running the upgrade step from version 953 to version 954.
2023-11-14 15:09:33.42 spid36s     Database 'master' running the upgrade step from version 954 to version 955.
2023-11-14 15:09:33.43 spid36s     Database 'master' running the upgrade step from version 955 to version 956.
2023-11-14 15:09:33.46 spid36s     Database 'master' running the upgrade step from version 956 to version 957.
2023-11-14 15:09:33.56 spid38s     [32767]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0
2023-11-14 15:09:33.56 spid38s     Starting up database 'mssqlsystemresource'.
2023-11-14 15:09:33.57 spid38s     The resource database build version is 16.00.4085. This is an informational message only. No user action is required.
2023-11-14 15:09:33.57 spid47s     Password policy update was successful.
2023-11-14 15:09:33.59 spid38s     [3]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0
2023-11-14 15:09:33.59 spid38s     Starting up database 'model'.
2023-11-14 15:09:33.60 spid36s     [32762]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0
2023-11-14 15:09:33.60 spid36s     Starting up database 'model_replicatedmaster'.
2023-11-14 15:09:33.62 spid38s     The tail of the log for database model is being rewritten to match the new sector size of 4096 bytes.  2048 bytes at offset 75776 in file /var/opt/mssql/data/modellog.ldf will be written.
2023-11-14 15:09:33.63 spid36s     The tail of the log for database model_replicatedmaster is being rewritten to match the new sector size of 4096 bytes.  2560 bytes at offset 99840 in file /var/opt/mssql/data/model_replicatedmaster.ldf will be written.
2023-11-14 15:09:33.63 spid38s     Converting database 'model' from version 927 to the current version 957.
2023-11-14 15:09:33.63 spid38s     Database 'model' running the upgrade step from version 927 to version 928.
2023-11-14 15:09:33.64 spid38s     Database 'model' running the upgrade step from version 928 to version 929.
2023-11-14 15:09:33.66 spid36s     Converting database 'model_replicatedmaster' from version 927 to the current version 957.
2023-11-14 15:09:33.66 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 927 to version 928.
2023-11-14 15:09:33.66 spid38s     Database 'model' running the upgrade step from version 929 to version 930.
2023-11-14 15:09:33.66 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 928 to version 929.
2023-11-14 15:09:33.67 spid38s     Database 'model' running the upgrade step from version 930 to version 931.
2023-11-14 15:09:33.67 spid38s     Database 'model' running the upgrade step from version 931 to version 932.
2023-11-14 15:09:33.68 spid38s     Database 'model' running the upgrade step from version 932 to version 933.
2023-11-14 15:09:33.68 Server      Common language runtime (CLR) functionality initialized.
2023-11-14 15:09:33.69 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 929 to version 930.
2023-11-14 15:09:33.69 spid38s     Database 'model' running the upgrade step from version 933 to version 934.
2023-11-14 15:09:33.69 spid38s     Database 'model' running the upgrade step from version 934 to version 935.
2023-11-14 15:09:33.70 spid38s     Database 'model' running the upgrade step from version 935 to version 936.
2023-11-14 15:09:33.70 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 930 to version 931.
2023-11-14 15:09:33.70 spid38s     Database 'model' running the upgrade step from version 936 to version 937.
2023-11-14 15:09:33.70 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 931 to version 932.
2023-11-14 15:09:33.71 spid38s     Database 'model' running the upgrade step from version 937 to version 938.
2023-11-14 15:09:33.71 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 932 to version 933.
2023-11-14 15:09:33.71 spid38s     Database 'model' running the upgrade step from version 938 to version 939.
2023-11-14 15:09:33.71 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 933 to version 934.
2023-11-14 15:09:33.72 spid38s     Database 'model' running the upgrade step from version 939 to version 940.
2023-11-14 15:09:33.72 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 934 to version 935.
2023-11-14 15:09:33.72 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 935 to version 936.
2023-11-14 15:09:33.72 spid38s     Database 'model' running the upgrade step from version 940 to version 941.
2023-11-14 15:09:33.73 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 936 to version 937.
2023-11-14 15:09:33.73 spid38s     Database 'model' running the upgrade step from version 941 to version 942.
2023-11-14 15:09:33.73 spid38s     Database 'model' running the upgrade step from version 942 to version 943.
2023-11-14 15:09:33.73 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 937 to version 938.
2023-11-14 15:09:33.74 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 938 to version 939.
2023-11-14 15:09:33.74 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 939 to version 940.
2023-11-14 15:09:33.74 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 940 to version 941.
2023-11-14 15:09:33.75 spid38s     Database 'model' running the upgrade step from version 943 to version 944.
2023-11-14 15:09:33.75 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 941 to version 942.
2023-11-14 15:09:33.75 spid38s     Database 'model' running the upgrade step from version 944 to version 945.
2023-11-14 15:09:33.75 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 942 to version 943.
2023-11-14 15:09:33.76 spid38s     Database 'model' running the upgrade step from version 945 to version 946.
2023-11-14 15:09:33.76 spid38s     Database 'model' running the upgrade step from version 946 to version 947.
2023-11-14 15:09:33.77 spid38s     Database 'model' running the upgrade step from version 947 to version 948.
2023-11-14 15:09:33.77 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 943 to version 944.
2023-11-14 15:09:33.77 spid38s     Database 'model' running the upgrade step from version 948 to version 949.
2023-11-14 15:09:33.77 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 944 to version 945.
2023-11-14 15:09:33.78 spid38s     Database 'model' running the upgrade step from version 949 to version 950.
2023-11-14 15:09:33.78 spid38s     Database 'model' running the upgrade step from version 950 to version 951.
2023-11-14 15:09:33.78 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 945 to version 946.
2023-11-14 15:09:33.79 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 946 to version 947.
2023-11-14 15:09:33.79 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 947 to version 948.
2023-11-14 15:09:33.79 spid38s     Database 'model' running the upgrade step from version 951 to version 952.
2023-11-14 15:09:33.80 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 948 to version 949.
2023-11-14 15:09:33.80 spid38s     Database 'model' running the upgrade step from version 952 to version 953.
2023-11-14 15:09:33.80 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 949 to version 950.
2023-11-14 15:09:33.80 spid38s     Database 'model' running the upgrade step from version 953 to version 954.
2023-11-14 15:09:33.80 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 950 to version 951.
2023-11-14 15:09:33.81 spid38s     Database 'model' running the upgrade step from version 954 to version 955.
2023-11-14 15:09:33.81 spid38s     Database 'model' running the upgrade step from version 955 to version 956.
2023-11-14 15:09:33.81 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 951 to version 952.
2023-11-14 15:09:33.82 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 952 to version 953.
2023-11-14 15:09:33.82 spid38s     Database 'model' running the upgrade step from version 956 to version 957.
2023-11-14 15:09:33.82 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 953 to version 954.
2023-11-14 15:09:33.83 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 954 to version 955.
2023-11-14 15:09:33.84 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 955 to version 956.
2023-11-14 15:09:33.85 spid36s     Database 'model_replicatedmaster' running the upgrade step from version 956 to version 957.
2023-11-14 15:09:33.89 spid36s     [32761]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0
2023-11-14 15:09:33.89 spid36s     Starting up database 'model_msdb'.
2023-11-14 15:09:33.92 spid36s     Converting database 'model_msdb' from version 927 to the current version 957.
2023-11-14 15:09:33.92 spid36s     Database 'model_msdb' running the upgrade step from version 927 to version 928.
2023-11-14 15:09:33.92 spid36s     Database 'model_msdb' running the upgrade step from version 928 to version 929.
2023-11-14 15:09:33.94 spid36s     Database 'model_msdb' running the upgrade step from version 929 to version 930.
2023-11-14 15:09:33.94 spid36s     Database 'model_msdb' running the upgrade step from version 930 to version 931.
2023-11-14 15:09:33.95 spid36s     Database 'model_msdb' running the upgrade step from version 931 to version 932.
2023-11-14 15:09:33.95 spid36s     Database 'model_msdb' running the upgrade step from version 932 to version 933.
2023-11-14 15:09:33.96 spid36s     Database 'model_msdb' running the upgrade step from version 933 to version 934.
2023-11-14 15:09:33.96 spid36s     Database 'model_msdb' running the upgrade step from version 934 to version 935.
2023-11-14 15:09:33.96 spid36s     Database 'model_msdb' running the upgrade step from version 935 to version 936.
2023-11-14 15:09:33.96 spid36s     Database 'model_msdb' running the upgrade step from version 936 to version 937.
2023-11-14 15:09:33.97 spid36s     Database 'model_msdb' running the upgrade step from version 937 to version 938.
2023-11-14 15:09:33.97 spid36s     Database 'model_msdb' running the upgrade step from version 938 to version 939.
2023-11-14 15:09:33.97 spid36s     Database 'model_msdb' running the upgrade step from version 939 to version 940.
2023-11-14 15:09:33.98 spid36s     Database 'model_msdb' running the upgrade step from version 940 to version 941.
2023-11-14 15:09:33.98 spid36s     Database 'model_msdb' running the upgrade step from version 941 to version 942.
2023-11-14 15:09:33.98 spid36s     Database 'model_msdb' running the upgrade step from version 942 to version 943.
2023-11-14 15:09:34.00 spid36s     Database 'model_msdb' running the upgrade step from version 943 to version 944.
2023-11-14 15:09:34.00 spid36s     Database 'model_msdb' running the upgrade step from version 944 to version 945.
2023-11-14 15:09:34.01 spid36s     Database 'model_msdb' running the upgrade step from version 945 to version 946.
2023-11-14 15:09:34.01 spid36s     Database 'model_msdb' running the upgrade step from version 946 to version 947.
2023-11-14 15:09:34.02 spid36s     Database 'model_msdb' running the upgrade step from version 947 to version 948.
2023-11-14 15:09:34.02 spid36s     Database 'model_msdb' running the upgrade step from version 948 to version 949.
2023-11-14 15:09:34.03 spid36s     Database 'model_msdb' running the upgrade step from version 949 to version 950.
2023-11-14 15:09:34.03 spid36s     Database 'model_msdb' running the upgrade step from version 950 to version 951.
2023-11-14 15:09:34.04 spid36s     Database 'model_msdb' running the upgrade step from version 951 to version 952.
2023-11-14 15:09:34.04 spid36s     Database 'model_msdb' running the upgrade step from version 952 to version 953.
2023-11-14 15:09:34.04 spid47s     A self-generated certificate was successfully loaded for encryption.
2023-11-14 15:09:34.04 spid47s     Server is listening on [ 'any' <ipv6> 1433] accept sockets 1.
2023-11-14 15:09:34.05 spid47s     Server is listening on [ 'any' <ipv4> 1433] accept sockets 1.
2023-11-14 15:09:34.05 spid36s     Database 'model_msdb' running the upgrade step from version 953 to version 954.
2023-11-14 15:09:34.05 Server      Server is listening on [ ::1 <ipv6> 1434] accept sockets 1.
2023-11-14 15:09:34.05 Server      Server is listening on [ 127.0.0.1 <ipv4> 1434] accept sockets 1.
2023-11-14 15:09:34.06 Server      Dedicated admin connection support was established for listening locally on port 1434.
2023-11-14 15:09:34.06 spid36s     Database 'model_msdb' running the upgrade step from version 954 to version 955.
2023-11-14 15:09:34.06 spid36s     Database 'model_msdb' running the upgrade step from version 955 to version 956.
2023-11-14 15:09:34.06 spid47s     Server is listening on [ ::1 <ipv6> 1431] accept sockets 1.
2023-11-14 15:09:34.07 spid47s     Server is listening on [ 127.0.0.1 <ipv4> 1431] accept sockets 1.
2023-11-14 15:09:34.07 spid47s     SQL Server is now ready for client connections. This is an informational message; no user action is required.
2023-11-14 15:09:34.08 spid36s     Database 'model_msdb' running the upgrade step from version 956 to version 957.
2023-11-14 15:09:34.10 spid49s     Attribute synchronization initialized
2023-11-14 15:09:34.10 spid49s     Attribute synchronization manager initialized
2023-11-14 15:09:34.10 spid36s     Resource governor reconfiguration succeeded.
2023-11-14 15:09:34.11 spid36s     SQL Server Audit is starting the audits. This is an informational message. No user action is required.
2023-11-14 15:09:34.11 spid36s     SQL Server Audit has started the audits. This is an informational message. No user action is required.
2023-11-14 15:09:34.11 spid36s     XE session 'system_health' started.
2023-11-14 15:09:34.13 spid36s     SQL Trace ID 1 was started by login "sa".
2023-11-14 15:09:34.14 spid36s     Server name is 'e4dddefee4a5'. This is an informational message only. No user action is required.
2023-11-14 15:09:34.16 spid38s     Clearing tempdb database.
2023-11-14 15:09:34.16 spid55s     Always On: The availability replica manager is starting. This is an informational message only. No user action is required.
2023-11-14 15:09:34.16 spid55s     Always On: The availability replica manager is waiting for the instance of SQL Server to allow client connections. This is an informational message only. No user action is required.
2023-11-14 15:09:34.16 spid58s     [4]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0
2023-11-14 15:09:34.17 spid58s     Starting up database 'msdb'.
2023-11-14 15:09:34.21 spid58s     Converting database 'msdb' from version 927 to the current version 957.
2023-11-14 15:09:34.21 spid58s     Database 'msdb' running the upgrade step from version 927 to version 928.
2023-11-14 15:09:34.23 spid58s     Database 'msdb' running the upgrade step from version 928 to version 929.
2023-11-14 15:09:34.24 spid38s     [2]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0
2023-11-14 15:09:34.24 spid38s     Starting up database 'tempdb'.
2023-11-14 15:09:34.25 spid58s     Database 'msdb' running the upgrade step from version 929 to version 930.
2023-11-14 15:09:34.26 spid58s     Database 'msdb' running the upgrade step from version 930 to version 931.
2023-11-14 15:09:34.27 spid58s     Database 'msdb' running the upgrade step from version 931 to version 932.
2023-11-14 15:09:34.27 spid58s     Database 'msdb' running the upgrade step from version 932 to version 933.
2023-11-14 15:09:34.28 spid38s     The tempdb database has 1 data file(s).
2023-11-14 15:09:34.28 spid58s     Database 'msdb' running the upgrade step from version 933 to version 934.
2023-11-14 15:09:34.28 spid55s     The Service Broker endpoint is in disabled or stopped state.
2023-11-14 15:09:34.28 spid55s     The Database Mirroring endpoint is in disabled or stopped state.
2023-11-14 15:09:34.29 spid58s     Database 'msdb' running the upgrade step from version 934 to version 935.
2023-11-14 15:09:34.29 spid58s     Database 'msdb' running the upgrade step from version 935 to version 936.
2023-11-14 15:09:34.30 spid55s     Service Broker manager has started.
2023-11-14 15:09:34.30 spid58s     Database 'msdb' running the upgrade step from version 936 to version 937.
2023-11-14 15:09:34.31 spid58s     Database 'msdb' running the upgrade step from version 937 to version 938.
2023-11-14 15:09:34.32 spid58s     Database 'msdb' running the upgrade step from version 938 to version 939.
2023-11-14 15:09:34.33 Logon       Error: 18456, Severity: 14, State: 38.
2023-11-14 15:09:34.33 Logon       Login failed for user 'sa'. Reason: Failed to open the explicitly specified database 'TrackingChain'. [CLIENT: 172.18.0.5]
2023-11-14 15:09:34.33 Logon       Error: 18456, Severity: 14, State: 38.
2023-11-14 15:09:34.33 Logon       Login failed for user 'sa'. Reason: Failed to open the explicitly specified database 'TrackingChain'. [CLIENT: 172.18.0.3]
trackingchain-aggregator-pool-1        | [15:09:34.343  ERR][]: An error occurred using the connection to database 'TrackingChain' on server 'trackingdb,1433'.  Microsoft.EntityFrameworkCore.Database.Connection 
2023-11-14 15:09:34.34 spid58s     Database 'msdb' running the upgrade step from version 939 to version 940.
2023-11-14 15:09:34.34 spid58s     Database 'msdb' running the upgrade step from version 940 to version 941.
trackingchain-aggregator-pool-1        | [15:09:34.351  ERR][]: An exception occurred while iterating over the results of a query for context type 'TrackingChain.TrackingChainCore.EntityFramework.Context.ApplicationDbContext'.
trackingchain-aggregator-pool-1        | System.InvalidOperationException: An exception has been raised that is likely due to a transient failure. Consider enabling transient error resiliency by adding 'EnableRetryOnFailure' to the 'UseSqlServer' call.
trackingchain-aggregator-pool-1        |  ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot open database "TrackingChain" requested by the login. The login failed.
trackingchain-aggregator-pool-1        | Login failed for user 'sa'.
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()
trackingchain-aggregator-pool-1        | --- End of stack trace from previous location ---
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        | ClientConnectionId:e890a20e-3001-42c8-bfdb-fccd209a6772
trackingchain-aggregator-pool-1        | Error Number:4060,State:1,Class:11
trackingchain-aggregator-pool-1        |    --- End of inner exception stack trace ---
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()  Microsoft.EntityFrameworkCore.Query 
trackingchain-aggregator-pool-1        | System.InvalidOperationException: An exception has been raised that is likely due to a transient failure. Consider enabling transient error resiliency by adding 'EnableRetryOnFailure' to the 'UseSqlServer' call.
trackingchain-aggregator-pool-1        |  ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot open database "TrackingChain" requested by the login. The login failed.
trackingchain-aggregator-pool-1        | Login failed for user 'sa'.
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()
trackingchain-aggregator-pool-1        | --- End of stack trace from previous location ---
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        | ClientConnectionId:e890a20e-3001-42c8-bfdb-fccd209a6772
trackingchain-aggregator-pool-1        | Error Number:4060,State:1,Class:11
trackingchain-aggregator-pool-1        |    --- End of inner exception stack trace ---
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
2023-11-14 15:09:34.34 spid58s     Database 'msdb' running the upgrade step from version 941 to version 942.
trackingchain-aggregator-pool-1        | [15:09:34.355  WRN][]: Waiting for database  TrackingChain.AggregatorPoolWorker.WaitingDBHostedService 
trackingchain-aggregator-pool-1        | System.InvalidOperationException: An exception has been raised that is likely due to a transient failure. Consider enabling transient error resiliency by adding 'EnableRetryOnFailure' to the 'UseSqlServer' call.
trackingchain-aggregator-pool-1        |  ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot open database "TrackingChain" requested by the login. The login failed.
trackingchain-aggregator-pool-1        | Login failed for user 'sa'.
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
2023-11-14 15:09:34.35 spid58s     Database 'msdb' running the upgrade step from version 942 to version 943.
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
trackingchain-aggregator-pool-1        |    at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()
trackingchain-aggregator-pool-1        | --- End of stack trace from previous location ---
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        | ClientConnectionId:e890a20e-3001-42c8-bfdb-fccd209a6772
trackingchain-aggregator-pool-1        | Error Number:4060,State:1,Class:11
trackingchain-aggregator-pool-1        |    --- End of inner exception stack trace ---
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at TrackingChain.AggregatorPoolWorker.WaitingDBHostedService.StartAsync(CancellationToken cancellationToken) in /src/src/AggregatorPool.Worker/WaitingDBHostedService.cs:line 35
2023-11-14 15:09:34.37 spid58s     Database 'msdb' running the upgrade step from version 943 to version 944.
2023-11-14 15:09:34.37 spid58s     Database 'msdb' running the upgrade step from version 944 to version 945.
2023-11-14 15:09:34.38 Logon       Error: 18456, Severity: 14, State: 38.
2023-11-14 15:09:34.38 Logon       Login failed for user 'sa'. Reason: Failed to open the explicitly specified database 'TrackingChain'. [CLIENT: 172.18.0.6]
trackingchain-transaction-generator-1  | [15:09:34.390  ERR][]: An error occurred using the connection to database 'TrackingChain' on server 'trackingdb,1433'.  Microsoft.EntityFrameworkCore.Database.Connection 
2023-11-14 15:09:34.38 spid58s     Database 'msdb' running the upgrade step from version 945 to version 946.
2023-11-14 15:09:34.39 spid58s     Database 'msdb' running the upgrade step from version 946 to version 947.
trackingchain-transaction-generator-1  | [15:09:34.396  ERR][]: An exception occurred while iterating over the results of a query for context type 'TrackingChain.TrackingChainCore.EntityFramework.Context.ApplicationDbContext'.
trackingchain-transaction-generator-1  | System.InvalidOperationException: An exception has been raised that is likely due to a transient failure. Consider enabling transient error resiliency by adding 'EnableRetryOnFailure' to the 'UseSqlServer' call.
trackingchain-transaction-generator-1  |  ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot open database "TrackingChain" requested by the login. The login failed.
trackingchain-transaction-generator-1  | Login failed for user 'sa'.
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
trackingchain-transaction-generator-1  |    at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
trackingchain-transaction-generator-1  |    at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
trackingchain-transaction-generator-1  |    at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
trackingchain-transaction-generator-1  |    at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
trackingchain-transaction-generator-1  |    at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()
trackingchain-transaction-generator-1  | --- End of stack trace from previous location ---
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  | ClientConnectionId:ef20c2a8-eacc-4fa2-85e7-1260ca5f4769
trackingchain-transaction-generator-1  | Error Number:4060,State:1,Class:11
trackingchain-transaction-generator-1  |    --- End of inner exception stack trace ---
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()  Microsoft.EntityFrameworkCore.Query 
trackingchain-transaction-generator-1  | System.InvalidOperationException: An exception has been raised that is likely due to a transient failure. Consider enabling transient error resiliency by adding 'EnableRetryOnFailure' to the 'UseSqlServer' call.
trackingchain-transaction-generator-1  |  ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot open database "TrackingChain" requested by the login. The login failed.
trackingchain-transaction-generator-1  | Login failed for user 'sa'.
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
trackingchain-transaction-generator-1  |    at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
trackingchain-transaction-generator-1  |    at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
trackingchain-transaction-generator-1  |    at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
trackingchain-transaction-generator-1  |    at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
trackingchain-transaction-generator-1  |    at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()
trackingchain-transaction-generator-1  | --- End of stack trace from previous location ---
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  | ClientConnectionId:ef20c2a8-eacc-4fa2-85e7-1260ca5f4769
trackingchain-transaction-generator-1  | Error Number:4060,State:1,Class:11
trackingchain-transaction-generator-1  |    --- End of inner exception stack trace ---
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
2023-11-14 15:09:34.39 spid58s     Database 'msdb' running the upgrade step from version 947 to version 948.
trackingchain-transaction-generator-1  | [15:09:34.401  WRN][]: Waiting for database  TrackingChain.TransactionGeneratorWorker.WaitingDBHostedService 
trackingchain-transaction-generator-1  | System.InvalidOperationException: An exception has been raised that is likely due to a transient failure. Consider enabling transient error resiliency by adding 'EnableRetryOnFailure' to the 'UseSqlServer' call.
trackingchain-transaction-generator-1  |  ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot open database "TrackingChain" requested by the login. The login failed.
trackingchain-transaction-generator-1  | Login failed for user 'sa'.
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
trackingchain-transaction-generator-1  |    at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
trackingchain-transaction-generator-1  |    at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
trackingchain-transaction-generator-1  |    at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
trackingchain-transaction-generator-1  |    at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
trackingchain-transaction-generator-1  |    at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()
trackingchain-transaction-generator-1  | --- End of stack trace from previous location ---
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  | ClientConnectionId:ef20c2a8-eacc-4fa2-85e7-1260ca5f4769
trackingchain-transaction-generator-1  | Error Number:4060,State:1,Class:11
trackingchain-transaction-generator-1  |    --- End of inner exception stack trace ---
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at TrackingChain.TransactionGeneratorWorker.WaitingDBHostedService.StartAsync(CancellationToken cancellationToken) in /src/src/TransactionGenerator.Worker/WaitingDBHostedService.cs:line 35
2023-11-14 15:09:34.40 spid58s     Database 'msdb' running the upgrade step from version 948 to version 949.
2023-11-14 15:09:34.40 spid58s     Database 'msdb' running the upgrade step from version 949 to version 950.
2023-11-14 15:09:34.41 spid58s     Database 'msdb' running the upgrade step from version 950 to version 951.
2023-11-14 15:09:34.42 spid58s     Database 'msdb' running the upgrade step from version 951 to version 952.
2023-11-14 15:09:34.42 spid58s     Database 'msdb' running the upgrade step from version 952 to version 953.
2023-11-14 15:09:34.43 spid58s     Database 'msdb' running the upgrade step from version 953 to version 954.
2023-11-14 15:09:34.43 spid58s     Database 'msdb' running the upgrade step from version 954 to version 955.
2023-11-14 15:09:34.44 spid58s     Database 'msdb' running the upgrade step from version 955 to version 956.
2023-11-14 15:09:34.46 spid58s     Database 'msdb' running the upgrade step from version 956 to version 957.
2023-11-14 15:09:34.48 spid75      [5]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0
2023-11-14 15:09:34.48 spid75      Starting up database 'TrackingChain'.
2023-11-14 15:09:34.48 spid75      RemoveStaleDbEntries: Cleanup of stale DB entries called for database ID: [5]
2023-11-14 15:09:34.48 spid75      RemoveStaleDbEntries: Cleanup of stale DB entries skipped because master db is not memory optimized. DbId: 5.
2023-11-14 15:09:34.51 spid75      Parallel redo is started for database 'TrackingChain' with worker pool size [6].
2023-11-14 15:09:34.52 spid75      Parallel redo is shutdown for database 'TrackingChain' with worker pool size [6].
2023-11-14 15:09:34.55 spid36s     Recovery is complete. This is an informational message only. No user action is required.
trackingchain-transaction-watcher-1    | [15:09:34.595  ERR][]: Failed executing DbCommand (32ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
trackingchain-transaction-watcher-1    | SELECT TOP(1) [a].[Id], [a].[ChainWatcherAddress], [a].[ChainWriterAddress], [a].[Name], [a].[PrivateKey]
trackingchain-transaction-watcher-1    | FROM [Accounts] AS [a]  Microsoft.EntityFrameworkCore.Database.Command 
trackingchain-transaction-watcher-1    | [15:09:34.604  ERR][]: An exception occurred while iterating over the results of a query for context type 'TrackingChain.TrackingChainCore.EntityFramework.Context.ApplicationDbContext'.
trackingchain-transaction-watcher-1    | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-transaction-watcher-1    |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-transaction-watcher-1    |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-transaction-watcher-1    |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-watcher-1    | --- End of stack trace from previous location ---
trackingchain-transaction-watcher-1    |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-watcher-1    |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-transaction-watcher-1    | --- End of stack trace from previous location ---
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-transaction-watcher-1    | ClientConnectionId:bcd9dffd-e6b3-4b7a-8ae8-c83a96b85428
trackingchain-transaction-watcher-1    | Error Number:208,State:1,Class:16  Microsoft.EntityFrameworkCore.Query 
trackingchain-transaction-watcher-1    | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-transaction-watcher-1    |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-transaction-watcher-1    |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-transaction-watcher-1    |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-watcher-1    | --- End of stack trace from previous location ---
trackingchain-transaction-watcher-1    |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-watcher-1    |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-transaction-watcher-1    | --- End of stack trace from previous location ---
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-transaction-watcher-1    | ClientConnectionId:bcd9dffd-e6b3-4b7a-8ae8-c83a96b85428
trackingchain-transaction-watcher-1    | Error Number:208,State:1,Class:16
trackingchain-transaction-watcher-1    | [15:09:34.608  WRN][]: Waiting for database  TrackingChain.TransactionWatcherWorker.WaitingDBHostedService 
trackingchain-transaction-watcher-1    | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-transaction-watcher-1    |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-transaction-watcher-1    |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-transaction-watcher-1    |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-watcher-1    | --- End of stack trace from previous location ---
trackingchain-transaction-watcher-1    |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-watcher-1    |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-transaction-watcher-1    | --- End of stack trace from previous location ---
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at TrackingChain.TransactionWatcherWorker.WaitingDBHostedService.StartAsync(CancellationToken cancellationToken) in /src/src/TransactionWatcher.Worker/WaitingDBHostedService.cs:line 35
trackingchain-transaction-watcher-1    | ClientConnectionId:bcd9dffd-e6b3-4b7a-8ae8-c83a96b85428
trackingchain-transaction-watcher-1    | Error Number:208,State:1,Class:16
trackingchain-monitor-1                | [15:09:34.732  ERR][]: Failed executing DbCommand (21ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
trackingchain-monitor-1                | SELECT TOP(1) [a].[Id], [a].[ChainWatcherAddress], [a].[ChainWriterAddress], [a].[Name], [a].[PrivateKey]
trackingchain-monitor-1                | FROM [Accounts] AS [a]  Microsoft.EntityFrameworkCore.Database.Command 
trackingchain-monitor-1                | [15:09:34.740  ERR][]: An exception occurred while iterating over the results of a query for context type 'TrackingChain.TrackingChainCore.EntityFramework.Context.ApplicationDbContext'.
trackingchain-monitor-1                | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-monitor-1                |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-monitor-1                |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-monitor-1                |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-monitor-1                | --- End of stack trace from previous location ---
trackingchain-monitor-1                |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-monitor-1                |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-monitor-1                | --- End of stack trace from previous location ---
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-monitor-1                | ClientConnectionId:39800fd5-78d3-41b3-abb6-0028a1ea8522
trackingchain-monitor-1                | Error Number:208,State:1,Class:16  Microsoft.EntityFrameworkCore.Query 
trackingchain-monitor-1                | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-monitor-1                |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-monitor-1                |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-monitor-1                |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-monitor-1                | --- End of stack trace from previous location ---
trackingchain-monitor-1                |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-monitor-1                |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-monitor-1                | --- End of stack trace from previous location ---
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-monitor-1                | ClientConnectionId:39800fd5-78d3-41b3-abb6-0028a1ea8522
trackingchain-monitor-1                | Error Number:208,State:1,Class:16
trackingchain-monitor-1                | [15:09:34.743  WRN][]: Waiting for database  TrackingChain.TransactionMonitorWorker.WaitingDBHostedService 
trackingchain-monitor-1                | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-monitor-1                |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-monitor-1                |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-monitor-1                |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-monitor-1                | --- End of stack trace from previous location ---
trackingchain-monitor-1                |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-monitor-1                |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-monitor-1                | --- End of stack trace from previous location ---
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at TrackingChain.TransactionMonitorWorker.WaitingDBHostedService.StartAsync(CancellationToken cancellationToken) in /src/src/TransactionMonitor.Worker/WaitingDBHostedService.cs:line 35
trackingchain-monitor-1                | ClientConnectionId:39800fd5-78d3-41b3-abb6-0028a1ea8522
trackingchain-monitor-1                | Error Number:208,State:1,Class:16
2023-11-14 15:09:34.84 spid45s     The default language (LCID 0) has been set for engine and full-text services.
2023-11-14 15:09:34.99 spid45s     The tempdb database has 8 data file(s).
trackingchain-aggregator-pool-1        | [15:09:39.442  ERR][]: Failed executing DbCommand (23ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
trackingchain-aggregator-pool-1        | SELECT TOP(1) [a].[Id], [a].[ChainWatcherAddress], [a].[ChainWriterAddress], [a].[Name], [a].[PrivateKey]
trackingchain-aggregator-pool-1        | FROM [Accounts] AS [a]  Microsoft.EntityFrameworkCore.Database.Command 
trackingchain-aggregator-pool-1        | [15:09:39.447  ERR][]: An exception occurred while iterating over the results of a query for context type 'TrackingChain.TrackingChainCore.EntityFramework.Context.ApplicationDbContext'.
trackingchain-aggregator-pool-1        | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-aggregator-pool-1        |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-aggregator-pool-1        |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-aggregator-pool-1        | --- End of stack trace from previous location ---
trackingchain-aggregator-pool-1        |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-aggregator-pool-1        |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-aggregator-pool-1        | --- End of stack trace from previous location ---
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-aggregator-pool-1        | ClientConnectionId:90f9466c-ede8-4ca0-84f1-0ffd0c113ceb
trackingchain-aggregator-pool-1        | Error Number:208,State:1,Class:16  Microsoft.EntityFrameworkCore.Query 
trackingchain-aggregator-pool-1        | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-aggregator-pool-1        |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-aggregator-pool-1        |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-aggregator-pool-1        | --- End of stack trace from previous location ---
trackingchain-aggregator-pool-1        |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-aggregator-pool-1        |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-aggregator-pool-1        | --- End of stack trace from previous location ---
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-aggregator-pool-1        | ClientConnectionId:90f9466c-ede8-4ca0-84f1-0ffd0c113ceb
trackingchain-aggregator-pool-1        | Error Number:208,State:1,Class:16
trackingchain-aggregator-pool-1        | [15:09:39.448  WRN][]: Waiting for database  TrackingChain.AggregatorPoolWorker.WaitingDBHostedService 
trackingchain-aggregator-pool-1        | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-aggregator-pool-1        |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-aggregator-pool-1        |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-aggregator-pool-1        |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-aggregator-pool-1        | --- End of stack trace from previous location ---
trackingchain-aggregator-pool-1        |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-aggregator-pool-1        |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-aggregator-pool-1        | --- End of stack trace from previous location ---
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
trackingchain-aggregator-pool-1        |    at TrackingChain.AggregatorPoolWorker.WaitingDBHostedService.StartAsync(CancellationToken cancellationToken) in /src/src/AggregatorPool.Worker/WaitingDBHostedService.cs:line 35
trackingchain-aggregator-pool-1        | ClientConnectionId:90f9466c-ede8-4ca0-84f1-0ffd0c113ceb
trackingchain-aggregator-pool-1        | Error Number:208,State:1,Class:16
trackingchain-transaction-generator-1  | [15:09:39.457  ERR][]: Failed executing DbCommand (19ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
trackingchain-transaction-generator-1  | SELECT TOP(1) [a].[Id], [a].[ChainWatcherAddress], [a].[ChainWriterAddress], [a].[Name], [a].[PrivateKey]
trackingchain-transaction-generator-1  | FROM [Accounts] AS [a]  Microsoft.EntityFrameworkCore.Database.Command 
trackingchain-transaction-generator-1  | [15:09:39.461  ERR][]: An exception occurred while iterating over the results of a query for context type 'TrackingChain.TrackingChainCore.EntityFramework.Context.ApplicationDbContext'.
trackingchain-transaction-generator-1  | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-transaction-generator-1  |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-transaction-generator-1  |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-generator-1  | --- End of stack trace from previous location ---
trackingchain-transaction-generator-1  |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-generator-1  |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-transaction-generator-1  | --- End of stack trace from previous location ---
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-transaction-generator-1  | ClientConnectionId:ef9b2cac-d0b6-4dee-a1c7-c15c5972dbfb
trackingchain-transaction-generator-1  | Error Number:208,State:1,Class:16  Microsoft.EntityFrameworkCore.Query 
trackingchain-transaction-generator-1  | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-transaction-generator-1  |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-transaction-generator-1  |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-generator-1  | --- End of stack trace from previous location ---
trackingchain-transaction-generator-1  |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-generator-1  |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-transaction-generator-1  | --- End of stack trace from previous location ---
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-transaction-generator-1  | ClientConnectionId:ef9b2cac-d0b6-4dee-a1c7-c15c5972dbfb
trackingchain-transaction-generator-1  | Error Number:208,State:1,Class:16
trackingchain-transaction-generator-1  | [15:09:39.462  WRN][]: Waiting for database  TrackingChain.TransactionGeneratorWorker.WaitingDBHostedService 
trackingchain-transaction-generator-1  | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-transaction-generator-1  |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-transaction-generator-1  |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-transaction-generator-1  |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-generator-1  | --- End of stack trace from previous location ---
trackingchain-transaction-generator-1  |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-generator-1  |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-transaction-generator-1  | --- End of stack trace from previous location ---
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
trackingchain-transaction-generator-1  |    at TrackingChain.TransactionGeneratorWorker.WaitingDBHostedService.StartAsync(CancellationToken cancellationToken) in /src/src/TransactionGenerator.Worker/WaitingDBHostedService.cs:line 35
trackingchain-transaction-generator-1  | ClientConnectionId:ef9b2cac-d0b6-4dee-a1c7-c15c5972dbfb
trackingchain-transaction-generator-1  | Error Number:208,State:1,Class:16
trackingchain-transaction-watcher-1    | [15:09:39.628  ERR][]: Failed executing DbCommand (6ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
trackingchain-transaction-watcher-1    | SELECT TOP(1) [a].[Id], [a].[ChainWatcherAddress], [a].[ChainWriterAddress], [a].[Name], [a].[PrivateKey]
trackingchain-transaction-watcher-1    | FROM [Accounts] AS [a]  Microsoft.EntityFrameworkCore.Database.Command 
trackingchain-transaction-watcher-1    | [15:09:39.629  ERR][]: An exception occurred while iterating over the results of a query for context type 'TrackingChain.TrackingChainCore.EntityFramework.Context.ApplicationDbContext'.
trackingchain-transaction-watcher-1    | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-transaction-watcher-1    |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-transaction-watcher-1    |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-transaction-watcher-1    |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-watcher-1    | --- End of stack trace from previous location ---
trackingchain-transaction-watcher-1    |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-watcher-1    |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-transaction-watcher-1    | --- End of stack trace from previous location ---
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-transaction-watcher-1    | ClientConnectionId:bcd9dffd-e6b3-4b7a-8ae8-c83a96b85428
trackingchain-transaction-watcher-1    | Error Number:208,State:1,Class:16  Microsoft.EntityFrameworkCore.Query 
trackingchain-transaction-watcher-1    | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-transaction-watcher-1    |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-transaction-watcher-1    |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-transaction-watcher-1    |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-watcher-1    | --- End of stack trace from previous location ---
trackingchain-transaction-watcher-1    |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-watcher-1    |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-transaction-watcher-1    | --- End of stack trace from previous location ---
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-transaction-watcher-1    | ClientConnectionId:bcd9dffd-e6b3-4b7a-8ae8-c83a96b85428
trackingchain-transaction-watcher-1    | Error Number:208,State:1,Class:16
trackingchain-transaction-watcher-1    | [15:09:39.630  WRN][]: Waiting for database  TrackingChain.TransactionWatcherWorker.WaitingDBHostedService 
trackingchain-transaction-watcher-1    | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-transaction-watcher-1    |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-transaction-watcher-1    |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-transaction-watcher-1    |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-watcher-1    | --- End of stack trace from previous location ---
trackingchain-transaction-watcher-1    |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-transaction-watcher-1    |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-transaction-watcher-1    | --- End of stack trace from previous location ---
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
trackingchain-transaction-watcher-1    |    at TrackingChain.TransactionWatcherWorker.WaitingDBHostedService.StartAsync(CancellationToken cancellationToken) in /src/src/TransactionWatcher.Worker/WaitingDBHostedService.cs:line 35
trackingchain-transaction-watcher-1    | ClientConnectionId:bcd9dffd-e6b3-4b7a-8ae8-c83a96b85428
trackingchain-transaction-watcher-1    | Error Number:208,State:1,Class:16
trackingchain-monitor-1                | [15:09:39.758  ERR][]: Failed executing DbCommand (3ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
trackingchain-monitor-1                | SELECT TOP(1) [a].[Id], [a].[ChainWatcherAddress], [a].[ChainWriterAddress], [a].[Name], [a].[PrivateKey]
trackingchain-monitor-1                | FROM [Accounts] AS [a]  Microsoft.EntityFrameworkCore.Database.Command 
trackingchain-monitor-1                | [15:09:39.758  ERR][]: An exception occurred while iterating over the results of a query for context type 'TrackingChain.TrackingChainCore.EntityFramework.Context.ApplicationDbContext'.
trackingchain-monitor-1                | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-monitor-1                |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-monitor-1                |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-monitor-1                |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-monitor-1                | --- End of stack trace from previous location ---
trackingchain-monitor-1                |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-monitor-1                |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-monitor-1                | --- End of stack trace from previous location ---
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-monitor-1                | ClientConnectionId:39800fd5-78d3-41b3-abb6-0028a1ea8522
trackingchain-monitor-1                | Error Number:208,State:1,Class:16  Microsoft.EntityFrameworkCore.Query 
trackingchain-monitor-1                | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-monitor-1                |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-monitor-1                |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-monitor-1                |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-monitor-1                | --- End of stack trace from previous location ---
trackingchain-monitor-1                |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-monitor-1                |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-monitor-1                | --- End of stack trace from previous location ---
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-monitor-1                | ClientConnectionId:39800fd5-78d3-41b3-abb6-0028a1ea8522
trackingchain-monitor-1                | Error Number:208,State:1,Class:16
trackingchain-monitor-1                | [15:09:39.759  WRN][]: Waiting for database  TrackingChain.TransactionMonitorWorker.WaitingDBHostedService 
trackingchain-monitor-1                | Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'Accounts'.
trackingchain-monitor-1                |    at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__208_0(Task`1 result)
trackingchain-monitor-1                |    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
trackingchain-monitor-1                |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-monitor-1                | --- End of stack trace from previous location ---
trackingchain-monitor-1                |    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
trackingchain-monitor-1                |    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
trackingchain-monitor-1                | --- End of stack trace from previous location ---
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
trackingchain-monitor-1                |    at TrackingChain.TransactionMonitorWorker.WaitingDBHostedService.StartAsync(CancellationToken cancellationToken) in /src/src/TransactionMonitor.Worker/WaitingDBHostedService.cs:line 35
trackingchain-monitor-1                | ClientConnectionId:39800fd5-78d3-41b3-abb6-0028a1ea8522
trackingchain-monitor-1                | Error Number:208,State:1,Class:16
^CGracefully stopping... (press Ctrl+C again to force)
Aborting on container exit...

@FedeC87p
Copy link
Contributor Author

Ok, from what I see in the logs however it is only one error for each service (it is the DB check that tries to verify if the database is ready, therefore it is correct that there is an error), the checkDb will ensure that as long as the Database is not ready will not start the service.

Unlike before you should no longer have any services shutting down.

You can see the warning from the logs:
[15:09:39.448 WRN][]: Waiting for database TrackingChain.AggregatorPoolWorker.WaitingDBHostedService

As soon as the databases are ready the messages should disappear. I preferred to continue logging these types of errors rather than hiding them even though the DB was being checked

@takahser takahser self-requested a review November 14, 2023 23:23
@takahser
Copy link
Contributor

@FedeC87p good call, let me have a look again tomorrow! I'll be back with feedback soon.

takahser
takahser approved these changes Nov 15, 2023
View reviewed changes
Copy link
Contributor

@takahser takahser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@FedeC87p you were right, the race conditions have been taken care of with your fix. I'm accepting this, you can find the final evaluation here.

@takahser takahser merged commit 6aec598 into w3f:master Nov 15, 2023
6 checks passed
@github-actions GitHub Actions
Copy link

🪙 Please fill out the invoice form in order to initiate the payment process. Thank you!

@github-actions GitHub Actions
Copy link

We noticed that this is the last milestone of your project. Congratulations on completing your grant! 🎊

So, where to from here? First of all, you should join our Grants Community chat, if you haven't already, so we can stay in touch.
If you are looking for continuative support for your project, there are quite a few options. The main goal of the W3F grants program is to support research as well as early-stage technical projects. If your project still falls under one of those categories, you might want to apply for a follow-up grant. However, depending on your goals and project status, there are other support programs in our ecosystem that might be better suited as the next step. For example, projects with a Business Case/Token should look into the Substrate Builders Program or VC Funding and Common Good projects have a good chance of receiving Treasury Funding. If you are looking for guidance, the team at https://substrate.io/ecosystem/square-one/ has compiled a list of ecosystem support sources and are happy to help you navigate it.

For a more comprehensive list, see our Alternative Funding page. Let us know if you have any questions regarding the above. We are more than happy to point you to additional resources and help you determine the best course of action.
Lastly, we hope your W3F grant was a success and we want to thank you for being part of the journey!

@RouvenP
Copy link

RouvenP commented Nov 29, 2023

hi @FedeC87p apologies for the delay in payment. We will settle the invoice before the end of the week and let you know as soon as done.

@FedeC87p
Copy link
Contributor Author

No problem, thanks for the update

@RouvenP
Copy link

RouvenP commented Dec 1, 2023

hi @FedeC87p we just paid the invoice. Thanks for your patience

@FedeC87p
Copy link
Contributor Author

FedeC87p commented Dec 1, 2023

Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@takahser takahser takahser approved these changes

Assignees

@takahser takahser

Labels
changes requested
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@FedeC87p @keeganquigley @takahser @RouvenP