New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Database schema optimizations #4731

Open

SergeyGaluzo wants to merge 51 commits into main from users/sergal/newschemarefsepFK

Contributor

SergeyGaluzo commented Nov 19, 2024 •

edited

Loading

Optimizations:

History resources split with removal of not needed indexes.
Raw resources in separate table.
Resource id integer mapping.

This is dependent on small schema change creating CurrentResource view.

SergeyGaluzo and others added 30 commits

September 27, 2024 08:42


          Raw resources in the Lake and Resource Id integer map

47c370f


          perf tweeks

5aca029


          Merge history separation with separated raw resources (#4685)

de5a156

* History separation V0

* Forgotten resource

* replace current by view

* TRUNCATE

* HardDelete

* delete history v0

* missed 64 and removed dead code from SQL query generator

* Removed history from current in merge

* Triggers

* tests and tran

* disable/enable indexes

* rollback hard delete and merge

* rollback delete invisible history

* TRUNCATE -> DELETE

* line

* leftovers

* PK check

* Cosmetic

* WHERE nanes

* Get resources with forced indexes

* Removed redundant where

* Adding feature flag for raw resource dedupping

* Enable invisible history by default

* parameters

* commit

* fixed typo

* 65 + more realistic diff

* adjusted update trigger

* right trigger

* next iteration

* special chars

* Added delete and update

* \r

* Adding script runner

* Added verification

* cosmetic

* HOLDLOCK hint

* removed incorrect comments

* RT

* lock timeout

* 180 and remove empty

* no changes in update resource search params

* blob rewriter tool

* Revert "no changes in update resource search params"

This reverts commit b2e0c38.

* comment

* Generic disable indexes and update search params

* Dummy resources

* fixes

* Dummy records based on surr id

* adjust test to filter dummy rows

* exclude history and current

* Adding PerfTest V-1

* Get asyn wrapper without type string to id function

* tool

* not exists on data copy

* packages back

* Added comments.

* Correct filtered index on resource current

* Added redundant IsHistory=0 to index

* deduping

* Removed dummy records

* pp-p

* history clause

* deduping

* testing parameters

* get resource by type and surr id range with many versions

* skip large databases

* Added calls to fhir

* added conflict

* max retries = 3

* database pings

* examples

* reverse

* start closer

* put logic

* Create

* Move 65 to 84

* Fixes after merge

* Adding RawResources table

* change capture tweeks

* Test fix

* Added MI

* inline insert

* Repeat on updates


          Merge resource id map

e37ffd4


          Fix after merge

5d8d03a


          Fix after merge

cc94008


          adjustments

d6ce380


          tests

2950bcf


          Merge from main

8e74731


          Merge fixes

c3ccf69


          Ignore NULL reference resource tyoe

b0ed1eb


          test fix

51a942d


          default number of columns = 13

38e869b


          commented 84 diff

bb535af


          set end date on enqueue


          rename secondary to adls

427cdfa


          defrag formatting

02a545f


          set timeout to 0 for getting frag

fbbaf20


          isnull on input param

3f5dcd9


          Merge branch 'main' into users/sergal/rawmaplakeandhistoryandresource…

e6e3fe9

…idmap


          after merge from main

5569ea1


          Merge branch 'users/sergal/rawmaplakeandhistoryandresourceidmap' into…

2648ecd

… users/sergal/newschemarefsep


          Reference separation

037061d


          file id

99c6620


          small things

0b226bf


          Do not write raw resource

7f286f9


          Added missing file id in history insert

873583e


          hard delete tests

ce28891


          change capture

848195c

FK

61b1c06

SergeyGaluzo requested a review from a team as a code owner

November 19, 2024 02:45

github-advanced-security bot found potential problems

View reviewed changes

src/Microsoft.Health.Fhir.SqlServer/Features/Storage/SqlAdlsStore.cs

+                      private static string _adlsAccountName;
+                      private static string _adlsAccountKey;
+                      private static Uri _adlsAccountUri;
+                      private static string _adlsAccountManagedIdentityClientId;

Check notice

Code scanning / CodeQL

Missed 'readonly' opportunity Note

Field '_adlsAccountManagedIdentityClientId' can be 'readonly'.

src/Microsoft.Health.Fhir.SqlServer/Features/Storage/SqlAdlsStore.cs

+                      private static string _adlsAccountKey;
+                      private static Uri _adlsAccountUri;
+                      private static string _adlsAccountManagedIdentityClientId;
+                      private static BlobContainerClient _adlsClient;

Check notice

Code scanning / CodeQL

Missed 'readonly' opportunity Note

Field '_adlsClient' can be 'readonly'.

src/Microsoft.Health.Fhir.SqlServer/Features/Storage/SqlAdlsStore.cs

+                      private static Uri _adlsAccountUri;
+                      private static string _adlsAccountManagedIdentityClientId;
+                      private static BlobContainerClient _adlsClient;
+                      private static bool _adlsIsSet;

Check notice

Code scanning / CodeQL

Missed 'readonly' opportunity Note

Field '_adlsIsSet' can be 'readonly'.

src/Microsoft.Health.Fhir.SqlServer/Features/Storage/SqlAdlsStore.cs

+                              {
+                                  if (!_adlsIsSet)
+                                  {
+                                      _adlsAccountName = GetStorageParameter(sqlRetryService, logger, "MergeResources.AdlsAccountName");

Check notice

Code scanning / CodeQL

Static field written by instance method Note

Write to static field from instance method, property, or constructor.

src/Microsoft.Health.Fhir.SqlServer/Features/Storage/SqlAdlsStore.cs

+                                  if (!_adlsIsSet)
+                                  {
+                                      _adlsAccountName = GetStorageParameter(sqlRetryService, logger, "MergeResources.AdlsAccountName");
+                                      _adlsConnectionString = GetStorageParameter(sqlRetryService, logger, "MergeResources.AdlsConnectionString");

Check notice

Code scanning / CodeQL

Static field written by instance method Note

Write to static field from instance method, property, or constructor.

src/Microsoft.Health.Fhir.SqlServer/Features/Storage/SqlAdlsStore.cs

+                                      _adlsConnectionString = GetStorageParameter(sqlRetryService, logger, "MergeResources.AdlsConnectionString");
+                                      if (_adlsAccountName != null)
+                                      {
+                                          _adlsAccountKey = GetStorageParameter(sqlRetryService, logger, "MergeResources.AdlsAccountKey");

Check notice

Code scanning / CodeQL

Static field written by instance method Note

Write to static field from instance method, property, or constructor.

src/Microsoft.Health.Fhir.SqlServer/Features/Storage/SqlAdlsStore.cs

+                                      if (_adlsAccountName != null)
+                                      {
+                                          _adlsAccountKey = GetStorageParameter(sqlRetryService, logger, "MergeResources.AdlsAccountKey");
+                                          _adlsConnectionString = $"DefaultEndpointsProtocol=https;AccountName={_adlsAccountName};AccountKey={_adlsAccountKey};EndpointSuffix=core.windows.net";

Check notice

Code scanning / CodeQL

Static field written by instance method Note

Write to static field from instance method, property, or constructor.

src/Microsoft.Health.Fhir.SqlServer/Features/Storage/SqlAdlsStore.cs

+                                      }
+                                      var db = sqlRetryService.Database.Length < 50 ? sqlRetryService.Database : sqlRetryService.Database.Substring(0, 50);
+                                      _adlsContainer = $"fhir-adls-{db.Replace("_", "-", StringComparison.InvariantCultureIgnoreCase).ToLowerInvariant()}";

Check notice

Code scanning / CodeQL

Static field written by instance method Note

Write to static field from instance method, property, or constructor.

src/Microsoft.Health.Fhir.SqlServer/Features/Storage/SqlAdlsStore.cs

+                                      var uriStr = GetStorageParameter(sqlRetryService, logger, "MergeResources.AdlsAccountUri");
+                                      if (uriStr != null)
+                                      {
+                                          _adlsAccountUri = new Uri(uriStr);

Check notice

Code scanning / CodeQL

Static field written by instance method Note

Write to static field from instance method, property, or constructor.

src/Microsoft.Health.Fhir.SqlServer/Features/Storage/SqlAdlsStore.cs

+                                      if (uriStr != null)
+                                      {
+                                          _adlsAccountUri = new Uri(uriStr);
+                                          _adlsAccountManagedIdentityClientId = GetStorageParameter(sqlRetryService, logger, "MergeResources.AdlsAccountManagedIdentityClientId");

Check notice

Code scanning / CodeQL

Static field written by instance method Note

Write to static field from instance method, property, or constructor.

SergeyGaluzo added 21 commits

November 19, 2024 08:52


          Comments

65b21dc


          resource list lake and 768

2b28ed4


          no lake on list

6c46e43


          no extra lines

2d8a618


          small cleanup

38e6c45


          84 in sync

90132fb


          diff

77d4950


          GetResources back

7f17c96


          Fix diff

8b660a0


          Use ResourceListLake correctly


          Added @ResourceListLake to update resource search params

daa5c53


          tweeking

cfa461c


          fix diff

5373db8


          resourceids

ad7576a


          filter out null ref resourec type


          handle search params updates in parallel with data move

e56f599


          Added id join in include queries

c59a7ce


          84 diff fix

83c812a


          no CR on IsResourceRef

3019c0f


          Removed not needed alias

46892b1


          no alias

4d65b4d

SergeyGaluzo changed the title ~~[Do not review] new schema ref sep fk~~ Database schema optimizations

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet