Sprint 62: Release Report (sprint-61 - sprint-62)

Major accomplishments:

• Cloud Scan (Replic) released to production
• Several enhancements for UC Berkeley LIT
  • 500 item ATOM feed default
  • Callback handler
  • New EZID retry logic
  • Local ID lookup via Search field
  • Isolate error jobs in batch and produce a new error report attachment
• Requeue and Delete API deployed in Ingest
• Added support for UC Davis to ETD Service

mrt-doc sprint-61-main..sprint-62-main

• 1b0f70c 2021-11-17 Update log with Sprint 63 goals
• e838be2 2021-11-17 refine slides, update mermaid
• c1592a5 2021-11-17 Merge branch 'main' of https://github.com/CDLUC3/mrt-doc into main
• 4f2a974 2021-11-17 diagram updates
• 2cd195c 2021-11-16 Fix broken links and transfer branch to main in URL references
• 1f5ffa1 2021-11-16 diagram updates

mrt-ingest sprint-61-main..sprint-62-main

• 7b8b731 2021-11-10 Merge pull request 36 from CDLUC3/errorAttachment
• 62a401d 2021-11-10 Add JOB ID to Callback pathname
• 3cb8796 2021-11-04 Merge pull request 35 from CDLUC3/errorAttachment
• 4ffbb88 2021-11-04 Support error attachment in CSV format
• 6d42ade 2021-11-01 Merge pull request 34 from CDLUC3/callback
• 5f810ae 2021-11-01 Provide Callback if failure occurs
• 6c47506 2021-10-31 Clean up Callback handler
• ba74da4 2021-10-27 Merge pull request 33 from CDLUC3/ezid504
• 2c1c9c6 2021-10-27 Support retry logic when EZID fails without an exception thrown (e.g. gateway error) #837

mrt-replic sprint-61..sprint-62

• 63b4fd7 2021-11-17 Fix Issue 860 - bug prevent match zero length files
• 64652f5 2021-11-10 close connection on completion of scan delete
• c6c4d84 2021-11-05 Ignore current keys (< 1 day)
• 38e3cb5 2021-11-03 Cleanup hanging connections
• fc09160 2021-11-02 Support scan cancel on scanid

mrt-dashboard sprint-61-main..sprint-62-main

• 2d4cd60f 2021-11-03 Merge pull request 108 from CDLUC3/atom
• 0a56ded1 2021-11-03 Merge pull request 109 from CDLUC3/localid
• d75f1787 2021-10-29 localid lookup
• 11852096 2021-10-29 refine per_page handling
• debe4849 2021-10-29 up default 500
• 66610265 2021-10-25 pagination params

mrt-admin-lambda sprint-61-main..sprint-62-main

• ef50ca9 2021-11-16 name scan inventory list
• 9a7ed91 2021-11-12 scan review details
• 2dc9d48 2021-11-10 del batch of keys
• a8c17ad 2021-11-09 bulk scan ops
• 6c950ec 2021-11-09 nav to a specific scan, improve pct display
• aad644c 2021-11-09 use prod scan tables
• e47fc5e 2021-11-05 fix files query
• 3f33cb4 2021-11-05 refine collid handling, nav bar creation
• d785a7e 2021-11-04 inv_storage_maints status
• 9d30e86 2021-11-04 Fix file count/file scan estimates
• ff525f3 2021-11-04 handle nodes buttons
• ccf8342 2021-11-04 fix queue date
• 6aa9f6f 2021-11-03 join on node id
• 1d856ce 2021-11-03 button controls
• b05f444 2021-11-03 fix js inline
• 71dff2c 2021-11-03 fix inline js, refine replic buttons
• c75198f 2021-11-02 refine replic buttons
• d841ddb 2021-11-02 fix button logic
• 01229a9 2021-11-02 add scan control buttons
• f33034c 2021-11-01 format child count
• 7bd52b3 2021-11-01 Merge pull request 36 from CDLUC3/inline
• 901f19c 2021-11-01 inline js/css
• 12449f1 2021-10-29 improve number formatting
• a679288 2021-10-29 use node_counts from billing db
• a4371b7 2021-10-29 daily billing node reports
• f0d8947 2021-10-28 enable requeue in prod
• 3a66f2d 2021-10-27 compute file counts per node

mrt-cron sprint-61-main..sprint-62-main

• 1e32616 2021-10-29 count primary/secondary
• 54f1d9e 2021-10-29 generate node counts

merritt-docker sprint-61-main..sprint-62-main

• 0b3bcc4 2021-11-10 Merge pull request 25 from CDLUC3/callback
• ec6bbb1 2021-11-10 Fix email notification
• 9a99ceb 2021-11-10 Callback
• 1bb0031 2021-11-10 ALter callback URL
• ed2a3ba 2021-10-31 Support Ingest Callback handler
• c5e3336 2021-10-28 Merge pull request 24 from CDLUC3/callback
• 050abb0 2021-10-27 Forgot to add callback container
• cbb231c 2021-10-27 Add callback service.

Deploy 2021-11-15B: ETD Release for UC Davis

#858

Deploy 2021-11-15A: Replication Scan Enhancements

#842

Release Report (sprint-60 - sprint-61)

Major accomplishments:

• SDSC issue investigated and resolved
• UI load issue investigated and appears to be resolved (monitoring)
• Collection Admin features enabled in production
• Cognito + SAML enabled in production
• Storage Scan implementation on Stage
• Queue visibility gained (Access, Inventory, Ingest)

mrt-doc sprint-60-main..sprint-61-main

• c112269 2021-10-27 Update log with sprint goals for Sprints 61 and 62
• 38893c5 2021-10-27 update nav
• 7ab445f 2021-10-27 index nav
• 2a17521 2021-10-27 add saml to cognito diagrams
• 2fedf5b 2021-10-27 refine nav/index
• ad19d26 2021-10-27 prev page nav
• 83c1b4d 2021-10-27 prev page nav
• bbb476d 2021-10-27 update highlight
• ef24ed5 2021-10-27 ui4
• c106e26 2021-10-27 update ui4
• b580eb3 2021-10-27 diagram updates
• f596104 2021-10-19 Refine store admin notes
• f50c4e2 2021-10-19 Refine Storage Admin Design

mrt-inventory sprint-60..sprint-61

• 18c0498 2021-10-21 Merge pull request 6 from CDLUC3/scan
• 5f61321 2021-10-21 Minor changes to status

mrt-ingest sprint-60-main..sprint-61-main

• 1e8bf47 2021-10-26 Merge pull request 32 from CDLUC3/requeueAPI
• ad7ce6d 2021-10-24 Support delete queue API
• 2e64615 2021-10-24 requeue API
• 8f9638d 2021-10-13 Merge pull request 31 from CDLUC3/zkQueuesAPI
• 1bc9b46 2021-10-13 Add que:queueNode and que:ID to Access and Inventory queries

mrt-replic sprint-60..sprint-61

• 7ec8648 2021-10-22 Merge pull request 6 from CDLUC3/scan
• b280b58 2021-10-21 Switch copy to CopyManifestVersion Add scan status command Add orphan copy

mrt-store sprint-60..sprint-61

• 718c951 2021-10-13 Merge pull request 20 from CDLUC3/queueAccess
• 105e990 2021-10-13 Fix overflow bug for large/small worker threshold

mrt-cloud sprint-60..sprint-61

• 0ac8054 2021-10-22 Reset logging for needsCopy
• 9ff729d 2021-10-21 CloudManifestCopyVersion replaces existing copy - block copy on matching meta sha256

mrt-zoo sprint-60..sprint-61

• dc43017 2021-10-27 Change cdl-zk-queue delete to deletep. Used on zookeeper exception

cdl-zk-queue sprint-60..sprint-61

• 7aa272c 2021-10-26 Merge pull request 4 from CDLUC3/requeueFail
• 3ff0edb 2021-10-24 Support requeue and delete

mrt-dashboard sprint-60-main..sprint-61-main

• f1aaf9e8 2021-10-22 Merge pull request 106 from CDLUC3/i724a
• f55de4a2 2021-10-21 force commit
• 5d9e5d3f 2021-10-21 update puma, add robots

mrt-admin-lambda sprint-60-main..sprint-61-main

• 36a9593 2021-10-27 Merge pull request 35 from CDLUC3/requeue
• 90a0db9 2021-10-27 use storage scan id
• 0e83834 2021-10-27 refine queue state handling
• f2526a3 2021-10-26 update reload handling
• f1e98b9 2021-10-26 ajax invoke of queue cmd
• 37af854 2021-10-25 requeue buttons
• ec6f736 2021-10-22 pull name when no mnemonic exists
• d5a1b7b 2021-10-22 Add link to collection info
• 3bb9a15 2021-10-22 coll info page
• 58a0005 2021-10-22 store obj page simplification
• 62101b6 2021-10-21 add buttons
• 1724605 2021-10-18 query scans table
• ecea5c8 2021-10-18 refine use of WARN
• c53c789 2021-10-15 datetime fileds
• 9e88268 2021-10-15 Queue report date filters
• 8adfe8b 2021-10-14 Identify large access server
• 1f2cb71 2021-10-14 add clarifying text to pages, limit admin object submission
• c062aa0 2021-10-14 Allow colladmin in prod
• 6e02a0f 2021-10-14 query on mod date
• 9cc9f5c 2021-10-13 categorize replic initiated
• ed35afc 2021-10-13 Add category to replic report
• 1372912 2021-10-12 add ip and az to server info
• d69a2f0 2021-10-12 add prior/next day nav
• 65d8f78 2021-10-11 report tidy
• 51335b3 2021-10-10 refine replic queries
• 8902cec 2021-10-08 clear stale audit batches
• e6bb8e4 2021-10-07 tidy store admin buttons
• 0463c4f 2021-10-07 support reaudit and rereplic in prod
• 0b3ba71 2021-10-07 Merge pull request 34 from CDLUC3/ingestSSM
• dd70dec 2021-10-01 Ingest SSM parms

mrt-cron sprint-60-main..sprint-61-main

• 31d190f 2021-10-18 call aggregate mult_coll
• 5b81b3b 2021-10-15 add queue reports

merritt-docker sprint-60-main..sprint-61-main

• 9d1ddb5 2021-10-27 update commits
• 5d128fd 2021-10-21 update commits

mrt-dashboard-config sprint-60..sprint-61

• 2a4b9e1 2021-10-12 #819
• a1aea00 2021-10-12 Fix error in creating Jack Miles papers
• 0f84a86 2021-10-12 #818
• 09dcadd 2021-10-12 #817

Deployment 2021-11-10: Ingest Callback Method

#844

Deployment 2021-11-09: Replication Scan

#842

Deployment 2021-11-03: Deploy Atom Feed Default Page Size Change

#846

Deployment 2021-10-28: Ingest Releases - EZID retry handling, Requeuing Endpoints

• #840
• #841

Deployment 2021-10-21: UI Release Puma Update to 5.5.1

#831

At the same time this change was deployed, Ashley modified our puppet config files to redirect /tmp to a larger disk.

Deployment 2021-10-19: Redeploy access server daemon control

#829