Deployment 2023-10-30 - Ingest opensearch support update

#1656

Deployment 2023-10-19 - Ingest opensearch support

#1646

Sprint 93: Release Report (sprint-92 - sprint-93)

Major accomplishments:

• UI: Retry logic for inv_object, presign file, atom feed render and methods with common 500 errors
• Inventory state and JSON logging for OpenSearch
• Audit state and JSON logging for OpenSearch
• Numerous new UCR Nuxeo harvests established
• Robin Collection drives 01 and 03 ingests

mrt-doc sprint-92..sprint-93

• c599979 2023-10-19 Update sprint-log.md

mrt-doc-private sprint-92..sprint-93

• 27bafef 2023-10-16 Update ezid-credential-rotate.md
• c173845 2023-10-13 Update dependency_scans.md
• 3f72cd3 2023-10-12 Update routine_librato_checks.md
• eaa0337 2023-10-10 Update db-credential-rotate.md
• 79cb4a8 2023-10-10 Update storage-cloud-s3cli.md
• fa88891 2023-10-10 Update storage-cloud-s3cli.md
• 0cc3078 2023-10-10 Update storage-cloud-s3cli.md
• 8624cc1 2023-10-09 Add ansible instructions
• 6f82967 2023-10-06 Update storage-cloud-s3cli.md
• b3cfd07 2023-09-20 Update db-credential-rotate.md

mrt-ingest sprint-92..sprint-93

• 4af2b44 2023-10-18 Merge pull request 89 from CDLUC3/jsonLog01
• 1c05a68 2023-10-18 Opensearch Storage requests
• 8f70ad0 2023-10-17 Fix Callback UTF-8 error.
• 968bcc9 2023-10-11 Opensearch logging for retrievals
• 345e98b 2023-10-09 Log4J - Callback requests
• 67d2cd6 2023-10-06 Change log4j loglevel on the fly
• 53e7bb3 2023-09-27 Initial opensearch logging test - EZID calls only

mrt-audit sprint-92..sprint-93

• 762e2c9 2023-10-18 Merge pull request 24 from CDLUC3/jsonstate
• 602eda1 2023-10-18 Add json state
• 4d964ae 2023-10-02 audit state log - fix exception; no entry processing except debug
• c33c2c3 2023-10-02 Merge pull request 23 from CDLUC3/auditlog
• c0a3ce1 2023-09-22 audit state log
• 663958f 2023-09-22 audit fix and add fileID

mrt-store sprint-92..sprint-93

• 365f8b4 2023-10-17 Merge pull request 42 from CDLUC3/cloudfix
• 597375f 2023-10-13 Add jsonstate, add log level to state

mrt-cloud sprint-92..sprint-93

• 9945c92 2023-10-18 Merge pull request 32 from CDLUC3/statedur
• d3c83fe 2023-10-18 Allow json state duration on missing node
• 34c59f8 2023-10-13 Merge pull request 31 from CDLUC3/nodel
• fec6b46 2023-10-13 Use TransferManager download only with AWS - errors minio, wasabi
• 169c038 2023-10-11 s3 - allow replace, use log4j2, provide class for json nodes states

mrt-core2 sprint-92..sprint-93

• a7bcc38 2023-09-25 Merge pull request 38 from CDLUC3/auditfix
• 4c7cf6b 2023-09-22 audit fix and add fileID

mrt-dashboard sprint-92..sprint-93

• 380adad1 2023-10-02 Merge pull request 152 from CDLUC3/retry4
• c7fa9e24 2023-10-02 retry
• a96fd7fe 2023-10-02 retry4 for inv_object
• 7f870bd9 2023-09-25 Merge pull request 151 from CDLUC3/retry3
• 23c33107 2023-09-25 presign try 2
• 858664f1 2023-09-25 atom cov
• 5122a751 2023-09-25 atom feed retry
• 999e1bbb 2023-09-25 Merge pull request 149 from CDLUC3/retry_log
• bb07200b 2023-09-22 rubocop fixes
• cef6e53f 2023-09-22 Add db retry logic

mrt-admin-lambda sprint-92..sprint-93

• 92a0af8 2023-10-18 jsonstate endpoint, change timer
• 69ac0a6 2023-10-17 ssm doc multiline
• c0be40f 2023-10-17 update param documentation
• ed79a06 2023-10-16 Update ssm.registry.yml
• d69432c 2023-10-16 note ETD copy of credential
• dda380f 2023-10-10 fix collad deps
• 4000a28 2023-10-02 obj health

mrt-cron sprint-92..sprint-93

• b74fde8 2023-10-19 pass user param to queries
• 3f40f27 2023-10-19 conditional updates
• 38b2842 2023-10-18 break obj handling into components
• 3072fe4 2023-10-18 Merge pull request 6 from CDLUC3/wip
• bec6b83 2023-10-17 save into separate cols
• 5ea452b 2023-10-16 refine
• 415481c 2023-10-16 clarify obj name
• 8555032 2023-10-16 clean up classes
• eb4c3aa 2023-10-13 save wip
• f8634cc 2023-10-13 add opensearch api
• 7eea0f8 2023-10-13 refine debug
• dac71bf 2023-10-12 obj queries
• ff8183c 2023-10-11 config classes in yaml
• d17bc71 2023-10-11 Merge branch 'main' of github.com:CDLUC3/mrt-cron
• 45a664f 2023-10-11 resolve sym in json
• 5a223f9 2023-10-11 Update README.md
• 358c79d 2023-10-09 set repo theme
• f8b274c 2023-10-06 cmd line args, read json
• 5cbbd28 2023-10-06 additional type categorization plus comments
• 9cb212a 2023-10-06 first pass mime type sort
• f26ff27 2023-10-05 mimes
• 7d5ec40 2023-10-05 enrich yaml
• 0fb5da4 2023-10-04 tests status
• 3b08dac 2023-10-04 sample tests
• c5e46de 2023-10-04 break up ruby classes
• 4368af7 2023-10-04 tidy obj health
• e35b5db 2023-10-04 tidy logic
• 31821b8 2023-10-04 fix container key
• 02493d8 2023-10-04 fix timestamp
• bc4702b 2023-10-03 obj loading
• 1a10c31 2023-09-26 obj health proto
• 6671629 2023-09-26 Update README.md
• 29c99ad 2023-09-26 Update README.md
• 7c8270d 2023-09-26 Update README.md
• a3e94f9 2023-09-26 Update README.md

merritt-docker sprint-92..sprint-93

• e0c93b8 2023-10-19 build config
• d8b3d45 2023-10-10 add --no-cache

mrt-dashboard-config sprint-92..sprint-93

• d44a606 2023-10-16 #1634
• 817ed8d 2023-10-11 Hiram has been processed after feed change
• 7607cee 2023-10-04 #1627
• e33f2fb 2023-10-03 #1626
• 8506570 2023-10-03 #1625
• 67ebe6e 2023-10-03 #1624
• 3890b7b 2023-10-03 #1623
• 54fa729 2023-10-02 #1622
• ee7d817 2023-10-02 Schedule production-UCI_Academic_Professional_Women.csh
• f9fad8f 2023-09-28 #1621
• 8f05d58 2023-09-21 #1607
• efdea73 2023-09-21 #1603

Deployment 2023-10-03 - Audit - State JSON logging

#1620

Deployment 2023-10-02 - UI - inv_object retry logic

#1629

Deployment 2023-09-25d - UI - Presign file retry

#1616

Deployment 2023-09-25c - UI - Retry logic for atom feed render

#1615

Deployment 2023-09-25b - UI - Retry for common 500 errors

#1612

Deployment 2023-09-25a - Inventory - State JSON logging

#1599

Sprint 92: Release Report (sprint-91 - sprint-92)

Major accomplishments:

• Storage state logs stage and production
• Storage access dashboard in OpenSearch
• Ingest manifest validation deployed
• Numerous new UCR Nuxeo harvests established and processed
• UI gem updates and GH Actions fix deployed

mrt-doc sprint-91..sprint-92

• a633579 2023-09-20 Update sprint-log.md

mrt-doc-private sprint-91..sprint-92

• 88d26d1 2023-09-19 Update dependency_scans.md
• 4318c82 2023-09-19 Update dependency_scans.md
• b26a976 2023-09-19 Update dependency_scans.md
• b82c89d 2023-09-19 Update dependency_scans.md
• fdee049 2023-09-19 Update dependency_scans.md
• 726bf71 2023-09-19 Update dependency_scans.md
• 12338ee 2023-09-19 Update dependency_scans.md
• 173794c 2023-09-19 Update dependency_scans.md
• 060145b 2023-09-14 Update logstash-customize.md
• 9d57385 2023-09-14 Create logstash-customize.md
• ee661a3 2023-09-14 Update system-admin.md
• a4ccf01 2023-09-11 Update dependency_scans.md

mrt-inventory sprint-91..sprint-92

• 3872f6d 2023-09-20 Merge pull request 28 from CDLUC3/invlog
• fac8745 2023-09-20 Clean up
• 562ff3c 2023-09-18 Standard log tags
• 683f4ff 2023-09-14 initial log state inventory

mrt-ingest sprint-91..sprint-92

• 1af52c7 2023-09-08 Merge pull request 88 from CDLUC3/validateManifest
• bbd61c6 2023-09-07 Memory cleanup
• a417220 2023-09-06 Preprocess object and batch manifests

mrt-store sprint-91..sprint-92

• f6c0570 2023-08-30 Use processNode with node json string

mrt-cloud sprint-91..sprint-92

• 508cd61 2023-09-13 Add Dryad dev node 3043

mrt-core2 sprint-91..sprint-92

• cfc3f3c 2023-09-20 Merge pull request 37 from CDLUC3/bom1.4
• 571b79a 2023-09-19 Remove unused jdom imports
• b3fbf4a 2023-09-19 BOM 1.4 - jackson-core
• 264bde2 2023-09-19 BOM 1.4
• c44ef48 2023-09-14 Add support for inventory
• 97b8e37 2023-08-30 Add processNode with node json string

mrt-dashboard sprint-91..sprint-92

• 4c85894a 2023-09-13 Merge pull request 148 from CDLUC3/gem-aug23
• 49cf5a9a 2023-08-30 gemfile lock change
• f1877ec6 2023-08-30 rubocop
• 8d566546 2023-08-30 elim webdrivers call
• f7b5f707 2023-08-30 match dryad approach
• 8b6e5aff 2023-08-30 hardcode chrome ver
• 5aaed419 2023-08-30 chrrome ver
• 15a49466 2023-08-30 add chrome install step to ghaction
• 0ecb6e98 2023-08-29 try
• 6a1d1cd7 2023-08-29 rubocop
• c27a4a90 2023-08-29 selenium webdrivers only
• def0a6b1 2023-08-29 specify chrome version
• c3c1f842 2023-08-29 rubocop fix
• ff36dbda 2023-08-29 update selenium
• b79a366f 2023-08-29 readd webdrivers
• 2227ea23 2023-08-29 update gems

mrt-admin-lambda sprint-91..sprint-92

• 103793c 2023-09-13 document

mrt-cron sprint-91..sprint-92

• 6cf5840 2023-09-13 tidy
• 13dcebb 2023-09-13 additional opt
• d5bb75d 2023-09-13 fix typo
• edf259c 2023-09-13 skip count
• 47688ef 2023-09-13 add logging
• d49ab21 2023-09-13 refine date logic
• 8eaf172 2023-09-13 add logging
• a691bef 2023-09-13 optimize
• fdd93b8 2023-09-12 Merge branch 'main' of github.com:CDLUC3/mrt-cron
• b04c7b1 2023-09-12 placeholder value creation
• 8ead49d 2023-09-12 placeholder value creation
• 7b1699c 2023-09-01 rename
• 3be3f29 2023-09-01 create placeholder vals

merritt-docker sprint-91..sprint-92

• 281ccc0 2023-09-13 update commits
• 6f61954 2023-09-13 comment image
• f842a24 2023-09-12 docker prune option
• ce124c1 2023-08-30 fix store for docker

mrt-dashboard-config sprint-91..sprint-92

• f6a94c5 2023-09-20 #1601
• b61f322 2023-09-20 Schedule: UCR Nuxeo feed: Andrae B. Nordskog papers
• 6d8649f 2023-09-20 #1600
• a3a826f 2023-09-18 Schedule: production-UCR_Esther_Klotz.csh
• d214e38 2023-09-18 Schedule: UCR Nuxeo feed: Edith Hipkins
• 8626f85 2023-09-18 schedule: production-UCR_Earl_R_Oatman_Papers.csh
• 8263bdb 2023-09-18 weekely scheduling: production-UCR_David_Bratman_Hour_25.csh
• de0484b 2023-09-18 Schedule Citrus collection
• bed68ed 2023-09-18 production-UCR_Catherine_Clark_Papers.csh weekly processing
• 5ad7bba 2023-09-18 production-UCR_Avery_E_Field.csh weekly processing
• b274ccb 2023-09-15 #1597
• 374a2ce 2023-09-15 #1596
• 7ea9151 2023-09-15 #1595
• 311ca32 2023-09-15 #1594
• f749e34 2023-09-15 #1593
• b11525f 2023-09-15 #1592
• 1ecfb10 2023-09-14 #1590
• 0965511 2023-09-14 SCHEDULE: UCR Nuxeo feed: Albert L. Page
• ed7d017 2023-09-14 #1589
• 3cbba97 2023-09-14 Schedule: UCR Nuxeo feed: Collection on race, ethnicity, and student activism
• bc72e3a 2023-09-14 Schedule UCR Nuxeo feed: Charles McGee
• 69fa72b 2023-09-14 #1588
• cacb996 2023-09-14 Schedule UCR Nuxeo feed: Alfred M. North
• 56db2b6 2023-09-14 #1587
• f694a82 2023-09-14 #1586
• 54250be 2023-09-11 Enable weekly processing of "27940-UCR_African_Student_Programs"