v2.6.1 New Delivery Config

PHDO Upload Release Notes - Version 2.6.1

Release Date: 2024-11-19

Version Number: Version 2.6.1

Release Changelog

API Swagger Documentation

DEX Upload API Github Releases

DEX Upload API Github Tags

Overview

The purpose of Upload API release version 2.6.1 is to introduce a new configuration YML file for defining manifest groups and mapping them to one or more delivery targets. This configuration was proviously defined in the JSON files within the upload-config
subproject of this repository. Now, those JSON file are only responsible for defining schema for manifest validation. All configuration for upload delivery has been moved to upload-server/configs/phdo/deliver.yml. The overall purpose of this change is
to make it easier to configure delivery targets for a given data stream and route.

In addition, several bug fixes and code cleanups were performed.

Enhancements

• Introduction of the deliver.yml file for delivery configuration

Bugfixes

• Fix for info endpoint returning null for delivery names for failed deliveries
• Fix for info endpoint timestamps to use nanosecond precision
• Fix for delivery date folder partitioning to use the dex_ingest_timestamp

Cleanup

• Removal of v1 configuration
• Removal of routing delivery target

Reporting Team

• DEX Upload Team Distribution List - [email protected]

v2.5.0 Direct Upload Delivery

DEX Upload Release Notes - Version 2.5.0

Release Date: 2024-09-26

Version Number: Version 2.5.0

Release Changelog

API Swagger Documentation

DEX Upload API Github Releases

DEX Upload API Github Tags

Overview

The purpose of Upload API release version 2.5.0 is to facilitate direct delivery of file uploads to program storage destinations. Instead of files being delivered indirectly though a separate routing microservice, the Upload API service will now deliver files directly to CDC program storage. To enable this, we've decoupled source and destination information from the delivery construct. This allows you to register sources and destinations separately, which increases flexibilty in configuring where the file is coming from and going to. In addition, we've increased the flexibilty of where files get dropped in destination storage. Before, files could be configured to be dropped in subfolders by datastream, route, and date. Now, the subfolder path can take on any pattern. This is accomplished through a new path_template field that can be specified in a particular upload config.

Enhancements

• Direct file delivery to CDC program storage
• Agnostic subfolder delivery path template

Reporting Team

• DEX Upload Team Distribution List - [email protected]

v2.4.1 Delivery Status and Routing Stability

DEX Upload Release Notes - Version 2.4.1

Release Date: 2024-09-20

Version Number: Version 2.4.1

Release Changelog

API Swagger Documentation

DEX Upload API Github Releases

DEX Upload API Github Tags

Overview

The purpose of Upload API release version 2.4.1 is to expand the /info endpoint to include file delivery status for a given upload ID. End users will be able to use this endpoint to verify whether or not their file got delivered to the intended destination. This release also includes improved stability to the in-memory eventing system, as well as configuration changes for the NDLP use case.

Enhancements

• Delivery status for uploads provided by the /info endpoint
• In-memory event system stability improvements

Reporting Team

• DEX Upload Team Distribution List - [email protected]

DEX Upload Release Notes - Version 2.3.0

Enhancements

• Improved reporting information
• Built-in file routing and post processing. Deprecation of function app and event grid
• Manual routing retry endpoint
• Improved redis connection pooling resiliency and recovery

Manifest Config Updates and Minor App Observability

• Update for ABCs
• Update for ED3N
• Internal /version endpoint showing git commit short sha and latest release tag

v2.2.0 Redis File Locker

DEX Upload Release Notes - Version 2.2.0

Release Date: 2024-06-24

Version Number: Version 2.2.0

Release Changelog

API Swagger Documentation

DEX Upload API Github Releases

DEX Upload API Github Tags

Overview

The purpose of Enterprise Data Exchange (DEX) Upload release version 2.2.0 is to release an improvement to how the Upload API prevents file corruption when the service horizontally scales. This improvement is implemented in this release via a distributed file locker that uses a Redis cache to keep track of which Tus instance is currently working on a file, and makes sure another Tus instance doesn't work on the same file. Before this improvement, we were relying on Azure's internal blob storage implementation to prevent clusters of tus servers from corrupting files. This implementation is obscured from us and completely out of our control. This custom distributed lock with Redis gives us full control, and has even been adopted by the Tusd maintainers.

Enhancements

• Improved reliability of horizontal scaling via a custom file locking mechanism that uses an external Redis cache.

Reporting Team

• DEX Upload Team Distribution List - [email protected]

v2.1.1

use websocket for upload processor service bus

v1.6.3

Overview

The purpose of Enterprise Data Exchange (DEX) Upload release version 1.6.3 is to apply a hotfix to not include appended tguid values to filenames copied to EDAV storage accounts.

New Features

• Remove appended tguid values from filenames routed to EDAV storage accounts.

v1.6.2

Overview

The purpose of Enterprise Data Exchange (DEX) Upload release version 1.6.2 is to update upload configurations for data streams, replace filename suffix clock ticks with upload id (tguid), hotfix for Routine Immunization v1 sender manifest config file, and adjust upload filename restrictions.

New Features

• Update upload filename character restrictions to only disallow forward slashes (/)
• Update Routine Immunization v1 sender manifest configuration file to remove incorrect metadata fields
• Add v2 sender manifest configuration files for NRSS and EHDI data streams
• Replace the filename suffix clock tick value with file upload id (tus guid) value

v1.5.0

Overview

The purpose of Enterprise Data Exchange (DEX) Upload release version 1.5.0 is to update upload configurations for data streams, add version 1 sender manifest metadata values to version 2 metadata fields, add the version 2 metadata configuration folder structure, add file copy functionality to routing storage, add Processing Status API integration, add Retry functionality, and add Replay functionality.

New Features

• Version 2 sender manifest configuration folder structure
• Version 1 sender manifest metadata values to version 2 metadata fields
• File copy functionality to routing storage accounts
• Processing Status API integration
• Retry functionality
• Replay functionality
• Upload configurations for the DAART data stream
• Remove file copy EDAV target from the CELR data stream