Modernize all the things #115
Conversation
| /** | ||
| * Parse lambda event kinesis records, and filter/log invalid | ||
| */ | ||
| export async function decodeRecords(event) { |
There was a problem hiding this comment.
This is a mix of what was in lib/get-records.js and the old shared index.js handler. Now that there are 4 separate handlers, much DRY-er to locate this all here.
The tests for this file likewise came out of the old tests for those 2 files.
| /** | ||
| * Utility class for inserted dynamodb redirect + segments-downloaded data | ||
| */ | ||
| export default class DynamoData { |
There was a problem hiding this comment.
This is a brand new class/abstraction. The DynamoDB lambda code was so dang confusing, I needed an abstraction to make sense of it.
The constructor takes in things from lib/dynamo.js upsert call, and figures out which downloads/impressions for this request are now "counted". And prevents double counting things.
Tests for this mostly existed elsewhere, just easier to unit-test things separately.
| logger.warn(`Error json parsing extras: ${err}`, { extras: oldExtras }); | ||
| } | ||
| } | ||
| export async function addFrequency({ client, listener, campaign, timestamp, maxSeconds } = {}) { |
There was a problem hiding this comment.
Above this line, this file is functionally the same. Just syntax changes and moving a few things into lib/dynamo-data.js helper class.
From here down, I ported the actual DDB commands out of lib/inputs/dovetail-frequency.js. Just so all the DDB commands are in one place.
| */ | ||
| export const format = (rec) => { | ||
| return { | ||
| timestamp: toISOExtendedZ(rec.timestamp || 0), |
There was a problem hiding this comment.
NOTE: we're now inserting milliseconds precision into BigQuery. Which you have to do with a timestamp string. Integers are epochs, so you can't get better granularity with them.
| listener: listenerId, | ||
| campaign: campaignId, | ||
| maxSeconds: maxSeconds, | ||
| timestamp: toEpochMilliseconds(timestamp || Date.now()), |
There was a problem hiding this comment.
NOTE: I made these milliseconds now. And this lambda can handle both epoch-seconds and milliseconds being in this field.
Will need to make sure DTR can also handle both, before deploying this.
Fixes #105, fixes #96, fixes #94, fixes #58, fixes #34. Parts of https://github.com/PRX/dovetail-router.prx.org/issues/349
Major refactor, changing to ES6, upgrading AWS/Bigquery SDKs by major versions, and changing to Biome for linting.
Ryan TODO before deploying:
My suggestion for reviewing this giant change:
/inputsdirectory, to be more straightforward and less scripty.lib/changes. Nearly all of that was ported as-is, converted via cjs-to-es6, biome formatted, and tests converted to Jest.test/files. Those were ported to Jest, that sit next to the file they're testing. Some changes to how things are mocked, but not too interesting.