Misc instrumentation and runtime TODOs

[Domiii]

NOTE: This issue is a bit of a septic tank of many small things that can be approved with several features. They generally require instrumentation and runtime fixes, and cannot be fixed solely in dbux-data and dbux-code.

Potentially Critical Bugs and Features

• fix jest support + javascript-algorithms
  • it's already mostly working, but there were issues (forgot what it was)
  • ref: https://github.com/Domiii/dbux/pull/481/files
• fix: await (async () => await A)(); (flat_vs_nested.js)
• fix proxy stuff
  • -> make sure that _readProperty is used for all prop access; there is a few places left in the code, especially in the monkey patchers
  • test w/ sequelize → ssce1: in chai's overwritingMethodWrapper (there used to be many unwanted proxy handler get context recordings)

DataNode TODOs

• fix: DataFlowNodeProvider.buildNodes might miss Value creation trace/ref
  • Some values are created without their own DataNode. E.g. var a = JSON.parse('[{ }, { }]'); implicitly creates three objects and three ValueRefs, but only a gets one DataNode. The other 2 share its nodeId.
    • but when selecting some trace x in var x = a[0];, Data Flow View does not list the actual creation trace.
• fix: async function calls do not propagate params<->args data flow
• fix spread arg DataNode order
  • Problem:
    • callExpressions.js -> buildTraceCall* first generates arguments, which traces all arguments
    • then RuntimeMonitor.traceBCE is called which adds the spread DataNodes
  • Sln: Make sure that arguments and spread arguments are all processed in order
• add data flow for promises
• add DataNode monkey patching: __defineGetter__, defineProperty, defineProperties etc.
  • test w/ hexo#4, select asset.path in hexo/lib/plugins/tag/asset_img.js:24
    • Actual: Data flow somehow connects with the value parameter of the cast function (line 61)
    • Expected: it should connect to the return value cache of the setGetter callback function (line 74)
    • -> In this case, the program first queried assest.path, add a getter on 'path' using __defineGetter__, then query again. Our heuristic links the two query results because they share the same accessId. See minimal reproducible example in getter2.js on branch issue#561
  • blocked by Misc instrumentation and runtime TODOs #577
• fix Data Flow ordering: in arrays-push.js, select a[2]. Why is ...a behind a[2] in data flow (even though it occurs beforehand)?
  • -> wrong recording order of DataNode
• fix UI: cannot select parameter traces because they have (but should not have) staticTrace.staticContextId !== staticContext.staticContextId

Other TODOs

• fix Program context for top-level await (with potential finalize/catch)
  • → blocked by ESM Support #562
  • also fix popProgram to call fixContext (in case of top-level await)?
  • test *.mjs files
• improve tracesDisabled config and add to docs
• enable prettyLogs for dbux-internal logging
  • make sure process.cwd polyfill won't cause problems: Tracing certain system polyfills can lead to infinite loops or incorrect results (e.g. process.cwd) #611
• fix: isTerminalNode designation
  • rewrite isTerminalNode: check whether current node has no Pre update, and no one else in CHAIN (via asyncEdges.to) has more than one Pre update
• Add callback link for callback assignments, e.g.: domEl.onclick = cb
  • Maybe add dynamic callback patching to also pick up on callback assignments. Consider heuristics:
    • name starts with on, target is a GlobalEventHandler, previous value is null (if not assigned before)
  • test w/ Editor.md
• fix: popFunction trace is thin, but should not be* [ ] fix: ThrowArgument always gets recorded twice
• add ACG edge for dynamic import: CHAIN/FORK the imported file's CGR to caller
• don't access module.exports (or somehow overcome circular reference warnings)?

Done

• Promise ctor <-> resolve linkage: CHAIN new promise to nesting post event
  • assign promiseCtorId to all events created when calling executor
  • if it only contains "singular chain of pre*AsyncEvents within the Promise ctor":
    • singular PreAwait or
    • singular PreCallback or
    • multiple PreThen, but only pre and post promiseIds reference each other
  • (NOTE: more convoluted than callback chain, since there can be multiple thens "pre-event"s and it would still become a CHAIN)
• don't enable prettyLogs inside of target application (or remove it's reading of runtime values)
  • -> it's happening in @dbux/cli, triggered by @dbux/babel-plugin/src/index.js
• find out why popProgram does not seem to work correctly (trace not selectable) when error was thrown
• fix samples/__samplesInput__/objectMethods2.js
• fix popProgram is not shown anymore?
  • [webpack/Project] if pw=.*
    • -> for now, add pb=v8-compile-cache, since we just cannot handle this type of multi-layered monkey patching well enough
    • e.g. var script = new vm.Script(wrapper, { in v8-compile-cache is unhappy with dynamic callback patching
• Console output correlation view: Global Analysis View: error + console output traces #618
• fix how VSCode sends/encodes strings to terminal
  • -> seems to add PS-type escaping on Windows (so we need ^^^^ instead of ^)
  • -> does not happen on Mac
  • -> for now, use workarounds:
    1. in Project.installPackages, use writeMergePackageJson + npmInstall() instead of yarn add x/npm i x
    2. if other places need it, maybe write script file instead?
• fix "cannot find node" bug on Mac
  • -> in Process.js -> inherit env, instead of overriding it
• fix express#1 again (woopsi)
• handle runtime connection timeout gracefully (might be caused by a too big workload starving out the net/IO queue)
  • -> increase timeout (Timeout error during handshake ambiguously reported as websocket error; stops re-connection attempts socketio/socket.io#4062)
  • -> increase pingTimeout (Socket.io disconnect with ping timeout randomly socketio/socket.io#2769)
  • -> re-connect upon hand-shake timeouts (others are harder to determine)
• fix notepack's encoding problems #570
• traces inside of a try block are not flagged as error, even if error was thrown #197
• The unary operator ! does not addDefaultTrace correctly -> trace of operand not recorded in if statement; e.g. if (!x) #602
• possible race condition in RuntimeClient when running node-fetch? (bugs out every few runs, sometimes persistently)
  • check if this is still a bug, or if its fixed with recent SendQueue fix
• Add support for Promise.race + Promise.any #591
• fix: Error.captureStackTrace callback problem
• fix: promisification (proper edges for callback in promise ctor executor)
• investigate FORK from convoluted Promise return linkage in hexo#4 (-> Promise.each not instrumented)
• improve Promisify handling
  • multi-nested promisify
  • basic SYNC w/ promisify
  • promisify without recorded root
  • test w/ async-js
• fix: add SYNC edges
  • SYNC via nesting
  • SYNC w/ Promise.all
  • test w/ async-js
  • test w/ sequelize#fOC
• fix: unwanted chaining on thread#1
• investigate: parallel webpack build
• fix: CallbackPatcher not patching call, apply, bind
• Editor.md sometimes does not send packets of certain event handlers
  • Observations
    • independent of Client's stayAwake
    • queue does not even seem to receive the data
    • cmUnbindScroll is one of the affected roots
  • steps:
    • diff to get set of unsent contexts -> identify and log all roots
    • manually patch those roots to determine why send is never called
  • Sln: Runtime client SendQueue had a race condition. Fixed. Also added data sanity validation to make sure we'll see it if it happens again.
• call gets undefined input (see hexo#4 -> shows up as a warning)
• delete cache folder when deleting project
• fix OptionalCallExpression
• fix (most of) callback and value propagation for bind, call, apply
• express#9: infinite loop from isClass (cannot reproduce?)