Some work on scripts internal/caravan/*

[SilasD]

• internal/caravan/common.lua was changed so that searching can match non-ASCII characters in items matching the search term. Basically, the search-word parser was changed to use dfhack.toSearchNormalized(string).
  No API changes. This change was noted in the changelog.

---

• The API for internal/caravan/common.lua obfuscate_value() was extended to allow for a pre-calculated threshold to be passed in instead of recomputing it on every call. This API change is backwards-compatible.

Additionally, get_threshold() and get_broker_skill() were exposed to users of the module.

• internal/caravan/movegoods.lua, internal/caravan/trade.lua, and internal/caravan/pedestal.lua were modified to pre-cache threshold and use the new API.

This change was made to reduce the startup time for internal/caravan/pedestal.lua and internal/caravan/movegoods.lua.

I profiled both of these scripts; both saw startup-time reductions of ~30%.

I found the repeated calls to get_broker_skill() to be the low-hanging fruit in these scripts. Further speed improvements will be more difficult, and are likely to only be incremental.

I intend to experiment with caching everything possible in common.lua, and will PR the code if I see further improvements.

(Later) implementing caching in most functions resulted in further speedups of only ~5% or so. I judge this to not be worth the trouble.

• lua/plugins/buildingplan/itemselection.lua was not modified to use the new API. This is basically because it lives in the main DFHack repository, not the scripts repository, so I didn't bother. Further, I judge potential gains to be minimal.

---

• internal/caravan/trade.lua was changed to use the trader's appraisal skill instead of the broker's.

Some commentary in each commit, but it basically repeats this comment.

[SilasD]

pedestal.lua profiling notes:

The fort is 15 years old currently has ~41000 items. Elven traders are at the depot. Cleanconst was used some time before the profiling; there are ~150 items in constructions. The fort is in text mode, i.e. Classic.

To prepare for each profiling run, Dwarf Fortress was killed and restarted, the fort was reloaded, the test pedestal was viewed, and the main window was minimized.

The profiling involved several 'runs' consisting of

• several 'reps' consisting of
• • forcing a full garbage collection cycle
• • stopping garbage collection
• • recording the current time (in seconds)
• • instantiating and discarding the main object
• • recording the current time and computing the elapsed time
• • restarting garbage collection.
• After each group of reps, the total elapsed time was recorded.
  At the end of all runs, the collected elapsed times were averaged.

NO LONGER RELEVANT: Note that there is likely a systematic error caused by using os.clock().
Measurement now uses dfhack.filesystem.mtime(), which in Windows has a resolution of 0.1 microseconds.

Note that there may be a systematic error where the elapsed time creeps up on every new execution of Dwarf Fortress. This should be explored.

Profiling was done by instantiating the AssignItems{} object.
Garbage collection time was not included (unless explicitly noted).

profiling run: stock, no gc: 40 runs of 1 reps:
mean average 1.378 seconds, deviation 0.013 seconds

profiling run: obfuscate_value caching, no gc: 40 runs of 1 reps:
mean average 0.991 seconds, deviation 0.007 seconds

So caching threshold lowered average startup time by 0.387 seconds, a speedup of 28%.
Not counting garbage collection or window creation.

profiling run: stock, full gc: 40 runs of 1 reps:
mean average 1.479 seconds, deviation 0.007 seconds

Garbage collection adds ~0.1 second if it kicks in during the object initialization.

[SilasD]

movegoods.lua profiling notes

The fort is 15 years old currently has ~41000 items. Elven traders are at the depot. Cleanconst was used some time before the profiling; there are ~150 items in constructions. The fort is in text mode, i.e. Classic.

To prepare for each profiling run, Dwarf Fortress was killed and restarted, the fort was reloaded, the depot was viewed, and the main window was minimized.

The profiling involved several 'runs' consisting of

• several 'reps' consisting of
• • forcing a full garbage collection cycle
• • stopping garbage collection
• • recording the current time (in seconds)
• • instantiating and discarding the main object
• • recording the current time and computing the elapsed time
• • restarting garbage collection.
• After each group of reps, the total elapsed time was recorded.
  At the end of all runs, the collected elapsed times were averaged.

NO LONGER RELEVANT: Note that there is likely a systematic error caused by using os.clock().
We now are using dfhack.filesystem.mtime(), which in Windows has a resolution of 0.1 microseconds.

Note that there may be a systematic error where the elapsed time creeps up on every new execution of Dwarf Fortress. This should be explored.

Profiling was done by instantiating the MoveGoods{} object.
Garbage collection time was not included (unless explicitly noted).

profiling run: stock, no gc: 50 runs of 1 reps:
mean average 1.065 seconds, deviation 0.007 seconds
Note: the run showed a general downward trend,
starting at 1.111 seconds and ending at 1.062 seconds.

profiling run: obfuscate_value caching, no gc: 50 runs of 1 reps:
mean average 0.733 seconds, deviation 0.007 seconds
Note: again, there was a notable downward trend, starting at 0.759 seconds,
dropping to 0.732 seconds on the second iteration, and ending at 0.727 seconds.
That first iteration likely caused most of the deviation from the mean.

So caching threshold lowered average startup time by 0.332 seconds, a speedup of 31%.
Not counting garbage collection or window creation.

profiling run: stock, full GC: 20 runs of 1 reps:
mean average 1.064 seconds, deviation 0.009 seconds

Garbage collection doesn't add any time at all; perhaps because not much data is
created and discarded during the object creation.

There was a bug in the testbed. These comparative runs without/with GC were not done with the exact same code, but is reasonably comparable.

profiling run: stock, no gc: 20 runs of 1 reps:
mean average 1.099 seconds, deviation 0.008 seconds
profiling run: stock, full gc: 20 runs of 1 reps:
mean average 1.211 seconds, deviation 0.009 seconds

GC adds ~0.1 seconds per run.

[SilasD]

internal/caravan/common.lua use trader over broker

When doing trading (i.e. the DF Trade window is open showing the two columns), it can happen that you have a broker, but you actually trade using a different unit. This can be triggered by opening the depot building view and choosing Anyone requested at depot, repeating this until some unit that is not the broker shows up todo the trading.

When this happens, the DFHack Select trade goods overlay shows different obfuscated values than the DF Trade window.

This patch fixes that case by using the trader's appraisal skill if trading is active.

This could use a lookover to verify that I haven't missed a necessary test.

[SilasD]

For discussion: Implicit de-obfuscation of obfuscated goods values.

When there is no broker, or the broker has less than Legendary Appraisal skill, items' values are obfuscated both by DF and by the DFHack Trade and Movegoods overlays.

However, the DFHack Trade and Movegoods overlays sort items by their un-obfuscated values.

This means that higher-valued items can be distinguished from lower-valued items, even if the broker appraises them at the same obfuscated value.

Should sorting be changed to sort by the obfuscated values? Or is this getting too picky?

[ab9rf]

Should sorting be changed to sort by the obfuscated values? Or is this getting too picky?

Yes, sorting should be by obfuscated value