Generate and Use Coverage at each run

[nordlow]

There is already

  --coverage DIR     Load .lst files corresponding to source files from DIR

. Why doesn't dub dustmite generate new coverage information "at each call" to dub run and use that via loadCoverage when deciding which code to remove prior to subsequent call?!!! Wouldn't this enable amazingly faster reduction speed for D especially in the reduction of dependencies.

I guess most of this logic should be implemented in dub.

Ping, @CyberShadow.

See https://github.com/CyberShadow/DustMite/blob/master/dustmite.d#L2405

[CyberShadow]

What are you trying to achieve with --coverage?

Can you describe your use case in more detail?

[nordlow]

I thought loadCoverage() gave hints to the reduction algorithm. Maybe I was wrong. What's the meaning of loadCoverage? Anyway, wouldn't information about unused lines and functions would be of exceptional value to the reduction algorithm?; the algorithm could start with remove the single lines that aren't covered typically inside conditionals and then try removing functions that were called in covered lines... Are you following?

[CyberShadow]

--coverage makes DustMite avoid reducing code that ran. Generally that set of code shouldn't change. It's possible that it does change, e.g. if DustMite reduces some non-code declarations that cause some code to no longer run, but I can't think of a practical use case for this scenario.

[CyberShadow]

the algorithm could start with remove the single lines that aren't covered typically inside conditionals and then try removing functions that were called in covered lines... Are you following?

In many cases it's just as useful to try reducing code that ran as everything else, so I can't really think of a good argument in favor of that.

[CyberShadow]

You can do that by running DustMite with --coverage, and then again without it.

[nordlow]

Ok, thanks. The administration of producing and reading the .lst-files needs to be added to dub dustmite, though.

[CyberShadow]

I don't even remember why it was added.

I'm not going to make any changes without some hard evidence that they're beneficial :)

[nordlow]

I don't even remember why it was added.

I'm not going to make any changes without some hard evidence that they're beneficial :)

What kinds of DustMite-changes are you thinking of; automatic generation and reading of .lst files when --coverage without arguments is passed?

[CyberShadow]

Oh, I misunderstood what you said. I thought you said that there was something that needed to be added to DustMite itself.

[nordlow]

Btw, my dustmite reduction takes too long time reducing packages. What strategy should I try then?

[CyberShadow]

Have you tried --strategy=careful yet?

[nordlow]

Now. I'll try that. Can I abort a DustMite with Ctrl-C and then use the the result as input to a new dustmite?

[CyberShadow]

Of course (assuming you didn't use --no-save).

[nordlow]

I first needed dlang/dub#2173.

[nordlow]

Btw, does dustmite utilize all the cores when checking if a set of tried splits succeeds in create a program the compiles? And does dub dustmite do the same?

[nordlow]

Btw, at https://dlang.org/blog/2020/04/13/dustmite-the-general-purpose-data-reduction-tool/ I read

"The parser DustMite uses for D source code is intentionally very simple because it needs to handle potentially invalid D code, and you don’t want your bug reduction tool to also crash on top of the compiler."

? If the compilation always passes after every reduction why does parser need to handle invalid D code? And what is meant by invalid code in this case

[nordlow]

Btw, is there a node deletion history written to disk during the call to DustMite?

[CyberShadow]

Btw, does dustmite utilize all the cores when checking if a set of tried splits succeeds in create a program the compiles?

I didn't understand that beyond "all cores".

You can ask DustMite to use all cores by using the -j flag.

No idea about the Dub side.

[nordlow]

No idea about the Dub side.

Can I figure that out by looking at the output of --trace?

[CyberShadow]

I don't understand what you're trying to achieve here, can you reword?

As for -j, you would have to look if dub allows passing that flag to DustMite somehow.

[CyberShadow]

? If the compilation always passes after every reduction why does parser need to handle invalid D code?

It doesn't know that. You might be reducing a bug where compilation fails due to the bug you're trying to create a test case for.

[CyberShadow]

Btw, is there a node deletion history written to disk during the call to DustMite?

There is --trace which will save verbose snapshots of every reduction attempt and whether it succeeded.

[nordlow]

Isn't it better to have a short description of what kind of reduction it tried?

[CyberShadow]

Why? What are you hoping to achieve by this?

[nordlow]

Say that you potentially have infinite set of possible reductions to try out of which only a small part compiles. If so and there are N cores at our disposal we should try syntax-checking (compiling) N of these pendings reductions in parallel and filter out the ones that compiles. If there's only a very small subset of these reductions that pass and D's parser is very fast this is gonna result in an enormous speedup. I assume this is already implemented, right? If so what parameters describe a reduction; deleted region R (start end end byte offset) in file F?

[CyberShadow]

I assume this is already implemented, right?

Yes, if you use -j.

If so what parameters describe a reduction; deleted region R (start end end byte offset) in file F?

The Reduction struct describes a reduction:

DustMite/dustmite.d

Line 58 in 6634109

struct Reduction

but I don't understand how that relates to the rest of your post.

[nordlow]

Ahh, I misinterpreted

-j[N] Use N look-ahead processes (48 by default)

as using 48 cores when even I don't specify -j. Thanks for clarifying this.

[nordlow]

According to my measurements my dub dustmite reduction is currently spending most of its time compiling successful programs where only a very small fraction of the number of lines are removed at every step. Making use of the last profiling information to guide the next reduction to remove lines not visited would afaict result in a much greater relative reduction of number of lines each time. I still don't understand why this isn't considered.

Would it be possible for you to experiment with running dub dustmite using dlang/dub#2174?

I'm currently using

dub dustmite -c it -b unittest --compiler=ldc2 --program-status=1 --program-regex='stack-buffer-overflow' --no-redirect -j32 --strategy=careful OUTPUT_DIR -- -s -d

as the error is triggered in unit-threaded unittest execution.

[nordlow]

Here's a snapshot of the contents of M-x proced on Emacs:

per   310427   2       0.0  0.0  16:47    00:00 /home/per/Work/dub/bin/dub dustmite -c it -b unittest --compiler=ldc2 --program-status=1 --program-regex=stack-buffer-overflow -j32 --strategy=careful ../PACKAGE_DIR.reduce.1 -- -s -d
per   310432    3      2.1  1.5  16:47    02:42 /home/per/.local/dlang/linux/bin64/dustmite /home/per/Work/PACKAGE_DIR.reduce.1 /home/per/Work/dub/bin/dub\ dustmite\ --test-package=PACKAGE_NAME\ --build=unittest\ --config=it\ "--compiler=ldc2"\ --program-status=1\ "--program-regex=stack-buffer-overflow"\ "--strategy=careful"\ "-j32"\ --vquiet
per  1133594     4     0.2  0.0  18:53    00:00 /bin/sh -c /home/per/Work/dub/bin/dub\ dustmite\ --test-package=PACKAGE_NAME\ --build=unittest\ --config=it\ "--compiler=ldc2"\ --program-status=1\ "--program-regex=stack-buffer-overflow"\ "--strategy=careful"\ "-j32"\ --vquiet
per  1133595      5    0.4  0.0  18:53    00:00 /home/per/Work/dub/bin/dub dustmite --test-package=PACKAGE_NAME --build=unittest --config=it --compiler=ldc2 --program-status=1 --program-regex=stack-buffer-overflow --strategy=careful -j32 --vquiet

. Do you see anything faulty?

The /home/per/Work/dub/bin/dub is the -j enabled dub.

[CyberShadow]

Yes. The second command has -j in the test command (dub passing -j back to itself), but not as a parameter to DustMite itself.

[CyberShadow]

BTW, the same is true with --strategy=careful, and I can see from the output you posted above that it's still using the default strategy.

[nordlow]

Thanks!

[nordlow]

This solves my problems dlang/dub#2175.

Output now looks more like ours:

Yes
############### ITERATION 0 ################
============= Depth 0 =============
[ 0.0%] Remove [] => No (lookahead-wait)
============= Depth 1 =============
[ 0.0%] Remove [1] => No (lookahead-wait)
[15.3%] Remove [0] => No (lookahead)
============= Depth 2 =============
[ 0.0%] Remove [11] => No (lookahead)
[ 8.1%] Remove [10] => Yes (lookahead-wait)
[15.3%] Remove [01] => No (lookahead)
[38.2%] Remove [00] => No (lookahead)
############### ITERATION 1 ################
============= Depth 0 =============
[ 0.0%] Remove [] => No (cached)
============= Depth 1 =============
[ 0.0%] Remove [1] => No (cached)
[ 8.8%] Remove [0] => No (lookahead)
============= Depth 2 =============
[ 0.0%] Remove [10] => No (cached)
[ 8.8%] Remove [01] => No (cached)
[33.5%] Remove [00] => No (cached)
============= Depth 3 =============
[ 0.0%] Remove [101] => Yes (lookahead)
[ 3.3%] Remove [100] => No (cached)
[ 8.8%] Remove [011] => No (lookahead)
[20.3%] Remove [010] => No (lookahead)
[33.5%] Remove [001] => No (lookahead)
[63.4%] Remove [000] => No (lookahead)
############### ITERATION 2 ################
============= Depth 0 =============
[ 0.0%] Remove [] => No (cached)
============= Depth 1 =============
[ 0.0%] Remove [1] => No (cached)
[ 5.6%] Remove [0] => No (lookahead)
============= Depth 2 =============
[ 0.0%] Remove [10] => No (cached)
[ 5.6%] Remove [01] => No (lookahead)
[31.1%] Remove [00] => No (lookahead)
============= Depth 3 =============
[ 0.0%] Remove [100] => No (cached)
[ 5.6%] Remove [011] => No (cached)
[17.6%] Remove [010] => No (cached)
[31.1%] Remove [001] => No (cached)
[62.1%] Remove [000] => No (cached)
============= Depth 4 =============
[ 0.0%] Remove [1001] => No (lookahead)
[ 3.9%] Remove [1000] => Yes (lookahead-wait)
[ 5.6%] Remove [0111] => Yes (lookahead-wait)
[10.4%] Remove [0110] => No (lookahead)
[17.6%] Remove [0101] => No (lookahead)
[24.4%] Remove [0100] => No (lookahead)
[31.1%] Remove [0011] => No (lookahead)
[39.8%] Remove [0010] => No (lookahead)
[62.1%] Remove [0001] => No (lookahead)

[nordlow]

Btw, have you seen creduce having activity as recent as 21 of June 2021.

[nordlow]

Moreover I believe the docs should provide a link to https://en.wikipedia.org/wiki/Delta_debugging and possibly also https://news.ycombinator.com/item?id=22855633.