Setting max number of lines doesn't appear to work for DataDistributor

[anjueappen]

For the the number of lines passed in via cmd line is 80. Regardless of this number, the output is always 5 lines.

dist = dist.setMaxNumberOfLines(80);

The default example (output to console) doesn't appear to set maxNumberOfLines for the distributor, yet it produces the same number of lines on each run. Is this variable set elsewhere in the code?

[mibrahim]

We'll try to reproduce it. Just as a general rule, the number of lines are the max so as long as you're generating fewer number of lines it won't stop. Also, it won't attempt to duplicate or repeat the number of lines to reach the max.

[anjueappen]

Thanks for your help!
We tried in increase the number of permutations across the columns and that fixed it. It was as you mentioned, the engine didn't duplicate any lines to meet the maximum and we had too few permutations in our code to do this.

We are finding, however, that as we introduce increase the line count into the 10s of millions, all the threads appear to block indefinitely, or until the JVM memory ran out.

The log file we had can be found here [https://raw.githubusercontent.com/anjueappen/391DataGeneration/master/hs_err_pid26610.log]

[mibrahim]

@anjueappen yes you are correct. If you're trying to generate data more than the heap allocated for you jvm, it will eventually run out of memory and error out.

As we understand now, you've reached a solution for this issue. I'll close this bug and please let us know if you have other questions or issues.
Thanks