comparison with python parser

[danielhrisca]

Hello Ingemar,

I was curious if you have performed a speed comaprison between your parser and the Python parser that I have developed (https://github.com/danielhrisca/asammdf) . It would be interesting to see how much the gap is between the natively compiled implementation and the interpreted language version.

All the best,
Daniel

[ihedvall]

Yes it could be interesting to check. My experience from the automotive industry is however that most of the "parsing" time is actually reading from the disc or in reality from a network disc (even worse). So there will be just minor differences in speed comparision between C++ and Python.

I have done a small MDF viewer application in the preliminary release install kit. It is similar to the Vector's MDF viewer. Unfortunately I only have small files (< 1GB) so it is hard to compare visually.

I can setup a C++/Python test for some MDF files.

Best Regards
Ingemar Hedvall

[danielhrisca]

I tried to use the demo app but it would not start complaining about missing dll files.

You can generate test files using this script https://github.com/danielhrisca/asammdf/blob/master/benchmarks/bench.py , more exactly this function
https://github.com/danielhrisca/asammdf/blob/1507f100bddc86c5f037db2770ec39e42bdfc306/benchmarks/bench.py#L114

[ihedvall]

Preliminary test result.

I tried to run the benchmark test on my machine but I failed to figure out all python dependencies. I did get the asammdf GUI running and its not any major difference in speed. I think its mostly a GUI thing.

I found your test files for
Test.mf4 (Your benchmark file) on AMD Ryzen 9 and from SSD storage.

Read measurement info (meta data): 9.088 ms
Read everything but raw data: 890.039 ms

I will play around little with testing diffrent file storage to check how NAS and iSCSI perform.

Best Regards
Ingemar Hedvall

[danielhrisca]

I tried to run the benchmark test on my machine but I failed to figure out all python dependencies. I did get the asammdf GUI running and its not any major difference in speed. I think its mostly a GUI thing.

to run the script bench.py you need to install additionally the following

pip install psutil mdfreader

[ihedvall]

Different storage. Note that iSCSI and TrueNAS is the same BSD machine running over the same network 1 GBit/s.

Read Measurement Info SSD: 6.7528 ms
Everything But Raw Data SSD: 884.517 ms

Read Measurement Info TrueNas: 256.303 ms
Everything But Raw Data TrueNas: 3970.01 ms

Read Measurement Info iSCSI: 14.6776 ms
Everything But Raw Data iSCSI: 824.295 ms

[danielhrisca]

Read Measurement Info SSD: 6.7528 ms Everything But Raw Data SSD: 884.517 ms

Read Measurement Info TrueNas: 256.303 ms Everything But Raw Data TrueNas: 3970.01 ms

Read Measurement Info iSCSI: 14.6776 ms Everything But Raw Data iSCSI: 824.295 ms

I guess this is with your library. I think the interesting timing is the second one, were you load everything but the actual samples

[ihedvall]

TestResults Python on the same machine. You should compare with the running with NAS values. Unfortunately is the parsing time more related to where the test file is stored, than any difference between parsing language.

I send over the result files but I'm little bit confused that all result files are timestamped 2019. I think I've done something wrong ?

results.zip

[ihedvall]

Forget the previously message. The bench.py fails due to string encoding problem. Why is it always string problems?

ldf is not supported
xls is not supported
xlsx is not supported
yaml is not supported
===== 2022.02.22 19:02:36 =====
Traceback (most recent call last):
File "C:\Users\ihedv\AppData\Local\Programs\Python\Python39\lib\idlelib\run.py", line 559, in runcode
exec(code, self.locals)
File "L:\asammdf\benchmarks\bench.py", line 1165, in
main(args.text_output, args.format)
File "L:\asammdf\benchmarks\bench.py", line 922, in main
generate_test_files(version)
File "L:\asammdf\benchmarks\bench.py", line 230, in generate_test_files
mdf.append(sigs, common_timebase=True)
File "C:\Users\ihedv\AppData\Local\Programs\Python\Python39\lib\site-packages\asammdf\blocks\mdf_v4.py", line 3190, in append
raise MdfException(f'wrong encoding "{encoding}" for string signal')
asammdf.blocks.utils.MdfException: wrong encoding "None" for string signal

[danielhrisca]

That's a bug in my code. I will push a fix to the development branch

[danielhrisca]

You can brag the fix now

[ihedvall]

The benchmark test hangs half way through the test. I stopped the benchmark after 3 hour. The test result so far is good. I don't think that the programming language (Python,C++) matters. Instead it is how the file is read and where it is stored.

Best Regards
Ingemar Hedvall

ldf is not supported
xls is not supported
xlsx is not supported
yaml is not supported

Benchmark environment

• 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)]
• Windows-10-10.0.22000-SP0
• AMD64 Family 23 Model 113 Stepping 0, AuthenticAMD
• numpy 1.22.2
• 64GB installed RAM

Notations used in the results

• compress = mdfreader mdf object created with compression=blosc
• nodata = mdfreader mdf object read with no_data_loading=True

Files used for benchmark:

• mdf version 3.30
  • 432 MB file size
  • 7 groups
  • 14007 channels
• mdf version 4.10
  • 481 MB file size
  • 7 groups
  • 14007 channels

================================================== ========= ========
Open file Time [ms] RAM [MB]
================================================== ========= ========
asammdf 7.0.7 mdfv3 721 337
mdfreader 4.1 mdfv3 81629 898
mdfreader 4.1 no_data_loading mdfv3 777 343
mdfreader 4.1 compress mdfv3 76534 738
asammdf 7.0.7 mdfv4 216 150
mdfreader 4.1 mdfv4 0* 0*
mdfreader 4.1 no_data_loading mdfv4 1361 227
mdfreader 4.1 compress mdfv4 0* 0*
================================================== ========= ========

================================================== ========= ========
Save file Time [ms] RAM [MB]
================================================== ========= ========
asammdf 7.0.7 mdfv3 5389 354
mdfreader 4.1 mdfv3 0* 0*
mdfreader 4.1 no_data_loading mdfv3 0* 0*
mdfreader 4.1 compress mdfv3 0* 0*
asammdf 7.0.7 mdfv4 10094 782

[danielhrisca]

The freeze is caused by the mdfreader functions.

[danielhrisca]

You can remove them by commenting out the functions that have reader in the name

[ihedvall]

Here is the preliminary result (Python/C++) for NAS storage.
Open File [ms]: 1376/73
Get All Channels but No Data [ms]: 60045/6601

I have tested with the new test files generated by the benchmark. I commented out most of the MDF3 test so the tests run faster.
I think the difference is too large (10x) so I propose that you also run the benchmark checking if you get the similar result. I tried to change the path to a local (SSD) path but without any luck.

Benchmark environment

• 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)]
• Windows-10-10.0.22000-SP0
• AMD64 Family 23 Model 113 Stepping 0, AuthenticAMD
• numpy 1.22.2
• 64GB installed RAM

Notations used in the results

• compress = mdfreader mdf object created with compression=blosc
• nodata = mdfreader mdf object read with no_data_loading=True

Files used for benchmark:

• mdf version 3.30
  • 432 MB file size
  • 7 groups
  • 14007 channels
• mdf version 4.10
  • 481 MB file size
  • 7 groups
  • 14007 channels

================================================== ========= ========
Open file Time [ms] RAM [MB]
================================================== ========= ========
asammdf 7.0.7 mdfv4 223 150
mdfreader 4.1 mdfv4 0* 0*
mdfreader 4.1 no_data_loading mdfv4 1376 228
================================================== ========= ========

================================================== ========= ========
Save file Time [ms] RAM [MB]
================================================== ========= ========
asammdf 7.0.7 mdfv4 9966 782
================================================== ========= ========

================================================== ========= ========
Get all channels (36424 calls) Time [ms] RAM [MB]
================================================== ========= ========
mdfreader 4.1 nodata mdfv4 17.703261314617706/s 60045 241
================================================== ========= ========

================================================== ========= ========
Convert file Time [ms] RAM [MB]
================================================== ========= ========
asammdf 7.0.7 v3 to v4 7604 538
asammdf 7.0.7 v4 to v410 2650 703
asammdf 7.0.7 v4 to v420 2281 866
================================================== ========= ========

================================================== ========= ========
Merge 3 files Time [ms] RAM [MB]
================================================== ========= ========
asammdf 7.0.7 v3 9455 604
mdfreader 4.1 v3 0* 0*
mdfreader 4.1 nodata v3 0* 0*
mdfreader 4.1 compress v3 0* 0*
asammdf 7.0.7 v4 7883 714
mdfreader 4.1 v4 0* 0*
mdfreader 4.1 nodata v4 0* 0*
mdfreader 4.1 compress v4 0* 0*
================================================== ========= ========

ERRORS

[danielhrisca]

I see 223ms for open , and the 60000ms for get is done with mdfreader not asammdf

[ihedvall]

I did change my test to read everything including all channel values. I think the Python/C++ comparing is now as close as we can get.

Reading everything with asammdf takes 60017 ms. With MdfRead (c++) it takes 20386 ms.

This looks more reasonable.

Best Regards

Files used for benchmark:

• mdf version 3.30
  • 432 MB file size
  • 7 groups
  • 14007 channels
• mdf version 4.10
  • 481 MB file size
  • 7 groups
  • 14007 channels

================================================== ========= ========
Open file Time [ms] RAM [MB]
================================================== ========= ========
asammdf 7.0.7 mdfv4 253 150
================================================== ========= ========

================================================== ========= ========
Save file Time [ms] RAM [MB]
================================================== ========= ========
asammdf 7.0.7 mdfv4 5537 782
================================================== ========= ========

================================================== ========= ========
Get all channels (36424 calls) Time [ms] RAM [MB]
================================================== ========= ========
asammdf 7.0.7 mdfv4 40.00525636744555/s 60017 302
================================================== ========= ========

================================================== ========= ========
Convert file Time [ms] RAM [MB]
================================================== ========= ========
asammdf 7.0.7 v3 to v4 4147 538
asammdf 7.0.7 v4 to v410 2632 702
asammdf 7.0.7 v4 to v420 2345 867
================================================== ========= ========

================================================== ========= ========
Merge 3 files Time [ms] RAM [MB]
================================================== ========= ========
asammdf 7.0.7 v4 8473 721
================================================== ========= ========

[danielhrisca]

Do you compute the scaled values in the C++ code (apply the channel conversion on the raw values to get the scaled values)?

In the python benchmark the scaled values are extracted, and also it's the worst case were for each get call the complete channel group raw samples are read. Other MDF class methods like select, convert .. use an optimized path were the raw channel group samples are read once and the corresponding channel values are extracted all together

[ihedvall]

No. The MdfReader stores each channel's raw sample value in an array. The conversions to engineering values are done afterwards by someapplication code. I added conversion to engineering values to the test and this added about 20 seconds.

The comparision between asammdf/MdfReader is 60s/43s.

Maybe the google test code might be more descriptive.

TEST_F(TestRead,` Benchmark) {
  {
    MdfReader oRead("K:/test/mdf/net/testfiles/test.mf4");
    const auto start = std::chrono::steady_clock::now();
    oRead.ReadMeasurementInfo();
    const auto stop = std::chrono::steady_clock::now();
    std::chrono::duration<double> diff = stop - start;
    std::cout << "Read Measurement Info TrueNas: " << diff.count() * 1000 << " ms" << std::endl;
  }
  {
    const auto start = std::chrono::steady_clock::now();
    MdfReader oRead("K:/test/mdf/net/testfiles/test.mf4");
    oRead.ReadEverythingButData();
    const auto* file = oRead.GetFile();
    DataGroupList dg_list;
    file->DataGroups(dg_list);
    for ( auto* dg : dg_list) {
      ChannelObserverList observer_list;
      auto cg_list = dg->ChannelGroups();
      for (const auto* cg : cg_list) {
        CreateChannelObserverForChannelGroup(*dg, *cg, observer_list);
      }
      oRead.ReadData(*dg);

      double eng_value = 0;
      bool valid = true;
      for (const auto& channel : observer_list) {
        size_t samples = channel->NofSamples();
        for (size_t sample = 0; sample < samples; ++sample) {
          valid = channel->GetEngValue(sample,eng_value);
        }
      }
    }

    const auto stop = std::chrono::steady_clock::now();
    std::chrono::duration<double> diff = stop - start;
    std::cout << "Everything + Conversion (TrueNas): " << diff.count() * 1000 << " ms" << std::endl;
  }

}

[danielhrisca]

The comparision between asammdf/MdfReader is 60s/43s.

The bench is stopped after 60s for asammdf, not all channels were extracted (40ch/s * 60 => 2400 channels otu of 14000)

[danielhrisca]

But if you look at the convert test asammdf 7.0.7 v4 to v410 2632 with the optimized extraction it takes 2.6s to get all the channels and append to a new MDF object

[ihedvall]

OK I didn't notice the 60s timeout. I patch the bench.py file to timeout in 10 min and rerun. It took 311s. I am little bit confused what we comparing. Note that I'm running against a NAS with 1 GBit network, so just transfer time is about 5s for a 500MB file.

def get_all_mdf4(output, fmt):


    with Timer("Get all channels", f"asammdf {asammdf_version} mdfv4", fmt) as timer:
        t = perf_counter()  
        x = MDF(r"test.mf4")
        counter = 0
        to_break = False
        for i, gp in enumerate(x.groups):
            if to_break:
                break
            for j in range(len(gp["channels"])):
                t2 = perf_counter()
                if t2 - t > 600:
                    timer.message += " {}/s".format(counter / (t2 - t))
                    to_break = True
                    break
                x.get(group=i, index=j, samples_only=True)
                counter += 1
    output.send([timer.output, timer.error])