Heartbeat duplicate detection potentially yielding false positives

[IgnisDa]

Describe the bug

I made an import from Wakatime, but the entire data in not imported. I have about 100hrs missing from each of my major worked on projects. I could not find any issues like this. Is this a known bug?

System information

Output of uname -ar

Linux main 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Hosted using dokku.

[muety]

Did you get any error messages in the server's console? For how long did you wait? If the amount of data is very large, it might well take a couple of hours, since it's currently downloaded in small batches (see #323).

[IgnisDa]

For how long did you wait?

About 10 minutes after which I got an email saying the import has completed successfully.

Here are the entire logs since server startup: http://sprunge.us/mbNHSN

Edit:

I do not see any failed logs. Could it happen that wakatime is not returning the entire data?
update: likely not. i downloaded their JSON dump, looks complete to me.

[muety]

It's probably related to #334 (comment). As already explained there, Wakapi and WakaTime calculate coding duration differently, so there will naturally be a discrepancy. Also, a lot of duplicates seem to have been filtered out during your import (only 373086 of 389859 downloaded heartbeats were actually persists).

Could you maybe check the WakaTime CSV dump if you can spot any duplicates or other irregularities? Might well be that there is a bug in Wakapi, that causes too many heartbeats to be filtered out. Would love to get your support on investigating this!

[muety]

As mentioned on that other issue, we hash every heartbeat object to check for duplicates. I just briefly reviewed the implementation of that again and there is a chance that hashing for a heartbeat's time attribute might not be working properly, due to mitchellh/hashstructure#38. I'll have to investigate deeper, but if that turns out true, then the "duplicate detection" might yield false positives.

Could you potentially send me a subset of your CSV export (pseudonymized, if you prefer that) that includes a portion of records where all relevant attributes (entity, type, project, language, ...) are identical, except for the timestamp? But don't worry, I can also just handcraft a bit of fake data for that!

Will keep you posted.

[IgnisDa]

@muety I am not sure what you want. Would the complete export work? Wakatime gives a download link, you can download it from there. It is about 250MBs.

[muety]

Complete export would help as well, but it obviously contains quite a lot of potentially personally identifiable information. So feel free to only take a portion of it and / or replace project names or so. Send it to ferdinand@muetsch.io.

But no worries if that's too much to ask for! I can probably also just write a quick script to generate test data for debugging the above!

[IgnisDa]

Looks like you will have to generate the data yourself, the files are just too large to upload. Sorry :(

[muety]

The CSVs can probably be compressed quite tremendously. But no worries if not! Thanks for help.

[IgnisDa]

I'm not in front of my system anymore. Will try to convert the json to CSV tomorrow and update here.