Routes not created correctly due to multiple devices tracking a the same time

[mcwieger]

OS & Hardware
Dawarich in Docker on Synology

Version
0.20.2

Describe the bug
Points are not correctly connected into the actual route that was travelled, leading to a spiderweb-like travel pattern and many additional kilomters traveled.

To Reproduce
Imported Records.json from Google. Navigate to Map.

Expected behavior
Correct connection of points.

Screenshots
See attachment. I actually travelled to two places (one south and one north in the map, from my home underneath the stats popup).

Logs
Can provide at your request.

Additional context
None.

UPD: Details of the issue: #635 (comment)

[shaman007]

@Freika, wouldn't it be better to set 'dots' in the interface by default? Can it be a user setting that could be saved?

[Freika]

@mcwieger If you visited the Map page when Import wasn't finished yet, I'd assume points were being created in a non-chronological order, thus forming this weird structure. The page reload would most likely solve that issue. If the importing process was completed and your routes are still looking like this after importing, I'd need a sample of, say, 100 points, so I could debug it.

@shaman007 it wouldn't be better because in current implementation big number of points on the map makes the page slow and at some point unusable, and enabling them by default or allowing user to save it to be enabled by default will inevitably lead to a situation where someone selected a time frame too wide for their browser to be able to handle, effectively constantly freezing the Map page. It's a subject to improvement in the future.

[mcwieger]

I have this on many days. How should I get the sample you need? Take it from Records.json or somewhere (how?) in Dawarich?

[Freika]

@mcwieger you can visit the Points page, select timeframe, search points and then click on "Export as" any of formats, then send me the resulting file

[mcwieger]

export_from_2016-11-30_to_2016-11-30.json

[Freika]

Hmmm I'm getting nice and clean picture:

[mcwieger]

That's a place I've never been! Think you may have used another file to create this?
First coordinate I see in the file is "5.274335","52.045901", which is definitely not in Spain.

[Freika]

@mcwieger huh, sorry, maybe my data overlapped... I'll have another look

[Freika]

@mcwieger I reproduced your issue, but to me it seems so far like your file contains points that have incorrect timestamps in them.

If you're willing to send me the original records.json file, I'd debug it

[mcwieger]

I can share the records.json file with you, but not here on Github. Can I share it with you privately?

[Freika]

@mcwieger send it to my discord PM, link to the server is in the readme

[mcwieger]

Provided via Discord.

[Freika]

Provided via Discord.

I think the problem is that you have more than one device, and on 30.11.2016 one of them was with you in your travels and the other was stationary somewhere (at home maybe), and they both were recording geodata. Since you not yet able to select which device points to render in Dawarich, they are all being rendered as if it was a single device teleporting between locations in short periods of time.

Running unique_device_ids = Point.distinct.pluck(Arel.sql("raw_data->>'deviceTag'")).compact returns 26 (!) different device ids that are found in your data set.

Since I can't really imagine removing one of them in hope Dawarich to render a nice picture, I'd say, you need to wait until I implement device management and a feature to select data from which device to render on the map.

I'll update original issue text and title to reflect the problem

[mcwieger]

That makes sense, thank you for investigating!

Alternative at this moment would be to use the Google Semantic history. Issue there is that it only shows the spots, not the routes that have the full detail provided in records.json.

I'll be patient :-)

[mcwieger]

Still looking at how I can filter out the devices, but in the source. Somehow I can't figure it out: all JSON to CSV scripts I can find only seem to provide the basic info (time + coordinates), but I would like to have the devicetag in the records as well. Do you see any way to get the JSON converted to a readable/editable table in which I can do the device filtering myself? Or is this something that is supported by e.g. owntracks?