UNIX timestamp s vs ms

[adobekan]

Hi All,

What would be the good way on handling UNIX timestamp in ms and s as well?

Right now we have following description

UNIX Timestamp	Time	Unix time is a system for describing a point in time. It is the number of seconds that have elapsed since the Unix epoch, excluding leap seconds.

[erikbosch]

Strictly speaking we do not specify encoding or datatype for the VSS units. So there is nothing that prevents us from using UNIX Timestamp together with float, and tooling will not even complain if you use it together with string or boolean. So indirectly we could get milliseconds by using float representation. (We actually do not use the UNIX timestamp unit at all today in standard VSS catalog)

But I have no problem if we would like to add a timestamp_ms unit. At the same time we should possibly rename UNIX Timestamp to something like timestamp_s to get a common pattern and get rid of the blank.

A related topic - if we start changing timestamp type, should we then also consider creating a unit for "ISO 8601 time". We have today some signals with datatype string, no unit, but in free text stated that the string shall be encoded according to ISO 8601. This could possibly be handled by a unit time_iso_8601. Then we could possibly also in the unit file specify which datatypes that are allowed for a unit. E.g. specify that timestamp_s shall be allowed for datatype: uint, int, float (i.e. all except boolean and string, even if supported time range most likely will be useless if you select uint8), and time_iso_8601 be allowed only for datatype: string. That would make it possible for tools to give errors if someone tries to define a timestamp signal using boolean as datatype. We could possibly even have recommended types, so that you get a warning if you use timestamp_s with int32, uint16 or some other datatypes that possibly does not fit your needs.

( By the way, Neil Puthuff also brought up some ideas on timestamp handling with additional fields, see https://docs.google.com/document/d/1dHHMIlIZpsbnVsTiiza4Suua-vVbfR1xm11KNKdAm38/edit#)

[slawr]

This is how Apache IoTDB handles it https://iotdb.apache.org/UserGuide/Master/Data-Concept/Data-Type.html#timestamp but of course specifying (input, output) is different from internal representation.

[erikbosch]

This is how Apache IoTDB handles it https://iotdb.apache.org/UserGuide/Master/Data-Concept/Data-Type.html#timestamp but of course specifying (input, output) is different from internal representation.

As a side note - I think the we can improve our documentation on the meaning of datatype/unit in VSS. We have a short sentence in documentation:

It must be noted that the selected unit does not imply that the value of Vehicle.Speed always needs to be sent or visualized as km/h (with float as datatype). A user interface or API may show or request vehicle speed in any unit it likes, and a transport protocol may send speed in another unit, possibly also involving scaling and offset. But in protocols not explicitly specifying data unit types (like VISS) it is expected that Vehicle.Speed is sent and received as km/h (without scaling or offset).

... but nothing similar for datatype, where we for example describe how datatypes typically are selected in VSS, and how implementations may handle it.

[adobekan]

But I have no problem if we would like to add a timestamp_ms unit. At the same time we should possibly rename UNIX Timestamp to something like timestamp_s to get a common pattern and get rid of the blank.

Yup this would make sense as well covering other types which we might have. But then question is how do we handle that between unit and datatype?

[SebastianSchildt]

So as unit I think something like time_iso_8601 is missing, and would be good. I don't think our tools should try to make sure it is combined with the "right" datatype (would we also do it for percentage? Will will also check min max there? At whatpoint would we run into corner cases?). Maybe this can be somebody else's problem?

Wrt Unix timestamp I think I have gone on record before, that I definitely don't like it as a datatype, but of course as Unit it makes sense if want to handle such timestamps in your system.

So dusting off my POSIX knowledge, for the classic "seconds since epoch" timestamp it used to be int32 (but remember that bites us 2039, so to do it well you better use uint64 in your VSS (I think POSIX-wise it is a typedef called time_tor similar).

For higher resolution timestamps the "POSIX-ish" way (Not sure what is the "POSIX law" here, but I think what I describe is common practice), it is actually normally a combination of seconds and nanoseconds (whereas in most systems, depending which API you use, you do not nearly use the nanosecond resolution), this is mainly represented as timespec struct, that in whereas one member is the seconds, second one the nanoseconds. That works in the "old world", because nanoseconds is long (used to be 32 bit in C), however the time_tfor the second is 64 bit on modern systems

tldr: For "standard" unix timestamp with sub-second resolution you need 64bit+32bit -> VSS has no native datatype to store that (uint8 array? Just kidding 😁 )

(Disclaimer: Please double check what I said, I did not "verify by Google", but am reasonable sure....)

Now for practical purposes in VSS you could invent a timestamp with 48bit seconds and 16 bits milliseconds, stuffed into a uint64. I can not imagine any application where it won't work. But if we put this as standard somewhere in VSS (catalogue), generations of programmers will hate us, and once VSS is an integral part of all vehicles and automotive ecosystems, all kind of interesting and exciting bugs due to faulty conversion will come up.

[erikbosch]

Meeting notes:

• Please review and add comments
• Adnan will consider comments/requests/proposals and create pull request.

[neil-rti]

A frequent approach to timestamps is to use integer values for seconds and nanoseconds since Unix Epoch, these are usually 32-bit values, resulting in a 64-bit struct.
This is used by ROS/ROS2: https://docs.ros2.org/foxy/api/builtin_interfaces/msg/Time.html
and in C++ (and other language) libs such as std::timespec: https://en.cppreference.com/w/cpp/chrono/c/timespec so the conversion would be straightforward. This also gives us plenty of resolution headroom, so we're not having this discussion again when we find we need microsecond resolution for something. :)
There's also plenty of standard library conversion functions to go between timespec, time_t, and tm struct formats, so the conversion between this and ISO 8601 should be easy.

[neil-rti]

As a word of caution: timestamps can be one of those things that can take a standards group down a rabbit hole of completeness. Here's one such example: https://gitlab.com/openfmb/psm/ops/idl/openfmb-ops-idl/blob/0d46e4fb734ce6a7fa5c44c4ffb911a3522788ce/OpenFMB-Operational-Model-IDLv4.1.idl#L709 that includes 32-bit fractional seconds (down to 250pS resolution), 64-bit seconds (will overflow in 584 billion years), time quality and error flagging.

[SebastianSchildt]

A frequent approach to timestamps is to use integer values for seconds and nanoseconds since Unix Epoch, these are usually 32-bit values, resulting in a 64-bit struct.

Wouldn't we then fall into the 2038 problem (https://en.wikipedia.org/wiki/Year_2038_problem)?

[erikbosch]

The ROS solution seems to have the 2038 problem, as it use int32. Well at least if you use 1970 as base, they do not really specify that, they say instead "relative to a clock's 0 point".

When we add struct support I see no theoretical problem to include a time struct and possibly use it for VSS signals. As long as we clearly document what ranges it support and how values shall be interpreted I am happy.

[neil-rti]

Good point, and thanks for the article link -- the solutions it shows are varied, but I like the idea of using a 64-bit count of nanoseconds from Unix epoch. It converts easily to/from other 32seconds:32nanoseconds formats, and won't roll over for a few hundred years.

[erikbosch]

A bit related to unit PR at #669, we could if wanted define a "1sec + millisec struct type" and list it as an allowed type for "unix-timestamp"

[erikbosch]

We regularly close old issues and pull requests that appear to be abandoned or completed, where there have not been any activity for a long time. If you think your item is still relevant please feel to add a comment and re-open it. You are also very welcome to bring up the topic at one of our weekly VSS meetings.