supplied instant is later than self

Meta

rustc --version --verbose:

rustc 1.52.1 (9bc8c42bb 2021-05-09)
binary: rustc
commit-hash: 9bc8c42bb2f19e745a63f3445f1ac248fb015e53
commit-date: 2021-05-09
host: x86_64-unknown-linux-gnu
release: 1.52.1
LLVM version: 12.0.0

Error output

thread 'rustc' panicked at 'supplied instant is later than self', library/std/src/time.rs:281:48
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.52.1 (9bc8c42bb 2021-05-09) running on x86_64-unknown-linux-gnu

note: compiler flags: -C opt-level=3 -C embed-bitcode=no --crate-type lib

note: some of the compiler flags provided by cargo are hidden

Backtrace

0: rust_begin_unwind
             at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panicking.rs:493:5
   1: core::panicking::panic_fmt
             at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/core/src/panicking.rs:92:14
   2: core::option::expect_failed
             at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/core/src/option.rs:1321:5
   3: core::option::Option<T>::expect
             at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/core/src/option.rs:349:21
   4: std::time::Instant::duration_since
             at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/time.rs:281:9
   5: <std::time::Instant as core::ops::arith::Sub>::sub
             at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/time.rs:406:9
   6: std::time::Instant::elapsed
             at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/time.rs:345:9
   7: rustc_codegen_llvm::base::compile_codegen_unit
   8: <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_ssa::traits::backend::CodegenBackend>::codegen_crate
   9: rustc_interface::passes::QueryContext::enter
  10: rustc_interface::queries::Queries::ongoing_codegen
  11: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter
  12: rustc_span::with_source_map
  13: rustc_interface::interface::create_compiler_and_run
  14: scoped_tls::ScopedKey<T>::set

This might be an LLVM bug, as it never occurs on Windows and occurs in every rust program that I've compiled on Linux, not when compiling, but running it.

[fee1-dead]

Can you check your system clock? That might be the issue. Also did you use RUST_BACKTRACE=full to get the full backtrace?

[the8472]

This usually is due to outdated kernels, hardware or hypervisor bugs (see #80674, #82606, #84448 and several others)

Can you provide the following information?

• OS and kernel version
• output for one core from /proc/cpuinfo
• hypervisor if the system is virtualized
• selected clock source (cat /sys/devices/system/clocksource/clocksource0/current_clocksource)

[tabokie]

@the8472 We encountered the same issue in a production service (TiKV) deployed on AWS EC2.

System information as requested:

• OS and kernel version

-bash-4.2$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
-bash-4.2$ uname -a
Linux ip-10-28-50-7.ap-northeast-1.compute.internal 4.14.154-128.181.amzn2.x86_64 #1 SMP Sat Nov 16 21:49:00 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• output for one core from /proc/cpuinfo

-bash-4.2$ cat /proc/cpuinfo
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 79
model name  : Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
stepping    : 1
microcode   : 0xb000038
cpu MHz     : 2700.001
cache size  : 46080 KB
physical id : 0
siblings    : 16
core id     : 0
cpu cores   : 8
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx xsaveopt
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit
bogomips    : 4599.99
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

• hypervisor if the system is virtualized

-bash-4.2$ cat /sys/hypervisor/
compilation/ guest_type   properties/  type         uuid         version/
-bash-4.2$ cat /sys/hypervisor/version/
extra  major  minor
-bash-4.2$ cat /sys/hypervisor/version/major
4
-bash-4.2$ cat /sys/hypervisor/version/minor
11
-bash-4.2$ cat /sys/hypervisor/version/extra
.amazon

• selected clock source (cat /sys/devices/system/clocksource/clocksource0/current_clocksource

-bash-4.2$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
xen

[the8472]

@tabokie thanks for the report.

4.14.154-128.181.amzn2

Looks like that does contain backported patches which query the hypervisor whether the pvclock is stable which then disables the clock monotonization in the kernel

So either the version of xen that amazon is using has a bug (advertising a stable clock when it isn't) or the backported patches are missing some critical pieces.

How reliably can you reproduce the issue? If it occurs frequently enough to make investigation easy, can you update your kernel to see if that solves it? That would indicate some incomplete backports.
If the issue persists after an update it would point at a problem at amazon, at least with that particular instance type. Based on the CPU type used I guess it's an m4 or r4 instance?

[tabokie]

@the8472 Thanks for looking into our case, we have only experienced the panic once. Will share more information if there's progress in reproducing it.

Since it's hard for us to lock-in certain "safe" kernel/hardware to our software, or check the safety of arbitrary kernel/hardware, we are leaning towards replacing standard Instant to our own with a non-panic version of elapsed(saturating_elapsed). Do you think the standard library could use such an addition too?

[the8472]

I'm primarily asking to figure out how to adjust std's built-in heuristics for the clock monotonization and where to report upstream bugs. Something is violating the posix CLOCK_MONOTONIC contract and it's hard to pinpoint the cause because the issue only affects specific hardware/software constellations.

we are leaning towards replacing standard Instant to our own with a non-panic version of elapsed(saturating_elapsed). Do you think the standard library could use such an addition too?

You can already get this with existing standard library methods via Instant::now().saturating_duration_since(old_instant). If you want a saturating_elapsed() as a shorthand that could be added to std or you could write your own extension trait for std::Instant.

For me, this error did not occur in a VM, it was on bare metal. Unfortunately said machine isn't working at the moment, but I do know the CPU was a Celeron n3060.