history_stats sensor does not reset until slightly after midnight for the next day #75903
Comments
|
history_stats documentation |
|
I may have the same issue. I have two history_stats entities which are based on an "on" condition (one for water heating, one for air conditioning.) I haven't had either one on for a couple of weeks. Today when I started them, one showed it had been on for 11.7 hours, the other 8.5. The history_stats values appear to have incremented normally after that, as the systems cycled on and off. Here's what one of them looked like today shortly after it was returned to service after being shut down for 2 weeks: |
|
I encountered the same problem. In the chart below, note how the sensor begins to accumulate time immediately at 00:00 on Aug 2. I only noticed the anomaly mid-afternoon the following day (the AC had definitely not been running non-stop for 15 hours). I restarted Home Assistant to clear the problem.
|
|
Today my two history_stats values are back to normal. So far, for me it's been a one-time thing; the value was zero for 8 days, then when it started accumulating again it started with a seemingly random value rather than starting with zero. I say random because this was not a previous value. In both cases the values were out of what would be a normal range. |
|
I have the exact same issue since updating to the newest versions HA Core. I have a Tablet uptime sensor based on a Tablet switch being on/off but where before the values were being reported accurately, now it shows that my daughter has been spending like 19hours in a day on her tablet (when I know it's been much much MUCH less than that. I can provide more details on this, just let me know what kind of info would be useful (have nothing wrong on the logs though).
|
|
This is still an issue on 2022.9.6 |
|
It happened again a few days ago. I have a history_stats sensor which tracks how long my 1st floor (zone 1) heating is “on” over the course of the day. Yesterday it only came on just once. After that, the “Zone 1 on today” sensor suddenly went to 13.3 hours:
In the second chart, notice that the binary “Heat 1st” sensor had no state before that. I haven’t needed heat for longer than the purge_keep_days in Recorder, so the previous state change had rolled off. The sensor changed to “on” at around 1302. It stayed on for about 15 minutes and changed to “off.” The 13.3 hours which were added to history_stats looks like the elapsed time since midnight, not since the sensor changed to “on.” |
|
Here's another occurrence. This time it suddenly jumped from around 2 hours to over 10.5 hours after restarting HA.
|
|
I have exactly the same issue. My heating turns off exactly at midnight, yet history stats assume it never turned off
As a temporary fix, I change my accumulation period to start from 0:00:01 (1 second after midninght) and this fixed mine. There must be a bag if the state changes exactly at the same time as the accumulation window starts. Below is the situation after the "fix"
|
|
I had the same problem this morning, I was about about to apply the 1 second fix, but after the HA update to 2022.12.7 my history stats was ok again (0min instead of 7h) Was this corrected ? |
|
I'm having issues with this. The history_stats sensor resets slightly after midnight, meaning that plotting any statistics graphs with'max' picks up the value from the previous day if it was higher than today's. In my case it's how long the heating has been on each day, but I get lots of double days with the same value.
|
|
My sensor showing currently time not sensor state time.
|
bdraco
changed the title
|
Same problem here. For a daily Pretty sure that would happen if it's a monthly time integration of a sensor and the sensor is |
|
There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. |
|
This definitely still needs attention as I'm pretty sure it hasn't been addressed. I have a workaround in place but it should just do what it says it does. |
@fenty17 You suggest that you've got a workaround; can you elaborate? Seems that this should not be allowed to go stale. Partially because I'm not aware of a way in the UI to fix the discrepancies. I've had to modify directly in the db, which is super ugly. |
|
@WoosterInitiative - I was following another similar GitHub issue and got the workaround from there: #84840 (comment) That issue has just been closed as 'not planned'. I'd much rather things like this got attention than voice assistant and Matter progress, but that's just my take. |
|
Hi all - I have a related/similar issue with history stats. See thread the root cause is that the 12am start does not happen until 30 seconds after midnight so the max for the new day holds the max from the previous day. strangely, if you chart using group_by max for the day, the data is correct but if using LTS (statistics) then max/day returns the max from the previous day (until the current day max > previous day max). Chart below shows the issue quite well. I need to use LTS (as I require more than 10 days of data) but the LTS data in not correct.
|
@fenty17 - thanks for the link and tip to reset the entity state. I have implemented this and will see how it works. Whilst setting up the python_script i noticed there is a HACS script that does the same thing and a bit more. You can just add it through the HACS GUI
|
|
I think I'm seeing the same issue, basically when I upgrade Home Assistant and it restarts, my history_stats type:time based on a binary sensor values are wrong for the next 24 hours, with odd spikes. See the screenshot, you can see the time period after I did the restart with all of the odd spikes:
My issue might actually be #80871 |
|
The fix i implemented a few weeks ago has stopped working in HA 2023.9. I use an automation to reset the state to 0 at 11.59.59. This is still working but the state is then updated again to the previous state before the reset. Since 2023.9 When working correctly, it looks like this... |
|
Still a problem |
|
having exactly same with mine history_stats
while the switch pc_s got turned off yesterday
|
|
I just re-tested in 2024.10.1, and it's still not fixed. Same exact behavior as 2 years ago. |
|
Bug still exist for me too. |
|
This bug is very frustrating - it's messing up my data and I've tried the plotly, apexcharts and standard statistics cards - they all depend on the statistics being properly recorded What has to be done to get this issue resolved? It's been open over 2 years |
There are at least 3 separate unrelated issues people keep throwing into this single thread and it makes it very unclear who is having an issue with what. Issue 1. Original OPs comment that history stats is ignoring an "off" state and keeps increasing even when it shouldn't. - I can't reproduce this. I would look into this if I could figure out how to make a reproduceable example. Most of the comments for this are old so I'm not clear if that might have been fixed. Issue 2. Complaint that history stats jumps to absurd values the moment a sensor becomes "on" for the first time after being off for a long time. This is a known issue with how history stats works in the absence of fresh recorder data, and would be fixed by this PR: #126271. Issue 3. Complaint that history stats doesn't reset until a few seconds after midnight and messes up max reading for daily Long Term Stats - this is pretty much just by design of how LTS works and I don't expect it would be fixed, but you can probably craft a start and end template that will mitigate the issue.
Which issue is your issue? |
|
thanks for the response @karwosts My issue is issue 3 - here is my image to demonstrate
Each of the 3 red outlined boxes represents an instance of this issue in just the last week - where the MAX value for day N-1 leaks into day N. This occurs (as mentioned above), in all the history graphs I've used but the one I prefer is plotly (apexcharts is slow and the built-in card has no features) - I will paste the code at the end of this comment. But I can't imagine why this is by design I would be very happy to craft a workaround if you can advise on how to do this, but I have no idea how to go about this and I imagine it could get messy! Here's the yaml...... as you might guess, I added the line entity to prove that the MAX value was wrong.... `
` |
|
@abandyb - Which of those source entities are from the |
|
Hi @karwosts I do apologise if I've wasted your (and other readers) time - I get that this issue page is related to the history_stats integration whereas my issue is with statistics generally - my initial post that you responded to was not complaining about history_stats, but the statistics in HA. To be honest, until I discovered this page recently, I was totally unaware of the existence of the history_stats integration, but would I be correct in assuming that it also is affected by the same issue I wrote about? From reading some of the posts describing what you've classified as Issue 3, my issue is exactly the same regardless of whther it's history_stats or standard HA statistics In any case, having just opened and skim-read the history_stats integration page, I will investigate whether I can build a workaround for my issue Again, apologies if I've wasted your time, and thanks |
|
No worries, it's easy to get them mixed up. History stats wouldn't be appropriate for energy sensors, as it doesn't sum an accumulation, it only counts the time a sensor is in a particular state. The only real way I can think of to fix the statistics accumulation is to make sure your sensor resets to 0 before midnight. So if you have a utility meter, trigger the reset at 23:59 instead of 00:00 should help with that. For making sure history stats resets to 0 before midnight, I would use something like: This will sum the time your sensor is on during a day, but it will reset to 0 at 23:59:?? instead of 00:00:??, so the max won't holdover into the next day. |
|
FWIW I’ve been following this issue for most of the 2.5 years since it was created. The only workaround that works for me is the python script technique mentioned earlier in this thread. What I don’t understand at all is why this issue still exists. The workaround simply should not be needed if the integration worked as intended. I suspect either there is an underlying complexity to a fix which impacts other things, or the HA team simply consider this not ‘popular’ enough to resource the issue resolution. In any case I’d love for one of the HA devs to explain a little more. |
|
My use case is to track how long a I'm not sure which sub-issue category this would fall into. |
|
#126271 did just get merged today, so hopefully that should fix some of these issues where the calculated value is unexpectedly high when a state first turns on. I can't tell if that is @Human issue without more info.
Most of the individual integrations are just volunteer driven. It's not as if Paulus runs around directing people what bugs to work on, things just get fixed when someone decides to work on it. I'll guess whatever issue here isn't severe enough to warrant attention of the top-tier maintainers, so everything else is just left to volunteers. |
|
I understand @karwosts thanks. I suppose it’s more about a bit of transparency around that - with maintainers chipping in on issues to let us know solutions need to come from the community. Otherwise we just wait, and threads like this suddenly come back to life when new people experience the issue and wonder why nothing has happened in years. I am unfortunately one of those who isn’t capable enough to actually dev any integrations but tries to contribute to the community in other ways where I can. I understand the core maintainers can’t do everything, but I think there is a perception that core integrations = HA team responsibility, custom integrations = responsibility of the dev who built it. Many folk who discover that history stats exists (like me) then expect it to work like the docs suggest. There’s no visibility of these types of issues there. I actually feel bad coming across a bit critical here. HA is unbelievably amazing (user of 4 years here) so I only get to this point when I feel there’s a genuine point of value to make. But of course I’m not sure any of the core HA team will actually see this! Meantime I will stick with my workaround knowing that it works, but with unnecessary additional stuff going on which I also know could be avoided. |
|
@karwosts wrote:
First thank you so much for stepping in to help and to cut through the accumulated cruft. My primary problem is issue 3, which what the title of this issue actually is: history_stats sensor does not reset until slightly after midnight for the next day.
Thank you for your workaround
I'll be testing it, but will need a number of days of the right conditions to verify it works. Assuming it does, should the History Stats documentation be updated to suggest using Regarding expectations and frustrations: @fenty17 - I understand your frustration and I agree that what @karwosts posted helps put things in perspective. Home Assistant's integration quality scale is a bit misleading and could probably use some thought. Internal integrations like history_stats are categorized on the documentation page as:
On the linked page Internal is ordered after platinum, implying internal integrations are the highest quality. Yes, if you look below that under Integration Owner it says it is community maintained. So the implied bit is a volunteer needs to step up if work is needed. But that's a topic for a different place. |
|
@karwosts , @rct note that the workaround does not work correctly - see my post above which has fix that works for statistics |
How so? I haven't tested yet but I thought that might work. |
Seems like you are in the same situation as me - content that your workaround works for you. BUT, that is some workaround to create multiple other HA entities just to make an internal ‘core’ integration do what it says it should do out of the box. @karwosts - reading the full thread, there are suggestions to reset slightly before and others for slightly after midnight. It depends on the use case. |
Doesn't this change the start time at, or after, 23:59:00 giving a new start - one minute before midnight? I'm assuming the intent of this is to make anything after 23:59:00 go into the next day. The other template based workarounds change the end of the interval, not the start. The python script workarounds that have been successful zero the entity at 23:59 |
|
I guess my suggestion would make anything from 23:59 - 00:00 not counted on any day. During that time the history stats range would be from |
I just wanted to say, I'm only one day in, but this appears to have fixed the issue of today showing the same max as where yesterday ended. This is the first time I actually had this graph work so I'm hopeful it was this easy. I agree with @fenty17 , would still be nice if the statistics just worked as intended so we didn't need any work arounds but this seems pretty painless to me.
|
Yes - with only 1 data point so far, this seems to have worked. The entity I tried the new start value with is showing the correct max stat for today. The old one shows yesterday’s max |
|
Hi everyone, I just want to report what is happening to me in case it helps to shed light on the issue. @karwosts wrote:
My issue is number 1. I am running HA 2024.10.4. Apologies is this has been fixed in a later release, but I cannot test them because of problems with another unrelated integration. I have a history stats sensor that is supposed to reset at 18:00 daily. The source for the history stats sensor is a binary sensor. If at the time of the reset the binary sensor is 'on' then the history stats sensor keeps on increasing until the binary sensor goes on-off again. It seems as if, instead of resetting at 18:00:00, the history sensor waits for the binary sensor to turn off to reset (at 18:00:17) but fails to stop the measure of time and keeps on measuring until the next 'off' event (at 19:30:36) of the binary sensor and then levels off. This is a graph that shows what is happening:
The definition of the history stats sensor is: I hope it helps. Javier EDIT: After changing the definition of the history stats sensor to: and restarting of HA, the history start sensor goes back to showing the correct duration: I do not know whether the correction happened because of the restart of HA, the change of the definition of the history stats sensor or both at the same time. |
|
This is still very much an issue. I've created new sensor to test on 2024.11.3
There is 2 minutes gap between sensor stopping recording and starting again - there should be no data in this period, but there is. |
This is not going to work the way you think. If you want the sensor to reset to 0 before midnight, you need to do what I suggested above: History stats only updates once per minute. Forcing the end to 23:59 does not achieve anything here, since when the sensor updates at 23:59:XX (or whatever arbitrary second), it will still be analyzing the period 00:01 - 23:59 of that day, and that will still have a nonzero value. So it will keep its nonzero value until the next update at ~00:00:XX, and statistics platform will see that and pick it up. |
|
Thanks @karwosts I will give it ago. |
This still won't work, no matter what minute you pick for end, because even if now is after the end period, history stats will still calculate a nonzero value until after the next day starts, because it's just looking at that start->end period. The intent of my suggestion was that once the time is 23:59 or later, the analyzed period becomes: Which is 0 time, and thus will have a result of 0. So then you're always starting the new day with a value of 0. |
@Kalma-House thanks for the details, I was able to reproduce your issue and submitted a PR for a fix in #133770 |
|
FWIW, I wasn't able to check my issue variant recently, but I did notice today that 2024.12.5 does not have the issue that I reported. |
and others
The problem
History_stats gives a wrong value if the sensor used as the base for the measurement is "on" over midnight and you want to measure a daily value.This problem has been reported before (#72357) but this issue was closed without action.
The following sensor is used to keep track of the daily usage of a water tap:
This speciIfic night switch.poolwater was open less than 5 minutes:
The history stat sensor should present 3m14s for July 27th and 22s for July 28th. But it continued to count all day ( today) even if the switch turned of 22s after midnight. In the morning it looked like this:
What version of Home Assistant Core has the issue?
2022.7.5
What was the last working version of Home Assistant Core?
No response
What type of installation are you running?
Home Assistant OS
Integration causing the issue
history_stats
Link to integration documentation on our website
No response
Diagnostics information
No response
Example YAML snippet
No response
Anything in the logs that might be useful for us?
No response
Additional information
Not sure if this problem occurs everytime the sensor is open over midnight. I've seen this problem before monitoring another sensor, but I thought it was corrected in a later release. See https://community.home-assistant.io/t/strange-history-statistics/433815 for additional details.
The text was updated successfully, but these errors were encountered: