64 bit recommendations discussion

[mstormi]

Hi @ecdye

Given the announcement to move to all-64 in OH5, I'm struggling to believe if our current handling of that is the right way or too aggressive for now. I have become sceptic about 64 bit.

There's at least one hard-to-track problem I experienced myself, too, that I believe to be related to the move to 64bit OS on Raspi, plus a number of signs. Java takes 4+ GB, saw even 5.5+ on some user boxes.
On mine, rule engine refused to run although the bundle says it's active.
I wouldn't be surprised if there's some issue looming and so far going unspotted that at times OH/Java cannot allocate all the memory in wants in a situation.

For now I'd recommend to throttle back.

• build another image that raises the HW minimum for the 64 bit version to 4G instead of 2G
• revoke openHABian documentation to recommend 64 bit image for now
• revoke or halt OH recommendation (4.3 release notes etc.) to install 64 ?
• others short term action?

[ecdye]

I'm not going to lie, I have not ever seen any significant issues with my decently large setup. I have been running 64bit on my system for 6 months now with no issue. I would note that it was really an experiment because I even have the rare 1GB model of the RPi 4. I consistently see about 75% usage of memory with zram enabled.

I know there are some bugs, but some are unrelated to 64bit directly and just need to be ironed out. I don't know that it's worth hiking the requirements so much if there is a pretty good chance for success with a setup like mine which is pretty much worst case scenario.

[mstormi]

Well. With my main setup, me neither.
But I myself am seeing problems that I have been unable to fix on some systems of customers of mine.
Several other OH users are seeing this. I'm not sure how to handle it or take action, but I'm not optimistic it'll sort out soon let alone that it's guaranteed to. Given 64bit has no direct advantages, I'm more on the "caution" side of the river.
See also openhab/openhab-core#4547

[ecdye]

I would suggest for the moment we continue as planned and monitor the issues, ideally these should get fixed before the OH 5 release and I should hope that there are not any major issues that would require a large spec bump to our baseline.

I do see your concerns and share some of the same reservations, but I don't want to make a big change when most people don't have a ton of issues, that makes me think it's more on the minor bug side that should be reported, investigated, and solved.

[mstormi]

While we might identify the remaining issues, what shocked me most about the issue referred to above is that on ALL systems
I had myself or heard from others to have or have had the problem, they have HUGE Java processes like 4 or more GB on a Raspi, and even 8GB on x86.

Take a second to think what that means to paging activity harming SD, or performance in general.
That's my main motivation to rethink our "go 64" recommendation.

[ecdye]

But thats just because of a bug, right? It didn't used to be that way IIRC

[mstormi]

As you know I'm providing a commercial EM system on top of openHAB(ian) so I have several systems rolled out that share the same config. Some I had to go back to 4.2.3 because of rules not running bug but meanwhile they all work.
However, I'm still seeing this huge mem usage of 4+G on every of them, including the one I updated to 4.3.2 some hours ago.
AFAIK there's no known or filed bug related to mem usage.

I believe that's just what you get on every (full-)64 bit system.
And I don't think that's a good thing in the end as 64 has no real advantages but drawbacks quite some, like increased paging to SD.
Now while this

[ecdye]

Yeah, I'm not sure what would cause that. I've never seen that happen on a machine, have you checked the Java memory settings to see if you can tweak those?

[mstormi]

You have never seen that large mem usage with recent 64 bit openHABian on RPi?
How many 64bit systems have you seen?
I'm using standard openHABian Java opts i.e. -Xms192m -Xmx768m.
(Xms is about the only one I'd think to affect mem usage. Without this, it would be even more)

[ecdye]

I've used upwards of 7-10 openHABian instances on 64bit for varying lengths of time.
In my use I've never seen large usage like that. I typically see around 700 MiB of memory usage and everything runs like normal.
This even applies to the 1 GB system I have which uses the lower defaults.

[mstormi]

Measured how?

Here's a typical line from top. 4.6G process size, 422MB resident, on a 1GB RPi3

658 openhab   20   0 4674408 422580   7732 S  50,0  43,1     19,26 java

Do you know of a means to determine which bundle/binding is using how much space? Maybe that's what makes the difference?

If you have a spare RPi, could you try installing my image from https://storm.house/download/storm.house-neu-202501061550.img.xz and check with that?

EDIT: I noticed java draws the full amount of mem right on start (when it doesn't know yet which bundles it is gonna start) so I don't believe that's the reason.

[ecdye]

OK I thought we were talking about Resident memory as that is what reflects memory usage on the host system. For virtual size mine is also ~4 GB but I don't notice any real impact as the system manages it pretty well.

1110699 openhab   20   0 4348580 218088   7328 S   0.0  22.3 275:00.39 java

As much as I wish I did, I don't know any magic special command to show that. I suspect you might be able to figure some statistics out by building a whole system and running it in debug mode, but that would probably also not accurately reflect a production system.

I don't currently have a RPi available but I'd be happy to test next time I have one open.

[mstormi]

ok so we're at least on the same page now.
Yes it's not necessarily a problem, I have customers running 64 image on 1 GB box.
Any virtual mem page not resident has to be paged out to SD at least once after java start however. Should ultimately be true even if zram is inbetween.
Paging now harms SD and increases startup times so should be minimized.

As my edit said allocation is right on java start so unrelated to what bindings are there. So it wont make sense analysing their mem use, right? Guess you don't need to mock with my images.
Rough-edged idea: as a cross check setup a barebones config to see how much that uses. If I was to guess it mmaps the kar file. Maybe check without? There is an option to download bindings on demand instead of using kar. Eventually uninstall the addons pkg first?
Then file this as a bug to openhab-core? Ask for a karaf command or other tool to find out what causes that allocation maybe some core dev knows.

[ecdye]

Sounds like a good approach to seeing if we can identify a root cause.

I don't currently have a ton of time to dedicate to testing, so if you want to take the lead on this, that would be great.

[mstormi]

I don't have much time either at the moment, but I think it's worth to dedicate more attention to this.

Just read on the forum that someone with a huge OH startup time (14 mins) moved from openHABian to DietPi and that reduced his startup to 1 min.
There may be more to it such as a change in OH version but it's a strong sign we should look after this

[mstormi]

BTW do we really need a 64bit OS to run a 64bit Java?
Do we really need a 64 bit Java to run openHAB5?

Isn't Java still Java, so eventually any 64bit variable would be 'cut' into 2 x 32 bit but in the end, the same java code would still work?
Note that "64bit via software" would be sort of okay given we don't do many long int or double float computations in openHAB so
that would not be a drama to be a little inefficient at times.

Have you ever tried a different Java 17 package than OpenJDK?

[ecdye]

I've used a couple of different JDKs but never really paid attention to performance. That would be interesting to do.

There are some libraries that openHAB uses both internally and for addons that have switched to only 64bit support for newer versions. In general your theory is correct but it's becoming less true as Java shifts to more specialized support (or at least developers do anyways).