H3 support

[loblik]

Type of issue

Question

H3 support

What steps are needed to have crust running on H3 based board?

As far as I uderstand it, currently only platforms with ATF are supported. Should u-boot be bringing up the SCP core from reset on platforms without ATF?

[smaeul]

Other than using a different CPU core, the H3 is practically the same SoC as the H5, which is already supported. So on the crust side, all that is needed is 1) adding a new platform, 2) hooking up existing drivers (everything but the CSS driver), and 3) adjusting the CSS driver for a Cortex-A7 (i.e. don't set AA64nAA32 in the resume path). I have pushed a WIP branch doing steps 1 and 2. That branch boots on my OPi One and I can make it successfully do power state transitions by poking system_state with U-Boot's mw.b command.

Even with only CPU core on/off control (no cluster reset or DRAM suspend), this is equivalent to what the U-Boot PSCI implementation provides today. If the existing A64/H5 cluster/DRAM suspend code does work out of the box, that's a nice bonus. If not, full suspend power savings will require a few days of figuring out what bits to twiddle from the BSP reference.

As you suggest, the hard part is actually the changes needed in U-Boot. It needs:

1. Configuration for SPL to load a FIT image containing both U-Boot and crust instead of a legacy image.
2. Code to program the reset vector and enable the AR100. Ideally, this means checking the DTB for SPL saying "I loaded scp.bin to $ADDRESS" (requires !SPL_FIT_TINY), and then verifying that $ADDRESS contains $MAGIC (see abi.md), and if so, starting the AR100.
3. A tiny mailbox driver so U-Boot can send SCPI commands to crust. This is not a normal DM_MAILBOX driver, because it goes in the secure monitor, not in U-Boot proper. Per abi.md, crust does all of the setup, so this can basically be "spin reading the status register" like the driver in TF-A.
4. A PSCI implementation that implements its methods by making SCPI calls. This isn't necessarily sunxi-specific, but it would be unless the mailbox is sufficiently abstracted out.
5. Also, somehow U-Boot needs to know where the SCPI shared memory area is. That can be hardcoded like in TF-A, or we can try to shove it in the DTB somewhere. Any improvements made here can be merged into TF-A as well.

Steps 1 and 2 can be emulated by a U-Boot script doing load and mw.l, so they don't really block steps 3 and 4.

Note that it's not really possible to work on the crust CSS/DRAM driver until we've replaced the U-Boot PSCI implementation (steps 3 and 4).

[smaeul]

Practically, there are a couple of other pieces needed for H3 board support, beyond H3 SoC support:

1. A GPIO regulator driver, so you can turn off PWR-STB and PWR-DRAM during shutdown.
2. A GPIO button driver, so you can turn the board back on after shutdown (gpio-keys only configures buttons for wakeup during suspend, not at shutdown).

Both of these are equally as useful for H5 boards, which use a similar design.

[smaeul]

WIP U-Boot support is here: https://github.com/smaeul/u-boot/commits/h3

[loblik]

Ok, now I feel confused a bit. Initially I thought H3 doesn't have TrustZone/ATF and it will be just Linux talking to AR100 using HW msgbox.

But you mentioned PSCI which normaly results in secure world switch I guess. Also not sure why all this is PSCI/SCPI needed in U-Boot as its not running after execution is handed over to Linux.

I probably need to read more on this stuff.

[smaeul]

U-Boot actually is still running. It provides a secure monitor containing a PSCI implementation, similar to what ATF does (although much more primitive). See this patch series which originally added it: https://lists.denx.de/pipermail/u-boot/2014-July/183610.html

Linux is not allowed (per the SCPI spec) to call the SCPI commands to turn CPUs on/off and suspend the system. Only the secure monitor can do that, so it has the opportunity to notify a trusted OS of the change (and possibly migrate it to a different CPU core). This applies even though we don't actually have a trusted OS running.

Because the SCPI shared memory area is in SRAM A2, that prohibition is actually enforced at the hardware level. SRAM A2 is secure-only, so Linux, running in NS SVC mode, cannot access it at all.

[loblik]

Thanks for the effort and clarification. I'm currently quite busy with my regular work, but I definetely want to look into this and try it on one of my H3 boards once I have some spare time.

[smaeul]

I just pushed initial H3 support to master. It is not complete, and it is not quite stable either, and it still requires the hacked-up U-Boot branch linked above. But it boots, SMP works, and suspend usually works.

[smaeul]

As of commit f690a19, I believe the Crust side of H3 support is feature-complete and ready for wider testing.

The U-Boot support is functional but still WIP for performance/reliability. I pushed a new branch, which is much closer to being upstreamable: https://github.com/smaeul/u-boot/commits/patch/h3-scp

I'll leave this issue open until U-Boot support is merged, so people can find the branch.

(ping @jernejsk, who has been using the H3 support already)

[gamelaster]

Hi @smaeul , thank you for your work. I have here available a few H3 devices, so I will test crust on them and get back to you about the results. 😊

[jernejsk]

Sure, I'll test this. I see that U-Boot commits are not applied on top of stable. Do I need to cherry pick any other patch beside yours to make it work on top of 2021.10?

[jernejsk]

Initial test on OrangePi Plus2E was successful, as it also was quick regression test on Tanix TX6 (H6). I'll test this on other H3 boards that I have and then roll out LibreELEC 11 PR, for wider testing. Although, to be honest, I didn't get any complaint about suspend/resume for a long time, even for your initial (previous, partial) H3 support.

[jernejsk]

H3 support nicely works on all my board that I currently have (OrangePi Plus2E, Beelink X2, BananaPi M2+). However, U-Boot commit [DO NOT MERGE] sunxi: psci: Delegate PSCI to SCPI breaks booting on R40, although it has #ifdefs for it.

[gamelaster]

Sorry, I just get to the testing of Crust on H3 only now.
I'm trying to use Crust with latest crust u-boot in patch/h3-scp branch, for some reason, this u-boot doesn't wants to boot my kernel (just stuck on Starting kernel...), where stock U-Boot 2022.01 boots my kernel just fine. Will try to investigate where is issue.

(I didn't had properly enabled SCP in u-boot configuration) So far, Crust works amazing on BPI M2+

[emrali361]

hello everyone
I want to make crust for OrangePi pc (H3) but it took some errors which i can't understand about this.

` $ make

CPP build/scp/arch/or1k/scp.ld.o
CC build/scp/arch/or1k/counter.o
arm-orangepc-linux-gnueabihf-gcc: error: unrecognized command-line option ‘-msfimm’
arm-orangepc-linux-gnueabihf-gcc: error: unrecognized command-line option ‘-mshftimm’
arm-orangepc-linux-gnueabihf-gcc: error: unrecognized command-line option ‘-msoft-div’
arm-orangepc-linux-gnueabihf-gcc: error: unrecognized command-line option ‘-msoft-mul’
make: *** [Makefile:205: build/scp/arch/or1k/counter.o] Error 1
`

please help me

[gamelaster]

@emrali361 you're using wrong compiler, you need or1k compiler.

[emrali361]

@gamelaster Thank you. I generated that cross compiler by crosstool-ng , therefore, I made u-boot and kernel with the cross compiler. right now, I need to bl31.bin for orangepi pc. I can't understand about the or1k compiler and don't know where can get or how to generate this cross compiler.

[gamelaster]

@emrali361 you can download it from your package manager, or compile it by yourself