Comparison table of various firmware already scored on the Dasharo page

[pietrushnic]

Code that can generate a comparison table of various firmware already scored on the Dasharo page

Looks like out of scope of what the utility does and is supposed to do. Sounds more like another utility to gather the data from docs.dasharo.com. Also does not seem to be related directly to focus on empty space and liberation from binary blobs (a separate issue for that would be better).

What exactly we would like to compare? Just the raw percentage of open/clsoed code ratio/liberation factor? Openness Score report contains much more data which does not necessarily needs to be compared, nor will give meaningful data. For now I don;t see what information could such comparison table provide, except openness/liberation percentage or raw closed source size change over the years of CPU evolution.

Another matter to solve is where to gather the data input for these comparisons. It should be something that is easily parsable and can be quickly imported to an array or something. Simplest would be CSV file...

Feeding it with the same platforms over and over does not increase the variety of results (e.g. feeding the utility with multiple releases of the same platform would just give neglectable differences).

Originally posted by @miczyg1 in #924 (comment)

[pietrushnic]

@miczyg1

What exactly we would like to compare? Just the raw percentage of open/clsoed code ratio/liberation factor? Openness Score report contains much more data which does not necessarily needs to be compared, nor will give meaningful data. For now I don;t see what information could such comparison table provide, except openness/liberation percentage or raw closed source size change over the years of CPU evolution.

Size of TCB in various market segments and versions of firmware. That may resonate with those who looking for balance between hardware cost and level of closed source they dealing with.

Another matter to solve is where to gather the data input for these comparisons. It should be something that is easily parsable and can be quickly imported to an array or something. Simplest would be CSV file...

It would be ideal, but do we have so much results that sorting would be needed? UX is important and paramount for such information. How we would achieve it with CSV?

Feeding it with the same platforms over and over does not increase the variety of results (e.g. feeding the utility with multiple releases of the same platform would just give neglectable differences).

Initial idea was to have it cross market segments. Do you envision that differently?

[miczyg1]

Size of TCB in various market segments and versions of firmware. That may resonate with those who looking for balance between hardware cost and level of closed source they dealing with.

So it basically means just closed source bytes vs open source bytes. The data is there, only need to put it in one place/table and show it. Make some ranking?

• sorting by total code size, the less the better
• sorting by closed-source size removed to total code size ratio, the more the better
• others?

It would be ideal, but do we have so much results that sorting would be needed? UX is important and paramount for such information. How we would achieve it with CSV?

CSV may be consumed by a different utility, oriented on the representation of data. Making a does-everything, good-for-nothing boilerplate from openness score source is not the way to go IMO.

Initial idea was to have it cross market segments. Do you envision that differently?

There is very little comparison of cross market. We don't have real servers for comparison. Embedded Intel segment is not that much different from client segment, in terms of FSP and microcode size, so differences in results would be neglectible. What we can surely observe is a cross microarchitectural closed source increase trend. Feeding the utility with scores made on platforms using different CPU/SOC microarchitecture will give interesting enough results.

[miczyg1]

Let's make some first steps towards that direction:

1. Have a switch in the utility to print output in a MD table entry/row form:

|Platform | Dasharo Firmware file | Proprietary Firmware file | closed-source diff [MB(%)] | data size diff [MB(%)] | empty space diff [MB(%)] |

The platform column would have to be filled manually, because the utility probably will not be able to determine it from the firmware file. or we can add another switch with platform name that will be used to print the Platform column. Firmware file will typically contain the version in the filename.

2. We can do a rundown for all binaries of interest to us and put them all in the table and commit it to docs.dasharo.com. We can add a bash script that will do this:

• download all binaries of interest
• run OpennessScore with the special switch
• save output to stdout or file

Very simplistic, but may be improved in the future. For now it should do its job. The convention would have to be as follows:

• each row should show a comparison of Dasharo firmware against its proprietary counterpart.
• invocation of OpennessScore would have to be done with two binaries (Dasharo and proprietary)

@pietrushnic let me know WDYT

[pietrushnic]

@miczyg1 sounds like a good plan. I think @artur-rs could look into it.

[PLangowski]

download all binaries of interest

Where should they be downloaded from?

[miczyg1]

download all binaries of interest

Where should they be downloaded from?

It depends. For Protectli all binaries can be sourced from: https://github.com/protectli-root/protectli-firmware-updater/tree/main/images (both Dasharo and AMI)

For MSI, simply check the vendor's board site (support section), e.g.: https://www.msi.com/Motherboard/PRO-Z690-A-DDR4/support (it has downloads for all AMI versions). Dasharo can be picked from our FTP: https://dl.3mdeb.com/open-source-firmware/Dasharo/msi_ms7d25/

ODROID:

• Dasharo: from MinIO only
• AMI: https://dn.odroid.com/ODROID-H4/bios/

For Novacustom:

• Insyde firmware: https://repo.palkeo.com/clevo-mirror/
• Dasharo: FTP https://dl.3mdeb.com/open-source-firmware/Dasharo/

I think that should be enough for now.

[PLangowski]

What exactly is closed-source diff [MB(%)]? Should it be calculated as the difference in percentages for both binaries or the difference in closed-source size (MB) converted to percent? If the latter, wouldn't the overall difference in binary sizes make it useless?

[miczyg1]

What exactly is closed-source diff [MB(%)]? Should it be calculated as the difference in percentages for both binaries or the difference in closed-source size (MB) converted to percent? If the latter, wouldn't the overall difference in binary sizes make it useless?

Binary sizes must be the same for a given platform. It is not possible to have different binary size for the same platform.

The Openness Score will give you the closed source size for Dasharo and proprietary firmware. So the percentage can be calculated as follows:

closed source diff % = (Dasharo closed source size - Proprietary closed source size) * 100 / Proprietary closed source size.

The assumption is that Proprietary firmware is always closed source (no open-source code). This metric should show hwo much more closed source is in proprietary firmware. This should always give negative percentage, because Dasharo should always have less closed source code than proprietary. Positive percentage will mean that Dasharo has more closed source than proprietary, which is very bad. So ideal value would be -100% (fully liberated, currently impossible).

Analogically empty space and data space:

empty space diff % = (Dasharo empty size - Proprietary empty size) * 100 / Proprietary empty size.

Here the metrics should show a positive percentage. Dasharo should have much less code than proprietary in general, so the more percent on the positive side the better. Negative percentage can mean that there is too much code in Dasharo (although there might be an exception where vboot is used, then we may have up to 3 full copies of the firmware, thus the switch to ignore vboot has been propsed in another issue).

data size diff % = (Dasharo data size - Proprietary data size) * 100 / Proprietary data size.

Here the percentage may vary. Probably negative percentage is good. If Dasharo uses less space for data than proprietary, it generally means it is better. But on the other hand, we have no extensive detection of data in proprietary images, so the statistics for Proprietary data size may be not accurate (there might be more data than detected). So if the percentage is positive, it is not a big deal.

[PLangowski]

Thank you, this helps a lot

[pietrushnic]

What exactly is closed-source diff [MB(%)]? Should it be calculated as the difference in percentages for both binaries or the difference in closed-source size (MB) converted to percent? If the latter, wouldn't the overall difference in binary sizes make it useless?

Binary sizes must be the same for a given platform. It is not possible to have different binary size for the same platform.

The Openness Score will give you the closed source size for Dasharo and proprietary firmware. So the percentage can be calculated as follows:

closed source diff % = (Dasharo closed source size - Proprietary closed source size) * 100 / Proprietary closed source size.

The assumption is that Proprietary firmware is always closed source (no open-source code). This metric should show hwo much more closed source is in proprietary firmware. This should always give negative percentage, because Dasharo should always have less closed source code than proprietary. Positive percentage will mean that Dasharo has more closed source than proprietary, which is very bad. So ideal value would be -100% (fully liberated, currently impossible).

Analogically empty space and data space:

empty space diff % = (Dasharo empty size - Proprietary empty size) * 100 / Proprietary empty size.

Here the metrics should show a positive percentage. Dasharo should have much less code than proprietary in general, so the more percent on the positive side the better. Negative percentage can mean that there is too much code in Dasharo (although there might be an exception where vboot is used, then we may have up to 3 full copies of the firmware, thus the switch to ignore vboot has been propsed in another issue).

data size diff % = (Dasharo data size - Proprietary data size) * 100 / Proprietary data size.

Here the percentage may vary. Probably negative percentage is good. If Dasharo uses less space for data than proprietary, it generally means it is better. But on the other hand, we have no extensive detection of data in proprietary images, so the statistics for Proprietary data size may be not accurate (there might be more data than detected). So if the percentage is positive, it is not a big deal.

I'm not sure if we have a dedicated task for documentation, but the above description is perfect. It could be improved by more marketing language, but it also describes my intention behind v0.2.

@PLangowski, please imagine a situation where we would like to write a social media post with one or maybe two graphics. Those graphics should clearly show the difference Dasharo introduces initially, and possibly later with subsequent versions. It should be very clear from the graphics:

• What property are we comparing?
• What platform are we referring to?
• What version of proprietary firmware and open-source firmware are we comparing?
• What is the result?

[pietrushnic]

@miczyg1 I'm looking at example @PLangowski sent me:

Proprietary:

* Image size: 33554432 (0x2000000)
* Number of entries: 2948
* Number of regions: 3
* Number of volumes: 8
* Total open-source files size: 0 (0x0)
* Total closed-source files size: 12526800 (0xbf24d0)
* Total data size: 13213712 (0xc9a010)
* Total empty size: 7813920 (0x773b20)

Dasharo:

* Image size: 33554432 (0x2000000)
* Number of regions: 37
* Number of CBFSes: 4
* Total open-source code size: 6370948 (0x613684)
* Total closed-source code size: 9233747 (0x8ce553)
* Total data size: 948981 (0xe7af5)
* Total empty size: 17000756 (0x1036934)

Is FSP so big, 9MB? It is a lot. Also, it's peculiar that to run open source, you need a total of 15MB of code, whereas for proprietary software, you only need 12MB. I would like to have better understanding of those things to create new issues for optimizations of those sizes I think this should be clear selling point, but it looks some minor decisions may stand on the road.

[PLangowski]

@miczyg1 https://dn.odroid.com/ODROID-H4/bios/ seems to be blocking downloads from the terminal. We could host those files on our FTP, wdyt?

Also, https://repo.palkeo.com/clevo-mirror/ requires credentials

[miczyg1]

@miczyg1 https://dn.odroid.com/ODROID-H4/bios/ seems to be blocking downloads from the terminal. We could host those files on our FTP, wdyt?

What a pity.

I didn't say it will be good for scripts.