Skip to content

Add sigs data and scripts from asmap/asmap.sigs repo#50

Merged
fjahr merged 6 commits into
bitcoin-core:mainfrom
jurraca:add-sigs-data
Jun 7, 2026
Merged

Add sigs data and scripts from asmap/asmap.sigs repo#50
fjahr merged 6 commits into
bitcoin-core:mainfrom
jurraca:add-sigs-data

Conversation

@jurraca

@jurraca jurraca commented May 19, 2026

Copy link
Copy Markdown
Collaborator

After consideration, we decided to move all attestation data close to the data itself, in this repository. This moves all attestations scripts and data (from run 1772726400) from https://github.com/asmap/asmap.sigs.

First 3 commits are direct copying of data. The last one f30ef69 adapts the asmap-attest and asmap-verify to the repo's structure, i.e. finding paths to write when attesting, and finding the right epochs and signature data when verifying attestations.

An existing year folder for asmap data now gets an attestations folder containing folders indexed by run timestamp e.g.:

2026
├── 1767888000_asmap.dat
├── 1767888000_asmap_unfilled.dat
├── 1772726400_asmap.dat
├── 1772726400_asmap_unfilled.dat
└── attestations
    └── 1772726400
        ├── fjahr
        │   ├── SHA256SUMS
        │   └── SHA256SUMS.asc
        ├── jurraca
        │   ├── SHA256SUMS
        │   └── SHA256SUMS.asc

While I considered changing the repo's structure to have {year}/{timestamp}/attestations and {year}/{timestamp}/data and think it would be cleaner, it would break the existing repo structure. i.e.:

2026
└── 1772726400
   └── data
      └── ...  
   └── attestations
      └── ...

If a user wanted to get a file and its attestations in one go, they could fetch the timestamp from the file name and find the attestation folder under attestations/{timestamp} which doesn't seem to burdensome.

This adds a year_from_epoch helper, which requires adding the date tool to required tools, and handle it differently for GNU vs BSD/Mac (though this was an LLM-provided suggestion, and I haven't tested on Mac). Otherwise scripts functionality is unchanged.

@fanquake

Copy link
Copy Markdown
Member

Concept ACK

@fjahr

fjahr commented May 26, 2026

Copy link
Copy Markdown
Collaborator

Concept ACK

I have just tested the verify script and it worked for me without issue.

I think I would have chosen a different folder structure. I would have done from top-level: attestations/year/timestamp/, so that the attestations are in the same repo but not right next to the maps. I prefer that because people looking to download maps casually may be confused by the attestations and the attestations are mostly interacted with through the script. But others may see it differently, just a nit to consider.

What is still missing is moving the documentation for the sigs stuff here. That may be in the readme or in a separate attestations doc. I don't have a preference for this.

@fjahr fjahr mentioned this pull request May 26, 2026
jurraca added 4 commits May 27, 2026 15:36
the attestations folder is nested under each year, so we update the
script paths to handle that.
add a helper function to identify the year from the unix epoch.
rename env vars to not mention old repo
@jurraca

jurraca commented May 27, 2026

Copy link
Copy Markdown
Collaborator Author

I would have done from top-level:

yea, that's cleaner, done. 799175f

moving the documentation for the sigs stuff here

done in d7a7e08

I also removed the part of the CLI help text that showed an example with NO_SIGN, which made the help text larger than a single terminal view, and imo is kind of unnecessary. The option is still indicated in the help text. 5a80d79

@fjahr

fjahr commented May 30, 2026

Copy link
Copy Markdown
Collaborator

tACK d7a7e08

Changes look good to me. I re-verified everything and created my own attestation for the latest run with the scripts included here: #52

@naiyoma

naiyoma commented Jun 4, 2026

Copy link
Copy Markdown

Concept ACK

@hodlinator hodlinator left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK d7a7e08

Compared with asmap.sigs at 5f23416f19bb955a3b76d5d2b3035e28ebfc6af0.

Both of these show "Folders have no differences":

meld ~/asmap-data/attestations/2026/1772726400 ~/asmap.sigs/1772726400
meld ~/asmap.sigs/builder-keys  ~/asmap-data/builder-keys

Confirmed that all files in asmap.sigs are now incorporated in asmap-data so that the former can be removed after this PR is merged.

Confirmed mine, sr-gi's and luisschwab's builder keys and attestations have not been tampered with since they were first merged as I was active around that time on the asmap.sigs repo.

The commit messages are very lowercase. :)
Changes to the scripts and README.md make sense given the prior discussion in the PR.
I would have leaned toward parallel data/ + attestations/ but totally fine with this too.

Comment thread asmap-attest

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nanonit in 5a80d79:
If you need to re-touch - would expand the commit description to include the justification for removal of the NO_SIGN=1 example (fitting it on a typical terminal screen).

Comment thread asmap-attest
# Required env vars should be non-empty
################

if [ -z "$SIGNER" ] || [ -z "$ASMAP_TXT" ] || [ -z "$ENCODED_FILLED" ] || [ -z "$ENCODED_UNFILLED" ] || [ -z "$EPOCH" ]; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've now also tested asmap-attest & asmap-verify.

Remarks for another PR:

  • Would also be nice if one could set EPOCH and get ASMAP_TXT+ENCODED_FILLED+ENCODED_UNFILLED calculated to default locations.

  • It's annoying that SIGNER is required even for NO_SIGN=1.

  • The output from asmap-verify comparing a file against itself is pointless:

Files /home/hodlinator/asmap-data/attestations/2026/1772726400/Hodlinator/SHA256SUMS and /home/hodlinator/asmap-data/attestations/2026/1772726400/Hodlinator/SHA256SUMS are identical

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's annoying that SIGNER is required even for NO_SIGN=1.

the reason it's like that is that we always want to know a valid signer is usable from the builder-keys. The NO_SIGN=1 just means "don't actually do the signing". It's a dry-run, but we do need to have a Signer set for it to be meaningful.

Would also be nice if one could set EPOCH and get ASMAP_TXT+ENCODED_FILLED+ENCODED_UNFILLED calculated to default locations.

yes it would, but I'm not sure what to provide as a useful default. The epoch is used in the asmap-data directory structure. We could find the final_result.txt location if we know the user's kartograf dir, but its not guaranteed the user put the encoded output files in that same dir (since the encoding script runs from bitcoin/contrib/asmap). It would be a bit of a blind guess imo. If you have any suggestions for defaults I'd take them.

The output from asmap-verify comparing a file against itself is pointless:

yes, actually that output is useless in general, i'll have only return differences

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree running without a SIGNER is of debatable value. Spawned #54, lets continue there as far as asmap-attest goes.

@fjahr

fjahr commented Jun 7, 2026

Copy link
Copy Markdown
Collaborator

Thanks @hodlinator for the review! I will go ahead and merge this. We don't have the next release coming up for another few months, the next asmap that will be embedded for the major release will be in September (maybe v31.1 is a bit earlier). So the sig tooling here should be used a few times before that critical point, and it's currently holding up the whole process of adding sigs for the last map and the map from this week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants