Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace git-clone with HTTP get of an archive #238

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

pstef
Copy link

@pstef pstef commented Oct 24, 2024

git-clone fetches the entire history of a repository. There is a GIT_SHALLOW option in CMake's ExternalProject, but reportedly it can only be used to get "named" commits (tags, branches).

ExternalProject provides an alternative way of fetching this, via URL combined with github's ability to generate a zip archive of any arbitrary commit hash.

DOWNLOAD_EXTRACT_TIMESTAMP must be set to avoid warnings from CMake. Set to True, the timestamps of the extracted files will match those in the archive.

I didn't add URL_HASH support mainly because this is more or less a PoC PR, but if you choose to go this route, it should be added at some point. For now I can provide md5 and sha256 hashes for your convenience.

$ md5sum */*/*/???????.zip
80c9724dbcf1fb52adac8e64ed02eef2  date-subbuild/date-populate-prefix/src/1ead671.zip
0d024cd5799e352ba3c67c3c38703b81  embed-binaries-subbuild/embed-binaries-populate-prefix/src/21f28ca.zip
0b0a558d95674da3313cf9faf62c186f  glm-subbuild/glm-populate-prefix/src/33b4a62.zip
da1af7599ff1c5b50ded237cbdf28870  libretro-common-subbuild/libretro-common-populate-prefix/src/8e2b884.zip
e8362bd91b0bbe85f139a569858231a8  libslirp-subbuild/libslirp-populate-prefix/src/e61dbd4.zip
0694331d14a5a828c59d7791ede785bc  melonds-subbuild/melonds-populate-prefix/src/824eb37.zip
8a2e1902508baa390e30d933c010e96a  pntr-subbuild/pntr-populate-prefix/src/650237a.zip
48d0528cbe3b42a9c8418aa2f1e7e97c  span-lite-subbuild/span-lite-populate-prefix/src/00afc28.zip
1609deb873eae02bfa636a929347ac69  yamc-subbuild/yamc-populate-prefix/src/4e015a7.zip
$ sha256sum */*/*/???????.zip
5523d5478e495911fa651687aff38c298c4f9e2e54cc99f160c007d1e94ff61f  date-subbuild/date-populate-prefix/src/1ead671.zip
ac0eed4126ce7f11b47b8777e848724212fa66f2aaf2d9a94b0c1022b4d86198  embed-binaries-subbuild/embed-binaries-populate-prefix/src/21f28ca.zip
79263e9526124b5b57c943f53e5e55b3e635aa9037f2a6326cd3193bba9816ec  glm-subbuild/glm-populate-prefix/src/33b4a62.zip
9e51b77da70d89cb86cef4e4a02d1344aa654a960b919965bd0c48f6ce96a46d  libretro-common-subbuild/libretro-common-populate-prefix/src/8e2b884.zip
82b7432192365d31b37bca9ae806e32bd62445f4ea17ccdc65162d1aad2d1d55  libslirp-subbuild/libslirp-populate-prefix/src/e61dbd4.zip
e5c1957a526234cf2acbfa7a65b6bd1fbb8bcf7d3f52cedfff3db0e9832759a9  melonds-subbuild/melonds-populate-prefix/src/824eb37.zip
506f1a9bfd8b0ea87ab8f9866ef307481f244f290ebcea27e694c0f4cc4f92c4  pntr-subbuild/pntr-populate-prefix/src/650237a.zip
29834cb5a0e988a73ab2331cdb3e7e0a939246af48392d472e24f791279bd2d0  span-lite-subbuild/span-lite-populate-prefix/src/00afc28.zip
7393a7454493c0acae06b7425c255bd75976b1790ca8be2e32e1bb43cadeef30  yamc-subbuild/yamc-populate-prefix/src/4e015a7.zip

git-clone fetches the entire history of a repository. There is a GIT_SHALLOW
option in CMake's ExternalProject, but reportedly it can only be used to get
"named" commits (tags, branches).

ExternalProject provides an alternative way of fetching this, via URL combined
with github's ability to generate a zip archive of any arbitrary commit hash.

DOWNLOAD_EXTRACT_TIMESTAMP must be set to avoid warnings from CMake. Set to
True, the timestamps of the extracted files will match those in the archive.

I didn't add URL_HASH support mainly because this is more or less a PoC PR,
but if you choose to go this route, it should be added at some point. For now
I can provide md5 and sha256 hashes for your convenience.
@pstef
Copy link
Author

pstef commented Oct 24, 2024

A side effect of this change is that you can no longer update repositories with git pull.

Copy link
Owner

@JesseTG JesseTG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for contributing! I do have some thoughts about this PR.

  • Please ensure that this PR targets dev, not main. You should be able to make that change easily, although I don't remember how off-hand.
  • I understand your desire to keep bandwidth usage down, but I don't want to risk disrupting the build process for a minority use case. Given that, would you mind making this opt-in with a cache variable? (Remember to prefix it with MELONDSDS_.)
    • If you go this route, I'd appreciate it if you could allow the whole URL to be customized; hardcoding /archive/${${VAR_NAME}_REPOSITORY_TAG}.zip into the URL presumes that dependency repos will always be hosted on GitHub, which I'd rather not do. However, using it as a common-case default is fine.

@pstef pstef changed the base branch from main to dev October 24, 2024 19:55
@pstef pstef changed the base branch from dev to main October 24, 2024 19:57
@pstef pstef marked this pull request as draft October 24, 2024 19:57
@pstef
Copy link
Author

pstef commented Oct 24, 2024

I'm converting this to draft because this was more of a PoC - kind of an upgraded issue. Please take it from here wherever you want.

@JesseTG
Copy link
Owner

JesseTG commented Oct 24, 2024

I'm converting this to draft because this was more of a PoC - kind of an upgraded issue. Please take it from here wherever you want.

No problem. When you're ready for me to take another look, unmark it as a draft.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants