sasquatch: 4.4 -> 4.5.1-1 #219169

vlaci · 2023-03-02T13:13:34Z

Required for #217836

Description of changes

Changed using ONEKEY fork of sasquatch which is up-to-date with squashfs-tools 4.5.1 and the Darwin compatibility patch shipped in Nixpkgs applies cleanly as well.

I have chosen to override the squashfsTools derivation, as this fork is entirely compatible with that version. I am open to critique on this. I wasn't sure copying from squashfstools or lifting a generic.nix base derivation out of it is the better solution. Existing NixOS test is also nice to have as a validation, although it may be overkill here.

Things done

vlaci · 2023-03-03T09:25:28Z

Ah, figured keeping the existing test doesn't make sense as it will just use squashfs-tools anyway.

AkechiShiro · 2023-05-20T19:40:05Z

Any news, any blockage ? Or any review from @Pamplemousse if he is still the right contact for this PR ?

vlaci · 2023-05-22T14:49:03Z

Some questions got answerred: I shouldn't depend on squashFsTools at all, as it got updated to 4.6.1, and it is unlikely I want to rebase sasquatch the near future.

vlaci · 2023-06-19T14:26:59Z

As squashfsTools derivation is now incompatible with previous versions, I have recreated the derivation from scratch.

Pamplemousse · 2023-06-19T15:24:36Z

@AkechiShiro Sorry I missed your message earlier.

I have stopped using this package in years... I am not well placed to test.
Also, could you remove me from the maintainers please?

Asked for their removal in NixOS#219169 (comment)

vlaci · 2023-06-19T16:07:24Z

@AkechiShiro Sorry I missed your message earlier.

I have stopped using this package in years... I am not well placed to test. Also, could you remove me from the maintainers please?

Done

qkaiser · 2023-12-24T10:32:34Z

This PR has been stuck for the last 6 months waiting for a review from someone who explicitly requested to be removed from the maintainers list.

Who should review this then ? How can we make it move forward ?

Asking cause it's the last item missing in out meta-issue to get unblob in nixpkgs #217836

AkechiShiro · 2023-12-27T10:24:17Z

This PR needs a rebase, first of all on master @vlaci could you please rebase?

Asked for their removal in NixOS#219169 (comment)

vlaci · 2024-01-01T22:43:47Z

Happy new year everyone :)
Rebased on master.

AkechiShiro · 2024-01-02T20:42:41Z

Thanks @vlaci, there seems to be a typo in the meta attribute of the package, please fix it, so that the ofborg evaluation test passes, see more details : https://gist.github.com/GrahamcOfBorg/6d11645476eaaef9b7fc87c573d4a188, meta.mainprogram should be meta.mainProgram.

I cannot help further as I'm inexperienced in reviews @qkaiser, maybe mentioning the pull request as ready for review here, should bring this PR to more experienced reviewers : https://discourse.nixos.org/t/prs-ready-for-review and hopefully help.

Asked for their removal in NixOS#219169 (comment)

qkaiser · 2024-01-03T07:25:49Z

@AkechiShiro thanks for the heads up, I sent a reply to the Discourse thread asking for a review on this PR

nixos-discourse · 2024-01-03T13:57:40Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/1016/87

tnias

I ran nixpkgs-review and sucessfully used both x86_64 and aarch64 to extract an example squashfs I found somewhere on my disk.
For me the diffoscope tests are broken on master on x86_64 even without this patch.

Result of nixpkgs-review pr 219169 run on x86_64-linux 1

3 packages failed to build:

diffoscope
diffoscope.dist
diffoscope.man

10 packages built:

binwalk (python311Packages.binwalk)
binwalk.dist (python311Packages.binwalk.dist)
python310Packages.binwalk
python310Packages.binwalk-full
python310Packages.binwalk-full.dist
python310Packages.binwalk.dist
python311Packages.binwalk-full
python311Packages.binwalk-full.dist
sasquatch
sasquatch-v4be

Result of nixpkgs-review pr 219169 run on aarch64-linux 1

13 packages built:

binwalk (python311Packages.binwalk)
binwalk.dist (python311Packages.binwalk.dist)
diffoscope
diffoscope.dist
diffoscope.man
python310Packages.binwalk
python310Packages.binwalk-full
python310Packages.binwalk-full.dist
python310Packages.binwalk.dist
python311Packages.binwalk-full
python311Packages.binwalk-full.dist
sasquatch
sasquatch-v4be

Mic92 · 2024-03-06T23:54:22Z

What does the bigendian variant do? Do we need both?

tilpner · 2024-03-06T23:59:42Z

pkgs/tools/filesystems/sasquatch/default.nix

-  pname = "sasquatch";
-  version = "4.4";
+  drv = stdenv.mkDerivation rec {
+    pname = "sasquatch${lib.optionalString bigEndian "-v4be"}";


This repeats the version number (v4), that's already appended to pname, so the full name would be sasquatch-v4be-4.5.1-4. Perhaps it should just be sasquatch-be?

The v4be indicates the squashfs filesystem version while the 4.5.1-4 indicates the squashfs-tools version sasquatch is based on. These versions are unrelated.

qkaiser · 2024-03-07T09:24:50Z

What does the bigendian variant do? Do we need both?

sasquatch is built on top of squashfs-tools to support non standard squashfs filesystems. By the standard, squashfs version 4.0 filesystems are fixed little-endian. There are no big-endian 4.0 filesystems. However, embedded vendors love to mess with standards. So instead of adapting squashfs-tools code, we build a version with __BYTE_ORDER swapped which is the sasquatch-v4be binary. This binary can handle non-standard squashfs v4 big-endian filesystems.

More information about the big-endian variant can be found at onekey-sec/sasquatch@0c65f00

Here's a small preview of what vendors do with squashfs. Cells with a star (*) are standard.

type	v2	v3	v4	DDWRT v3	Broadcom v3	??? v3	AVM	Netgear
magic (LE)	hsqs*	hsqs*	hsqs*	hsqt	shsq	zlqs	Header in BE, but chunks in LE.	Let‘s use a non standard LZMA + XZ compression !
magic (BE)	sqsh*	sqsh*	sqsh	tqsh	qshs	sqlz	/	/

smancill · 2024-08-19T03:30:17Z

What's the status of this? Can it be merged?

AkechiShiro · 2024-09-01T17:51:25Z

Hello @smancill I think there was a review that approved the package, I guess running nixpkgs-review on this PR and testing the tool and leaving a comment stating how it went could help move the PR forward.

I don't know however if a rebase is needed to make sure that the current package still build with the latest upstream in Nixpkgs.

AkechiShiro · 2024-09-01T17:53:43Z

@vlaci there might be a dead link as a source for the current version :

I cannot find the repo on GitHub, maybe there is new unblob that is being worked on that doesn't depend on sasquatch ?

I only found this repo : https://github.com/onekey-sec/sasquatch

EDIT :

nixpkgs-review result :

1 package failed to build:
sasquatch-v4be

14 packages built:
binwalk binwalk.dist diffoscope diffoscope.dist diffoscope.man python310Packages.binwalk python310Packages.binwalk-full python310Packages.binwalk-full.dist python310Packages.binwalk.dist python311Packages.binwalk python311Packages.binwalk-full python311Packages.binwalk-full.dist python311Packages.binwalk.dist sasquatch

error: build log of '/nix/store/40nfx0yjl4q7965p05dkqgavd9xjbh7j-sasquatch-v4be-4.5.1-4.drv^*' is not available
error: build log of '/nix/store/bw99kzndaqc2r8bykzcjc34z91a964j0-sasquatch-v4be-4.5.1-4' is not available

vlaci · 2024-09-09T07:43:43Z

@vlaci there might be a dead link as a source for the current version :

I cannot find the repo on GitHub, maybe there is new unblob that is being worked on that doesn't depend on sasquatch ?

I only found this repo : https://github.com/onekey-sec/sasquatch

EDIT :

nixpkgs-review result :
1 package failed to build:
sasquatch-v4be

14 packages built:
binwalk binwalk.dist diffoscope diffoscope.dist diffoscope.man python310Packages.binwalk python310Packages.binwalk-full python310Packages.binwalk-full.dist python310Packages.binwalk.dist python311Packages.binwalk python311Packages.binwalk-full python311Packages.binwalk-full.dist python311Packages.binwalk.dist sasquatch

error: build log of '/nix/store/40nfx0yjl4q7965p05dkqgavd9xjbh7j-sasquatch-v4be-4.5.1-4.drv^*' is not available
error: build log of '/nix/store/bw99kzndaqc2r8bykzcjc34z91a964j0-sasquatch-v4be-4.5.1-4' is not available

It must have been a github api error, as the source is there...

Asked for their removal in NixOS#219169 (comment)

vlaci · 2024-09-09T07:52:08Z

IDK if I should move these from all-packages accorving to nixpkgs-vet.

AkechiShiro · 2024-09-11T17:06:56Z

I believe there is a typo in the package @vlaci

       error: evaluation aborted with the following error message: 'Failed to evaluate sasquatch-4.5.1-4: «unknown-meta»: has an invalid meta attrset:
         - key 'meta.mainprogram' is unrecognized; expected one of:
         ['available', 'badPlatforms', 'branch', 'broken', 'changelog', 'description', 'downloadPage', 'executables', 'homepage', 'hydraPlatforms', 'insecure', 'isBuildPythonPackage', 'isFcitxEngine', 'isGutenprint', 'isHydraChannel', 'isIbusEngine', 'knownVulnerabilities', 'license', 'longDescription', 'mainProgram', 'maintainers', 'maxSilent', 'name', 'outputsToInstall', 'pkgConfigModules', 'platforms', 'position', 'priority', 'schedulingPriority', 'sourceProvenance', 'tag', 'tests', 'timeout', 'unfree', 'unsupported', 'version']
       '

mainprogram -> mainProgram, if you can also solve the conflict and rebase the PR please ?

Asked for their removal in NixOS#219169 (comment)

vlaci · 2024-09-11T17:19:40Z

I believe there is a typo in the package @vlaci

       error: evaluation aborted with the following error message: 'Failed to evaluate sasquatch-4.5.1-4: «unknown-meta»: has an invalid meta attrset:
         - key 'meta.mainprogram' is unrecognized; expected one of:
         ['available', 'badPlatforms', 'branch', 'broken', 'changelog', 'description', 'downloadPage', 'executables', 'homepage', 'hydraPlatforms', 'insecure', 'isBuildPythonPackage', 'isFcitxEngine', 'isGutenprint', 'isHydraChannel', 'isIbusEngine', 'knownVulnerabilities', 'license', 'longDescription', 'mainProgram', 'maintainers', 'maxSilent', 'name', 'outputsToInstall', 'pkgConfigModules', 'platforms', 'position', 'priority', 'schedulingPriority', 'sourceProvenance', 'tag', 'tests', 'timeout', 'unfree', 'unsupported', 'version']
       '

mainprogram -> mainProgram, if you can also solve the conflict and rebase the PR please ?

Thanks, this error did not emerge on my Nix version.
Also, I botched up a previous patch, hence the conflict. Adjusted.

AkechiShiro · 2024-09-11T18:45:23Z

Tested a build (nixpkgs-review seems a bit confused however) with the new changes, seems to be working alright.

Output of nixpkgs-review :

8 packages removed:
binwalk (†2.4.2) diffoscope (†277) ghidraninja-ghidra-scripts-unstable (†2020-10-07) python3.11-binwalk (†2.4.2) python3.11-binwalk-full (†2.4.2) python3.12-binwalk (†2.4.2) python3.12-binwalk-full (†2.4.2) sasquatch (†4.4)

Nothing to be built.

I ran the build manually.

AkechiShiro

Just waiting for sasquatch-v4be to be moved from top-level/all-packages.nix into by-name/sa/sasquatch-v4be, then I will approve this PR on my side, I can't really test however on nix darwin

pkgs/by-name/sa/sasquatch/package.nix

pkgs/top-level/all-packages.nix

NOTE: as of now, it requires overriding nixpkgs to NixOS/nixpkgs#219169

vlaci · 2024-09-11T19:22:31Z

Just waiting for sasquatch-v4be to be moved from top-level/all-packages.nix into by-name/sa/sasquatch-v4be, then I will approve this PR on my side, I can't really test however on nix darwin

I've created a draft PR for unblob pointing to this branch, so its tests will run for darwin as well: https://github.com/onekey-sec/unblob/pull/946/checks

Changed using ONEKEY fork of sasquatch which is up-to-date with squashfs-tools 4.5.1 and the Darwin compatibility patch used to be shipped in Nixpkgs applies cleanly as well. The updated version also provides a big-endian compatible build as well to support even more insane squashfs dialects.

Asked for their removal in NixOS#219169 (comment)

AkechiShiro · 2024-09-12T18:52:28Z

I don't think the ghidra failure is related :

1 package failed to build:
ghidra-extensions.ghidraninja-ghidra-scripts

13 packages built:
binwalk binwalk.dist diffoscope diffoscope.dist diffoscope.man python311Packages.binwalk python311Packages.binwalk-full python311Packages.binwalk-full.dist python311Packages.binwalk.dist python312Packages.binwalk-full python312Packages.binwalk-full.dist sasquatch sasquatch-v4be

nixos-discourse · 2024-09-14T13:28:25Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/4576

vlaci mentioned this pull request Mar 2, 2023

Package request: unblob #217836

Open

7 tasks

ofborg bot added the 8.has: package (new) label Mar 2, 2023

ofborg bot requested a review from Pamplemousse March 2, 2023 13:32

ofborg bot added 11.by: package-maintainer 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 labels Mar 2, 2023

vlaci force-pushed the sasquatch-4.5.1 branch from 407d774 to 8738804 Compare June 19, 2023 14:25

ofborg bot removed the 10.rebuild-darwin: 1 label Jun 19, 2023

vlaci added a commit to vlaci/nixpkgs that referenced this pull request Jun 19, 2023

sasquatch: removing pamplemousse from maintainers

ce47f62

Asked for their removal in NixOS#219169 (comment)

Pamplemousse removed their request for review December 27, 2023 10:52

vlaci force-pushed the sasquatch-4.5.1 branch from ce47f62 to e5b1f7b Compare January 1, 2024 22:39

vlaci added a commit to vlaci/nixpkgs that referenced this pull request Jan 1, 2024

sasquatch: removing pamplemousse from maintainers

e5b1f7b

Asked for their removal in NixOS#219169 (comment)

vlaci force-pushed the sasquatch-4.5.1 branch from e5b1f7b to 754543d Compare January 1, 2024 22:43

vlaci added a commit to vlaci/nixpkgs that referenced this pull request Jan 1, 2024

sasquatch: removing pamplemousse from maintainers

754543d

Asked for their removal in NixOS#219169 (comment)

vlaci force-pushed the sasquatch-4.5.1 branch from 754543d to 4832e2d Compare January 2, 2024 20:57

vlaci added a commit to vlaci/nixpkgs that referenced this pull request Jan 2, 2024

sasquatch: removing pamplemousse from maintainers

4832e2d

Asked for their removal in NixOS#219169 (comment)

tnias approved these changes Mar 6, 2024

View reviewed changes

tilpner reviewed Mar 6, 2024

View reviewed changes

wegank added the 12.approvals: 1 label Mar 9, 2024

vlaci force-pushed the sasquatch-4.5.1 branch from 4832e2d to f8b0cee Compare September 9, 2024 07:48

vlaci added a commit to vlaci/nixpkgs that referenced this pull request Sep 9, 2024

sasquatch: removing pamplemousse from maintainers

f8b0cee

Asked for their removal in NixOS#219169 (comment)

vlaci force-pushed the sasquatch-4.5.1 branch from f1438af to b5f6276 Compare September 11, 2024 17:18

vlaci added a commit to vlaci/nixpkgs that referenced this pull request Sep 11, 2024

sasquatch: removing pamplemousse from maintainers

99e918e

Asked for their removal in NixOS#219169 (comment)

AkechiShiro suggested changes Sep 11, 2024

View reviewed changes

pkgs/by-name/sa/sasquatch/package.nix Outdated Show resolved Hide resolved

pkgs/top-level/all-packages.nix Show resolved Hide resolved

vlaci force-pushed the sasquatch-4.5.1 branch from b5f6276 to 1a2f7f1 Compare September 11, 2024 19:18

vlaci pushed a commit to onekey-sec/unblob that referenced this pull request Sep 11, 2024

nix: move to sasquatch from nixpkgs

4738205

NOTE: as of now, it requires overriding nixpkgs to NixOS/nixpkgs#219169

vlaci pushed a commit to onekey-sec/unblob that referenced this pull request Sep 11, 2024

DROP ME: override nixpkgs to NixOS/nixpkgs#219169

faf2be8

vlaci mentioned this pull request Sep 11, 2024

Move sasquatch dependency to nixpkgs onekey-sec/unblob#946

Draft

vlaci added 4 commits September 12, 2024 10:19

sasquatch: removing pamplemousse from maintainers

3253a7c

Asked for their removal in NixOS#219169 (comment)

sasquatch: move to pkgs/by-name

a61172b

sasquatch: format package.nix to match RFC style

465008e

vlaci force-pushed the sasquatch-4.5.1 branch from 1a2f7f1 to 465008e Compare September 12, 2024 08:20

AkechiShiro approved these changes Sep 14, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sasquatch: 4.4 -> 4.5.1-1 #219169

sasquatch: 4.4 -> 4.5.1-1 #219169

vlaci commented Mar 2, 2023

vlaci commented Mar 3, 2023

AkechiShiro commented May 20, 2023

vlaci commented May 22, 2023

vlaci commented Jun 19, 2023

Pamplemousse commented Jun 19, 2023

vlaci commented Jun 19, 2023

qkaiser commented Dec 24, 2023

AkechiShiro commented Dec 27, 2023 •

edited

Loading

vlaci commented Jan 1, 2024

AkechiShiro commented Jan 2, 2024 •

edited

Loading

qkaiser commented Jan 3, 2024

nixos-discourse commented Jan 3, 2024

tnias left a comment

Mic92 commented Mar 6, 2024

tilpner Mar 6, 2024 •

edited

Loading

qkaiser Mar 7, 2024

qkaiser commented Mar 7, 2024

smancill commented Aug 19, 2024

AkechiShiro commented Sep 1, 2024

AkechiShiro commented Sep 1, 2024 •

edited

Loading

vlaci commented Sep 9, 2024

vlaci commented Sep 9, 2024

AkechiShiro commented Sep 11, 2024

vlaci commented Sep 11, 2024

AkechiShiro commented Sep 11, 2024

AkechiShiro left a comment

vlaci commented Sep 11, 2024

AkechiShiro commented Sep 12, 2024

nixos-discourse commented Sep 14, 2024

sasquatch: 4.4 -> 4.5.1-1 #219169

Are you sure you want to change the base?

sasquatch: 4.4 -> 4.5.1-1 #219169

Conversation

vlaci commented Mar 2, 2023

Description of changes

Things done

vlaci commented Mar 3, 2023

AkechiShiro commented May 20, 2023

vlaci commented May 22, 2023

vlaci commented Jun 19, 2023

Pamplemousse commented Jun 19, 2023

vlaci commented Jun 19, 2023

qkaiser commented Dec 24, 2023

AkechiShiro commented Dec 27, 2023 • edited Loading

vlaci commented Jan 1, 2024

AkechiShiro commented Jan 2, 2024 • edited Loading

qkaiser commented Jan 3, 2024

nixos-discourse commented Jan 3, 2024

tnias left a comment

Choose a reason for hiding this comment

Mic92 commented Mar 6, 2024

tilpner Mar 6, 2024 • edited Loading

Choose a reason for hiding this comment

qkaiser Mar 7, 2024

Choose a reason for hiding this comment

qkaiser commented Mar 7, 2024

smancill commented Aug 19, 2024

AkechiShiro commented Sep 1, 2024

AkechiShiro commented Sep 1, 2024 • edited Loading

vlaci commented Sep 9, 2024

vlaci commented Sep 9, 2024

AkechiShiro commented Sep 11, 2024

vlaci commented Sep 11, 2024

AkechiShiro commented Sep 11, 2024

AkechiShiro left a comment

Choose a reason for hiding this comment

vlaci commented Sep 11, 2024

AkechiShiro commented Sep 12, 2024

nixos-discourse commented Sep 14, 2024

AkechiShiro commented Dec 27, 2023 •

edited

Loading

AkechiShiro commented Jan 2, 2024 •

edited

Loading

tilpner Mar 6, 2024 •

edited

Loading

AkechiShiro commented Sep 1, 2024 •

edited

Loading