-
Notifications
You must be signed in to change notification settings - Fork 13
feat: track package descriptions when loading #2639
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
| Some(generators.join(", ")) | ||
| } | ||
|
|
||
| /// Format a generator value from the metadata. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
moved to describe.rs
| /// Read a Package in json format from an io reader. | ||
| /// Returns package and the combined extension registry | ||
| /// of the provided registry and the package extensions. | ||
| pub(super) fn from_json_reader( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
moved to reader.rs
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2639 +/- ##
==========================================
- Coverage 83.37% 83.36% -0.02%
==========================================
Files 259 261 +2
Lines 50652 51015 +363
Branches 46213 46576 +363
==========================================
+ Hits 42233 42527 +294
- Misses 6051 6120 +69
Partials 2368 2368
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| assert_matches!(check(&hugr, ®istry), Ok(())); | ||
| } | ||
|
|
||
| #[test] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
error is no longer thrown
8ebd2e7 to
1792a90
Compare
CodSpeed Performance ReportMerging #2639 will improve performances by 23.17%Comparing Summary
Benchmarks breakdown
|
5f2ab49 to
2ebcbe3
Compare
beed3c2 to
c3c4b64
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR introduces a describe CLI subcommand that provides high-level descriptions of HUGR packages for debugging purposes. It tracks package descriptions during loading, enabling partial descriptions even when errors occur. The implementation centralizes package loading logic through a new Reader struct and deprecates the existing read_envelope function in favor of read_described_envelope, which returns both a description and the package.
Key changes:
- New
describeCLI command with table and JSON output formats - Package description tracking via
PackageDesc,ModuleDesc, andExtensionDescstructures - Refactored envelope reading to use
EnvelopeReaderfor unified loading and description tracking
Reviewed Changes
Copilot reviewed 21 out of 22 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| hugr-core/src/envelope/description.rs | Defines description structures for packages, modules, and extensions |
| hugr-core/src/envelope/reader.rs | New reader implementation that tracks descriptions during loading |
| hugr-core/src/envelope.rs | Deprecates read_envelope, introduces read_described_envelope and ReadError |
| hugr-core/src/import.rs | Adds import_described_hugr to return module descriptions alongside results |
| hugr-cli/src/describe.rs | Implements the describe CLI subcommand with table and JSON output |
| hugr-cli/src/hugr_io.rs | Updates input handling to support described envelopes |
Comments suppressed due to low confidence (1)
hugr-cli/src/describe.rs:1
- Debug macro
dbg!()should not be used in production code. Remove thedbg!()wrapper and use.assert()directly.
//! Convert between different HUGR envelope formats.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
c3c4b64 to
d429940
Compare
| let items = values | ||
| .iter() | ||
| .zip(variant.iter()) | ||
| .map(|(value, typ)| self.import_value(*value, *typ)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typos
6709cbc to
d83cebc
Compare
d83cebc to
35bf8fa
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
35bf8fa to
5b2ae6c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is big, I'll do another pass.
I like this a lot
| use crate::import::ImportError; | ||
| use crate::{Extension, import::import_package}; | ||
|
|
||
| // TODO centralise all core metadata keys in one place. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please make an issue for this and link it.
hugr-core/src/envelope.rs
Outdated
| flag_ids: Vec<usize>, | ||
| }, | ||
| /// Error raised while checking for breaking extension version mismatch. | ||
| // no longer used |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we deprecate this variant?
| } | ||
|
|
||
| /// Get a string representation of an OpType for description purposes. | ||
| pub fn op_string(op: &OpType) -> String { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think these are user visible strings.
I wonder if it would be better to just use Display here? that's mostly what you're doing, I wonder if tweaking you do should be done in Display. That would probably be breaking.
I suggest a snapshot test of this. We have experienced buggy Display getting to users before which is pretty embarrassing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I didn't want to mess with Display for optype, this is specialised to "descriptions", where I've made the choice to show function names and signatures. I will rename and add unit test.
| } | ||
| } | ||
|
|
||
| fn extend_option_vec<T: Clone>(vec: &mut Option<Vec<T>>, items: impl IntoIterator<Item = T>) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is called a partial vec above. I prefer option vec. If you want a crate for this check out stable-vec
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
renamed to OptionVec
f6d8675 to
d6350c9
Compare
d6350c9 to
f32406d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not obvious to me that Described is worth it.
This isn't a Monad as written, the key monadic method is like and_then
hugr-core/src/envelope.rs
Outdated
| Ok(_) => Ok(res.map(Result::unwrap)), | ||
| Err(_) => Err(ReadError::Payload { | ||
| inner: res.map(|r| Box::new(r.expect_err("matched on error"))), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you need
fn transpose(from: Described<Result<T,E>>) -> Result<Described<T>,Described<E>>
f32406d to
f51da3c
Compare
Agreed - which is why I've put the whole thing in a private module and only exposed Edit: now reverted |
f51da3c to
fe20cc4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some nits, but looks great.
hugr-core/src/envelope.rs
Outdated
| /// Error raised while checking for breaking extension version mismatch. | ||
| #[deprecated(since = "0.24.1")] | ||
| #[error(transparent)] | ||
| #[allow(deprecated)] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's just always better to #[expect(deprecated)]?
| } | ||
|
|
||
| /// High-level description of a HUGR package. | ||
| #[derive(Debug, Clone, PartialEq, Default, serde::Serialize)] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's weird to have serialize but no deserialize. I guess we don't need it right now but this is a public type, I prefer to give deserialize too
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm happy to add it for cases where derive just works but there are a couple of to-string serialisations I use for which I don't think it's writing a parser, so won't add for those.
|
|
||
| /// Returns the generator(s) of the package modules, if any. | ||
| /// Concatenates multiple generators with commas. | ||
| pub fn generator(&self) -> Option<String> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe it doesn't matter, but this will "break" for generators that include commas.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it doesn't matter, this string is not intended to be structured
| } | ||
|
|
||
| /// Returns an iterator over the packaged extension descriptions. | ||
| pub fn packaged_extensions(&self) -> impl Iterator<Item = &Option<ExtensionDesc>> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't this be more useful with Item = &ExtensionDesc?
| } | ||
| } | ||
|
|
||
| impl<E: AsRef<crate::Extension>> From<&E> for ExtensionDesc { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we are using AsRef on our interface we should really impl AsRef<Extension> for Extension e.g. https://doc.rust-lang.org/std/primitive.str.html#impl-AsRef%3Cstr%3E-for-str
| } | ||
| } | ||
|
|
||
| #[derive(Debug, Clone, PartialEq, serde::Serialize)] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would like Deserialize
| serializer.serialize_str(op_string(op_type).as_str()) | ||
| } | ||
|
|
||
| #[derive(Debug, Clone, PartialEq, Default, serde::Serialize)] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
plz deserialize
| Err(e) => Err(ReadError::Payload { | ||
| source: Box::new(e), | ||
| partial_description: desc, | ||
| }), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is the main point of the whole operation, give a partial description on error.
Is it tested? (coverage checker broken for me right now).
This behaviour isn't described in docs (maybe docs on ReadError are enough)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there is coverage, see https://app.codecov.io/gh/CQCL/hugr/pull/2639?src=pr&el=tree&filepath=hugr-core/src/envelope.rs#1b6c34164c5688f367beb900f3954593-R182
a full integration test of the behaviour is included in the follow up PR #2650 via cli tests
I will add docs, thanks
fe20cc4 to
334e96b
Compare
Closes #2603 See base PR for more context: #2639 The `describe` CLI command loads the package with a description and: - If successful prints the description as tables to output - If error prints partial description to output and error to stderror - If --json flag is passed outputs as json instead, for use with other tools ## Help <img width="762" height="932" alt="image" src="https://github.com/user-attachments/assets/adf53b06-652a-4967-afcb-88e9cabfaad8" /> ## Summary <img width="1056" height="337" alt="image" src="https://github.com/user-attachments/assets/f0882eb3-2b80-421d-9de7-614dc4e42c7c" /> ## JSON (nushell render) <img width="1449" height="261" alt="image" src="https://github.com/user-attachments/assets/7a2728cc-a0d4-417c-b938-06b26d409d47" /> --------- Co-authored-by: Alec Edgington <[email protected]>
## 🤖 New release * `hugr-model`: 0.24.0 -> 0.24.1 (✓ API compatible changes) * `hugr-core`: 0.24.0 -> 0.24.1 (✓ API compatible changes) * `hugr-llvm`: 0.24.0 -> 0.24.1 (✓ API compatible changes) * `hugr-passes`: 0.24.0 -> 0.24.1 (✓ API compatible changes) * `hugr-persistent`: 0.3.1 -> 0.3.2 (✓ API compatible changes) * `hugr`: 0.24.0 -> 0.24.1 (✓ API compatible changes) * `hugr-cli`: 0.24.0 -> 0.24.1 (✓ API compatible changes) <details><summary><i><b>Changelog</b></i></summary><p> ## `hugr-model` <blockquote> ## [0.24.1](hugr-model-v0.24.0...hugr-model-v0.24.1) - 2025-11-03 ### Bug Fixes - Correct conversion of `table::Term::Tuple` to `ast::Term` ([#2653](#2653)) </blockquote> ## `hugr-core` <blockquote> ## [0.24.1](hugr-core-v0.24.0...hugr-core-v0.24.1) - 2025-11-03 ### Bug Fixes - validation outside entrypoint, normalize_cfgs w/ nonlocal edges ([#2633](#2633)) - SiblingSubgraph::try_from_nodes for non-entrypoint region ([#2655](#2655)) - Correct conversion of `table::Term::Tuple` to `ast::Term` ([#2653](#2653)) ### New Features - track package descriptions when loading ([#2639](#2639)) - *(cli)* describe sub-command ([#2650](#2650)) </blockquote> ## `hugr-llvm` <blockquote> ## [0.24.0](hugr-llvm-v0.23.0...hugr-llvm-v0.24.0) - 2025-10-13 ### New Features - LLVM lowering for borrow arrays using bitmasks ([#2574](#2574)) - *(py, core, llvm)* add `is_borrowed` op for BorrowArray ([#2610](#2610)) ### Refactor - [**breaking**] consistent inout order in borrow array ([#2621](#2621)) </blockquote> ## `hugr-passes` <blockquote> ## [0.24.1](hugr-passes-v0.24.0...hugr-passes-v0.24.1) - 2025-11-03 ### Bug Fixes - validation outside entrypoint, normalize_cfgs w/ nonlocal edges ([#2633](#2633)) </blockquote> ## `hugr-persistent` <blockquote> ## [0.3.2](hugr-persistent-v0.3.1...hugr-persistent-v0.3.2) - 2025-11-03 ### New Features - *(persistent)* More efficient HugrView iterators for PersistentHugr ([#2595](#2595)) </blockquote> ## `hugr` <blockquote> ## [0.24.1](hugr-v0.24.0...hugr-v0.24.1) - 2025-11-03 ### Bug Fixes - validation outside entrypoint, normalize_cfgs w/ nonlocal edges ([#2633](#2633)) - SiblingSubgraph::try_from_nodes for non-entrypoint region ([#2655](#2655)) - Correct conversion of `table::Term::Tuple` to `ast::Term` ([#2653](#2653)) ### New Features - track package descriptions when loading ([#2639](#2639)) - *(cli)* describe sub-command ([#2650](#2650)) </blockquote> ## `hugr-cli` <blockquote> ## [0.24.1](hugr-cli-v0.24.0...hugr-cli-v0.24.1) - 2025-11-03 ### New Features - track package descriptions when loading ([#2639](#2639)) - *(cli)* describe sub-command ([#2650](#2650)) </blockquote> </p></details> --- This PR was generated with [release-plz](https://github.com/release-plz/release-plz/).
## 🤖 New release * `hugr-model`: 0.24.1 -> 0.24.2 * `hugr-core`: 0.24.1 -> 0.24.2 * `hugr-llvm`: 0.24.1 -> 0.24.2 * `hugr-passes`: 0.24.1 -> 0.24.2 (✓ API compatible changes) * `hugr`: 0.24.1 -> 0.24.2 (✓ API compatible changes) * `hugr-cli`: 0.24.1 -> 0.24.2 (✓ API compatible changes) * `hugr-persistent`: 0.3.2 -> 0.3.3 <details><summary><i><b>Changelog</b></i></summary><p> ## `hugr-model` <blockquote> ## [0.24.1](hugr-model-v0.24.0...hugr-model-v0.24.1) - 2025-11-03 ### Bug Fixes - Correct conversion of `table::Term::Tuple` to `ast::Term` ([#2653](#2653)) </blockquote> ## `hugr-core` <blockquote> ## [0.24.1](hugr-core-v0.24.0...hugr-core-v0.24.1) - 2025-11-03 ### Bug Fixes - validation outside entrypoint, normalize_cfgs w/ nonlocal edges ([#2633](#2633)) - SiblingSubgraph::try_from_nodes for non-entrypoint region ([#2655](#2655)) - Correct conversion of `table::Term::Tuple` to `ast::Term` ([#2653](#2653)) ### New Features - track package descriptions when loading ([#2639](#2639)) - *(cli)* describe sub-command ([#2650](#2650)) </blockquote> ## `hugr-llvm` <blockquote> ## [0.24.0](hugr-llvm-v0.23.0...hugr-llvm-v0.24.0) - 2025-10-13 ### New Features - LLVM lowering for borrow arrays using bitmasks ([#2574](#2574)) - *(py, core, llvm)* add `is_borrowed` op for BorrowArray ([#2610](#2610)) ### Refactor - [**breaking**] consistent inout order in borrow array ([#2621](#2621)) </blockquote> ## `hugr-passes` <blockquote> ## [0.24.2](hugr-passes-v0.24.1...hugr-passes-v0.24.2) - 2025-11-03 ### Bug Fixes - ReplaceTypes: operate on whole Hugr, with set_regions ([#2662](#2662)) </blockquote> ## `hugr` <blockquote> ## [0.24.2](hugr-v0.24.1...hugr-v0.24.2) - 2025-11-03 ### Bug Fixes - ReplaceTypes: operate on whole Hugr, with set_regions ([#2662](#2662)) </blockquote> ## `hugr-cli` <blockquote> ## [0.24.1](hugr-cli-v0.24.0...hugr-cli-v0.24.1) - 2025-11-03 ### New Features - track package descriptions when loading ([#2639](#2639)) - *(cli)* describe sub-command ([#2650](#2650)) </blockquote> ## `hugr-persistent` <blockquote> ## [0.3.2](hugr-persistent-v0.3.1...hugr-persistent-v0.3.2) - 2025-11-03 ### New Features - *(persistent)* More efficient HugrView iterators for PersistentHugr ([#2595](#2595)) </blockquote> </p></details> --- This PR was generated with [release-plz](https://github.com/release-plz/release-plz/).
## 🤖 New release * `hugr-model`: 0.24.2 -> 0.24.3 (✓ API compatible changes) * `hugr-core`: 0.24.2 -> 0.24.3 * `hugr-llvm`: 0.24.2 -> 0.24.3 (✓ API compatible changes) * `hugr-passes`: 0.24.2 -> 0.24.3 (✓ API compatible changes) * `hugr`: 0.24.2 -> 0.24.3 (✓ API compatible changes) * `hugr-cli`: 0.24.2 -> 0.24.3 (✓ API compatible changes) * `hugr-persistent`: 0.3.3 -> 0.3.4 <details><summary><i><b>Changelog</b></i></summary><p> ## `hugr-model` <blockquote> ## [0.24.1](hugr-model-v0.24.0...hugr-model-v0.24.1) - 2025-11-03 ### Bug Fixes - Correct conversion of `table::Term::Tuple` to `ast::Term` ([#2653](#2653)) </blockquote> ## `hugr-core` <blockquote> ## [0.24.1](hugr-core-v0.24.0...hugr-core-v0.24.1) - 2025-11-03 ### Bug Fixes - validation outside entrypoint, normalize_cfgs w/ nonlocal edges ([#2633](#2633)) - SiblingSubgraph::try_from_nodes for non-entrypoint region ([#2655](#2655)) - Correct conversion of `table::Term::Tuple` to `ast::Term` ([#2653](#2653)) ### New Features - track package descriptions when loading ([#2639](#2639)) - *(cli)* describe sub-command ([#2650](#2650)) </blockquote> ## `hugr-llvm` <blockquote> ## [0.24.3](hugr-llvm-v0.24.2...hugr-llvm-v0.24.3) - 2025-11-06 ### Bug Fixes - BorrowArray discard handler allows elements to be borrowed ([#2666](#2666)) </blockquote> ## `hugr-passes` <blockquote> ## [0.24.3](hugr-passes-v0.24.2...hugr-passes-v0.24.3) - 2025-11-06 ### Bug Fixes - BorrowArray discard handler allows elements to be borrowed ([#2666](#2666)) </blockquote> ## `hugr` <blockquote> ## [0.24.3](hugr-v0.24.2...hugr-v0.24.3) - 2025-11-06 ### Bug Fixes - BorrowArray discard handler allows elements to be borrowed ([#2666](#2666)) </blockquote> ## `hugr-cli` <blockquote> ## [0.24.1](hugr-cli-v0.24.0...hugr-cli-v0.24.1) - 2025-11-03 ### New Features - track package descriptions when loading ([#2639](#2639)) - *(cli)* describe sub-command ([#2650](#2650)) </blockquote> ## `hugr-persistent` <blockquote> ## [0.3.2](hugr-persistent-v0.3.1...hugr-persistent-v0.3.2) - 2025-11-03 ### New Features - *(persistent)* More efficient HugrView iterators for PersistentHugr ([#2595](#2595)) </blockquote> </p></details> --- This PR was generated with [release-plz](https://github.com/release-plz/release-plz/).
Goals
describecli subcommand for high-level descriptions of a package, primarily for debug purposes feat(cli): describe sub-command #2650. Since we want a description even if some loading stage fails (e.g. extension resolution), tracking partial descriptions is a requirement.Secondary goals
Approach
In order to make this PR non-breaking I have deprecated existing reading functions, and introduced new functions that return a full description on successful read, and a partial when error. The deprecated functions are most of the missing test coverage.
Reading naturally falls in to two stages: header and payload. If the header fails no description is valid, so that error does not come with a description. Payload errors can come with descriptions.
In order to build descriptions while loading I introduce the
Readerstruct to track data across the reading stages. This is handed off to the modelContextwhen that stage takes over.Partial descriptions
Descriptions are defined in description.rs. They are intended to be small amounts of simple data, so all fields are pub. All fields except the envelope header are optional, a value of None indicates the field has not been set yet.
Appendix: JSON
Since the JSON format is effectively legacy I did not focus much on incremental description tracking, it is more difficult since serde likes to read all at once.
I found a surprise performance regression in JSON loading. In commit af01043 I attempted to address this by removing an unnecessary two stage serde deser, which has swung things in the other direction, improving performance from prior.
Appendix: EnvelopeError
EnvelopeError is used as a catch all for all reading and writing errors. This PR goes some way to introducing new more specific errors for reaeding. EnvelopeError should really in future have the reading errors be taken out of it and renamed to
WriteErroror similar, then all the conversions from the reading errors can be removed. Removing the deprecated read functions is a pre-requisite for this. The merging of this PR should be accompanied by issues for these follow ups.