Skip to content

Add Bulk create and delete operation support#223

Open
gmicol wants to merge 59 commits intoCiscoDevNet:developfrom
gmicol:nd42_add_bulk_operations
Open

Add Bulk create and delete operation support#223
gmicol wants to merge 59 commits intoCiscoDevNet:developfrom
gmicol:nd42_add_bulk_operations

Conversation

@gmicol
Copy link
Copy Markdown
Collaborator

@gmicol gmicol commented Apr 8, 2026

Summary of Changes (backwards compatible with existing modules)

Introduced optional bulk operation support across three layers:

1. Orchestrator Layer (NDBaseOrchestrator)

  • Added supports_bulk_create / supports_bulk_delete class variable flags (default False)
  • Added create_bulk / delete_bulk methods guarded by a requires_bulk_support decorator — calling them on an unsupported orchestrator raises AttributeError
  • Added create_bulk_endpoint / delete_bulk_endpoint as conditionally required fields — validated at instantiation via @model_validator, only required when their respective flag is True
  • Subclasses opt-in by setting the flag and defining the endpoint:
class MyOrchestrator(NDBaseOrchestrator[MyModel]):
    supports_bulk_create: ClassVar[bool] = True
    create_bulk_endpoint = EpMyModelBulkPost

2. Collection Layer (NDConfigCollection)

  • Added add_many() / delete_many() to support batch mutations
  • delete_many() performs a single index rebuild instead of one per deletion, reducing complexity from O(D×N) to O(N)

3. State Machine Layer (NDStateMachine)

  • Adopted a categorize-then-execute pattern: items are first classified (create/update/delete), then executed in batch
  • When bulk is supported → single API call for all creates/deletes
  • When bulk is not supported → falls back to individual API calls transparently
  • self.sent tracking deferred until after successful API execution to ensure correctness

Bulk support is fully opt-in — existing orchestrators require zero changes.

allenrobel and others added 30 commits March 9, 2026 09:45
…nherited from future class models. Modify class models for local_user.
…e.py based on comments. Add a get method and get_identifier_value function to NDBaseModel.
…tion of core design adding new methods which will be used in NDConfigCollection and NDNetworkResourceModule classes as well as basic error handling and simple docstrings.
…ign changing existing methods and adding new ones which will be used in NDNetworkResourceModule class as well as basic error handling and simple docstrings.
…of core design changing existing methods and adding new ones which will be used in future as a based for ND network resource modules as well as basic error handling and simple docstrings.
…s for orchestrating crud api operations with model instances and endpoints.
… built-in functionalities. Slightly modify models/base.py to enforce identifiers definitions in NDBaseModel subclasses. Added multiple notes to assert next steps.
…rt endpoints and Pydantic models modification (works for merge and replace states). Add comments for next steps.
…nts and changes to models/local_user.py and api_endpoints/base.py
…ons methods that work for single_identifier strategy (meant to be overridden if needed).
…or NDNestedModel. Add types.file. Various Renaming and small Modifications across the repo. WIP.
…i_endpoints. Adapt api_endpoints, models, orchestrators accordingly. Integration Tests passing for nd_local_user module. Still WIP.
…teration of (Mock Pydantic objects/methods) to pass sanity checks for Pydantic importation.
…NDStateMachineand add custom Exception for it in common/exceptions dir. Set json mode for to_diff_dict method in NDBaseModel.
@gmicol gmicol force-pushed the nd42_add_bulk_operations branch from 11b6d37 to 6e07737 Compare April 8, 2026 06:27
shrsr
shrsr previously approved these changes Apr 8, 2026
Copy link
Copy Markdown
Collaborator

@shrsr shrsr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mikewiebe
Copy link
Copy Markdown
Collaborator

mikewiebe commented Apr 8, 2026

@gmicol Are there any changes required to modules that are using this infrastructure currently without bulk APIs? Is it backwards compatible?

@gmicol
Copy link
Copy Markdown
Collaborator Author

gmicol commented Apr 8, 2026

@mikewiebe there is no change needed to modules that are using this infrastructure currently without bulk APIs. I made sure it was backwards compatible.

Copy link
Copy Markdown
Collaborator

@allenrobel allenrobel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to @mikewiebe, would be good for reviewers if there were a more detailed description of the changes and design decisions in the PR.


@requires_bulk_support("supports_bulk_create")
def create_bulk(self, model_instances: List[ModelType], **kwargs) -> ResponseType:
pass
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since subclasses are expected to implement these, should we consider raising NotImplementedError rather than pass for create_bulk and delete_bulk? This way, if they are called, but not implemented, we'll catch it right away.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a great idea! I will update these methods

items_to_create_bulk.append(final_item)
else:
self.model_orchestrator.create(final_item)
self.sent.add(final_item)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the expected behavior if create_bulk() fails and ignore_errors is True? Are the items in self.sent considered sent and would still appear in the output? Wondering if self.sent.add() should be moved to after the bulk call succeeds?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, in the case of a bulk operation when ignore_errors is True, the items would still appear in self.sent in the output even though the request failed. self.sent.add() should be moved after the bulk call succeeds.

self.model_orchestrator.delete(item)

# Remove from collection
self.existing.delete(identifier)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as with self.sent.add() being called before create_bulk. What is the expected behavior when ignore_errors is True and the delete_bulk call fails?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same answer as before #223 (comment)

for proposed_item in self.proposed:
try:
identifier = proposed_item.get_identifier_value()
identifier = proposed_item.get_identifier_value()
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the expected behavior if get_identifier_value raises and ignore_errors is True? Previously, this was wrapped in a try/except with the except branch containing a conditional for ignore_errors.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error would not be catch correctly, will add it back to the try/except

# Configuration
self.model_orchestrator = model_orchestrator(sender=self.nd_module)
# Accept either an orchestrator instance or a class.
if isinstance(model_orchestrator, type):
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, but anything that's a class will pass this check, not just orchestrator classes. Fine as long as folks are careful.

Copy link
Copy Markdown
Collaborator Author

@gmicol gmicol Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You definitely right. Therefore, I will make the condition more specific to avoid any mistake by devs on the orchestrator class type

raise NDStateMachineError(error_msg) from e

# Log deletion
self.output.assign(after=self.existing)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this needed given the same on line 139? Just asking since I'm not sure.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is needed as this is for delete operation output (_manage_override_deletions and _manage_delete_state methods), whereas line 139 is for saving _manage_create_update_state() output.

But you could argue that this repetitive code that could be set directly in the broader mnage_state function.

gmicol added 4 commits April 8, 2026 23:16
…l fields for NDBaseOrchestrator. Add Validation for these fields.
…g clear distinction between NDBaseOrchestrator class and instance in the NDStateMachine initialization.
…ions:

	- Fix premature sent tracking by deferring until after API execution
	- Add add_many/delete_many to NDConfigCollection to batch index rebuilds
	- Extract _execute_operation helper to centralize error handling
	- Reduce delete complexity from O(D×N) to O(N) via batch deletion
	- DRY delete logic into shared _delete_items method
	- Fix bulk delete indentation bug in _manage_override_deletions
@gmicol gmicol force-pushed the nd42_add_bulk_operations branch from 6e07737 to 4c9031a Compare April 9, 2026 05:01
@gmicol gmicol changed the title [ignore] Add Bulk API handlers for create and delete operations Add Bulk create and delete operation support Apr 9, 2026
…e_create_update_state method. Modify error message.
@gmicol
Copy link
Copy Markdown
Collaborator Author

gmicol commented Apr 9, 2026

Similar to @mikewiebe, would be good for reviewers if there were a more detailed description of the changes and design decisions in the PR.

I agree, I added a concise description of the main changes.

@gmicol gmicol requested review from allenrobel and shrsr April 9, 2026 05:40
@mtarking mtarking deleted the branch CiscoDevNet:develop April 9, 2026 15:41
@mtarking mtarking closed this Apr 9, 2026
@mtarking mtarking reopened this Apr 9, 2026
@mtarking mtarking changed the base branch from nd42_integration to develop April 9, 2026 15:45
@mtarking mtarking dismissed shrsr’s stale review April 9, 2026 15:45

The base branch was changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants