Add Bulk create and delete operation support#223
Add Bulk create and delete operation support#223gmicol wants to merge 59 commits intoCiscoDevNet:developfrom
Conversation
… Nexus Dashboard v4.1.0 and higher.
…nherited from future class models. Modify class models for local_user.
…e.py based on comments. Add a get method and get_identifier_value function to NDBaseModel.
…tion of core design adding new methods which will be used in NDConfigCollection and NDNetworkResourceModule classes as well as basic error handling and simple docstrings.
…ign changing existing methods and adding new ones which will be used in NDNetworkResourceModule class as well as basic error handling and simple docstrings.
…of core design changing existing methods and adding new ones which will be used in future as a based for ND network resource modules as well as basic error handling and simple docstrings.
…anges added to NDNetworkResourceModule.
…s for orchestrating crud api operations with model instances and endpoints.
… built-in functionalities. Slightly modify models/base.py to enforce identifiers definitions in NDBaseModel subclasses. Added multiple notes to assert next steps.
…rt endpoints and Pydantic models modification (works for merge and replace states). Add comments for next steps.
…nts and changes to models/local_user.py and api_endpoints/base.py
…ons methods that work for single_identifier strategy (meant to be overridden if needed).
…or NDNestedModel. Add types.file. Various Renaming and small Modifications across the repo. WIP.
…i_endpoints. Adapt api_endpoints, models, orchestrators accordingly. Integration Tests passing for nd_local_user module. Still WIP.
…ome sanity issues.
…teration of (Mock Pydantic objects/methods) to pass sanity checks for Pydantic importation.
…NDStateMachineand add custom Exception for it in common/exceptions dir. Set json mode for to_diff_dict method in NDBaseModel.
… Add Exception for bulk operations.
11b6d37 to
6e07737
Compare
|
@gmicol Are there any changes required to modules that are using this infrastructure currently without bulk APIs? Is it backwards compatible? |
|
@mikewiebe there is no change needed to modules that are using this infrastructure currently without bulk APIs. I made sure it was backwards compatible. |
allenrobel
left a comment
There was a problem hiding this comment.
Similar to @mikewiebe, would be good for reviewers if there were a more detailed description of the changes and design decisions in the PR.
|
|
||
| @requires_bulk_support("supports_bulk_create") | ||
| def create_bulk(self, model_instances: List[ModelType], **kwargs) -> ResponseType: | ||
| pass |
There was a problem hiding this comment.
Since subclasses are expected to implement these, should we consider raising NotImplementedError rather than pass for create_bulk and delete_bulk? This way, if they are called, but not implemented, we'll catch it right away.
There was a problem hiding this comment.
That's a great idea! I will update these methods
| items_to_create_bulk.append(final_item) | ||
| else: | ||
| self.model_orchestrator.create(final_item) | ||
| self.sent.add(final_item) |
There was a problem hiding this comment.
What's the expected behavior if create_bulk() fails and ignore_errors is True? Are the items in self.sent considered sent and would still appear in the output? Wondering if self.sent.add() should be moved to after the bulk call succeeds?
There was a problem hiding this comment.
You are right, in the case of a bulk operation when ignore_errors is True, the items would still appear in self.sent in the output even though the request failed. self.sent.add() should be moved after the bulk call succeeds.
| self.model_orchestrator.delete(item) | ||
|
|
||
| # Remove from collection | ||
| self.existing.delete(identifier) |
There was a problem hiding this comment.
Same comment as with self.sent.add() being called before create_bulk. What is the expected behavior when ignore_errors is True and the delete_bulk call fails?
| for proposed_item in self.proposed: | ||
| try: | ||
| identifier = proposed_item.get_identifier_value() | ||
| identifier = proposed_item.get_identifier_value() |
There was a problem hiding this comment.
What's the expected behavior if get_identifier_value raises and ignore_errors is True? Previously, this was wrapped in a try/except with the except branch containing a conditional for ignore_errors.
There was a problem hiding this comment.
The error would not be catch correctly, will add it back to the try/except
| # Configuration | ||
| self.model_orchestrator = model_orchestrator(sender=self.nd_module) | ||
| # Accept either an orchestrator instance or a class. | ||
| if isinstance(model_orchestrator, type): |
There was a problem hiding this comment.
Nit, but anything that's a class will pass this check, not just orchestrator classes. Fine as long as folks are careful.
There was a problem hiding this comment.
You definitely right. Therefore, I will make the condition more specific to avoid any mistake by devs on the orchestrator class type
| raise NDStateMachineError(error_msg) from e | ||
|
|
||
| # Log deletion | ||
| self.output.assign(after=self.existing) |
There was a problem hiding this comment.
Is this needed given the same on line 139? Just asking since I'm not sure.
There was a problem hiding this comment.
It is needed as this is for delete operation output (_manage_override_deletions and _manage_delete_state methods), whereas line 139 is for saving _manage_create_update_state() output.
But you could argue that this repetitive code that could be set directly in the broader mnage_state function.
…l fields for NDBaseOrchestrator. Add Validation for these fields.
…g clear distinction between NDBaseOrchestrator class and instance in the NDStateMachine initialization.
…ions: - Fix premature sent tracking by deferring until after API execution - Add add_many/delete_many to NDConfigCollection to batch index rebuilds - Extract _execute_operation helper to centralize error handling - Reduce delete complexity from O(D×N) to O(N) via batch deletion - DRY delete logic into shared _delete_items method - Fix bulk delete indentation bug in _manage_override_deletions
6e07737 to
4c9031a
Compare
…e_create_update_state method. Modify error message.
I agree, I added a concise description of the main changes. |
Summary of Changes (backwards compatible with existing modules)
Introduced optional bulk operation support across three layers:
1. Orchestrator Layer (
NDBaseOrchestrator)supports_bulk_create/supports_bulk_deleteclass variable flags (defaultFalse)create_bulk/delete_bulkmethods guarded by arequires_bulk_supportdecorator — calling them on an unsupported orchestrator raisesAttributeErrorcreate_bulk_endpoint/delete_bulk_endpointas conditionally required fields — validated at instantiation via@model_validator, only required when their respective flag isTrue2. Collection Layer (
NDConfigCollection)add_many()/delete_many()to support batch mutationsdelete_many()performs a single index rebuild instead of one per deletion, reducing complexity from O(D×N) to O(N)3. State Machine Layer (
NDStateMachine)self.senttracking deferred until after successful API execution to ensure correctnessBulk support is fully opt-in — existing orchestrators require zero changes.