|
| 1 | +# GCP Dynamic Adapter Framework |
| 2 | + |
| 3 | +The GCP Dynamic Adapter Framework is a powerful system for automatically generating GCP resource adapters by making simple HTTP requests to GCP APIs instead of using versioned SDKs. This framework eliminates the need to manually implement GET/SEARCH/LIST methods and handles all the complex wiring, validation, and error handling automatically. |
| 4 | + |
| 5 | +## What is a Dynamic Adapter? |
| 6 | + |
| 7 | +Instead of using versioned SDKs for GCP, we make simple HTTP requests and generate resource adapters dynamically. The framework provides several key advantages: |
| 8 | + |
| 9 | +- **No Manual Method Implementation**: Instead of creating GET/SEARCH/LIST methods manually, we define only the endpoints in the adapter type definition |
| 10 | +- **Automatic Link Detection**: We identify the linked items, but the framework handles all the wiring depending on the adapter metadata |
| 11 | +- **Centralized Framework Logic**: All adapter metadata, query validations, error handling, iterations, and caching are handled by the framework |
| 12 | +- **AI-Assisted Development**: With Cursor instructions, all we need to do is provide links for the resource type definition and GET endpoint. Cursor does a good job generating the code, but the output should be thoroughly inspected. The author should not allow good-looking verbose unnecessary code since every line of code is a liability. Focus on concise, essential implementations and comprehensive test coverage |
| 13 | + |
| 14 | +## Why Dynamic? |
| 15 | + |
| 16 | +We don't use fixed SDKs. We always use the dynamic API response. With comprehensive logging in place, we can identify potential links even after creating adapters, which was not possible before. We do this by checking the structure of an attribute - if it looks like a resource name but we don't have a link for it, then we log it as a potential adapter. |
| 17 | + |
| 18 | +This approach provides several benefits: |
| 19 | + |
| 20 | +- **Future-Proof**: No dependency on SDK versions that may change |
| 21 | +- **Consistent**: All adapters follow the same patterns and behaviors |
| 22 | +- **Discoverable**: Automatic detection of new potential links from API responses |
| 23 | +- **Maintainable**: Centralized logic means updates apply to all adapters |
| 24 | + |
| 25 | +## Resource Requirements |
| 26 | + |
| 27 | +For a resource to be compatible with the dynamic adapter framework, it should follow standard naming conventions and API response types. See BigQuery as an example of a non-standard adapter that required a manual implementation due to its unique API response format and naming conventions. |
| 28 | + |
| 29 | +**Standard Requirements:** |
| 30 | +- Consistent resource naming in API responses |
| 31 | +- Standard REST API patterns (GET, LIST endpoints) |
| 32 | +- Predictable response structures |
| 33 | +- Standard GCP resource URL patterns |
| 34 | + |
| 35 | +**Non-Standard Examples (Require Manual Adapters):** |
| 36 | +- BigQuery resources with composite IDs (`projectID:datasetID.tableID`) |
| 37 | +- Resources with attributes referencing multiple resource types |
| 38 | +- APIs with non-standard response formats |
| 39 | + |
| 40 | +## Linker: How It Works |
| 41 | + |
| 42 | +The linker is a critical component that finds the adapter metadata for linked items and creates linked item queries by their definition. This standardizes how a certain adapter is linked across the entire source and prevents code duplication. |
| 43 | + |
| 44 | +**Key Benefits:** |
| 45 | +- **Standardization**: Ensures consistent linking patterns across all adapters |
| 46 | +- **Centralized Updates**: If a linked item adapter changes, the update applies to all existing adapters automatically |
| 47 | +- **No Find/Replace**: Eliminates the need to manually update multiple files when linked item logic changes |
| 48 | +- **Manual Adapter Compatibility**: It's possible to link manual adapters to dynamic adapters seamlessly |
| 49 | + |
| 50 | +## Flow: GET Request to SDP Adapter |
| 51 | + |
| 52 | +The complete flow from making a GET request to creating an SDP adapter follows these steps: |
| 53 | + |
| 54 | +1. **Adapter Definition**: Define the adapter metadata in the adapter file (see [dynamic-adapter-creation.mdc](adapters/.cursor/rules/dynamic-adapter-creation.mdc)) |
| 55 | +2. **Adapter Creation**: Framework creates the appropriate adapter type based on metadata configuration |
| 56 | +3. **GET Request Processing**: Validate scope, check cache, construct URL, make HTTP request, convert to SDP item |
| 57 | +4. **External Response to SDP Conversion**: Extract attributes, apply blast propagation rules, generate linked item queries |
| 58 | +5. **Unit Test Coverage**: Test GET functionality and static tests for blast propagation |
| 59 | + |
| 60 | +For detailed implementation patterns and code examples, refer to the [dynamic adapter creation rules](adapters/.cursor/rules/dynamic-adapter-creation.mdc). |
| 61 | + |
| 62 | +## AI Tools Available |
| 63 | + |
| 64 | +We have helper scripts that benefit from Linear and Cursor integration to streamline adapter development: |
| 65 | + |
| 66 | +### Generate Adapter Ticket |
| 67 | +```bash |
| 68 | +# Generate implementation ticket for new adapter |
| 69 | +go run ai-tools/generate-adapter-ticket-cmd/main.go -name compute-subnetwork -api-ref "https://cloud.google.com/compute/docs/reference/rest/v1/subnetworks/get" |
| 70 | +``` |
| 71 | + |
| 72 | +### Generate Test Ticket |
| 73 | +```bash |
| 74 | +# Generate test ticket for existing adapter |
| 75 | +go run ai-tools/generate-test-ticket-cmd/main.go compute-global-address |
| 76 | +``` |
| 77 | + |
| 78 | +**Benefits:** |
| 79 | +- **Automated Ticket Creation**: Generates Linear tickets with proper context and requirements |
| 80 | +- **Cursor Integration**: Works seamlessly with Cursor rules for consistent implementation |
| 81 | +- **Comprehensive Context**: Includes API references, implementation checklists, and testing requirements |
| 82 | + |
| 83 | +For detailed usage instructions, see the [AI Tools README](ai-tools/README.md). |
| 84 | + |
| 85 | +## Cursor Integration |
| 86 | + |
| 87 | +It is highly recommended to use Cursor for creating adapters. There are comprehensive rules available that guide the implementation process. After creating an adapter, the author MUST perform the following checks: |
| 88 | + |
| 89 | +### Adapter Validation |
| 90 | + |
| 91 | +1. **Terraform Mappings GET/Search**: Check from Terraform registry that the mappings are correct |
| 92 | +2. **Blast Propagations**: Verify they are comprehensive and attribute values follow standards |
| 93 | +3. **Item Selector**: If the item identifier in the API response is something other than `name`, define it properly |
| 94 | +4. **Unique Attribute Keys**: Investigate the GET endpoint format and ensure it's correct |
| 95 | + |
| 96 | +### Test Completeness |
| 97 | + |
| 98 | +1. **Blast Propagation/Linked Item Queries**: Verify they work as expected |
| 99 | +2. **Unique Attribute**: Ensure it matches the GET call response |
| 100 | +3. **Terraform Mapping for Search**: Confirm it exists if search is supported |
| 101 | + |
| 102 | +## Post-Implementation Steps |
| 103 | + |
| 104 | +After adding a new adapter, follow the comprehensive post-implementation checklist in the [main adapter documentation](../README.md#post-implementation-steps). This includes updating documentation, IAM permissions, and enabling required APIs. |
| 105 | + |
| 106 | +## Adapter Types |
| 107 | + |
| 108 | +The framework supports four types of adapters based on their capabilities: |
| 109 | + |
| 110 | +- **Standard**: GET only |
| 111 | +- **Listable**: GET + LIST |
| 112 | +- **Searchable**: GET + SEARCH |
| 113 | +- **SearchableListable**: GET + LIST + SEARCH |
| 114 | + |
| 115 | +The adapter type is automatically determined based on the metadata configuration: |
| 116 | + |
| 117 | +```go |
| 118 | +func adapterType(meta gcpshared.AdapterMeta) typeOfAdapter { |
| 119 | + if meta.ListEndpointFunc != nil && meta.SearchEndpointFunc == nil { |
| 120 | + return Listable |
| 121 | + } |
| 122 | + if meta.SearchEndpointFunc != nil && meta.ListEndpointFunc == nil { |
| 123 | + return Searchable |
| 124 | + } |
| 125 | + if meta.ListEndpointFunc != nil && meta.SearchEndpointFunc != nil { |
| 126 | + return SearchableListable |
| 127 | + } |
| 128 | + return Standard |
| 129 | +} |
| 130 | +``` |
| 131 | + |
| 132 | +## Benefits of Dynamic Adapters |
| 133 | + |
| 134 | +1. **Consistency**: All adapters follow the same patterns and behaviors |
| 135 | +2. **Efficiency**: Reduces boilerplate code and speeds up development |
| 136 | +3. **Maintainability**: Centralized logic makes updates and bug fixes easier |
| 137 | +4. **Scalability**: Simplifies the process of adding new resources |
| 138 | +5. **Quality**: Automatic validation and error handling ensure reliability |
| 139 | +6. **Discoverability**: Automatic detection of potential new links from API responses |
| 140 | + |
| 141 | +## Getting Started |
| 142 | + |
| 143 | +1. **Use AI Tools**: Generate tickets using the helper scripts |
| 144 | +2. **Follow Cursor Rules**: Apply the comprehensive rules for consistent implementation |
| 145 | +3. **Review Thoroughly**: Check all validation points before considering complete |
| 146 | +4. **Update Documentation**: Ensure all related documentation is updated |
| 147 | +5. **Test Extensively**: Verify all functionality works as expected |
| 148 | + |
| 149 | +The dynamic adapter framework represents a significant advancement in how we handle GCP resource discovery, providing a robust, scalable, and maintainable solution for infrastructure mapping. |
0 commit comments