fix: Improve IO Abstractions and cleanup

.cursor/rules/documentation.mdc

@@ -1,25 +1,20 @@
 ---
-description:
+description: Update conceptual documentation when SDK code changes to keep docs synchronized with implementation
 globs: application_sdk/**/*.py
 alwaysApply: false
 ---
-Update conceptual documentation when SDK code changes.
-  A file within the Application SDK has been modified: `{trigger_file_path}`.
+# Documentation Updates
 
-  **Action Required:** Please review the changes in this file and update the corresponding conceptual documentation under `docs/docs/concepts/` to ensure it accurately reflects the current code. Keeping documentation synchronized with the code is crucial for maintainability.
+When modifying SDK code, update corresponding conceptual documentation under `docs/docs/concepts/` to reflect changes.
 
-  **Module to Concept Doc Mapping:**
-  *   `application_sdk/application/**` -> `docs/docs/concepts/application.md` (Covers both simple and specialized workflow applications. The `BaseApplication` class is the generic entry point for all workflow-driven applications.)
-  *   `application_sdk/server/**` -> `docs/docs/concepts/server.md` (Covers server abstractions, FastAPI integration, and API endpoint patterns.)
-  *   `application_sdk/handlers/**` -> `docs/docs/concepts/handlers.md` (Covers handler interfaces and custom handler logic.)
-  *   `application_sdk/clients/**` -> `docs/docs/concepts/clients.md` (Covers client interfaces, SQL clients, and connection management.)
-  *   `application_sdk/activities/**` -> `docs/docs/concepts/activities.md` (Covers activity interfaces and activity patterns.)
-  *   `application_sdk/common/**` or `application_sdk/constants.py` -> `docs/docs/concepts/common.md` (Covers shared utilities, logging, and constants.)
-  *   `application_sdk/inputs/**` -> `docs/docs/concepts/inputs.md` (Covers input abstractions and input handling.)
-  *   `application_sdk/outputs/**` -> `docs/docs/concepts/outputs.md` (Covers output abstractions and output handling.)
-  *   `application_sdk/transformers/**` -> `docs/docs/concepts/transformers.md` (Covers transformer interfaces and entity mapping.)
-  *   `application_sdk/workflows/**` -> `docs/docs/concepts/workflows.md` (Covers workflow interfaces, workflow patterns, and orchestration.)
-  *   `application_sdk/worker.py` -> `docs/docs/concepts/worker.md` (Covers worker orchestration and Temporal worker management.)
-  *   `docs/docs/concepts/application_sql.md` -> Specialized SQL workflow application patterns and examples.
-
-  *(Note: Carefully examine the code changes to determine the exact impact and which documentation file(s) require updates. For application changes, review both simple and specialized application patterns.)*
+**Module to Concept Doc Mapping:**
+*   `application_sdk/application/**` -> `docs/docs/concepts/application.md`
+*   `application_sdk/server/**` -> `docs/docs/concepts/server.md`
+*   `application_sdk/handlers/**` -> `docs/docs/concepts/handlers.md`
+*   `application_sdk/clients/**` -> `docs/docs/concepts/clients.md`
+*   `application_sdk/activities/**` -> `docs/docs/concepts/activities.md`
+*   `application_sdk/common/**` or `application_sdk/constants.py` -> `docs/docs/concepts/common.md`
+*   `application_sdk/io/**` -> `docs/docs/concepts/inputs.md` (for readers) AND `docs/docs/concepts/outputs.md` (for writers)
+*   `application_sdk/transformers/**` -> `docs/docs/concepts/transformers.md`
+*   `application_sdk/workflows/**` -> `docs/docs/concepts/workflows.md`
+*   `application_sdk/worker.py` -> `docs/docs/concepts/worker.md`

.cursor/rules/exception-handling.mdc

@@ -146,22 +146,56 @@ except Exception as e:
     # Don't re-raise for non-critical operations
 ```
 
+## Valid Exception Swallowing in This Codebase
+
+The following patterns are acceptable and used in the codebase:
+
+### **✅ VALID: Signal Handlers and Cleanup Operations**
+```python
+# From observability.py - signal handler cleanup
+except Exception as e:
+    logging.error(f"Error during signal handler flush: {e}")
+    # Don't re-raise - cleanup failures shouldn't crash the application
+```
+
+### **✅ VALID: Observability Operations (Metrics, Traces, Events)**
+```python
+# From observability_decorator.py - trace recording
+try:
+    traces.record_trace(...)
+except Exception as trace_error:
+    logger.error(f"Failed to record trace: {str(trace_error)}")
+    # Don't re-raise - observability failures shouldn't break business logic
+
+# From interceptors/events.py - event publishing
+except Exception as publish_error:
+    logger.warning(f"Failed to publish workflow end event: {publish_error}")
+    # Don't re-raise - event publishing is non-critical
+```
+
+### **✅ VALID: Cleanup in Finally Blocks**
+```python
+# From interceptors/cleanup.py - workflow cleanup
+finally:
+    try:
+        await workflow.execute_activity(cleanup, ...)
+    except Exception as e:
+        logger.warning(f"Failed to cleanup artifacts: {e}")
+        # Don't re-raise - cleanup failures shouldn't fail the workflow
+```
+
 ## Module-Specific Guidelines
 
 ### **Handlers (`application_sdk/handlers/`)**
 - **Critical**: Always re-raise exceptions after logging
 - **Include operation context** in error messages
 - **Use specific exception types** for different error scenarios
 
-### **Outputs (`application_sdk/outputs/`)**
-- **Critical**: Re-raise exceptions for data writing operations
-- **Non-critical**: May swallow exceptions for metrics/logging operations
-- **Resource cleanup**: Ensure files/connections are properly closed
-
-### **Inputs (`application_sdk/inputs/`)**
-- **Critical**: Re-raise exceptions for data reading operations
+### **I/O Module (`application_sdk/io/`)**
+- **Critical**: Re-raise exceptions for data reading/writing operations
 - **Include file/connection context** in error messages
 - **Handle specific I/O exceptions** appropriately
+- **Resource cleanup**: Ensure files/connections are properly closed
 
 ### **Observability (`application_sdk/observability/`)**
 - **Non-critical**: May swallow exceptions to prevent cascading failures

.cursor/rules/guidelines.mdc

@@ -1,5 +1,5 @@
 ---
-description:
+description: Meta-guidelines for writing and maintaining cursor rule files with proper structure and formatting
 globs: .cursor/rules/*.mdc
 alwaysApply: false
 ---

.cursor/rules/logging.mdc

@@ -1,10 +1,14 @@
 ---
-description:
+description: Logging standards and best practices using AtlanLoggerAdapter for structured logging
 globs: application_sdk/**/*.py
 alwaysApply: false
 ---
 # Logging Guidelines
 
+## Exceptions
+
+- **Observability Module**: The `application_sdk/observability/` module uses the standard `logging` module directly since it implements the `get_logger` wrapper. All other modules must use `get_logger`.
+
 - **Logger Configuration**
     - Use `AtlanLoggerAdapter` for all logging
     - Configure loggers using `get_logger(__name__)`

.cursor/rules/performance.mdc

@@ -186,29 +186,32 @@ for query in queries:
 #### **JSON Serialization Optimization**
 - **Rule**: Use efficient serialization libraries and avoid unnecessary conversions
 - **Anti-pattern**: Using default json module for large datasets
-- **Correct pattern**: Use orjson, ujson, or custom serializers
+- **Correct pattern**: Use orjson (already used throughout this codebase)
 
-**✅ DO: Efficient JSON serialization**
+**✅ DO: Efficient JSON serialization (from objectstore.py)**
 ```python
 import orjson  # Much faster than json
 
-# Use orjson for better performance
-data = orjson.dumps(large_dict, option=orjson.OPT_APPEND_NEWLINE)
+# Fast JSON parsing from object store response
+file_list = orjson.loads(response_data.decode("utf-8"))
 
-# Use streaming for large datasets
-def serialize_large_dataset(data_iterator):
-    for item in data_iterator:
-        yield orjson.dumps(item) + b'\n'
+# Efficient JSON encoding
+data = json.dumps({"prefix": prefix}).encode("utf-8") if prefix else ""
+```
+
+**✅ DO: orjson in I/O operations (from io/json.py)**
+```python
+# Fast DataFrame to JSON conversion
+for chunk in df.to_pandas(batch_size=chunk_size):
+    for record in chunk.to_dict(orient="records"):
+        json_line = orjson.dumps(record).decode("utf-8")
 ```
 
 **❌ DON'T: Inefficient serialization**
 ```python
 # REJECT: Using default json for large datasets
 import json
 data = json.dumps(large_dict)  # Slower than orjson
-
-# REJECT: Converting to string unnecessarily
-data = str(orjson.dumps(large_dict))  # Unnecessary conversion
 ```
 
 

.cursor/rules/standards.mdc

@@ -1,6 +1,6 @@
 ---
-description:
-globs:
+description: Core development standards including code formatting, documentation, testing, logging, exception handling, and performance
+globs: application_sdk/**/*.py
 alwaysApply: true
 ---
 # Development Standards
@@ -58,6 +58,12 @@ This document outlines the development standards for the Application SDK, includ
 - **Logging Standards**
     - logging standards are defined in [logging.mdc](mdc:.cursor/rules/logging.mdc)
 
+- **Exception Handling**
+    - exception handling standards are defined in [exception-handling.mdc](mdc:.cursor/rules/exception-handling.mdc)
+
+- **Performance Standards**
+    - performance standards are defined in [performance.mdc](mdc:.cursor/rules/performance.mdc)
+
 ## Security, Performance, and Resource Management
 
 - **Security Considerations**