Scaling and Securing Passport: Enhancing Infrastructure for High-Impact Growth

[erichfi]

Scaling and Securing Passport: Enhancing Infrastructure for High-Impact Growth

Overview

This milestone focuses on enhancing the security measures and scaling the infrastructure of our project to handle increased load efficiently. The timeline allocated for this phase is from April 25th to May 7th, totaling 19 Dev Days.

Revised Tasks and Priorities

Automating Data Sync

1. Automating Data Sync: Parquet File Export Integration for Passport Scores with Open Source Observer #2353: Automate the synchronization process to enhance reliability and timeliness of data exchanges.

Monitoring Enhancements

2. Infra Week: Monitoring of Public and Private APIs #2402: Ensure robust monitoring to detect disruptions or performance issues.
3. Review & Replace Existing Alarms #2228: Update and streamline alarm systems to improve responsiveness and relevance.
4. Monitoring of Cron Jobs #2404: Set up monitoring for all scheduled tasks to ensure they run as expected without failures.
5. Monitoring Database Performance #2403: Implement detailed database performance monitoring to anticipate and mitigate potential issues.

Database Model Changes

6. Change to db models for data models #2412: Change to db models for data models.

Security and Access Management

6. Investigate Passport IAM Errors and AWS Alarm  #2183: Address identity and access management errors to maintain system integrity.

Comprehensive Testing

7. End-to-End Testing of Passport App #2405: Conduct thorough testing to ensure all components function seamlessly together.

Resource Management

8. Resource Tagging #2406: Improve resource tracking and cost management through systematic tagging.

Load Testing

• Load Testing (Pending Tim's Availability): When Tim gets back online, he will prioritize load testing to ensure the infrastructure's capability to handle expected traffic. This is crucial to verify our system’s resilience and performance under high load conditions.

Future Considerations

These tasks are critical for long-term strategic goals but are not prioritized in the current milestone due to either complexity or lower immediate impact:

• Set Up Aurora Serverless DB Settings: Postponed due to the need for thorough evaluation and potential risks.
• Clean Up Pager Duty Alarms: Integrated into the review of existing alarms but specific clean-up tasks may be deferred.
• Pulumi Security Updates: Including WAF setup and AWS Shield Basic—pending further discussion on implications for partners and testing methodologies.
• Restrict SSH Access: Important for security but not critical for immediate implementation.
• Fix Docker Access: Essential for operational integrity but can be addressed after more pressing security tasks.
• Document Processes and Coding Guidelines
• Preview Step in Pipeline for Pulumi
• Automate Running Migrations
• Manage Access Keys
• Document Deployment Processes
• Map Out Architecture
• Map Out DevOps Cycle
• Rollback Github Actions

Strategic Focus

The focus remains on leveraging the 80/20 principle to prioritize tasks that offer the most significant impact on security and scalability with minimal resource expenditure. The immediate goal is to ensure the infrastructure is robust enough to handle anticipated traffic surges without compromising security or performance.

Expected Outcomes

• Enhanced security across all levels of the infrastructure.
• Improved monitoring and response capabilities.
• Streamlined and efficient data synchronization with open-source observers.
• Comprehensive testing to ensure functionality and readiness.

This updated milestone draft aims to reflect the current priorities and strategic adjustments necessary for successful project execution, with a clear path outlined for future enhancements and optimizations.

[erichfi]

Next Steps:

• Review and Planning: @nutrina and @larisa17 are tasked with reviewing the notes thoroughly. Please collaborate closely with the engineering team to determine the feasibility and priority of the proposed tasks within the allocated timeframe. Should additional time be necessary, we may consider splitting this up into two chunks and doing the second / third sprint on this in the future.
• Implementation and Tracking: Following the above discussions, @erichfi will formalize this document into a structured checklist. This checklist will be integrated into GitHub with linked issues, organized according to the agreed-upon priorities in the development backlog.

[nutrina]

@erichfi we have prioritised the list and created issues.

So in the order of priorities we have the following list:

Monitoring

Also, as part of each ticket, update this Notion to reflect the latest state: https://www.notion.so/gitcoin/Passport-Monitors-PD-Alarms-444bfbe603d146ecbdd54211e1646957
The Notion page should provide an overview of the monitoring topic.

• Infra Week: Monitoring of Public and Private APIs #2402
• Review & Replace Existing Alarms #2228
• Monitoring of Cron Jobs #2404
• Monitoring Database Performance #2403
• Investigate Passport IAM Errors and AWS Alarm  #2183

End-to-end testing

• End-to-End Testing of Passport App #2405

Tags

• Resource Tagging #2406

[erichfi]

Prio

Top priority
#2402
#2228
#2404
#2403
OSS

#2183
#2405
#2406

Tim Prio:
Load Testing
Then rest of backlog

[erichfi]

Aligned to not inlcude #2406, #2403, #2405, #2409

[nutrina]

Putting #2402 back into the backlog for now.
The current work is merge but it is not finalised yet.

[nutrina]

Open topics for next infra week:

1. finalise Infra Week: Monitoring of Public and Private APIs #2402
2. Investigating private endpoints / APIs in backend #2470
3. [LOAD TEST] Move APIs updating the passport metadata in ceramic_cache to lambdas #2471
4. [LOAD TEST] Enable load testing to be run as ECS Task #2472
5. [LOAD TEST] Investigate timeouts for EVM flow in load testing of IAM service #2473
6. [LOAD TEST] add metadata requests (setting composedb stream_id) to the load tests #2474
7. Switch to AWS Aurora #2475