Skip to content

Commit

Permalink
Merge branch 'main' of github.com:xumoyan/routing-api
Browse files Browse the repository at this point in the history 
* 'main' of github.com:xumoyan/routing-api: (155 commits)
  fix: bump lambda version to pick up new zksync secret (Uniswap#730)
  feat: support routing on zksync (Uniswap#729)
  feat: double sampling tenderly node estimate gas api (Uniswap#728)
  chore: remove quicknodetest_1 sampling (Uniswap#727)
  feat: shadow tenderly new node endpoint (Uniswap#726)
  fix: add quicknodetest_1 into the secret fetch list (Uniswap#725)
  fix: bump lambda version to pick up secrets in env var (Uniswap#724)
  fix: quicknode test key naming (Uniswap#723)
  chore: shadow sampling mainnet traffic to quicknode test key (Uniswap#722)
  chore(subgraph-cache): Increase retries on decentralized network endpoints (Uniswap#721)
  chore: remove nirvana from sampling and 5x alchemy sampling traffic (Uniswap#720)
  chore(subgraph): Use decentralized network arbitrum subgraph (Uniswap#719)
  chore: increase alchemy shadow sampling by 10x (Uniswap#718)
  fix(cache job): Switch to the uniswap-v2-dev subgraph (Uniswap#717)
  fix(caching): Try older version of subgraph (Uniswap#716)
  fix(caching): Use decentralized network v2 subgraph on Mainnet (Uniswap#715)
  chore(cron): Migrate Polygon to Alchemy (Uniswap#714)
  Revert "fix: manually filter out fantom cached routes (Uniswap#712)" (Uniswap#713)
  fix: manually filter out fantom cached routes (Uniswap#712)
  fix: bump lambda version to pick up the new s3 subgraph pool files (Uniswap#711)
  ...

# Conflicts:
#	bin/app.ts
#	bin/stacks/routing-api-stack.ts
  • Loading branch information
xumoyan committed Aug 20, 2024
2 parents 2d23165 + 45f2160 commit 4bbac49
Show file tree
Hide file tree
Showing 49 changed files with 3,636 additions and 1,621 deletions.
7 changes: 5 additions & 2 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ The best way to develop and test the API is to deploy your own instance to AWS.
2. Create .env file in the root directory of the project with :
```
THROTTLE_PER_FIVE_MINS = '' # Optional
JSON_RPC_PROVIDER_{CHAIN ID} = { RPC Provider}
WEB3_RPC_{CHAIN ID} = { RPC Provider}
# RPC Providers must be set for the following CHAIN IDs:
# MAINNET = 1
# ROPSTEN = 3
Expand All @@ -32,9 +32,12 @@ The best way to develop and test the API is to deploy your own instance to AWS.
# BNB = 56
# BASE = 8453
# BLAST = 81457
# ZORA = 7777777
# ZKSYNC = 324
TENDERLY_USER = '' # For enabling Tenderly simulations
TENDERLY_PROJECT = '' # For enabling Tenderly simulations
TENDERLY_ACCESS_KEY = '' # For enabling Tenderly simulations
TENDERLY_NODE_API_KEY = '' # For enabling Tenderly node-level RPC access
```
3. Install and build the package
```
Expand Down Expand Up @@ -91,7 +94,7 @@ Integration tests run against a local DynamoDB node deployed using [dynamodb-loc

The end-to-end tests fetch quotes from your deployed API, then execute the swaps on a Hardhat mainnet fork.

1. First deploy your test API using the intructions above. Then update your `.env` file with the URL of the API, and the RPC URL of an archive node:
1. First deploy your test API using the instructions above. Then update your `.env` file with the URL of the API, and the RPC URL of an archive node:

```
UNISWAP_ROUTING_API='...'
Expand Down
67 changes: 53 additions & 14 deletions bin/app.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,10 @@ export class RoutingAPIStage extends Stage {
tenderlyUser: string
tenderlyProject: string
tenderlyAccessKey: string
tenderlyNodeApiKey: string
unicornSecret: string
alchemyQueryKey?: string
decentralizedNetworkApiKey?: string
}
) {
super(scope, id, props)
Expand All @@ -52,7 +55,10 @@ export class RoutingAPIStage extends Stage {
tenderlyUser,
tenderlyProject,
tenderlyAccessKey,
tenderlyNodeApiKey,
unicornSecret,
alchemyQueryKey,
decentralizedNetworkApiKey,
} = props

const { url } = new RoutingAPIStack(this, 'RoutingAPI', {
Expand All @@ -69,7 +75,10 @@ export class RoutingAPIStage extends Stage {
tenderlyUser,
tenderlyProject,
tenderlyAccessKey,
tenderlyNodeApiKey,
unicornSecret,
alchemyQueryKey,
decentralizedNetworkApiKey,
})
this.url = url
}
Expand Down Expand Up @@ -110,35 +119,48 @@ export class RoutingAPIPipeline extends Stack {
// Load RPC provider URLs from AWS secret (for RPC Gateway)
const RPC_GATEWAY_PROVIDERS = [
// Optimism
'INFURA_10',
// 'INFURA_10',
'QUICKNODE_10',
'ALCHEMY_10',
// Polygon
'QUICKNODE_137',
'INFURA_137',
// 'INFURA_137',
'ALCHEMY_137',
// Celo
'QUICKNODE_42220',
'INFURA_42220',
// 'INFURA_42220',
// Avalanche
'INFURA_43114',
// 'INFURA_43114',
'QUICKNODE_43114',
'NIRVANA_43114',
// BNB
'QUICKNODE_56',
// Base
'QUICKNODE_8453',
'INFURA_8453',
// 'INFURA_8453',
'ALCHEMY_8453',
'NIRVANA_8453',
// Sepolia
'INFURA_11155111',
// 'INFURA_11155111',
'ALCHEMY_11155111',
// Arbitrum
'INFURA_42161',
// 'INFURA_42161',
'QUICKNODE_42161',
'NIRVANA_42161',
'ALCHEMY_42161',
// Ethereum
// 'INFURA_1',
'QUICKNODE_1',
'NIRVANA_1',
'ALCHEMY_1',
'QUICKNODERETH_1',
// Blast
'QUICKNODE_81457',
// 'INFURA_81457',
// ZORA
'QUICKNODE_7777777',
// ZkSync
'QUICKNODE_324',
]
for (const provider of RPC_GATEWAY_PROVIDERS) {
jsonRpcProviders[provider] = "https://morning-alien-card.quiknode.pro/54d7a389bc802b3e771e92a514d961ddcd9c349a"
Expand All @@ -162,6 +184,7 @@ export class RoutingAPIPipeline extends Stack {
tenderlyProject: "",
tenderlyAccessKey: "",
unicornSecret: "",
tenderlyNodeApiKey: "",
})

const betaUsEast2AppStage = pipeline.addStage(betaUsEast2Stage)
Expand All @@ -185,6 +208,7 @@ export class RoutingAPIPipeline extends Stack {
tenderlyProject: '',
tenderlyAccessKey: '',
unicornSecret: '',
tenderlyNodeApiKey: '',
})

const prodUsEast2AppStage = pipeline.addStage(prodUsEast2Stage)
Expand Down Expand Up @@ -256,37 +280,51 @@ const jsonRpcProviders = {
WEB3_RPC_43114: process.env.WEB3_RPC_43114!,
WEB3_RPC_56: process.env.WEB3_RPC_56!,
WEB3_RPC_8453: process.env.WEB3_RPC_8453!,
WEB3_RPC_324: process.env.WEB3_RPC_324!,
// The followings are for RPC Gateway
// Optimism
INFURA_10: process.env.INFURA_10!,
// INFURA_10: process.env.INFURA_10!,
QUICKNODE_10: process.env.QUICKNODE_10!,
ALCHEMY_10: process.env.ALCHEMY_10!,
// Polygon
QUICKNODE_137: process.env.QUICKNODE_137!,
INFURA_137: process.env.INFURA_137!,
// INFURA_137: process.env.INFURA_137!,
ALCHEMY_137: process.env.ALCHEMY_137!,
// Celo
QUICKNODE_42220: process.env.QUICKNODE_42220!,
INFURA_42220: process.env.INFURA_42220!,
// INFURA_42220: process.env.INFURA_42220!,
// Avalanche
INFURA_43114: process.env.INFURA_43114!,
// INFURA_43114: process.env.INFURA_43114!,
QUICKNODE_43114: process.env.QUICKNODE_43114!,
NIRVANA_43114: process.env.NIRVANA_43114!,
// BNB
QUICKNODE_56: process.env.QUICKNODE_56!,
// Base
QUICKNODE_8453: process.env.QUICKNODE_8453!,
INFURA_8453: process.env.INFURA_8453!,
// INFURA_8453: process.env.INFURA_8453!,
ALCHEMY_8453: process.env.ALCHEMY_8453!,
NIRVANA_8453: process.env.NIRVANA_8453!,
// Sepolia
INFURA_11155111: process.env.INFURA_11155111!,
// INFURA_11155111: process.env.INFURA_11155111!,
ALCHEMY_11155111: process.env.ALCHEMY_11155111!,
// Arbitrum
INFURA_42161: process.env.INFURA_42161!,
// INFURA_42161: process.env.INFURA_42161!,
QUICKNODE_42161: process.env.QUICKNODE_42161!,
NIRVANA_42161: process.env.NIRVANA_42161!,
ALCHEMY_42161: process.env.ALCHEMY_42161!,
// Ethereum
// INFURA_1: process.env.INFURA_1!,
QUICKNODE_1: process.env.QUICKNODE_1!,
NIRVANA_1: process.env.NIRVANA_1!,
ALCHEMY_1: process.env.ALCHEMY_1!,
// Blast
QUICKNODE_81457: process.env.QUICKNODE_81457!,
// INFURA_81457: process.env.INFURA_81457!,
// Zora
QUICKNODE_7777777: process.env.QUICKNODE_7777777!,
// ZkSync
QUICKNODE_324: process.env.QUICKNODE_324!,
ALCHEMY_324: process.env.ALCHEMY_324!,
}

// Local dev stack
Expand All @@ -305,6 +343,7 @@ new RoutingAPIStack(app, 'RoutingAPIStack', {
tenderlyUser: process.env.TENDERLY_USER!,
tenderlyProject: process.env.TENDERLY_PROJECT!,
tenderlyAccessKey: process.env.TENDERLY_ACCESS_KEY!,
tenderlyNodeApiKey: process.env.TENDERLY_NODE_API_KEY!,
unicornSecret: process.env.UNICORN_SECRET!,
})

Expand Down
24 changes: 20 additions & 4 deletions bin/stacks/routing-api-stack.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import { RoutingDatabaseStack } from './routing-database-stack'
import { RpcGatewayDashboardStack } from './rpc-gateway-dashboard'
import { REQUEST_SOURCES } from '../../lib/util/requestSources'
import { TESTNETS } from '../../lib/util/testNets'
import { RpcGatewayFallbackStack } from './rpc-gateway-fallback-stack'

export const CHAINS_NOT_MONITORED: ChainId[] = TESTNETS
export const REQUEST_SOURCES_NOT_MONITORED = ['unknown']
Expand All @@ -43,7 +44,10 @@ export class RoutingAPIStack extends cdk.Stack {
tenderlyUser: string
tenderlyProject: string
tenderlyAccessKey: string
tenderlyNodeApiKey: string
unicornSecret: string
alchemyQueryKey?: string
decentralizedNetworkApiKey?: string
}
) {
super(parent, name, props)
Expand All @@ -63,13 +67,19 @@ export class RoutingAPIStack extends cdk.Stack {
tenderlyUser,
tenderlyProject,
tenderlyAccessKey,
tenderlyNodeApiKey,
unicornSecret,
alchemyQueryKey,
decentralizedNetworkApiKey,
} = props

const {
poolCacheBucket,
poolCacheBucket2,
poolCacheBucket3,
poolCacheKey,
poolCacheGzipKey,
poolCacheLambdaNameArray,

Check failure on line 82 in bin/stacks/routing-api-stack.ts

View workflow job for this annotation

GitHub Actions / Run tests 

'poolCacheLambdaNameArray' is declared but its value is never read.
tokenListCacheBucket,
} = new RoutingCachingStack(this, 'RoutingCachingStack', {
chatbotSNSArn,
Expand All @@ -78,6 +88,8 @@ export class RoutingAPIStack extends cdk.Stack {
pinata_key,
pinata_secret,
hosted_zone,
alchemyQueryKey,
decentralizedNetworkApiKey,
})

const {
Expand All @@ -88,13 +100,15 @@ export class RoutingAPIStack extends cdk.Stack {
cachedV3PoolsDynamoDb,
cachedV2PairsDynamoDb,
tokenPropertiesCachingDynamoDb,
rpcProviderStateDynamoDb,
rpcProviderHealthStateDynamoDb,
} = new RoutingDatabaseStack(this, 'RoutingDatabaseStack', {})

const { routingLambdaAlias } = new RoutingLambdaStack(this, 'RoutingLambdaStack', {
poolCacheBucket,
poolCacheBucket2,
poolCacheBucket3,
poolCacheKey,
poolCacheGzipKey,
jsonRpcProviders,
tokenListCacheBucket,
provisionedConcurrency,
Expand All @@ -103,14 +117,15 @@ export class RoutingAPIStack extends cdk.Stack {
tenderlyUser,
tenderlyProject,
tenderlyAccessKey,
tenderlyNodeApiKey,
routesDynamoDb,
routesDbCachingRequestFlagDynamoDb,
cachedRoutesDynamoDb,
cachingRequestFlagDynamoDb,
cachedV3PoolsDynamoDb,
cachedV2PairsDynamoDb,
tokenPropertiesCachingDynamoDb,
rpcProviderStateDynamoDb,
rpcProviderHealthStateDynamoDb,
unicornSecret,
})

Expand Down Expand Up @@ -165,8 +180,8 @@ export class RoutingAPIStack extends cdk.Stack {
priority: 0,
statement: {
rateBasedStatement: {
// Limit is per 5 mins, i.e. 120 requests every 5 mins
limit: throttlingOverride ? parseInt(throttlingOverride) : 120,
// Limit is per 5 mins, i.e. 200 requests every 5 mins
limit: throttlingOverride ? parseInt(throttlingOverride) : 200,
// API is of type EDGE so is fronted by Cloudfront as a proxy.
// Use the ip set in X-Forwarded-For by Cloudfront, not the regular IP
// which would just resolve to Cloudfronts IP.
Expand Down Expand Up @@ -224,6 +239,7 @@ export class RoutingAPIStack extends cdk.Stack {
})

new RpcGatewayDashboardStack(this, 'RpcGatewayDashboardStack')
new RpcGatewayFallbackStack(this, 'RpcGatewayFallbackStack', { rpcProviderHealthStateDynamoDb })

const lambdaIntegration = new aws_apigateway.LambdaIntegration(routingLambdaAlias)

Expand Down
27 changes: 25 additions & 2 deletions bin/stacks/routing-caching-stack.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import { Construct } from 'constructs'
import * as path from 'path'
import { chainProtocols } from '../../lib/cron/cache-config'
import { STAGE } from '../../lib/util/stage'
import { PoolCachingFilePrefixes } from '../../lib/util/poolCachingFilePrefixes'

export interface RoutingCachingStackProps extends cdk.NestedStackProps {
stage: string
Expand All @@ -24,29 +25,39 @@ export interface RoutingCachingStackProps extends cdk.NestedStackProps {
pinata_secret?: string
hosted_zone?: string
chatbotSNSArn?: string
alchemyQueryKey?: string
decentralizedNetworkApiKey?: string
}

export class RoutingCachingStack extends cdk.NestedStack {
public readonly poolCacheBucket: aws_s3.Bucket
public readonly poolCacheBucket2: aws_s3.Bucket
public readonly poolCacheBucket3: aws_s3.Bucket
public readonly poolCacheKey: string
public readonly poolCacheGzipKey: string
public readonly tokenListCacheBucket: aws_s3.Bucket
public readonly ipfsPoolCachingLambda: aws_lambda_nodejs.NodejsFunction
public readonly ipfsCleanPoolCachingLambda: aws_lambda_nodejs.NodejsFunction
public readonly poolCacheLambdaNameArray: string[] = []
public readonly alchemyQueryKey: string | undefined = undefined
public readonly decentralizedNetworkApiKey: string | undefined = undefined

constructor(scope: Construct, name: string, props: RoutingCachingStackProps) {
super(scope, name, props)

const { chatbotSNSArn } = props
const { chatbotSNSArn, alchemyQueryKey, decentralizedNetworkApiKey } = props

const chatBotTopic = chatbotSNSArn ? aws_sns.Topic.fromTopicArn(this, 'ChatbotTopic', chatbotSNSArn) : undefined

this.alchemyQueryKey = alchemyQueryKey
this.decentralizedNetworkApiKey = decentralizedNetworkApiKey

// TODO: Remove and swap to the new bucket below. Kept around for the rollout, but all requests will go to bucket 2.
this.poolCacheBucket = new aws_s3.Bucket(this, 'PoolCacheBucket', {

})
this.poolCacheBucket2 = new aws_s3.Bucket(this, 'PoolCacheBucket2')
this.poolCacheBucket3 = new aws_s3.Bucket(this, 'PoolCacheBucket3')

this.poolCacheBucket2.addLifecycleRule({
enabled: true,
Expand All @@ -62,7 +73,14 @@ export class RoutingCachingStack extends cdk.NestedStack {
expiration: cdk.Duration.days(365 * 10),
})

this.poolCacheKey = 'poolCache.json'
this.poolCacheBucket3.addLifecycleRule({
enabled: true,
// See the comment above for the reasoning behind this TTL.
expiration: cdk.Duration.days(365 * 10),
})

this.poolCacheKey = PoolCachingFilePrefixes.PlainText
this.poolCacheGzipKey = PoolCachingFilePrefixes.GzipText

const { stage, route53Arn, pinata_key, pinata_secret, hosted_zone } = props

Expand Down Expand Up @@ -107,7 +125,11 @@ export class RoutingCachingStack extends cdk.NestedStack {
environment: {
POOL_CACHE_BUCKET: this.poolCacheBucket.bucketName,
POOL_CACHE_BUCKET_2: this.poolCacheBucket2.bucketName,
POOL_CACHE_BUCKET_3: this.poolCacheBucket3.bucketName,
POOL_CACHE_KEY: this.poolCacheKey,
POOL_CACHE_GZIP_KEY: this.poolCacheGzipKey,
ALCHEMY_QUERY_KEY: this.alchemyQueryKey ?? '',
DCN_API_KEY: this.decentralizedNetworkApiKey ?? '',
chainId: chainId.toString(),
protocol,
timeout: timeout.toString(),
Expand All @@ -119,6 +141,7 @@ export class RoutingCachingStack extends cdk.NestedStack {
targets: [new aws_events_targets.LambdaFunction(lambda)],
})
this.poolCacheBucket2.grantReadWrite(lambda)
this.poolCacheBucket3.grantReadWrite(lambda)
const lambdaAlarmErrorRate = new aws_cloudwatch.Alarm(
this,
`RoutingAPI-SEV4-PoolCacheToS3LambdaErrorRate-ChainId${chainId}-Protocol${protocol}`,
Expand Down
Loading

0 comments on commit 4bbac49

Please sign in to comment.