From bb3c2a57a33a5008d9b088695f7228789233d049 Mon Sep 17 00:00:00 2001 From: Fan Su Date: Thu, 25 Apr 2024 13:26:02 +0800 Subject: [PATCH] docs(others) batch proxy data source creation --- docs/en/user-guide/appendix-database-proxy.md | 108 ++++++++++++++---- docs/zh/user-guide/appendix-database-proxy.md | 97 +++++++++++++--- 2 files changed, 164 insertions(+), 41 deletions(-) diff --git a/docs/en/user-guide/appendix-database-proxy.md b/docs/en/user-guide/appendix-database-proxy.md index 1c078379..2e2dfa94 100644 --- a/docs/en/user-guide/appendix-database-proxy.md +++ b/docs/en/user-guide/appendix-database-proxy.md @@ -1,35 +1,97 @@ -## Using EC2 to configure database agents +## Configuring Database Proxy on EC2 -Create one EC2 instance as the database proxy. Install Nginx to set port forwarding. For example: +### Create and log in to the proxy EC2 machine, configure forwarding ports +Some users' database Security Groups have restrictions that only allow fixed IP access. In this case, users need an EC2 as a Proxy to provide a fixed IP. +Next, we will create an EC2 instance as a database proxy, install Nginx software, and set up port forwarding. When making a data source connection, SDPS connects to EC2 and makes a JDBC connection to the database through this EC2. -### Step 1:Install -`sudo yum install nginx nginx-mod-stream` -### Step 2:Start -`sudo systemctl start nginx` -### Step 3:View status -`systemctl status nginx` -### Step 4:Edit /etc/nginx/nginx.conf -`sudo vim /etc/nginx/nginx.conf` -Add content similar to the following at the end of the file -``` +##### Step 1: Create an EC2 Instance +- In the EC2 console. Create an EC2 in the VPC where SDP is located, to be used as a proxy server. +- Configure the EC2 Security Group: Add an Inbound Rule, allowing all TCP entries from the following two security groups: SDPS-CustomDB, Stack Name-RDSRDSClient + +##### Step 2: Install Nginx software on EC2 +- Copy the EC2's .pem file to the Bastion host for logging into the proxy server. +- From your Bastion host, log in to EC2 using SSH, for example: + `ssh -i /path/to/your/key.pem ec2-user@ec2-private-ip` +- Run the following commands in sequence to install and start Nginx. + - Installation: `sudo yum install nginx nginx-mod-stream` + - Start: `sudo systemctl start nginx` + - Check status: `systemctl status nginx` + +##### Step 3: Configure Nginx software + - Open the configuration file: `vim /etc/nginx/nginx.conf` + - Edit the configuration file: +```python +# Replace the default nginx.conf file content with code. You need to make necessary adjustments. stream { upstream backend1 { - server 10.0.34.171:3306 max_fails=3 fail_timeout=30s; # Server address can use domain name + server 10.0.34.171:3306 max_fails=3 fail_timeout=30s; + # You need to modify the server address to the IP:Port of your target database, you can also use DomainName:Port format. } server { - listen 3306; - proxy_connect_timeout 1s; + listen 7001; # This EC2 port is used for forwarding requests (Port) + proxy_connect_timeout 2s; + proxy_timeout 3s; proxy_pass backend1; } } ``` -### Step 5: Reload configuration file -`sudo nginx -s reload` -### Step 6: Add 2 security groups to the instance -Add Rule to the Proxy security group to allow all TCP entries from the following two security groups:`SDPS-CustomDB`、`StackName-RDSRDSClient` -### Step 7: (Optional) Is the local testing agent effective -``` -sudo yum install telnet -telnet 127.0.0.1 7001 +!!! Info How to edit the configuration file when there are many databases? + If you need to configure multiple port forwarding, you can use the SDP **batch create data source** feature, and create the Nginx configuration file through the template. See below Appendix. + +##### Step 5: Reload the configuration file +Save the configuration file and reload it to take effect: `sudo nginx -s reload` + +##### Step 7: Test if the proxy EC2 port forwarding is effective (Optional) +On EC2, install telnet, and test if the local 7001 port can be pinged. +`sudo yum install telnet` +`telnet 127.0.0.1 7001` +If configured correctly, you should see the following log: +```java + Trying 127.0.0.1... + Connected to 127.0.0.1. ``` +Now, you have completed the configuration of the proxy server, you can go back to the SDP UI to manually add or batch add data sources. + +--- +### Appendix: Batch create data sources forwarded from the proxy server + +##### Step 1: Download the template +From the SDP UI, download the template for batch creating data sources. + +##### Step 2: Edit the excel file +Fill in the data sources you need to scan. + +| InstanceName | SSL | Description | JDBC_URL | JDBC_Databases | SecretARN | Username | Password | AccountID | Region | ProviderID | +|---------------------|-----|--------------------------------------------------------------------|----------------------------------------------|----------------|-----------|----------|------------|----------------------|----------------|------------| +| test-instance-7001 | 1 | xxxx1.sql.db.com:23297 | jdbc:mysql://172.31.48.6:7001 | | | root | Temp123456! | 123456789 | ap-guangzhou-1 | 4 | +| test-instance-7002 | 1 | xxxx2.sql.db.com:3306 | jdbc:mysql://172.31.48.6:7002 | | | root | Temp123456! | 123456789 | ap-guangzhou-1 | 4 | + +##### Step 3: Generate the Nginx software's config file +(On your local machine) Open the Excel software, in the menu bar click Tools → Macro → Visual Basic Editor. + +Click the run button, and a config.txt file will be generated in the directory where the Excel file is located. + +```java +// This is a sample. +// Forward through EC2's 7001 port to xxxx1.sql.db.com:23297 database. +// Forward through EC2's 7002 port to xxxx2.sql.xxdb.com:3306 database. +stream { + upstream backend1 { + server xxxx1.sql.db.com:23297 max_fails=3 fail_timeout=30s; + } + server { + listen 7001; + proxy_connect_timeout 2s; + proxy_pass backend1; + } + upstream backend2 { + server xxxx2.sql.db.com:3306 max_fails=3 fail_timeout=30s; + } + server { + listen 7002; + proxy_connect_timeout 2s; + proxy_pass backend2; + } +} +``` \ No newline at end of file diff --git a/docs/zh/user-guide/appendix-database-proxy.md b/docs/zh/user-guide/appendix-database-proxy.md index 23ee86fd..95e1d5d6 100644 --- a/docs/zh/user-guide/appendix-database-proxy.md +++ b/docs/zh/user-guide/appendix-database-proxy.md @@ -1,18 +1,27 @@ ## 使用EC2配置数据库代理 +g +### 创建并登录到代理EC2机器，配置转发端口 +有一些用户的数据库Security Group设置了限制，只允许固定IP访问。这个时候，用户需要一个EC2作为Proxy来提供固定的IP。 -创建1个EC2实例作为数据库代理。安装Nginx 设置端口转发。例如: +接下来，我们将创建1个EC2实例作为数据库代理，并安装Nginx软件并设置端口转发。在做数据源连接时，SDPS连接EC2，并通过这台EC2目标对数据库进行JDBC连接。 +##### Step 1:创建EC2实例 +- 在EC2控制台。在SDP所在的VPC创建一台EC2机器, 作为代理服务器。 +- 配置EC2的安全组（Security Group）：添加Inbound Rule，允许以下2个安全组的所有TCP进入：SDPS-CustomDB、堆栈名-RDSRDSClient -### Step 1:安装 -`sudo yum install nginx nginx-mod-stream` -### Step 2:启动 -`sudo systemctl start nginx` -### Step 3:查看状态 -`systemctl status nginx` -### Step 4:编辑/etc/nginx/nginx.conf文件 -`sudo vim /etc/nginx/nginx.conf` -在文件末尾添加类似以下内容 -``` +##### Step 2:在EC2上安装Nginx软件 +- 将EC2的pem文件拷贝到Bastion host上，用于登陆代理服务器。 +- 从你的Bastion host上，使用SSH方式登陆到EC2。例如： + `ssh -i /path/to/your/key.pem ec2-user@ec2-private-ip` +- 依次运行下面的命令，安装并启动Nginx。 + - 安装： `sudo yum install nginx nginx-mod-stream` + - 启动：`sudo systemctl start nginx` + - 查看状态：`systemctl status nginx` +##### Step 3:配置Nginx软件 + - 打开配置文件：`vim /etc/nginx/nginx.conf` + - 编辑配置文件： +```python +# 用代码替换默认的nginx.conf文件内容。您需要进行必要的调整。 stream { upstream backend1 { server 10.0.34.171:3306 max_fails=3 fail_timeout=30s; # server地址可以使用域名 @@ -24,12 +33,64 @@ stream { } } ``` -### Step 5: 重新加载配置文件 -`sudo nginx -s reload` -### Step 6: 为实例添加安全组 -Proxy安全组添加Rule，允许以下2个安全组的所有TCP进入：`SDPS-CustomDB`、`堆栈名-RDSRDSClient` -### Step 7: （可选）本地测试代理是否生效 +!!! Info 数据库太多时，如何编辑配置文件？ + 如果您需要配置多个端口转发，可以使用SDP **批量创建数据源**功能，并通过模版来创建Nginx配置文件。见下面附录。 + +##### Step 5: 重新加载配置文件 +保存配置文件，并重新加载使其生效：`sudo nginx -s reload` + +##### Step 7: 测试代理EC2端口转发是否生效 （可选） +在EC2上，安装telnet，并测试本机的7001端口是否可以ping通。 +`sudo yum install telnet` +`telnet 127.0.0.1 7001` +如果已经正确配置，您应该看到如下log： +```java + Trying 127.0.0.1... + Connected to 127.0.0.1. ``` -sudo yum install telnet -telnet 127.0.0.1 7001 +至此，您已经配置完代理服务器的配置，可以回到SDP UI上手动添加或者批量添加数据源了。 + +--- +### 附录：批量创建从代理服务器转发的数据源 + +##### Step 1: 下载模版 +从SDP UI上面，下载批量创建数据源的模版。 + +##### Step 2: 编辑excel文件 +填入您所需要扫描的数据源。 + +| InstanceName | SSL | Description | JDBC_URL | JDBC_Databases | SecretARN | Username | Password | AccountID | Region | ProviderID | +|---------------------|-----|--------------------------------------------------------------------|----------------------------------------------|----------------|-----------|----------|------------|----------------------|----------------|------------| +| test-instance-7001 | 1 | xxxx1.sql.db.com:23297 | jdbc:mysql://172.31.48.6:7001 | | | root | Temp123456! | 123456789 | ap-guangzhou-1 | 4 | +| test-instance-7002 | 1 | xxxx2.sql.db.com:3306 | jdbc:mysql://172.31.48.6:7002 | | | root | Temp123456! | 123456789 | ap-guangzhou-1 | 4 | + + +##### Step 3: 生成Nginx软件的config文件 +（在本地）打开excel软件，菜单栏点击 Tools → Marco → Visual Basic Editor 功能。 + + +点击运行按钮，会看到excel文件所在目录下生成一个config.txt文件。 + +```java +// 这个是一个样例。 +// 通过EC2的7001端口转发至xxxx1.sql.db.com:23297数据库。 +// 通过EC2的7002端口转发至xxxx2.sql.xxdb.com:3306数据库。 +stream { + upstream backend1 { + server xxxx1.sql.db.com:23297 max_fails=3 fail_timeout=30s; + } + server { + listen 7001; + proxy_connect_timeout 2s; + proxy_pass backend1; + } + upstream backend2 { + server xxxx2.sql.db.com:3306 max_fails=3 fail_timeout=30s; + } + server { + listen 7002; + proxy_connect_timeout 2s; + proxy_pass backend2; + } +} ```