DataSphereStudio installation involves both frontend and backend separately. You should prepare the installation environment before installation.
Due to the richness of its components, DataSphereStudio can be installed as a simple version or a standard version, different with other softwares. The differences are mentioned as below:
-
Simple Version Minimal environment dependencies and only basic components need to be prepared：Linkis、JAVA、MYSQL、Nginx. Then you can immediately enjoy the integrated Scriptis(for data development), real-time workflow execution, visualization, email delivery, datacheck and eventcheck services. Click to enter the simple version DSS environment configuration preparation
-
Standard Version DSS also implements the integration of many external systems, such as Qualitis and Azkaban. If you want to use these external systems, you need to install and start the Qualities and Azkaban services before installing the simple version, and ensure that they can provide services properly. The IP and port of the corresponding service should be specified in the configuration. Click me to enter the standard version DSS environment configuration preparation
DSS environment configuration can be divided into three steps, including basic software installation, backend environment configuration, and frontend environment configuration. The details are as below:
Linkis standard version (above 0.9.1). How to install Linkis
JDK (above 1.8.0_141). How to install JDK
MySQL (5.5+). How to Install MySQL
Nginx. How to Install Nginx
- For example: the deployment user is hadoop. Create a deployment user on the deployment machine and use this user for installation.
sudo useradd hadoop
(Note: It is better for users to gain sudo permission and login to the machine without password). How to configure SSH passwordless login
On the DSS release page click here to enter the download page, download the corresponding installation package. First decompress the installation package to the installation directory and then modify the configuration of the decompressed file
tar -xvf wedatasphere-dss-x.x.x-dist.tar.gz
Note: If the installation package was compiled by the user, you need to copy the visualis-server installation package to the share/visualis-server folder in the DSS installation directory for automated installation.
vi conf/config.sh
<!--Note: The following are the basic DSS configuration items. If you want to experience more enterprise features, please install the standard DSS.-->
deployUser=hadoop #Specify the deployment user
DSS_INSTALL_HOME=/appcom/Install/DSS #Specify the DSS installation directory
EUREKA_INSTALL_IP=127.0.0.1 #Linkis EUREKA Service host IP address
EUREKA_PORT=20303 #Linkis EUREKA Service port number
WORKSPACE_USER_ROOT_PATH=file:///tmp/Linkis #Specify the user's root directory to store the user's script files and log files. This is the user's working space.
RESULT_SET_ROOT_PATH=hdfs:///tmp/linkis #Result set file path, used to store the result set file of the job
#Used for DATACHECK
HIVE_META_URL=jdbc:mysql://127.0.0.1:3306/linkis?characterEncoding=UTF-8
HIVE_META_USER=xxx
HIVE_META_PASSWORD=xxx vi conf/db.sh #Set the database connection information of DSS-Server and Eventchecker AppJoint.
MYSQL_HOST=
MYSQL_PORT=
MYSQL_DB=
MYSQL_USER=
MYSQL_PASSWORD=Click release to download the corresponding installation package and extract it to the installation directory:
unzip wedatasphere-dss-web-x.x.x-dist.zipNote: If the DSS frontend installation package was compiled by the user, you need to copy the visualis frontEnd installation package to the dss/visualis folder in the DSS frontEnd-end installation directory for automated installation.
Enter the frontend working directory and edit the file in this directory
vi conf/config.sh
Change the dss frontend port and backend linkis gateway IP address/port
# Configuring frontEnd-end ports
dss_port="8088"
# URL of the backend linkis gateway
linkis_url="http://127.0.0.1:9001"
# dss ip address
dss_ipaddr=$(ip addr | awk '/^[0-9]+: / {}; /inet.*global/ {print gensub(/(.*)\/(.*)/, "\\1", "g", $2)}')
The environment is ready, click me to enter **4. Installation and use
The standard DSS environment preparation is also divided into three parts, the frontEnd-end and back-end basic software installation, back-end environment preparation, and frontEnd-end environment preparation. The details are as follows:
Linkis standard version (above 0.9.1), How to install Linkis
JDK (above 1.8.0_141), How to install JDK
MySQL (5.5+), How to Install MySQL
Nginx, How to Install Nginx
Qualitis, How to Install Qualitis
Azkaban, How to Install Azkaban
Note: To support Azkaban scheduling, linkis-jobtype needs to be installed. Please click the Linkis jobType installation document.
- For example: the deployment user is hadoop account Create a deployment user on the deployment machine and use that user for installation.
sudo useradd hadoop
Note: It is best for users to have sudo permissions and log in to this machine without password. How to configure SSH passwordless login
From the DSS released click here to enter the download page, download the corresponding installation package. First decompress the installation package to the installation directory and modify the configuration of the decompressed file
tar -xvf wedatasphere-dss-x.x.x-dist.tar.gz
Note: If the installation package is compiled by the user, you need to copy the visualis-server installation package to the share/visualis-server folder in the DSS installation directory for automated installation.
vi conf/config.sh
<!--Note: The following are mandatory configuration items for DSS. Please ensure that external services are available.-->
deployUser=hadoop #Specify the deployment user
DSS_INSTALL_HOME=/appcom/Install/DSS #Specify DSS installation directory
EUREKA_INSTALL_IP=127.0.0.1 #Linkis EUREKA Service host IP address
EUREKA_PORT=20303 #Linkis EUREKA Service port number
WORKSPACE_USER_ROOT_PATH=file:///tmp/Linkis #Specify the user's root directory and store the user's script files and log files. This is the user's working space.
RESULT_SET_ROOT_PATH=hdfs:///tmp/linkis # Result set file path, used to store the result set file of the job
WDS_SCHEDULER_PATH=file:///appcom/tmp/wds/scheduler #Azkaban project storage directory
#1、for DATACHECK
HIVE_META_URL=jdbc:mysql://127.0.0.1:3306/linkis?characterEncoding=UTF-8
HIVE_META_USER=xxx
HIVE_META_PASSWORD=xxx
#2、for Qualitis
QUALITIS_ADRESS_IP=127.0.0.1 #QUALITIS service IP address
QUALITIS_ADRESS_PORT=8090 #QUALITIS service port number
#3、for AZKABAN
AZKABAN_ADRESS_IP=127.0.0.1 #AZKABAN service IP address
AZKABAN_ADRESS_PORT=8091 #AZKABAN service port number vi conf/db.sh # Set the connection information between the DSS-Server and the EventChecker AppJoint database。
MYSQL_HOST=
MYSQL_PORT=
MYSQL_DB=
MYSQL_USER=
MYSQL_PASSWORD=Click releaseto select the installation package to download and extract it in the installation directory:
unzip wedatasphere-DataSphereStudio-x.x.x-dist.zip
Note: If the DSS frontEnd-end installation package is compiled by the user, you need to copy the visualis frontEnd installation package to the dss/visualis folder in the DSS frontEnd installation directory for automated installation.
Enter the frontEnd working directory and edit in this directory
vi conf/config.sh
Change the dss frontEnd port and BackEnd linkis gateway IP address and port
# Configuring frontEnd-end ports
dss_port="8088"
# URL of the backend linkis gateway
linkis_url="http://127.0.0.1:9001"
# dss ip address
dss_ipaddr=$(ip addr | awk '/^[0-9]+: / {}; /inet.*global/ {print gensub(/(.*)\/(.*)/, "\\1", "g", $2)}')
The environment is ready, click me to enter Four Installation and use
sh bin/install.sh-
The install.sh script will ask you about the installation mode. The installation mode is the simple mode or the standard mode. Please select the appropriate installation mode according to your prepared environment. Both the simple mode and the standard mode will check the mysql service. The standard mode will also check the Qualities service and the Azkaban service.
-
The install.sh script will ask if you need to initialize the database and import metadata. Because I am worried that the user repeatedly executes the install.sh script to clear the user data in the database, when the install.sh is executed, the user will be asked whether to initialize the database and import metadata. You must select Yes for the first installation.
-
The install.sh script will ask you if you need to initialize the library tables that davinci depends on. If you have not installed davinci, you need to initialize the table. If you have already installed davinci, you don't need to initialize it again. Because I worry that the user will clear the davinci data already installed in mysql, so when install.sh is executed, the user will be asked if it needs to be initialized. You must select Yes for the first installation.
Check whether the installation is successful by viewing the log information printed on the console. If there is an error message, you can view the specific cause of the error. You can also get answers to your questions by viewing our FAQ.
Execute the following command in the installation directory to start all services:
./bin/start-all.sh > start.log 2>start_error.logYou can view the success of the service startup on the Eureka interface, and check the method:
-
Use http://${EUREKA_INSTALL_IP}:${EUREKA_PORT}, Open it in your browser to see if the service is successfully registered.
-
If you did not specify EUREKA_INSTALL_IP and EUREKA_INSTALL_IP in config.sh, the HTTP address is: http://127.0.0.1:20303
-
As shown in the figure below, if the following microservices appear on your Eureka homepage, the services are successfully started and services can be provided normally:
Click release to select the installation package to download and extract it in the installation directory:
unzip wedatasphere-dss-web-x.x.x-dist.zipEnter the frontEnd-end working directory and execute the installation script
sh bin/install.shAfter execution can be accessed directly through Google Chrome：
http://dss_ipaddr:dss_portdss_port is the port configured in config.sh, and dss_ipaddr is the IP of the installation machine.
If the access failed: You can check the log of install.log to see which step went wrong
If you want to manually install the frontend, the steps are as follows:
sudo vi /etc/nginx/conf.d/dss.confAdd the following configurations:
server {
listen 8080;# Access port
server_name localhost;
#charset koi8-r;
#access_log /var/log/nginx/host.access.log main;
location /dss/visualis {
root /appcom/Install/DSS/frontEnd; # visualis Static file directory
autoindex on;
}
location / {
root /appcom/Install/DSS/frontEnd/dist; # Directory for unpacking the frontEnd package
index index.html index.html;
}
location /ws {#webSocket configuration support
proxy_pass http://192.168.xxx.xxx:9001;#linkis-gateway Service ip port
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
location /api {
proxy_pass http://192.168.xxx.xxx:9001; # linkis-gateway Service ip port
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header x_real_ipP $remote_addr;
proxy_set_header remote_addr $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_connect_timeout 4s;
proxy_read_timeout 600s;
proxy_send_timeout 12s;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection upgrade;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
/appcom/Install/DSS/frontEnd; # frontend package installation directory
Note: To manually install the DSS frontend , you need to go to the dss/visualis folder of the DSS frontend installation directory and decompress the visualis frontEnd installation package.
sudo systemctl restart nginx
http://nginx_ip:nginx_port
How to use DSS in detail, click me to enter DSS user manual.
(1)User token is empty
sudo vi dss-server/conf/token.properties
Add login user
xxx=xxx
(2)visualis execution error
Caused by: java.lang.Exception: /data/DSSInstall/visualis-server/bin/phantomjsis not executable!
Download the driver and put the phantomjs binary file in the bin directory of visualis-server.
(3)Upload file size limit
sudo vi /etc/nginx/nginx.conf
Change upload size
client_max_body_size 200m
(4)Interface timed out
sudo vi /etc/nginx/conf.d/dss.conf
Change the interface timeout
proxy_read_timeout 600s