From b73f02f3a790d5741090b382b1bd8d972f332c5d Mon Sep 17 00:00:00 2001 From: Rui Liu Date: Fri, 5 Jul 2024 18:39:39 +0800 Subject: [PATCH] Update doc (#50) --- README.md | 145 ++----------------------------------------------- host/README.md | 98 ++++++++++++++++++++++++--------- llm/README.md | 6 +- rdb/README.md | 1 - 4 files changed, 82 insertions(+), 168 deletions(-) diff --git a/README.md b/README.md index 852e3cc..9f831a0 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# ODCD (OpenTelemetry based Data Collector for Telemetry Data) +# ODCD (OpenTelemetry Data Collector for Telemetry Data) **[Semantic Convention](docs/semconv)** | **[Support](docs/support/README.md)** | @@ -7,146 +7,13 @@ **[License](LICENSE)** --- -ODCD (OpenTelemetry based Data Collector for Telemetry Data) is a collection of stanalone OpenTelemetry receivers for databases, systems, and apps. All implementations are based on predefined OpenTelemetry Semantic Conventions. A standard OTLP exporter is provided to forward the data from this "Data Collector" to a Telemetry backend or an OpenTelemetry Collector. +ODCD (OpenTelemetry Data Collector for Telemetry Data) is a collection of stanalone OpenTelemetry receivers for databases, systems, and apps. All implementations are based on predefined OpenTelemetry Semantic Conventions. A standard OTLP exporter is provided to forward the data from this "Data Collector" to a Telemetry backend or an OpenTelemetry Collector.

-# Data Collectors +# Data Collectors provided -## -### Requirements -- Data Collectors for Host **Java 11+** -- Data Collectors for Databases **Java 8+** -- Data Collectors for LLM **Java 11+** +- [OTel Data Collectors for Relational Databases](rdb/README.md) (**Java 8+**) +- [OTel Data Collectors for Host](host/README.md) (**Java 11+**) +- [OTel Data Collectors for LLM](llm/README.md) (**Java 11+**) -## Build & Installation - -1) Make sure Java SDK is installed. -```bash -java -version -``` - -2) Get the source code from `github.com`. -```bash -git clone https://github.com/instana/otel-dc.git -``` -- Data Collectors for Host -```bash -cd otel-dc/host -``` -- Data Collectors for Databases -```bash -cd otel-dc/rdb -``` -- Data Collectors for LLM -```bash -cd otel-dc/llm -``` -1) Build with Gradle -```bash -./gradlew clean build -``` -*Note: gradle 7.4 will be installed if you do not have it.* - -## Run - -- ### Run with source code (Take the database plugin as an example) - -1) Make sure code is built with Java SDK required version. - -2) Refine configuration file (config/config.yaml) according to your own database. Right now we provide Data Collector for following databases: - - DaMeng database - - Oceanbase - - Informix - - >*Note: We respect the OTel community conventions and consider more about popular database engines for conventionss. While we may create implementations from those databases without OTel support based on user requirements or developer contributions. * - -3) Start up your OTLP backend which accept OTLP connections. Right now we support following protocols: -- otlp/grpc -- otlp/http - -4) Option-1: Run the Data Collector with gradle -```bash -./gradlew run -``` -5) Option-2: Find the deployment package (e.g.:otel-dc-rdb-*.tar/otel-dc-host-*.tar/otel-dc-host-*.tar) generated by gradle in the "build/distributions/" directory, extract deployment files: -```bash -cd build/distributions/ -tar vxf otel-dc-rdb-*.tar -rm -f *.tar *.zip -cd otel-dc-rdb-* -``` -Refer to following chapter to run the Data Collector. - -- ### Run with release -Download the latest [Release package](https://github.com/instana/otel-dc/releases/tag/Release) according to the operating system. - -### Modify the configuration file. -Then, make sure following configuration files are correct for your environment.: - - config/config.yaml - - config/logging.properties - -Run the Data Collector with following command according to your current implentation: -- **database** -```bash -export DC_CONFIG=config/config.yaml -export JAVA_OPTS=-Dconfig/logging.properties -bin/otel-dc-rdb -``` -Or run Data Collector in background -```bash -export DC_CONFIG=config/config.yaml -export JAVA_OPTS=-Dconfig/logging.properties -nohup bin/otel-dc-rdb > /dev/null 2>&1 & -``` - - -Run the Data Collector with following command according to your current implentation: -- **host** -```bash -export DC_CONFIG=config/config.yaml -export JAVA_OPTS=-Dconfig/logging.properties -bin/otel-dc-host -``` -Or run Data Collector in background -```bash -export DC_CONFIG=config/config.yaml -export JAVA_OPTS=-Dconfig/logging.properties -nohup bin/otel-dc-host > /dev/null 2>&1 & -``` - -- **llm** -```bash -export DC_CONFIG=config/config.yaml -export JAVA_OPTS=-Dconfig/logging.properties -bin/otel-dc-llm -``` -Or run Data Collector in background -```bash -export DC_CONFIG=config/config.yaml -export JAVA_OPTS=-Dconfig/logging.properties -nohup bin/otel-dc-llm > /dev/null 2>&1 & -``` - -*Note:* -1) You can omit this line to set DC_CONFIG because the default configuration file is "config/config.yaml", you can also use environment variable "DC_CONFIG" to speficy the configuration file, for example: -```bash -export DC_CONFIG=config/config-oceanbase.yaml -export JAVA_OPTS=-Dconfig/logging.properties -bin/otel-dc-rdb -``` - -2) Query DataCollector -```bash -ps -ef | grep otel-dc | grep -v grep -``` - -3) Stop DataCollector -```bash -ps -ef | grep otel-dc | grep -v grep | awk '{printf " "$2" "}' | xargs kill -9 -``` - -4) Check logging file -```bash -ls -l ~/*-dc*.log* -``` diff --git a/host/README.md b/host/README.md index 58f5b19..6ca4cff 100644 --- a/host/README.md +++ b/host/README.md @@ -1,4 +1,4 @@ -# ODCD (OpenTelemetry based Data Collector for Telemetry Data) +# ODCD (OpenTelemetry Data Collector for Telemetry Data) **[Semantic Convention](../docs/semconv)** | **[Support](../docs/support/README.md)** | @@ -7,7 +7,7 @@ **[License](../LICENSE)** --- -ODCD (OpenTelemetry based Data Collector for Telemetry Data) is a collection of stanalone OpenTelemetry receivers for databases, systems, and apps. All implementations are based on predefined OpenTelemetry Semantic Conventions. A standard OTLP exporter is provided to forward the data from this "Data Collector" to a Telemetry backend or an OpenTelemetry Collector. +ODCD (OpenTelemetry Data Collector for Telemetry Data) is a collection of stanalone OpenTelemetry receivers for databases, systems, and apps. All implementations are based on predefined OpenTelemetry Semantic Conventions. A standard OTLP exporter is provided to forward the data from this "Data Collector" to a Telemetry backend or an OpenTelemetry Collector.

@@ -16,11 +16,57 @@ ODCD (OpenTelemetry based Data Collector for Telemetry Data) is a collection of ## Requirements -- Java 8+ +- Java 11+ -## Build & Installation +Ensure that Java SDK 11+ is installed. +```bash +java -version +``` + + +## Run Data Collector with release (for end users) +Download the latest [Release package](https://github.com/instana/otel-dc/releases/tag/Release) according to the operating system. + + +1) Download the installation package: +```bash +wget https://github.com/instana/otel-dc/releases/download/v1.0.1/otel-dc-host-0.2.2.tar +``` + +2) Extract the package to the desired deployment location: +```bash +tar vxf otel-dc-host-0.2.2.tar +cd otel-dc-host-0.2.2 +``` + +3) Make sure following configuration files are correct for your environment: + - config/config.yaml + - config/logging.properties + +Refine configuration file (config/config.yaml) according to your host. Right now we provide Data Collector for following hosts: + - SNMP Host + - IBM MQ appliance + - Simple Linux Host + +*Note:* The default configuration file config/config.yaml is for linux host. If you want to monitor an IBM MQ appliance, you can copy config/config-mqappliance.yaml to config/config.yaml, or you can also use environment variable "DC_CONFIG" to specify the configuration file, for example: +```bash +export DC_CONFIG=config/config-mqappliance.yaml +``` + +Notes for some parameters： +- `otel.backend.url`：The OTel gRPC address of the OTel backends, for example Instana Agent (as OTel Backend): http://localhost:4317 +- `otel.service.name`：The Data Collector name, which can be any string you choose. + +4) Run Data Collector +Run the Data Collector with the following command according to your current system: +```bash +nohup ./bin/otel-dc-host +``` + + +## Build & Run (for develpers) -1) Make sure Java SDK 8+ is installed. +1) Make sure Java SDK 11+ is installed. ```bash java -version ``` @@ -37,30 +83,32 @@ cd otel-dc/host ``` *Note: gradle 7.4 will be installed if you do not have it.* -## Run - -1) Make sure code is built with Java SDK 8+. +4) Make sure following configuration files are correct for your environment: + - config/config.yaml + - config/logging.properties -2) Refine configuration file (config/config.yaml) according to your host. Right now we provide Data Collector for following hosts: - - Linux Host +Refine configuration file (config/config.yaml) according to your host. Right now we provide Data Collector for following hosts: + - SNMP Host - IBM MQ appliance + - Simple Linux Host *Note:* The default configuration file config/config.yaml is for linux host. If you want to monitor an IBM MQ appliance, you can copy config/config-mqappliance.yaml to config/config.yaml, or you can also use environment variable "DC_CONFIG" to specify the configuration file, for example: - ```bash export DC_CONFIG=config/config-mqappliance.yaml ``` -3) Start up your OTLP backend which accept OTLP connections. Right now we support following protocols: + +5) Start up your OTLP backend which accept OTLP connections. Right now we support following protocols: - otlp/grpc - otlp/http -4) Option-1: Run the Data Collector with gradle ```bash ./gradlew run ``` *Note:* If you want to monitor IBM MQ appliance, you need to install **expect** before starting the Data Collector. You can run the command `which expect` to check if **expect** is already installed. -5) Option-2: Find the deployment package (otel-dc-host-*.tar) generated by gradle in the "build/distributions/" directory, extract deployment files: + +6) *Appendix:* Run binary in "build/distributions" +Find the deployment package (e.g.:otel-dc-rdb-*.tar/otel-dc-host-*.tar/otel-dc-host-*.tar) generated by gradle in the "build/distributions/" directory, extract deployment files: ```bash cd build/distributions/ tar vxf otel-dc-host-*.tar @@ -68,20 +116,20 @@ rm -f *.tar *.zip cd otel-dc-host-* ``` -Then, make sure following configuration files are correct for your environment: +Modify the configuration files. +Then, make sure following configuration files are correct for your environment.: - config/config.yaml - config/logging.properties -*Note:* The default configuration file config/config.yaml is for linux host. If you want to monitor an IBM MQ appliance, you can copy config/config-mqappliance.yaml to config/config.yaml, or you can also use environment variable "DC_CONFIG" to specify the configuration file, for example: +Run the Data Collector with following command according to your current implentation: ```bash -export DC_CONFIG=config/config-mqappliance.yaml +export DC_CONFIG=config/config.yaml +export JAVA_OPTS=-Dconfig/logging.properties +bin/otel-dc-rdb ``` -Run the Data Collector with following command according to your current OS: +Or run Data Collector in background ```bash -bin/otel-dc-host -``` -*Note:* If you want to monitor IBM MQ appliance, you need to install **expect** before starting the Data Collector. You can run the command `which expect` to check if **expect** is already installed. - -## Create Data Collector implementation for a new type of host - -Please refer to "[How to create data collector implementation for a new database](docs/developer/new-db.md)". \ No newline at end of file +export DC_CONFIG=config/config.yaml +export JAVA_OPTS=-Dconfig/logging.properties +nohup bin/otel-dc-host > /dev/null 2>&1 & +``` \ No newline at end of file diff --git a/llm/README.md b/llm/README.md index 256a788..5065ddc 100644 --- a/llm/README.md +++ b/llm/README.md @@ -1,6 +1,6 @@ -# ODCL (OTel based Data Collector for LLM) +# ODCL (OTel Data Collector for LLM) -ODCL (OTel based Data Collector for LLM) is the tool or template to generate OpenTelemetry metrics for various LLM and LLM Applications. All implementation are based on predefined OpenTelemetry Semantic Conventions. +ODCL (OTel Data Collector for LLM) is the tool or template to generate OpenTelemetry metrics for various LLM and LLM Applications. All implementation are based on predefined OpenTelemetry Semantic Conventions. ## Requirements @@ -28,7 +28,7 @@ cd otel-dc-llm-1.0.0 vi config/config.yaml ``` The following options are required： -- `otel.backend.url`：The OTel gRPC address of the agent, for example: http://localhost:4317 +- `otel.backend.url`：The OTel gRPC address of the OTel backends, for example Instana Agent (as OTel Backend): http://localhost:4317 - `otel.service.name`：The Data Collector name, which can be any string you choose. - `*.price.prompt.tokens.per.kilo`：The unit price per thousand prompt tokens. - `*.price.complete.tokens.per.kilo`：The unit price per thousand complete tokens. diff --git a/rdb/README.md b/rdb/README.md index 88e9b42..5a0c756 100644 --- a/rdb/README.md +++ b/rdb/README.md @@ -118,7 +118,6 @@ Then, make sure following configuration files are correct for your environment.: - config/logging.properties Run the Data Collector with following command according to your current implentation: -- **database** ```bash export DC_CONFIG=config/config.yaml export JAVA_OPTS=-Dconfig/logging.properties