Skip to content

Commit

Permalink
Update doc (#50)
Browse files Browse the repository at this point in the history
  • Loading branch information
Rui Liu committed Jul 5, 2024
1 parent 489d25f commit b73f02f
Show file tree
Hide file tree
Showing 4 changed files with 82 additions and 168 deletions.
145 changes: 6 additions & 139 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# ODCD (OpenTelemetry based Data Collector for Telemetry Data)
# ODCD (OpenTelemetry Data Collector for Telemetry Data)

**[Semantic Convention](docs/semconv)** |
**[Support](docs/support/README.md)** |
Expand All @@ -7,146 +7,13 @@
**[License](LICENSE)**

---
ODCD (OpenTelemetry based Data Collector for Telemetry Data) is a collection of stanalone OpenTelemetry receivers for databases, systems, and apps. All implementations are based on predefined OpenTelemetry Semantic Conventions. A standard OTLP exporter is provided to forward the data from this "Data Collector" to a Telemetry backend or an OpenTelemetry Collector.
ODCD (OpenTelemetry Data Collector for Telemetry Data) is a collection of stanalone OpenTelemetry receivers for databases, systems, and apps. All implementations are based on predefined OpenTelemetry Semantic Conventions. A standard OTLP exporter is provided to forward the data from this "Data Collector" to a Telemetry backend or an OpenTelemetry Collector.

<br><br>

# Data Collectors
# Data Collectors provided

##
### Requirements
- Data Collectors for Host **Java 11+**
- Data Collectors for Databases **Java 8+**
- Data Collectors for LLM **Java 11+**
- [OTel Data Collectors for Relational Databases](rdb/README.md) (**Java 8+**)
- [OTel Data Collectors for Host](host/README.md) (**Java 11+**)
- [OTel Data Collectors for LLM](llm/README.md) (**Java 11+**)

## Build & Installation

1) Make sure Java SDK is installed.
```bash
java -version
```

2) Get the source code from `github.com`.
```bash
git clone https://github.com/instana/otel-dc.git
```
- Data Collectors for Host
```bash
cd otel-dc/host
```
- Data Collectors for Databases
```bash
cd otel-dc/rdb
```
- Data Collectors for LLM
```bash
cd otel-dc/llm
```
1) Build with Gradle
```bash
./gradlew clean build
```
*Note: gradle 7.4 will be installed if you do not have it.*

## Run

- ### Run with source code (Take the database plugin as an example)

1) Make sure code is built with Java SDK required version.

2) Refine configuration file (config/config.yaml) according to your own database. Right now we provide Data Collector for following databases:
- DaMeng database
- Oceanbase
- Informix

>*Note: We respect the OTel community conventions and consider more about popular database engines for conventionss. While we may create implementations from those databases without OTel support based on user requirements or developer contributions. *
3) Start up your OTLP backend which accept OTLP connections. Right now we support following protocols:
- otlp/grpc
- otlp/http

4) Option-1: Run the Data Collector with gradle
```bash
./gradlew run
```
5) Option-2: Find the deployment package (e.g.:otel-dc-rdb-*.tar/otel-dc-host-*.tar/otel-dc-host-*.tar) generated by gradle in the "build/distributions/" directory, extract deployment files:
```bash
cd build/distributions/
tar vxf otel-dc-rdb-*.tar
rm -f *.tar *.zip
cd otel-dc-rdb-*
```
Refer to following chapter to run the Data Collector.

- ### Run with release
Download the latest [Release package](https://github.com/instana/otel-dc/releases/tag/Release) according to the operating system.

### Modify the configuration file.
Then, make sure following configuration files are correct for your environment.:
- config/config.yaml
- config/logging.properties

Run the Data Collector with following command according to your current implentation:
- **database**
```bash
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
bin/otel-dc-rdb
```
Or run Data Collector in background
```bash
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
nohup bin/otel-dc-rdb > /dev/null 2>&1 &
```


Run the Data Collector with following command according to your current implentation:
- **host**
```bash
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
bin/otel-dc-host
```
Or run Data Collector in background
```bash
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
nohup bin/otel-dc-host > /dev/null 2>&1 &
```

- **llm**
```bash
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
bin/otel-dc-llm
```
Or run Data Collector in background
```bash
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
nohup bin/otel-dc-llm > /dev/null 2>&1 &
```

*Note:*
1) You can omit this line to set DC_CONFIG because the default configuration file is "config/config.yaml", you can also use environment variable "DC_CONFIG" to speficy the configuration file, for example:
```bash
export DC_CONFIG=config/config-oceanbase.yaml
export JAVA_OPTS=-Dconfig/logging.properties
bin/otel-dc-rdb
```

2) Query DataCollector
```bash
ps -ef | grep otel-dc | grep -v grep
```

3) Stop DataCollector
```bash
ps -ef | grep otel-dc | grep -v grep | awk '{printf " "$2" "}' | xargs kill -9
```

4) Check logging file
```bash
ls -l ~/*-dc*.log*
```
98 changes: 73 additions & 25 deletions host/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# ODCD (OpenTelemetry based Data Collector for Telemetry Data)
# ODCD (OpenTelemetry Data Collector for Telemetry Data)

**[Semantic Convention](../docs/semconv)** |
**[Support](../docs/support/README.md)** |
Expand All @@ -7,7 +7,7 @@
**[License](../LICENSE)**

---
ODCD (OpenTelemetry based Data Collector for Telemetry Data) is a collection of stanalone OpenTelemetry receivers for databases, systems, and apps. All implementations are based on predefined OpenTelemetry Semantic Conventions. A standard OTLP exporter is provided to forward the data from this "Data Collector" to a Telemetry backend or an OpenTelemetry Collector.
ODCD (OpenTelemetry Data Collector for Telemetry Data) is a collection of stanalone OpenTelemetry receivers for databases, systems, and apps. All implementations are based on predefined OpenTelemetry Semantic Conventions. A standard OTLP exporter is provided to forward the data from this "Data Collector" to a Telemetry backend or an OpenTelemetry Collector.

<br><br>

Expand All @@ -16,11 +16,57 @@ ODCD (OpenTelemetry based Data Collector for Telemetry Data) is a collection of

## Requirements

- Java 8+
- Java 11+

## Build & Installation
Ensure that Java SDK 11+ is installed.
```bash
java -version
```


## Run Data Collector with release (for end users)
Download the latest [Release package](https://github.com/instana/otel-dc/releases/tag/Release) according to the operating system.


1) Download the installation package:
```bash
wget https://github.com/instana/otel-dc/releases/download/v1.0.1/otel-dc-host-0.2.2.tar
```

2) Extract the package to the desired deployment location:
```bash
tar vxf otel-dc-host-0.2.2.tar
cd otel-dc-host-0.2.2
```

3) Make sure following configuration files are correct for your environment:
- config/config.yaml
- config/logging.properties

Refine configuration file (config/config.yaml) according to your host. Right now we provide Data Collector for following hosts:
- SNMP Host
- IBM MQ appliance
- Simple Linux Host

*Note:* The default configuration file config/config.yaml is for linux host. If you want to monitor an IBM MQ appliance, you can copy config/config-mqappliance.yaml to config/config.yaml, or you can also use environment variable "DC_CONFIG" to specify the configuration file, for example:
```bash
export DC_CONFIG=config/config-mqappliance.yaml
```

Notes for some parameters：
- `otel.backend.url`：The OTel gRPC address of the OTel backends, for example Instana Agent (as OTel Backend): http://localhost:4317
- `otel.service.name`：The Data Collector name, which can be any string you choose.

4) Run Data Collector
Run the Data Collector with the following command according to your current system:
```bash
nohup ./bin/otel-dc-host
```


## Build & Run (for develpers)

1) Make sure Java SDK 8+ is installed.
1) Make sure Java SDK 11+ is installed.
```bash
java -version
```
Expand All @@ -37,51 +83,53 @@ cd otel-dc/host
```
*Note: gradle 7.4 will be installed if you do not have it.*

## Run

1) Make sure code is built with Java SDK 8+.
4) Make sure following configuration files are correct for your environment:
- config/config.yaml
- config/logging.properties

2) Refine configuration file (config/config.yaml) according to your host. Right now we provide Data Collector for following hosts:
- Linux Host
Refine configuration file (config/config.yaml) according to your host. Right now we provide Data Collector for following hosts:
- SNMP Host
- IBM MQ appliance
- Simple Linux Host

*Note:* The default configuration file config/config.yaml is for linux host. If you want to monitor an IBM MQ appliance, you can copy config/config-mqappliance.yaml to config/config.yaml, or you can also use environment variable "DC_CONFIG" to specify the configuration file, for example:

```bash
export DC_CONFIG=config/config-mqappliance.yaml
```
3) Start up your OTLP backend which accept OTLP connections. Right now we support following protocols:

5) Start up your OTLP backend which accept OTLP connections. Right now we support following protocols:
- otlp/grpc
- otlp/http

4) Option-1: Run the Data Collector with gradle
```bash
./gradlew run
```
*Note:* If you want to monitor IBM MQ appliance, you need to install **expect** before starting the Data Collector. You can run the command `which expect` to check if **expect** is already installed.

5) Option-2: Find the deployment package (otel-dc-host-*.tar) generated by gradle in the "build/distributions/" directory, extract deployment files:

6) *Appendix:* Run binary in "build/distributions"
Find the deployment package (e.g.:otel-dc-rdb-*.tar/otel-dc-host-*.tar/otel-dc-host-*.tar) generated by gradle in the "build/distributions/" directory, extract deployment files:
```bash
cd build/distributions/
tar vxf otel-dc-host-*.tar
rm -f *.tar *.zip
cd otel-dc-host-*
```

Then, make sure following configuration files are correct for your environment:
Modify the configuration files.
Then, make sure following configuration files are correct for your environment.:
- config/config.yaml
- config/logging.properties

*Note:* The default configuration file config/config.yaml is for linux host. If you want to monitor an IBM MQ appliance, you can copy config/config-mqappliance.yaml to config/config.yaml, or you can also use environment variable "DC_CONFIG" to specify the configuration file, for example:
Run the Data Collector with following command according to your current implentation:
```bash
export DC_CONFIG=config/config-mqappliance.yaml
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
bin/otel-dc-rdb
```
Run the Data Collector with following command according to your current OS:
Or run Data Collector in background
```bash
bin/otel-dc-host
```
*Note:* If you want to monitor IBM MQ appliance, you need to install **expect** before starting the Data Collector. You can run the command `which expect` to check if **expect** is already installed.

## Create Data Collector implementation for a new type of host

Please refer to "[How to create data collector implementation for a new database](docs/developer/new-db.md)".
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
nohup bin/otel-dc-host > /dev/null 2>&1 &
```
6 changes: 3 additions & 3 deletions llm/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# ODCL (OTel based Data Collector for LLM)
# ODCL (OTel Data Collector for LLM)

ODCL (OTel based Data Collector for LLM) is the tool or template to generate OpenTelemetry metrics for various LLM and LLM Applications. All implementation are based on predefined OpenTelemetry Semantic Conventions.
ODCL (OTel Data Collector for LLM) is the tool or template to generate OpenTelemetry metrics for various LLM and LLM Applications. All implementation are based on predefined OpenTelemetry Semantic Conventions.

## Requirements

Expand Down Expand Up @@ -28,7 +28,7 @@ cd otel-dc-llm-1.0.0
vi config/config.yaml
```
The following options are required：
- `otel.backend.url`：The OTel gRPC address of the agent, for example: http://localhost:4317
- `otel.backend.url`：The OTel gRPC address of the OTel backends, for example Instana Agent (as OTel Backend): http://localhost:4317
- `otel.service.name`：The Data Collector name, which can be any string you choose.
- `*.price.prompt.tokens.per.kilo`：The unit price per thousand prompt tokens.
- `*.price.complete.tokens.per.kilo`：The unit price per thousand complete tokens.
Expand Down
1 change: 0 additions & 1 deletion rdb/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,6 @@ Then, make sure following configuration files are correct for your environment.:
- config/logging.properties

Run the Data Collector with following command according to your current implentation:
- **database**
```bash
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
Expand Down

0 comments on commit b73f02f

Please sign in to comment.