Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update doc #50

Merged
merged 1 commit into from
Jul 5, 2024
Merged

Update doc #50

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 6 additions & 139 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# ODCD (OpenTelemetry based Data Collector for Telemetry Data)
# ODCD (OpenTelemetry Data Collector for Telemetry Data)

**[Semantic Convention](docs/semconv)** |
**[Support](docs/support/README.md)** |
Expand All @@ -7,146 +7,13 @@
**[License](LICENSE)**

---
ODCD (OpenTelemetry based Data Collector for Telemetry Data) is a collection of stanalone OpenTelemetry receivers for databases, systems, and apps. All implementations are based on predefined OpenTelemetry Semantic Conventions. A standard OTLP exporter is provided to forward the data from this "Data Collector" to a Telemetry backend or an OpenTelemetry Collector.
ODCD (OpenTelemetry Data Collector for Telemetry Data) is a collection of stanalone OpenTelemetry receivers for databases, systems, and apps. All implementations are based on predefined OpenTelemetry Semantic Conventions. A standard OTLP exporter is provided to forward the data from this "Data Collector" to a Telemetry backend or an OpenTelemetry Collector.

<br><br>

# Data Collectors
# Data Collectors provided

##
### Requirements
- Data Collectors for Host **Java 11+**
- Data Collectors for Databases **Java 8+**
- Data Collectors for LLM **Java 11+**
- [OTel Data Collectors for Relational Databases](rdb/README.md) (**Java 8+**)
- [OTel Data Collectors for Host](host/README.md) (**Java 11+**)
- [OTel Data Collectors for LLM](llm/README.md) (**Java 11+**)

## Build & Installation

1) Make sure Java SDK is installed.
```bash
java -version
```

2) Get the source code from `github.com`.
```bash
git clone https://github.com/instana/otel-dc.git
```
- Data Collectors for Host
```bash
cd otel-dc/host
```
- Data Collectors for Databases
```bash
cd otel-dc/rdb
```
- Data Collectors for LLM
```bash
cd otel-dc/llm
```
1) Build with Gradle
```bash
./gradlew clean build
```
*Note: gradle 7.4 will be installed if you do not have it.*

## Run

- ### Run with source code (Take the database plugin as an example)

1) Make sure code is built with Java SDK required version.

2) Refine configuration file (config/config.yaml) according to your own database. Right now we provide Data Collector for following databases:
- DaMeng database
- Oceanbase
- Informix

>*Note: We respect the OTel community conventions and consider more about popular database engines for conventionss. While we may create implementations from those databases without OTel support based on user requirements or developer contributions. *

3) Start up your OTLP backend which accept OTLP connections. Right now we support following protocols:
- otlp/grpc
- otlp/http

4) Option-1: Run the Data Collector with gradle
```bash
./gradlew run
```
5) Option-2: Find the deployment package (e.g.:otel-dc-rdb-*.tar/otel-dc-host-*.tar/otel-dc-host-*.tar) generated by gradle in the "build/distributions/" directory, extract deployment files:
```bash
cd build/distributions/
tar vxf otel-dc-rdb-*.tar
rm -f *.tar *.zip
cd otel-dc-rdb-*
```
Refer to following chapter to run the Data Collector.

- ### Run with release
Download the latest [Release package](https://github.com/instana/otel-dc/releases/tag/Release) according to the operating system.

### Modify the configuration file.
Then, make sure following configuration files are correct for your environment.:
- config/config.yaml
- config/logging.properties

Run the Data Collector with following command according to your current implentation:
- **database**
```bash
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
bin/otel-dc-rdb
```
Or run Data Collector in background
```bash
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
nohup bin/otel-dc-rdb > /dev/null 2>&1 &
```


Run the Data Collector with following command according to your current implentation:
- **host**
```bash
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
bin/otel-dc-host
```
Or run Data Collector in background
```bash
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
nohup bin/otel-dc-host > /dev/null 2>&1 &
```

- **llm**
```bash
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
bin/otel-dc-llm
```
Or run Data Collector in background
```bash
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
nohup bin/otel-dc-llm > /dev/null 2>&1 &
```

*Note:*
1) You can omit this line to set DC_CONFIG because the default configuration file is "config/config.yaml", you can also use environment variable "DC_CONFIG" to speficy the configuration file, for example:
```bash
export DC_CONFIG=config/config-oceanbase.yaml
export JAVA_OPTS=-Dconfig/logging.properties
bin/otel-dc-rdb
```

2) Query DataCollector
```bash
ps -ef | grep otel-dc | grep -v grep
```

3) Stop DataCollector
```bash
ps -ef | grep otel-dc | grep -v grep | awk '{printf " "$2" "}' | xargs kill -9
```

4) Check logging file
```bash
ls -l ~/*-dc*.log*
```
99 changes: 74 additions & 25 deletions host/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# ODCD (OpenTelemetry based Data Collector for Telemetry Data)
# ODCD (OpenTelemetry Data Collector for Telemetry Data)

**[Semantic Convention](../docs/semconv)** |
**[Support](../docs/support/README.md)** |
Expand All @@ -7,7 +7,7 @@
**[License](../LICENSE)**

---
ODCD (OpenTelemetry based Data Collector for Telemetry Data) is a collection of stanalone OpenTelemetry receivers for databases, systems, and apps. All implementations are based on predefined OpenTelemetry Semantic Conventions. A standard OTLP exporter is provided to forward the data from this "Data Collector" to a Telemetry backend or an OpenTelemetry Collector.
ODCD (OpenTelemetry Data Collector for Telemetry Data) is a collection of stanalone OpenTelemetry receivers for databases, systems, and apps. All implementations are based on predefined OpenTelemetry Semantic Conventions. A standard OTLP exporter is provided to forward the data from this "Data Collector" to a Telemetry backend or an OpenTelemetry Collector.

<br><br>

Expand All @@ -16,11 +16,57 @@ ODCD (OpenTelemetry based Data Collector for Telemetry Data) is a collection of

## Requirements

- Java 8+
- Java 11+

## Build & Installation
Ensure that Java SDK 11+ is installed.
```bash
java -version
```


## Run Data Collector with release (for end users)
Download the latest [Release package](https://github.com/instana/otel-dc/releases/tag/Release) according to the operating system.


1) Download the installation package:
```bash
wget https://github.com/instana/otel-dc/releases/download/v1.0.1/otel-dc-host-0.2.2.tar
```

2) Extract the package to the desired deployment location:
```bash
tar vxf otel-dc-host-0.2.2.tar
cd otel-dc-host-0.2.2
```

3) Make sure following configuration files are correct for your environment:
- config/config.yaml
- config/logging.properties

Refine configuration file (config/config.yaml) according to your host. Right now we provide Data Collector for following hosts:
- SNMP Host
- IBM MQ appliance
- Simple Linux Host

*Note:* The default configuration file config/config.yaml is for linux host. If you want to monitor an IBM MQ appliance, you can copy config/config-mqappliance.yaml to config/config.yaml, or you can also use environment variable "DC_CONFIG" to specify the configuration file, for example:
```bash
export DC_CONFIG=config/config-mqappliance.yaml
```

Notes for some parameters：
- `otel.backend.url`：The OTel gRPC address of the OTel backends, for example Instana Agent (as OTel Backend): http://localhost:4317
- `otel.service.name`：The Data Collector name, which can be any string you choose.

4) Run Data Collector
Run the Data Collector with the following command according to your current system:
```bash
nohup ./bin/otel-dc-host
```


## Build & Run (for develpers)

1) Make sure Java SDK 8+ is installed.
1) Make sure Java SDK 11+ is installed.
```bash
java -version
```
Expand All @@ -37,51 +83,54 @@ cd otel-dc/host
```
*Note: gradle 7.4 will be installed if you do not have it.*

## Run

1) Make sure code is built with Java SDK 8+.
4) Make sure following configuration files are correct for your environment:
- config/config.yaml
- config/logging.properties

2) Refine configuration file (config/config.yaml) according to your host. Right now we provide Data Collector for following hosts:
- Linux Host
Refine configuration file (config/config.yaml) according to your host. Right now we provide Data Collector for following hosts:
- SNMP Host
- IBM MQ appliance
- Simple Linux Host

*Note:* The default configuration file config/config.yaml is for linux host. If you want to monitor an IBM MQ appliance, you can copy config/config-mqappliance.yaml to config/config.yaml, or you can also use environment variable "DC_CONFIG" to specify the configuration file, for example:

```bash
export DC_CONFIG=config/config-mqappliance.yaml
```
3) Start up your OTLP backend which accept OTLP connections. Right now we support following protocols:

5) Start up your OTLP backend which accept OTLP connections. Right now we support following protocols:
- otlp/grpc
- otlp/http

4) Option-1: Run the Data Collector with gradle
```bash
./gradlew run
```
*Note:* If you want to monitor IBM MQ appliance, you need to install **expect** before starting the Data Collector. You can run the command `which expect` to check if **expect** is already installed.

5) Option-2: Find the deployment package (otel-dc-host-*.tar) generated by gradle in the "build/distributions/" directory, extract deployment files:

6) *Appendix:* Run binary in "build/distributions"
Find the deployment package (e.g.:otel-dc-rdb-*.tar/otel-dc-host-*.tar/otel-dc-host-*.tar) generated by gradle in the "build/distributions/" directory, extract deployment files:
```bash
cd build/distributions/
tar vxf otel-dc-host-*.tar
rm -f *.tar *.zip
cd otel-dc-host-*
```

Then, make sure following configuration files are correct for your environment:
Modify the configuration files.
Then, make sure following configuration files are correct for your environment.:
- config/config.yaml
- config/logging.properties

*Note:* The default configuration file config/config.yaml is for linux host. If you want to monitor an IBM MQ appliance, you can copy config/config-mqappliance.yaml to config/config.yaml, or you can also use environment variable "DC_CONFIG" to specify the configuration file, for example:
Run the Data Collector with following command according to your current implentation:
- **database**
```bash
export DC_CONFIG=config/config-mqappliance.yaml
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
bin/otel-dc-rdb
```
Run the Data Collector with following command according to your current OS:
Or run Data Collector in background
```bash
bin/otel-dc-host
```
*Note:* If you want to monitor IBM MQ appliance, you need to install **expect** before starting the Data Collector. You can run the command `which expect` to check if **expect** is already installed.

## Create Data Collector implementation for a new type of host

Please refer to "[How to create data collector implementation for a new database](docs/developer/new-db.md)".
export DC_CONFIG=config/config.yaml
export JAVA_OPTS=-Dconfig/logging.properties
nohup bin/otel-dc-host > /dev/null 2>&1 &
```
6 changes: 3 additions & 3 deletions llm/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# ODCL (OTel based Data Collector for LLM)
# ODCL (OTel Data Collector for LLM)

ODCL (OTel based Data Collector for LLM) is the tool or template to generate OpenTelemetry metrics for various LLM and LLM Applications. All implementation are based on predefined OpenTelemetry Semantic Conventions.
ODCL (OTel Data Collector for LLM) is the tool or template to generate OpenTelemetry metrics for various LLM and LLM Applications. All implementation are based on predefined OpenTelemetry Semantic Conventions.

## Requirements

Expand Down Expand Up @@ -28,7 +28,7 @@ cd otel-dc-llm-1.0.0
vi config/config.yaml
```
The following options are required：
- `otel.backend.url`：The OTel gRPC address of the agent, for example: http://localhost:4317
- `otel.backend.url`：The OTel gRPC address of the OTel backends, for example Instana Agent (as OTel Backend): http://localhost:4317
- `otel.service.name`：The Data Collector name, which can be any string you choose.
- `*.price.prompt.tokens.per.kilo`：The unit price per thousand prompt tokens.
- `*.price.complete.tokens.per.kilo`：The unit price per thousand complete tokens.
Expand Down
Loading