Merge pull request #5 from felipe-loka/feat/instrumentation

add instrumentation to the code

.gitignore

@@ -1,6 +1,8 @@
 # Created by https://www.toptal.com/developers/gitignore/api/node,visualstudiocode,macos
 # Edit at https://www.toptal.com/developers/gitignore?templates=node,visualstudiocode,macos
 
+.vscode/
+
 ##############################
 ###### macOS
 ##############################

Dockerfile

@@ -17,4 +17,4 @@ RUN pnpm install --prod
 COPY --from=build /app/dist ./
 COPY src/config/database/migrations ./src/config/database/migrations
 EXPOSE 3000
-CMD ["node", "./src/index.js"]
+CMD ["node", "--require", "./src/config/instrumentation.js", "./src/index.js"]

README.md

@@ -21,9 +21,9 @@ The architecture of this project is simple: an API that uses MySQL as the databa
 
 ## Getting Started
 
-This project uses [pnpm](https://pnpm.io/) as the package manager, so this must be installed in your machine in order to start contributing to this project.
+This project uses [pnpm](https://pnpm.io/) as the package manager, so this must be installed in your machine to start contributing to this project.
 
-Several helpful script are present in the `package.json` file, such as: create the infrastructure needed (MySQL database), run tests, run locally the code, build the code...
+Several helpful scripts are present in the `package.json` file, such as: creating the infrastructure needed (MySQL database), running tests, running the code, build the code...
 
 A step-by-step guide on how to run this project is the following:
 
@@ -50,27 +50,44 @@ DB_NAME=api
 DB_HOST=localhost
 DB_PORT=3306
 DATABASE_URL=mysql://user:password123@localhost:3306/api
-LOGGER_FILENAME=poll.log
+OTEL_COLLECTOR_URL=http://otel-collector:4318
 ```
 
-5. Setup database infrastructure
+Besides these environment variables (related to the application) some others need to be set (OpenTelemetry):
 
-In order to setup the infrastructure the following steps are executed: MySQL Docker container is created, migrations are execute, database is seeded, observability stack is created. In order to simplify all these steps a script was created for you, so you can simply run:
+```shell
+OTEL_SERVICE_NAME=poll-api
+```
+
+5. Setup database infrastructure
+To set up the infrastructure the following steps are executed: MySQL Docker container is created, migrations are executed, the database is seeded, and the observability stack is created. To simplify all these steps a script was created for you, so you can simply run:
 
 ```
 pnpm infra:up
 ```
 
-We are using [dbmate](https://github.com/amacneil/dbmate), an agnostic tool used for database migration. For MySQL databases it expect the [DATABASE_URL](https://github.com/amacneil/dbmate#mysql) environment variable to be set with the string URL used to connect to the database, that is why we are creating this environment variable in the `.env` file given above.
+We are using [dbmate](https://github.com/amacneil/dbmate), an agnostic tool used for database migration. For MySQL databases it expects the [DATABASE_URL](https://github.com/amacneil/dbmate#mysql)(https://github.com/amacneil/dbmate#mysql) environment variable to be set with the string URL used to connect to the database, that is why we are creating this environment variable in the `.env` file given above.
 
 6. Run the code
 
-You can run the code in a watch mode, which means the API will auto-refresh with new changes everytime you modify and save any files, this helps you increase your productivity. You can run the API with the following script:
+You can run the code in a watch mode, which means the API will auto-refresh with new changes every time you modify and save any files, this helps you increase your productivity. You can run the API with the following script:
 
 ```
 pnpm start:dev
 ```
 
+If you want to use the observability stack (log, traces, metrics) you can't simply run locally, you will need to run the container version in the same network as all the other components (MySQL, Loki, Grafana, Tempo, Prometheus). So instead of running `pnpm start:dev` you can run the following:
+
+```
+pnpm container:up
+```
+
+Notice that it won't auto-refresh, so if you want to test your change you will need to:
+
+```
+pnpm container:down
+```
+
 7. Run tests
 
 Always create tests for new features. The tests are not mocking the database, so it will run tests against the MySQL docker database created! For it to work perfectly it expects the database to be seeded!
@@ -85,6 +102,7 @@ You can destroy all the created infrastructure with the following command:
 
 ```
 pnpm infra:down
+pnpm container:down
 ```
 
 ## Database Modeling
@@ -97,26 +115,26 @@ This project fullfills the following requirements:
 
 ### Poll Creation
 
-- User should be able to create a poll and get a link to share with people
-- User can choose if the poll allows more than one choice in the same vote (multi choice poll)
+- Users should be able to create a poll and get a link to share with people
+- Users can choose if the poll allows more than one choice in the same vote (multi choice poll)
 
 ### Poll Voting System
-
-- User should be able to vote in a given poll
-- User should be able to see the vote results of a given poll after voting
+- Users should be able to vote in a given poll
+- Users should be able to see the vote results of a given poll after voting
 
 ## Observability
 
-It's beeing implemented a whole observability stack to monitor the application correctly.
+It's being implemented a whole observability stack to monitor the application correctly.
 
-- **Logs**: are being collected by [FluentBit](https://fluentbit.io/) and forwarded to Loki.
+- **Logs**: are being collected by [FluentBit](https://fluentbit.io/) and forwarded to [Loki](https://grafana.com/docs/loki/latest/).
+- **Traces**: Traces are automatically generated (instrumentation) using OpenTelemetry and are collected by [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) and forwarded to [Tempo](https://grafana.com/oss/tempo/) using [OTLP protocol](https://opentelemetry.io/docs/specs/otel/protocol/) via HTTP requests
 
-In order to fully test this locally you will need to build the local application using Docker. To make it easier a script was created and added in `package.json`, so you can simply run `pnpm container`
+To fully test this locally you will need to build the local application using Docker. To make it easier a script was created and added in `package.json`, so you can simply run `pnpm container`
 
 ## Next Steps
 
-The following requirements should be implemented in next versions:
+The following requirements should be implemented in the next versions:
 
-- Add observability to the application: metrics, traces
+- Add observability to the application: metrics
 - Add cache layer (e.g. Redis) to minimize latency time
-- Validate the need of using a poll of connections to manage the connection to the database
+- Validate the need to use a poll of connections to manage the connection to the database

config/grafana.yaml

@@ -1,7 +1,12 @@
 apiVersion: 1
 datasources:
+  # https://grafana.com/docs/grafana/latest/datasources/loki/#provision-the-loki-data-source
   - name: Loki
     type: loki
     access: proxy
     url: http://loki:3100
-    basicAuth: false
+    basicAuth: false
+  # https://grafana.com/docs/grafana/latest/datasources/tempo/configure-tempo-data-source/#provision-the-data-source
+  - name: Tempo
+    type: tempo
+    url: http://tempo:3200

config/otel-collector.yaml

@@ -0,0 +1,20 @@
+# https://opentelemetry.io/docs/collector/configuration/#receivers
+receivers:
+  otlp:
+    protocols:
+      http:
+        endpoint: 0.0.0.0:4318
+      grpc:
+        endpoint: 0.0.0.0:4317
+
+# https://opentelemetry.io/docs/collector/configuration/#exporters
+exporters:
+  otlphttp:
+    endpoint: http://tempo:4318
+
+# https://opentelemetry.io/docs/collector/configuration/#exporters
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      exporters: [otlphttp]

config/tempo.yaml

@@ -0,0 +1,30 @@
+stream_over_http_enabled: true
+
+# https://grafana.com/docs/tempo/latest/configuration/#server
+server:
+  http_listen_port: 3200
+  log_level: info
+
+# https://grafana.com/docs/tempo/latest/configuration/#query-frontend
+query_frontend:
+  search:
+    duration_slo: 5s
+  trace_by_id:
+    duration_slo: 5s
+
+# https://grafana.com/docs/tempo/latest/configuration/#distributor
+distributor:
+  receivers:
+    otlp:
+      protocols:
+        http:
+        grpc:
+
+# https://grafana.com/docs/tempo/latest/configuration/#storage
+storage:
+  trace:
+    backend: local
+    wal:
+      path: /tmp/tempo/wal             # where to store the the wal locally
+    local:
+      path: /tmp/tempo/blocks

docker-compose.yaml

@@ -76,6 +76,28 @@ services:
       timeout: 5s
       retries: 5
 
+  otel-collector:
+    image: otel/opentelemetry-collector:0.95.0
+    container_name: otel-collector
+    command: [ "--config=/etc/otel-collector.yaml" ]
+    networks:
+      - pollNetwork
+    volumes:
+      - ./config/otel-collector.yaml:/etc/otel-collector.yaml
+
+  tempo:
+    image: grafana/tempo:2.3.1
+    container_name: tempo
+    networks:
+      - pollNetwork
+    ports:
+      - 3200:3200
+      - 4317:4317 # OTLP gRPC receiver
+      - 4318:4318 # OTLP http receiver
+    volumes:
+      - ./config/tempo.yaml:/etc/tempo/config.yaml
+    command: -config.file=/etc/tempo/config.yaml
+
 networks:
   pollNetwork:
     name: pollNetwork

package.json

@@ -11,8 +11,8 @@
     "db:migrate:up": "pnpm exec dbmate --migrations-dir ./src/config/database/migrations up",
     "db:migrate:down": "pnpm exec dbmate --migrations-dir ./src/config/database/migrations down",
     "db:seed": "pnpm exec dbmate --migrations-dir ./src/config/database/seeds up",
-    "start": "ts-node src/index.ts",
-    "start:dev": "nodemon src/index.ts",
+    "start": "npx ts-node src/index.ts",
+    "start:dev": "npx nodemon --require ./src/config/instrumentation.ts src/index.ts",
     "container:up": "docker build -t poll-api . && docker container run -d -p 3000:3000 --name poll --env-file .env -e DB_HOST='mysql' --network pollNetwork --log-driver fluentd --log-opt fluentd-address=127.0.0.1:24224 --log-opt fluentd-async-connect=true --log-opt tag=poll-api poll-api",
     "container:down": "docker rm -f poll"
   },
@@ -42,6 +42,12 @@
     "typescript": "^5.2.2"
   },
   "dependencies": {
+    "@opentelemetry/api": "^1.7.0",
+    "@opentelemetry/auto-instrumentations-node": "^0.41.1",
+    "@opentelemetry/exporter-trace-otlp-proto": "^0.48.0",
+    "@opentelemetry/sdk-metrics": "^1.21.0",
+    "@opentelemetry/sdk-node": "^0.48.0",
+    "@opentelemetry/sdk-trace-node": "^1.21.0",
     "body-parser": "^1.20.2",
     "compression": "^1.7.4",
     "cors": "^2.8.5",
@@ -52,6 +58,7 @@
     "iconv-lite": "^0.6.3",
     "jest": "^29.7.0",
     "mysql2": "^3.6.3",
+    "npx": "^10.2.2",
     "supertest": "^6.3.3",
     "uuid": "^9.0.1",
     "winston": "^3.11.0",