Add compose monitoring profile

[bwbroersma]

• Fixes Options to only create main part of internet.nl Batch API (Django) #1588

[mxsasha]

The number of profiles seems to be rapidly expanding. We've had a dev misconfiguration for them already. I think we need one place in the deployment docs where they are all listed, along with defaults and recommendations. Particularly as misconfiguration may not be immediately obvious.

[bwbroersma]

Agree, also wonder if all services with the monitoring profile should also have the alertmanager profile?

It seems after this addition there are these profiles defined in the compose files:

• alertmanager
• build
• connectiontest
• cron
• monitoring
• routinator
• run-tests

I see batch used in 2 .env files:

Internet.nl/docker/build.env

Line 5 in 0e5a6d1

COMPOSE_PROFILES=connectiontest,cron,routinator,batch,build

Internet.nl/docker/batch-test.env

Line 11 in 0e5a6d1

COMPOSE_PROFILES=batch,cron

However, its unused in all compose files.

[aequitas]

I think we should refrain from giving these kinds of instructions for COMPOSE_PROFILES completely (for which I'm guilty of as well). And go with @mxsasha idea #1589 (comment) and have 1 place in the documentation to refer to with a table with all possible permutations of COMPOSE_PROFILES (that are sane at least) and have them copy pasteable so there is no ambiguity.

[bwbroersma]

@aequitas I think we discusses this also, and some compose profiles (batch) are indeed no longer used and can be deleted right? Is there still an use case to keep run-tests?

[bwbroersma]

Idea:

• add more documentation to the compose file
• generate container list (and which are crucial) from this
• include in which profiles containers are

[bwbroersma]

$ yq -r '"container|profiles|description","-|-|-",.services|to_entries|map([.key,(.value.profiles //[]|join(", "),(.key|head_comment))]|join("|"))[]' compose.yaml|sed 's/$/|/'|column -ts\| -o' | '|sed 's/^/| /;s/ $//;2{s/ /-/g}'

using yq, sed and column (maybe also add -W3 -c120 to column call).
Alternatively markdown-table could be used:

$ yq -r '"container|profiles|description",.services|to_entries|map([.key,(.value.profiles //[]|join(", "),(.key|head_comment))]|join("|"))[]' compose.yaml|markdown-table -s\|

container	profiles	description
webserver		nginx proxy container
app		django container
db-migrate		django DB migrations
worker		worker for queue
worker-nassl		worker for queue with potential memory leak
worker-slow		worker for slow and long running tasks that could require a lot of memory (eg: hof update)
beat		celery task queue
redis		redis cache / state
rabbitmq		rabbitmq message-broker
postgres		PostgreSQL database
routinator	routinator	Routinator for RPKI
unbound	connectiontest	unbound DNS server used for connection test
resolver-permissive		unbound resolver used for all DNS queries by app, worker, etc that needs to ignore DNSSEC errors and pass them on to the client
resolver-validating		unbound resolver used for ldns-dane that require DNSSEC validation
cron	cron	cron with periodic tasks
cron-docker	cron	cron daemon with access to Docker socket but no networking
grafana	monitoring	grafana to visualize the monitoring
prometheus	monitoring	prometheus for monitoring
alertmanager	alertmanager	alertmanager for sending monitoring alerts (also requires monitoring profile)
postgresql-exporter	monitoring	provide PostgreSQL metrics for monitoring
redis-exporter	monitoring	provide redis metrics for monitoring
statsd-exporter	monitoring	provide statsd metrics for monitoring
celery-exporter	monitoring	provide celery metrics for monitoring
node-exporter	monitoring	provide node metrics for monitoring
docker_stats_exporter	monitoring	provide docker container metrics for monitoring
nginx_logs_exporter	monitoring	provide nginx metrics for monitoring

Questions:

• Where/when to write this table?
• Add table with profile description?
• If so, where to make this ideally single profile comment?