feat: Add configurable health check endpoint to online ingestor #207
Conversation
…ect/scicat-ingestor into feat-integration-test
…ynamic resource provisioning
…nce setup scripts
…roposals and instruments
| def _check_kafka(self) -> bool: | ||
| """Check if Kafka is reachable.""" | ||
| try: | ||
| self.consumer.list_topics(timeout=5) |
There was a problem hiding this comment.
I'm not sure if kafka consumer is thread safe.
Should we use threading.Lock?
There was a problem hiding this comment.
Hmm... maybe it's okay as we're not consuming messages here...
There was a problem hiding this comment.
Good catch, I will do some research on this and keep you updated.
There was a problem hiding this comment.
So the confluent-kafka-python library uses librdkafka, which is thread safe according to its documentation. The list_topics() calls librdkafka.rd_kafka_metadata() function which is unrelated to reading messages.
There was a problem hiding this comment.
Great! Thank you for letting me know : D
YooSunYoung
left a comment
YooSunYoung
left a comment
There was a problem hiding this comment.
It looks good to me, apart from the minor concer about using the consumer in the other thread.
Hi @YooSunYoung, I added a new “stress” test script that sends Kafka messages to the online ingestor and, in parallel, continuously calls the health endpoint to see how the app handles it. Based on what I saw in the kafka library and from this test, I’m more confident in the threadsafety. ref: #207 (comment) |
This PR introduces a health check HTTP endpoint to the online ingestor, allowing external monitoring of the service's connectivity to Kafka, storage, and SciCat.