Adding health reports to microservices

Note	This repository contains the guide documentation source. To view the guide in published form, view it on the Open Liberty website.

Explore how to report and check the health of a microservice with MicroProfile Health.

What you’ll learn

You will learn how to use MicroProfile Health to report the health status of microservices and take appropriate actions based on this report.

MicroProfile Health allows services to report their health, and it publishes the overall health status to a defined endpoint. A service reports UP if it is available and reports DOWN if it is unavailable. MicroProfile Health reports an individual service status at the endpoint and indicates the overall status as UP if all the services are UP. A service orchestrator can then use the health statuses to make decisions.

A service checks its own health by performing necessary self-checks and then reports its overall status by implementing the API provided by MicroProfile Health. A self-check can be a check on anything that the service needs, such as a dependency, a successful connection to an endpoint, a system property, a database connection, or the availability of required resources. MicroProfile offers checks for startup, liveness, and readiness.

You will add startup, liveness, and readiness checks to the system and inventory services, that are provided for you, and implement what is necessary to report health status by using MicroProfile Health.

https://raw.githubusercontent.com/OpenLiberty/guides-common/prod/gitclone.adoc

https://raw.githubusercontent.com/OpenLiberty/guides-common/prod/twyb-intro.adoc

The system and inventory services can be found at the following URLs:

• http://localhost:9080/system/properties

• http://localhost:9080/inventory/systems

Visit the http://localhost:9080/health URL to see the overall health status of the application, as well as the aggregated data of the startup, liveness and readiness checks. Three checks show the state of the system service, and the other three checks show the state of the inventory service. As you might expect, all services are in the UP state, and the overall health status of the application is in the UP state.

Access the /health/started endpoint by visiting the http://localhost:9080/health/started URL to view the data from the startup health checks. You can also access the /health/live endpoint by visiting the http://localhost:9080/health/live URL to view the data from the liveness health checks. Similarly, access the /health/ready endpoint by visiting the http://localhost:9080/health/ready URL to view the data from the readiness health checks.

https://raw.githubusercontent.com/OpenLiberty/guides-common/prod/twyb-end.adoc

Adding health checks to microservices

Navigate to the start directory to begin.

https://raw.githubusercontent.com/OpenLiberty/guides-common/prod/devmode-lmp33-start.adoc

A health report will be generated automatically for all services that enable MicroProfile Health. The mpHealth feature has already been enabled for you in the src/main/liberty/config/server.xml file.

All services must provide an implementation of the HealthCheck interface, which is used to verify their health. MicroProfile Health offers health checks for startup, liveness, and readiness. A startup check allows applications to define startup probes that are used for initial verification of the application before the Liveness probe takes over. For example, a startup check might check which applications require additional startup time on their first initialization. A liveness check allows third-party services to determine whether a microservice is running. If the liveness check fails, the application can be terminated. For example, a liveness check might fail if the application runs out of memory. A readiness check allows third-party services, such as Kubernetes, to determine whether a microservice is ready to process requests. For example, a readiness check might check dependencies, such as database connections.

server.xml

link:finish/src/main/liberty/config/server.xml[role=include]

Adding health checks to the system service

Create the SystemStartupCheck class.
src/main/java/io/openliberty/guides/system/SystemStartupCheck.java

SystemStartupCheck.java

link:finish/src/main/java/io/openliberty/guides/system/SystemStartupCheck.java[role=include]

The @Startup annotation indicates that this class is a startup health check procedure. In this case, you are checking the cpu usage. If more than 95% of the cpu is being used, a status of DOWN is returned.

Create the SystemLivenessCheck class.
src/main/java/io/openliberty/guides/system/SystemLivenessCheck.java

SystemLivenessCheck.java

link:finish/src/main/java/io/openliberty/guides/system/SystemLivenessCheck.java[role=include]

The @Liveness annotation indicates that this class is a liveness health check procedure. In this case, you are checking the heap memory usage. If more than 90% of the maximum memory is being used, a status of DOWN is returned.

Create the SystemReadinessCheck class.
src/main/java/io/openliberty/guides/system/SystemReadinessCheck.java

SystemReadinessCheck.java

link:finish/src/main/java/io/openliberty/guides/system/SystemReadinessCheck.java[role=include]

The @Readiness annotation indicates that this class is a readiness health check procedure. By pairing this annotation with the ApplicationScoped context from the Contexts and Dependency Injections API, the bean is discovered automatically when the http://localhost:9080/health endpoint receives a request.

The call() method is used to return the health status of a particular service. In this case, you are checking if the server name is defaultServer and returning UP if it is, and DOWN otherwise. This example is a very simple implementation of the call() method. In a real environment, you would orchestrate more meaningful health checks.

Adding health checks to the inventory service

Create the InventoryStartupCheck class.
src/main/java/io/openliberty/guides/inventory/InventoryStartupCheck.java

InventoryStartupCheck.java

link:finish/src/main/java/io/openliberty/guides/inventory/InventoryStartupCheck.java[role=include]

This startup check verifies that the cpu usage is below 95%. If more than 95% of the cpu is being used, a status of DOWN is returned.

Create the InventoryLivenessCheck class.
src/main/java/io/openliberty/guides/inventory/InventoryLivenessCheck.java

InventoryLivenessCheck.java

link:finish/src/main/java/io/openliberty/guides/inventory/InventoryLivenessCheck.java[role=include]

As with the system liveness check, you are checking the heap memory usage. If more than 90% of the maximum memory is being used, a DOWN status is returned.

Create the InventoryReadinessCheck class.
src/main/java/io/openliberty/guides/inventory/InventoryReadinessCheck.java

InventoryReadinessCheck.java

link:finish/src/main/java/io/openliberty/guides/inventory/InventoryReadinessCheck.java[role=include]

In the isHealthy() method, you report the inventory service as not ready if the service is in maintenance or if its dependant service is unavailable.

For simplicity, the custom io_openliberty_guides_inventory_inMaintenance MicroProfile Config property, which is defined in the resources/CustomConfigSource.json file, indicates whether the service is in maintenance. This file was already created for you.

Moreover, the readiness health check procedure makes an HTTP GET request to the system service and checks its status. If the request is successful, the inventory service is healthy and ready because its dependant service is available. Otherwise, the inventory service is not ready and an unhealthy readiness status is returned.

If you are curious about the injected inventoryConfig object or if you want to learn more about MicroProfile Config, see Configuring microservices.

CustomConfigSource.json

link:finish/resources/CustomConfigSource.json[role=include]

https://raw.githubusercontent.com/OpenLiberty/guides-common/prod/devmode-build.adoc

While the Liberty is running, navigate to the http://localhost:9080/health URL to find the aggregated startup, liveness, and readiness health reports on the two services.

You can also navigate to the http://localhost:9080/health/started URL to view the startup health report, to the http://localhost:9080/health/live URL to view the liveness health report or the http://localhost:9080/health/ready URL to view the readiness health report.

Put the inventory service in maintenance by setting the io_openliberty_guides_inventory_inMaintenance property to true in the CustomConfigSource.json file.

Replace the CustomConfigSource.json file.
resources/CustomConfigSource.json

CustomConfigSource.json

link:finish/resources/CustomConfigSource2.json[role=include]

Because this configuration file is picked up dynamically, simply refresh the http://localhost:9080/health URL to see that the state of the inventory service changed to DOWN. The overall state of the application also changed to DOWN as a result. Go to the http://localhost:9080/inventory/systems URL to verify that the inventory service is indeed in maintenance. Set the io_openliberty_guides_inventory_inMaintenance property back to false after you are done.

Testing health checks

You will implement several test methods to validate the health of the system and inventory services.

Create the HealthIT class.
src/test/java/it/io/openliberty/guides/health/HealthIT.java

HealthIT.java

link:finish/src/test/java/it/io/openliberty/guides/health/HealthIT.java[role=include]

Let’s break down the test cases:

• The testStartup() test case compares the generated health report for the startup checks with the actual status of the services.

• The testLiveness() test case compares the generated health report for the liveness checks with the actual status of the services.

• The testReadiness() test case compares the generated health report for the readiness checks with the actual status of the services.

• The testHealth() test case compares the generated health report with the actual status of the services. This test also puts the inventory service in maintenance by setting the io_openliberty_guides_inventory_inMaintenance property to true and comparing the generated health report with the actual status of the services.

A few more tests were included to verify the basic functionality of the system and inventory services. They can be found under the src/test/java/it/io/openliberty/guides/inventory/InventoryEndpointIT.java and src/test/java/it/io/openliberty/guides/system/SystemEndpointIT.java files. If a test failure occurs, then you might have introduced a bug into the code. These tests run automatically as a part of the integration test suite.

CustomConfigSource.json

link:finish/resources/CustomConfigSource.json[role=include]

InventoryEndpointIT.java

link:finish/src/test/java/it/io/openliberty/guides/inventory/InventoryEndpointIT.java[role=include]

SystemEndpointIT.java

link:finish/src/test/java/it/io/openliberty/guides/system/SystemEndpointIT.java[role=include]

https://raw.githubusercontent.com/OpenLiberty/guides-common/prod/devmode-test.adoc

You see the following output:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running it.io.openliberty.guides.health.HealthIT
[INFO] [WARNING ] CWMMH0052W: The class io.openliberty.microprofile.health30.impl.HealthCheck30ResponseImpl implementing HealthCheckResponse in the guide-microprofile-health application in module guide-microprofile-health.war, reported a DOWN status with data Optional[{}].
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.706 s - in it.io.openliberty.guides.health.HealthIT
[INFO] Running it.io.openliberty.guides.system.SystemEndpointIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in it.io.openliberty.guides.system.SystemEndpointIT
[INFO] Running it.io.openliberty.guides.inventory.InventoryEndpointIT
[INFO] [WARNING ] Interceptor for {http://client.inventory.guides.openliberty.io/}SystemClient has thrown exception, unwinding now
[INFO] Could not send Message.
[INFO] [err] The specified host is unknown.
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.171 s - in it.io.openliberty.guides.inventory.InventoryEndpointIT
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0

The warning messages are expected. The first warning results from a request to a service that is under maintenance. This request is made in the testHealth() test from the InventoryEndpointIT integration test. The second warning and error results from a request to a bad or an unknown hostname. This request is made in the testUnknownHost() test from the InventoryEndpointIT integration test.

The tests might fail if your system CPU or memory use is high. The status of the system is DOWN if the CPU usage is over 95%, or the memory usage is over 90%.

To see whether the tests detect a failure, manually change the configuration of io_openliberty_guides_inventory_inMaintenance from false to true in the resources/CustomConfigSource.json file. Rerun the tests to see a test failure occur. The test failure occurs because the initial status of the inventory service is DOWN.

https://raw.githubusercontent.com/OpenLiberty/guides-common/prod/devmode-quit-ctrlc.adoc

Great work! You’re done!

You just learned how to add health checks to report the states of microservices by using MicroProfile Health in Open Liberty. Then, you wrote tests to validate the generated health report.

Feel free to try one of the related MicroProfile guides. They demonstrate additional technologies that you can learn and expand on top of what you built here.

https://raw.githubusercontent.com/OpenLiberty/guides-common/prod/attribution.adoc