Skip to content

Latest commit

 

History

History
699 lines (548 loc) · 22.5 KB

README.adoc

File metadata and controls

699 lines (548 loc) · 22.5 KB
Raw

Neo4j from the JVM ecosystem.

Examples of how to connect to Neo4j on the JVM, reading and writing data.

The idea of this project is to show how to access the standard movie graph and create a web api for it on different platforms running on the JVM.

All projects have been created either via their official "starter" services or Maven archetypes.

Note
 The order in which the frameworks is alphabetical and doesn’t represent any preference. As I prefer Maven, I chose to create all projects as Maven projects.

Spring Data Neo4j 6 can be used inside a CDI container as well and is known to work in Liberty and Helidon. We provide a CDI extension for that. You need to bring in an application scoped instance of the Neo4j Driver. The portable CDI extension won’t work in Quarkus. So in case you really want to have great Object Mapping but cannot decide to use Spring as a container, you might want to try out the CDI extension.

Neo4j-OGM can work on GraalVM native since version 3.2.19 when you provide an index of the node entities. The index must be in META-INF/resources/name/of/your/package/neo4j-ogm.index. Also the entities in question must be registered for reflection (either through pure GraalVM reflection-config.json or through the means of your selected framework.)

The goal of this repository is to check and demonstrate what is necessary to achieve the following inside the current Java application frameworks and runtimes:

  • Support a couple of "standard" use cases presented as HTTP Api, transporting JSON. This is a hard requirement

  • See if they provide automatic setup of the Neo4j driver or at least a dynamic configuration mechanism to do this

  • Check if there’s an officially supported Data Mapping framework available for Neo4j. Having a data mapping framework layer is really nice to have, but if this is not possible, the Neo4j Java driver plays very well with custom mapping functions or things like MapStruct.

  • Built-in dev-mode with reload / restart feature

  • Dedicated support for optimized docker images (Preferable via the chosen build tool but a two step process via a dedicated docker file is ok)

  • See if a Reactive Streams implementation is present and whether it works with the Neo4j Java driver

  • See if the application can be turned into a GraalVM native image via the chosen build tool without fiddling around with native-image parameters

A red cross doesn’t mean a given Framework is bad in anyway. It is just an indicator that something either doesn’t work without additional configuration or programming or is just not there. An assessment whether a particular thing is needed or not is up to the reader and must be reasoned about in the context of the framework and it’s actually present features.

All the frameworks used and tested here support the actual "business" logic.

Shared Api

All projects provide the following API:

POST /api/people

Example request
$ curl 'http://localhost:8080/api/people' -i -X POST \
    -H 'Content-Type: application/json' \
    -d '{"name":"Lieschen Müller","born":2020}'
Example response
HTTP/1.1 201 Created
Content-Type: application/json
Content-Length: 64

{
  "born" : 2020,
  "name" : "Lieschen Müller",
  "id" : 339
}

GET /api/movies

Example request
$ curl 'http://localhost:8080/api/movies' -i -X GET
Example response
HTTP/1.1 200 OK
transfer-encoding: chunked
Content-Type: text/event-stream;charset=UTF-8
Content-Length: 1270

[ {
  "title" : "A Few Good Men",
  "description" : "In the heart of the nation's capital, in a courthouse of the U.S. government, one man will stop at nothing to keep his honor, and one will stop at nothing to find the truth.",
  "actors" : [ {
    "roles" : [ "Capt. Jack Ross" ],
    "name" : "Kevin Bacon"
  }, {
    "roles" : [ "Lt. Cdr. JoAnne Galloway" ],
    "name" : "Demi Moore"
  }, {
    "roles" : [ "Man in Bar" ],
    "name" : "Aaron Sorkin"
  }, {
    "roles" : [ "Lt. Col. Matthew Andrew Markinson" ],
    "name" : "J.T. Walsh"
  }, {
    "roles" : [ "Lt. Jonathan Kendrick" ],
    "name" : "Kiefer Sutherland"
  }, {
    "roles" : [ "Lt. Sam Weinberg" ],
    "name" : "Kevin Pollak"
  }, {
    "roles" : [ "Cpl. Jeffrey Barnes" ],
    "name" : "Noah Wyle"
  }, {
    "roles" : [ "Cpl. Carl Hammaker" ],
    "name" : "Cuba Gooding Jr."
  }, {
    "roles" : [ "Dr. Stone" ],
    "name" : "Christopher Guest"
  }, {
    "roles" : [ "Lt. Daniel Kaffee" ],
    "name" : "Tom Cruise"
  }, {
    "roles" : [ "Col. Nathan R. Jessup" ],
    "name" : "Jack Nicholson"
  }, {
    "roles" : [ "Pfc. Louden Downey" ],
    "name" : "James Marshall"
  } ],
  "directors" : [ {
    "born" : 1947,
    "name" : "Rob Reiner",
    "id" : 194
  } ],
  "released" : 1992
} ]

Status / Health checks

Liveness

General Liveness of the application:

The “Liveness” state of an application tells whether its internal state allows it to work correctly, or recover by itself if it’s currently failing.

— Spring Boot Documentation
Example request
$ curl 'http://localhost:8080/management/health/liveness' -i -X GET
Example response
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Date: Mon, 16 Nov 2020 12:51:24 GMT
Content-Length: 21

{
  "status" : "UP"
}

Readiness

The “Readiness” state of an application tells whether the application is ready to handle traffic. A failing “Readiness” state tells the platform that it should not route traffic to the application for now.

— Spring Boot Documentation

We expect the Neo4j database connection to be taken into consideration for readiness.

Example Request
$ curl 'http://localhost:8080/management/health/readiness' -i -X GET \
    -H 'Accept: application/json'
Example response
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Date: Mon, 16 Nov 2020 12:54:29 GMT
Content-Length: 21

{
  "status" : "UP"
}

Helidon SE

  • ✅ Automatic setup of the Neo4j driver (A configuration framework however is provided)

  • ⚠️ Officially supported Data Mapping framework available (Neo4j-OGM can be used outside a Spring environment and can use the Driver as is, without CDI.)

  • ❌ Built-in dev-mode with reload / restart feature

  • ⚠️ Dedicated support for optimized docker images (A Docker file is provided)

  • ✅ Reactive Streams: Helidon’s own implementation

  • ✅ GraalVM native compilation

  • ✅ Health (Liveness, Readiness and detailed status)

Demo projects provided:

  • helidon-se-reactive

Created via mvn -U archetype:generate -DinteractiveMode=false -DarchetypeGroupId=io.helidon.archetypes-DarchetypeArtifactId=helidon-quickstart-se -DarchetypeVersion=2.3.0, version tested 2.3.0.

Configuration of the Neo4j Java Driver

No manual work is required. Any namespace in a YAML or properties file can be used. The integration is provided as an artifact under those coordinates:

<dependency>
    <groupId>io.helidon.integrations.neo4j</groupId>
    <artifactId>helidon-integrations-neo4j</artifactId>
</dependency>

This is an example how to use Neo4j properties under the key neo4j:

var driver = config.get("neo4j").as(Neo4j::create).map(Neo4j::driver).orElseThrow();

The driver can than be used in the SE application / microservice.

Running

mvn clean package
java -jar target/helidon-se-reactive.jar

Testing

Manual work required, can be solved with JUnit 5 means and the easy to use Helidon SE api.

Create docker images

Two steps required, a Dockerfile is provided.

mvn clean package
docker build .

Create native images

# For your current system, GraalVM 11 is required
mvn clean package -Pnative-image
# As a native docker image
mvn clean package
docker build -f Dockerfile.native .

Helidon offers io.helidon.common.Reflected for classes that needs to be included in the image and require reflection based access.

Health

Health infrastructure provided with Helidon Health Checks io.helidon.health:helidon-health and io.helidon.health:helidon-health-checks, support for Neo4j included via io.helidon.integrations.neo4j:helidon-integrations-neo4j-health.

However, the build-in support in 2.3.0 is broken for native compilation, see GH-3060.

Format of the individual Neo4j status:

Request
$ curl 'http://localhost:8080/management/health/readiness' -i -X GET
{
  "outcome": "UP",
  "status": "UP",
  "checks": [
    {
      "name": "Neo4j connection health check",
      "state": "UP",
      "status": "UP",
      "data": {
        "database": "neo4j",
        "server": "Neo4j/4.2.0@localhost:7687"
      }
    }
  ]
}

Micronaut

  • ⚠️ Automatic setup of the Neo4j driver (Only URL, credentials and TLS settings supported as properties)

  • ⚠️ Officially supported Data Mapping framework available (GORM, in beta for Neo4j, SDN 6 might work as well)

  • ✅ Built-in dev-mode with reload / restart feature

  • ✅ Dedicated support for optimized docker images

  • ✅ Reactive Streams: RxJava2 and 3

  • ✅ GraalVM native compilation

  • ✅ Health (Liveness, Readiness and detailed status)

Demo projects provided:

  • micronaut-reactive

Created via: https://micronaut.io/launch/, version tested 2.2.0.

Configuration of the Neo4j Java Driver

Support of the some 4.0.x config options under the namespace neo4j.* Basic setup:

neo4j:
  username: neo4j
  password: secret
  uri: bolt://localhost:7687

Running

./mvnw mn:run

Testing

Directly supported only with an older, not reactive capable version of Neo4j embedded.

Create docker images

The packaging in the pom.xml must be set via a property and the io.micronaut.build:micronaut-maven-plugin Maven plugin must be defined. Both is done via the generator by default. The Maven plugin uses Googles jib-maven-plugin under the hhod.

./mvnw clean package -Dpackaging=docker

The plugin can also be used to generate a Docker file for further editing. Similar tooling with the same features is available for Gradle.

Create native images

# For your current system, GraalVM 11 20.3 is required
./mvnw clean package -Dpackaging=native-image
# As a native docker image
./mvnw clean package -Dpackaging=docker-native

Micronaut offers io.micronaut.core.annotation.Introspected for classes that should be instrospected as Beans on compile time. There is also io.micronaut.core.annotation.ReflectiveAccess to mark constructors, methods and fields for reflective access..

The Maven setup requires the io.micronaut:micronaut-graal annotation processor to correctly produce native image calls:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-compiler-plugin</artifactId>
  <configuration>
    <annotationProcessorPaths combine.children="append">
      <path>
        <groupId>io.micronaut:micronaut-graal</groupId>
        <artifactId>micronaut-graal</artifactId>
        <version>${micronaut.version}</version>
      </path>
    </annotationProcessorPaths>
  </configuration>
</plugin>

Health

Health infrastructure provided with Micronaut Management (io.micronaut:micronaut-management), Neo4j included ootb.

Note
 Path to individual endpoints for liveness and readiness seems not to be changeable.

Format of the individual Neo4j status:

Request
$ curl 'http://localhost:8080/management/health' -i -X GET
{
  "name": "micronaut-reactive",
  "status": "UP",
  "details": {
    "neo4j": {
      "name": "micronaut-reactive",
      "status": "UP",
      "details": {
        "server": "Neo4j/4.1.0@localhost:7687"
      }
    }
  }
}

Quarkus

  • ✅ Automatic setup of the Neo4j driver

  • ✅ Officially supported Data Mapping framework available (Neo4j-OGM, JVM and dev modes work ootb, native mode needs an index, which is just an enumeration of the mapped entities.)

  • ✅ Built-in dev-mode with reload / restart feature

  • ✅ Dedicated support for optimized docker images

  • ✅ Reactive Streams: Smallrye Mutiny

  • ✅ GraalVM native compilation

  • ✅ Health (Liveness, Readiness and detailed status)

Demo projects provided:

  • quarkus-imperative

  • quarkus-ogm (Take note of the entity index here.)

  • quarkus-reactive

Created via: https://code.quarkus.io, version tested: 1.9.2.Final.

Configuration of the Neo4j Java Driver

Support of the all relevant 4.1.x config options under the namespace quarkus.neo4j.*, including TLS: Basic setup:

quarkus.neo4j.uri=bolt://localhost:7687
quarkus.neo4j.authentication.username=neo4j
quarkus.neo4j.authentication.password=secret

Running

./mvnw quarkus:dev

Testing

Easy setup of test connections (via a custom QuarkusTestResourceLifecycleManager). Can be used with Neo4j embedded test harness (as in the example) or with Neo4j Test-Containers.

Create docker images

(Extension container-image-docker must be provided once, via ./mvnw quarkus:add-extension -Dextensions="container-image-docker").

./mvnw clean package -Dquarkus.container-image.build=true

Create native images

# For your current system, GraalVM 11 is required
./mvnw clean package -Pnative
# As a native docker image
./mvnw package -Pnative -Dquarkus.native.container-build=true -Dquarkus.container-image.build=true

Quarkus offers io.quarkus.runtime.annotations.RegisterForReflection for classes that needs to be included in the image and require reflection based access.

Health

Health infrastructure provided with Quarkus Smallrye Health (io.quarkus:quarkus-smallrye-health), Neo4j included ootb. All paths can be easily configured (health separate from liveness and readiness).

Format of the individual Neo4j status:

Request
$ curl 'http://localhost:8080/management/health' -i -X GET
{
  "status": "UP",
  "checks": [
    {
      "name": "Neo4j connection health check",
      "status": "UP",
      "data": {
        "server": "Neo4j/4.1.0@localhost:7687",
        "database": "neo4j"
      }
    }
  ]
}

Spring

  • ✅ Automatic setup of the Neo4j driver

  • ✅ Officially supported Data Mapping framework available (SDN 6 for current, SDN5+OGM for older version)

  • ✅ Built-in dev-mode with reload / restart feature

  • ✅ Dedicated support for optimized docker images

  • ✅ Reactive Streams: Project Reactor

  • ⚠️ GraalVM native compilation (Currently in beta, not part of a standard setup)

  • ✅ Health (Liveness, Readiness and detailed status)

Demo projects provided:

  • spring-plain-imperative

  • spring-plain-reactive

  • spring-data-imperative

  • spring-data-reactive

Created via: https://start.spring.io, version tested: 2.5.1.

There is an additional project, spring-boot24-with-sdn-ogm that does a bit work on the dependencies so that people can use SDN+OGM with the most recent versions of Spring Boot. The preferred way of using SDN+OGM however is Spring Boot prior to 2.4.

Configuration of the Neo4j Java Driver

Full support of all official 4.1.x config options under the namespace spring.neo4j.*. Basic setup:

spring.neo4j.uri=bolt://localhost:7687
spring.neo4j.authentication.username=neo4j
spring.neo4j.authentication.password=secret

Running

./mvnw spring-boot:run

Testing

Easy setup of test connections (via @DataNeo4jTest and a custom @DynamicPropertySource). Can be used with Neo4j embedded test harness (as in the example) or with Neo4j Test-Containers.

Create docker images

./mvnw -DskipTests clean spring-boot:build-image

Similar tooling with the same features is available for Gradle.

Create native images

Provide the compile time dependency

<dependency>
	<groupId>org.springframework.experimental</groupId>
	<artifactId>spring-graalvm-native</artifactId>
	<version>0.8.3</version>
	<scope>compile</scope>
</dependency>

Adapt the build config like this

<build>
	<plugins>
		<plugin>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-maven-plugin</artifactId>
			<configuration>
				<image>
					<builder>paketobuildpacks/builder:tiny</builder>
					<env>
						<BP_BOOT_NATIVE_IMAGE>1</BP_BOOT_NATIVE_IMAGE>
						<BP_BOOT_NATIVE_IMAGE_BUILD_ARGUMENTS>
							-Dspring.spel.ignore=true
							-Dspring.native.remove-yaml-support=true
						</BP_BOOT_NATIVE_IMAGE_BUILD_ARGUMENTS>
					</env>
				</image>
			</configuration>
		</plugin>
	</plugins>
</build>

And ./mvnw -DskipTests clean spring-boot:build-image will create a native image. Tested with Spring Boot 2.4 and Spring GraalVM Native 0.8.3.

Health

Health infrastructure provided with Spring Boot Actuator (org.springframework.boot:spring-boot-starter-actuator), Neo4j included ootb.

Note
 Path to individual endpoints for liveness and readiness seems to be changeable with management.endpoint.health.group.live.include=livenessState, but that feels not very intuitive.

Format of the individual Neo4j status:

Request
$ curl 'http://localhost:8080/management/health' -i -X GET
{
  "status": "UP",
  "components": {
    "neo4j": {
      "status": "UP",
      "details": {
        "server": "Neo4j/4.1.0@localhost:7687",
        "edition": "enterprise",
        "database": "neo4j"
      }
    }
  }
}

TCK

I have created a TCK - basically a glorified end-to-end-test - that brings up each application and ensure it’s expected behaviour. You need bash, Docker and JDK 16 to run it:

./runTck.sh

It will bring up a Neo4j docker instance and take each project, build a docker image, start it and than executes a couple of requests against it. The script is tested currently only under macOS.

Some numbers

All scripts to build, verify and benchmark the images are in this repository. The path of least resistance (or effort) has been chosen to build the images, JVM and native one, following the official instructions.

All tests have been conducted with apib: API Bench using the following options:

apib -c20 -d60 http://localhost:$EXPOSED_PORT/api/movies

The application was connected against a Neo4j container running on the same host: The Neo4j container has been shutdown and restarted between each benchmark run.

apib fully supports HTTP 1.1 including keep-alives and chunked encoding. Apache Bench cannot reliable benchmark the reactive infrastructure without it.

Time to readiness is the time from container start until the container reports UP in a GET /management/health/readiness.

Table 1. Some numbers
Framework Image size Time to healthy Memory usage (before) Throughput Average latency Memory usage (after )

helidon-se-reactive

231.0MB

0m2.016s🥉

148.1MiB

885.052 requests/second

25.628 milliseconds

788.3MiB

helidon-se-reactive-native

33.6MB🥇

0m0.676s🥈

60.73MiB🥉

729.026 requests/second

32.379 milliseconds

853.7MiB

micronaut-imperative

362.0MB

0m2.581s

169.9MiB

1006.241 requests/second

19.904 milliseconds🥈

799.7MiB

micronaut-reactive

362.0MB

0m2.579s

172.00MiB

147.376 requests/second

135.795 milliseconds

693.0MiB

micronaut-reactive-native

91.2MB🥈

0m0.594s🥇

68.30MiB

135.618 requests/second

147.520 milliseconds

962.3MiB

quarkus-imperative

464.0MB

0m4.283s

103.8MiB

1095.802 requests/second🥉

18.272 milliseconds🥇

666.6MiB

quarkus-imperative-native

150.0MB

0m3.316s

26.09MiB🥇

996.390 requests/second

20.096 milliseconds🥉

699.3MiB

quarkus-ogm

467.0MB

0m4.335s

111.7MiB

622.312 requests/second

32.175 milliseconds

908.1MiB

quarkus-reactive

466.0MB

0m4.411s

117.0MiB

487.273 requests/second

41.087 milliseconds

665.8MiB

quarkus-reactive-native

151.0MB

0m3.353s

26.42MiB🥈

446.196 requests/second

44.868 milliseconds

650.8MiB

spring-boot23-with-sdn-ogm

290.0MB

0m4.719s

197.40MiB

49.789 requests/second

400.856 milliseconds

411.4MiB🥇

spring-boot24-with-sdn-ogm

286.0MB

0m4.843s

206.10MiB

42.680 requests/second

468.021 milliseconds

610.5MiB

spring-data-imperative

300.0MB

0m6.105s

236.90MiB

709.336 requests/second

28.226 milliseconds

797.1MiB

spring-data-imperative-native

134.0MB🥉

0m3.462s

71.08MiB

605.255 requests/second

33.078 milliseconds

722.0MiB

spring-data-reactive

302.0MB

0m6.268s

241.50MiB

673.214 requests/second

29.762 milliseconds

738MiB

spring-plain-imperative

297.0MB

0m6.066s

197.80MiB

1258.016 requests/second🥈

15.917 milliseconds

414.6MiB🥈

spring-plain-reactive

299.0MB

0m5.945s

253.40Mi B

1272.513 requests/second🥇

15.738 milliseconds

588.4MiB🥉

Numbers taken on a MacBook Pro with 2.4Ghz Intel Core i9 and 32 GB Ram. Docker set to use at Max 8 "CPUs" and 8GiB memory in total. The numbers for quarkus-ogm reflect the state prior to commit 7e74c8e1c373cb72715a2e095b15f1c95b9d109f.