Java/jOOQ: Add basic demo application and testing rig #9

amotl · 2023-01-20T14:27:19Z

Hi there,

after learning that evaluating CrateDB together with jOOQ is a long cherished wish, this patch finally stages a corresponding testing rig.

The program and test suite can be invoked like:

./gradlew run
./gradlew test

With kind regards,
Andreas.

/cc @hlcianfagna, @matriv, @mkleen, @proddata, @seut, @hammerhead

About

The idea of jOOQ is to generate typesafe code based on the SQL schema. Then, accessing a database table using the jOOQ DSL API looks like this:

cratedb-examples/by-language/java-jooq/src/main/java/io/crate/demo/jooq/Application.java

Lines 110 to 115 in 278b0f8

    
           // Fetch records, with filtering and sorting. 
        
           Result<Record> result = db.select() 
        
                   .from(AUTHOR) 
        
                   .where(AUTHOR.NAME.like("Ja%")) 
        
                   .orderBy(AUTHOR.NAME) 
        
                   .fetch();

Caveat

jOOQ's code generator takes your database schema and reverse-engineers it into a set of Java classes.

This feature currently does not work with CrateDB yet. The code provided within the src/generated directory has not been derived by reflecting the database schema from CrateDB. In order to make it work out of the box using jOOQ's PostgreSQL dialect, CrateDB would need to support WITH RECURSIVE CTEs.

Demo application using CrateDB with jOOQ and the PostgreSQL JDBC driver.

jOOQ's code generator [1] takes your database schema and reverse- engineers it into a set of Java classes. This feature currently does not work with CrateDB yet. The code provided within the `src/generated` directory has not been derived by reflecting the database schema from CrateDB. Hereby, it is provided statically. [1] https://www.jooq.org/doc/latest/manual/code-generation/

mkleen · 2023-01-20T14:52:25Z

@amotl Are you sure that you want to check in autogenerated code ? Would it be easier to generate the code add-hoc as part of the build process ?

amotl · 2023-01-20T15:00:43Z

Are you sure that you want to check in autogenerated code ? Would it be easier to generate the code add-hoc as part of the build process?

@mkleen: Thanks for your suggestion. However, it is currently not possible to generate the corresponding code, see the "Caveat" section above. However, I will do it quickly if you can fix crate/crate#12544 right away? ;]

mkleen · 2023-01-20T15:03:39Z

Are you sure that you want to check in autogenerated code ? Would it be easier to generate the code add-hoc as part of the build process?

@mkleen: Thanks for your suggestion. However, it is currently not possible to generate the corresponding code, see the "Caveat" section above. However, I will do it quickly if you can fix crate/crate#12544 right away? ;]

I am referring to the generated domain model such as

cratedb-examples/by-language/java-jooq/src/generated/java/io/crate/demo/jooq/model/tables/records/AuthorRecord.java

Lines 18 to 23 in eb951b9

    
           @Generated( 
        
                   value = { 
        
                       "https://www.jooq.org", 
        
                       "jOOQ version:3.17.7" 
        
                   }, 
        
                   comments = "This class is generated by jOOQ."

amotl · 2023-01-20T15:06:29Z

It is currently not possible to generate the corresponding code.

jOOQ's code generator runs a CTE with WITH RECURSIVE to reflect the database schema. We are not there yet. The corresponding code has been assembled from other sources, and adjusted by hand.

mkleen · 2023-01-20T15:08:52Z

Why so serious? Here we go, the intention is clearly stated in the README:

I am so serious about that, because I am tired of example apps which don't follow best practices. They are painful to maintain and also painful for the users because they will start on a weak foundation.

mkleen · 2023-01-20T15:09:45Z

What's the best practice with jOOQ ? Is it possible to use it without source generation with a normal domain model ?

amotl · 2023-01-20T15:10:17Z

I am not sure if this a good idea to provide this as a sample app then if it's not possible to use straight away.

Why so serious? Here we go, the intention is clearly stated in the README:

It is intended as a basic example to demonstrate what currently works, and as a testing rig for eventually growing a full-fledged CrateDB dialect, or at least making the code generator work. Contributions are welcome.

amotl · 2023-01-20T15:11:39Z

What's the best practice with jOOQ? Is it possible to use it without source generation with a normal domain model?

This patch is here to demonstrate just that, right. And to open the door for subsequent refinements. Best practice, and main intention, is definitively to use it with code generation, according to upstream docs. Personally, I think the other details of jOOQ are equally powerful, even without using code generation.

mkleen · 2023-01-20T15:26:20Z

What's the best practice with jOOQ? Is it possible to use it without source generation with a normal domain model?

This patch is here to demonstrate just that, right. And to open the door for subsequent refinements. Best practice, and main intention, is definitively to use it with code generation, according to upstream docs. Personally, I think the other details of jOOQ are equally powerful, even without using code generation.

I think if we provide an example with CrateDB and jOOQ then this should work out of the box. If the standard workflow is to generate the domain model from the database schema then this should work too. Imagine to explain to a user that he first has to generate the domain model from another database and then modify it by hand to make it work with CrateDB.

amotl · 2023-01-20T15:32:49Z

I am so serious about that, because I am tired of example apps which don't follow best practices. They are painful to maintain and also painful for the users because they will start on a weak foundation.

I am not sure that committing the generated domain model is not following best practices in this particular case. In order to generate the code, you will need a running database upfront. That would defeat using the domain model within unit tests which use mocking-based techniques right away, essentially rendering the whole application test suite completely useless.

When assembling this code, I followed the same practice of providing the generated domain model classes within the source code repository like seen at ¹. This was invaluable information for me to get started directly using CrateDB, without being able to use the code generator.

I think it is a good start into this topic and will help others to get started quickly as well. I think it does not mislead the user in any way, and follows the style of the example code as provided by the upstream author(s). I would clearly like to see any gaps being closed on behalf of subsequent iterations, in order to leverage more of what jOOQ has to offer.

I think if we provide an example with CrateDB and jOOQ then this should work out of the box.

It does! ;]

If the standard workflow is to generate the domain model from the database schema then this should work too.

I don't think it has to be strictly like that, based on my arguments. However, your question will clearly spawn a corresponding research.

https://github.com/jOOQ/jOOQ-mcve/tree/main/jOOQ-mcve-java/src/main/java/org/jooq/mcve/java ↩

amotl · 2023-01-20T15:38:15Z

Imagine to explain to a user that he first has to generate the domain model from another database and then modify it by hand to make it work with CrateDB.

Please also have a look at the "dynamic schema" example code, which also demonstrates a well-supported subset of jOOQ's feature set.

cratedb-examples/by-language/java-jooq/src/main/java/io/crate/demo/jooq/Application.java

Lines 132 to 191 in eb951b9

    
               /** 
        
                * jOOQ as a standalone SQL builder without code generation [1] 
        
                * 
        
                * If you have a dynamic schema, you don't have to use the code generator. 
        
                * This is the simplest of all use cases, allowing for construction of 
        
                * valid SQL for any database. In this use case, you will not use jOOQ's 
        
                * code generator and maybe not even jOOQ's query execution facilities. 
        
                * 
        
                * Instead, you'll use jOOQ's query DSL API to wrap strings, literals and 
        
                * other user-defined objects into an object-oriented, type-safe AST 
        
                * modelling your SQL statements. 
        
                * 
        
                * [1] https://www.jooq.org/doc/latest/manual/getting-started/use-cases/jooq-as-a-sql-builder-without-codegeneration/ 
        
                * 
        
                */ 
        
               public void exampleWithDynamicSchema() throws IOException, SQLException { 
        
                   DSLContext db = getDSLContext(); 
        
                   Table<Record> BOOK = table("\"testdrive\".\"book\""); 
        
                   Field<Object> BOOK_ID = field("id"); 
        
                   Field<Object> BOOK_TITLE = field("title"); 
        
                   // Create table. 
        
                   String bootstrap_sql = Tools.readTextFile("bootstrap.sql"); 
        
                   db.query(bootstrap_sql).execute(); 
        
                   // Truncate table. 
        
                   db.delete(BOOK).where(DSL.trueCondition()).execute(); 
        
                   db.query(String.format("REFRESH TABLE %s", BOOK)).execute(); 
        
                   // Insert records. 
        
                   InsertSetMoreStep<Record> new_record1 = db.insertInto(BOOK).set(BOOK_ID, 1).set(BOOK_TITLE, "Foo"); 
        
                   InsertSetMoreStep<Record> new_record2 = db.insertInto(BOOK).set(BOOK_ID, 2).set(BOOK_TITLE, "Bar"); 
        
                   new_record1.execute(); 
        
                   new_record2.execute(); 
        
                   db.query(String.format("REFRESH TABLE %s", BOOK)).execute(); 
        
                   // Fetch records, with filtering and sorting. 
        
                   Result<Record> result = db.select() 
        
                           .from(BOOK) 
        
                           .where(BOOK_TITLE.like("B%")) 
        
                           .orderBy(BOOK_TITLE) 
        
                           .fetch(); 
        
                   // Display result. 
        
                   // System.out.println("Result:"); 
        
                   // System.out.println(result); 
        
                   // Iterate and display records. 
        
                   System.out.println("By record:"); 
        
                   for (Record record : result) { 
        
                       // TODO: How can we know about the index positions of the corresponding columns? 
        
                       Integer id = (Integer) record.getValue(0); 
        
                       String title = (String) record.getValue(1); 
        
                       System.out.println("id: " + id + ", title: " + title); 
        
                   } 
        
                   System.out.println(); 
        
               }

mkleen · 2023-01-20T15:38:25Z

Ok, seems like the sample app from jooq also checks in the generated sources
https://github.com/jOOQ/jOOQ-mcve/tree/main/jOOQ-mcve-java/src/main/java/org/jooq/mcve/java

proddata · 2023-01-20T15:46:34Z

However, I will do it quickly if you can fix crate/crate#12544 right away? ;]

Out of curiosity, is this the only missing feature?
Also is this coming from the pg jdbc driver? have you tried the crate variant?

also wondering if generation would work, if specified as org.jooq.meta.jdbc.JDBCDatabase ❓
https://www.jooq.org/doc/latest/manual/code-generation/codegen-advanced/codegen-config-database/codegen-database-name/

amotl · 2023-01-20T16:00:26Z

Out of curiosity, is this the only missing feature?

Code generation is advertised as one of the major features of jOOQ. Personally, I think using jOOQ will become really powerful when eventually supporting CrateDB's container data types OBJECT and ARRAY in a jOOQish way, which is in a slightly different area, and would certainly be desired in the long run.

Working on those details would also be an exercise for subsequent iterations, this patch is only intended to set the stage, by exercising very basic operations.

Also is this coming from the pg jdbc driver? have you tried the crate variant?

I haven't, but I don't think that pgjdbc generates this query, never heard about that. If you think it will give more insights, I can also try the other driver.

amotl · 2023-01-20T16:02:05Z

Also wondering if generation would work, if specified as org.jooq.meta.jdbc.JDBCDatabase?

Thanks for the suggestion, I will try.

amotl · 2023-01-20T16:51:52Z

Also wondering if code generation would work when using the org.jooq.meta.jdbc.JDBCDatabase generator?

Thanks for the suggestion, I will try.

It also croaks, see #10 (comment).

mkleen · 2023-01-23T09:56:42Z

@amotl I spent a bit more time on reading on jOQL. There is the option to use it without code generation. I think I would prefer if we set up an example for jOQL to use an option which the user can reproduce it without any hacks. I would also spend a little bit more time on advanced functionality to see if this works end-to-end. Let's make sure this ages well.

mkleen · 2023-01-23T11:07:45Z

However, if crate/crate#12544 is the blocker to get jOQl properly working, I would probably add an issue and then wait for feedback from the community. Then we can support the feature once it gets upvotes.

amotl · 2023-01-23T11:38:30Z

Hi @mkleen,

thanks for your feedback. Maybe GH-10 already addresses some of your suggestions.

I hope that this patch will only be the baseline, and would like to evaluate other features and more advanced operations than just the basic ones demonstrated here, on behalf of subsequent patches. The two most prominent topics for me would be those:

DDLDatabase: Code generation from SQL files ¹
Using JDBC batch operations (with jOOQ) ²

With kind regards,
Andreas.

amotl · 2023-01-24T11:48:13Z

by-language/java-jooq/src/generated/java/io/crate/demo/jooq/model/DemoDatabase.java

+/**
+ * This class is generated by jOOQ.
+ */
+@Generated(
+        value = {
+                "https://www.jooq.org",
+                "jOOQ version:3.17.7"
+        },
+        comments = "This class is generated by jOOQ."
+)


@seut mentioned that the pseudo-generated classes should not contain comments like this.

Instead, the comments should be more clear about that they have been hand-written / derived from other examples, eventually citing the README:

cratedb-examples/by-language/java-jooq/README.rst

Lines 46 to 52 in eb951b9

Caveats

=======

- `jOOQ's code generator`_ takes your database schema and reverse-engineers it

into a set of Java classes. This feature currently does not work with CrateDB

yet. The code provided within the ``src/generated`` directory has not been

derived by reflecting the database schema from CrateDB.

How would a solution look like without generated code ? Would that result in the same kind of classes ?

@mkleen Afaik this can be seen here

cratedb-examples/by-language/java-jooq/src/main/java/io/crate/demo/jooq/Application.java

Line 132 in eb951b9

/**

.

GH-11 takes a better approach on this. Instead of leaving us or any reader hanging on this detail, it uses the existing SQL DDL scripts to reflect the database schema, see DDLDatabase: Code generation from SQL files ¹. Let me know what you think about it.

As outlined within the upstream documentation, and accordingly cited within the README now, this fits well with the very reasonable scenario that your SQL DDL statements are usually maintained in form of multiple incremental migration scripts anyway, when writing an application where schema changes are important to be tracked within the source code repository. Within this scenario, there is no need to connect to an existing database for reflecting its schema.

jOOQ even promises it could cope well with CrateDB-specific extensions to the CREATE TABLE DDL statements, which definitively matters.

While the script uses pretty standard SQL constructs, you may well use some vendor-specific extensions, and even DML statements in between to set up your schema - it doesn't matter. By ignoring unsupported content, the jOOQ SQL parser supports parsing everything that is representable through the jOOQ API, as well as some well known vendor-specific syntax, [and even unknown vendor-specific syntax by adding correspdonding hints].

Footnotes

https://www.jooq.org/doc/latest/manual/code-generation/codegen-ddl/ ↩

amotl force-pushed the amo/jooq/baseline branch from 73313b3 to 00dd60a Compare January 20, 2023 14:35

amotl requested review from hlcianfagna, matriv, mkleen, proddata, seut and hammerhead and removed request for proddata and hlcianfagna January 20, 2023 14:35

amotl added 3 commits January 20, 2023 15:44

Java/jOOQ: Add basic demo application

b60cf0d

Demo application using CrateDB with jOOQ and the PostgreSQL JDBC driver.

Java/jOOQ: Add Gradle wrapper

eb951b9

amotl force-pushed the amo/jooq/baseline branch from 00dd60a to eb951b9 Compare January 20, 2023 14:44

amotl changed the title ~~Add basic jOOQ demo application and testing rig~~ Java/jOOQ: Add basic demo application and testing rig Jan 20, 2023

amotl mentioned this pull request Jan 20, 2023

Java/jOOQ: Add code generation add-on for Gradle, with reflection from the database #10

Draft

amotl commented Jan 24, 2023

View reviewed changes

amotl mentioned this pull request Jan 24, 2023

Java/jOOQ: Add code generation add-on for Gradle, with reflection from SQL DDL files #11

Merged

amotl merged commit da6efeb into main Jan 25, 2023

amotl deleted the amo/jooq/baseline branch January 25, 2023 10:42

amotl mentioned this pull request Jan 25, 2023

[STUB] Java/Jinq: Add README #13

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Java/jOOQ: Add basic demo application and testing rig #9

Java/jOOQ: Add basic demo application and testing rig #9

amotl commented Jan 20, 2023 •

edited

Loading

mkleen commented Jan 20, 2023

amotl commented Jan 20, 2023 •

edited

Loading

mkleen commented Jan 20, 2023 •

edited

Loading

amotl commented Jan 20, 2023 •

edited

Loading

mkleen commented Jan 20, 2023 •

edited

Loading

mkleen commented Jan 20, 2023

amotl commented Jan 20, 2023

amotl commented Jan 20, 2023 •

edited

Loading

mkleen commented Jan 20, 2023

amotl commented Jan 20, 2023 •

edited

Loading

amotl commented Jan 20, 2023

mkleen commented Jan 20, 2023

proddata commented Jan 20, 2023 •

edited

Loading

amotl commented Jan 20, 2023

amotl commented Jan 20, 2023

amotl commented Jan 20, 2023

mkleen commented Jan 23, 2023 •

edited

Loading

mkleen commented Jan 23, 2023 •

edited

Loading

amotl commented Jan 23, 2023 •

edited

Loading

amotl Jan 24, 2023

mkleen Jan 24, 2023

seut Jan 24, 2023

amotl Jan 24, 2023

	// Fetch records, with filtering and sorting.
	Result<Record> result = db.select()
	.from(AUTHOR)
	.where(AUTHOR.NAME.like("Ja%"))
	.orderBy(AUTHOR.NAME)
	.fetch();

	Caveats
	=======

	- `jOOQ's code generator`_ takes your database schema and reverse-engineers it
	into a set of Java classes. This feature currently does not work with CrateDB
	yet. The code provided within the ``src/generated`` directory has not been
	derived by reflecting the database schema from CrateDB.

Java/jOOQ: Add basic demo application and testing rig #9

Java/jOOQ: Add basic demo application and testing rig #9

Conversation

amotl commented Jan 20, 2023 • edited Loading

About

Caveat

mkleen commented Jan 20, 2023

amotl commented Jan 20, 2023 • edited Loading

mkleen commented Jan 20, 2023 • edited Loading

amotl commented Jan 20, 2023 • edited Loading

mkleen commented Jan 20, 2023 • edited Loading

mkleen commented Jan 20, 2023

amotl commented Jan 20, 2023

amotl commented Jan 20, 2023 • edited Loading

mkleen commented Jan 20, 2023

amotl commented Jan 20, 2023 • edited Loading

Footnotes

amotl commented Jan 20, 2023

mkleen commented Jan 20, 2023

proddata commented Jan 20, 2023 • edited Loading

amotl commented Jan 20, 2023

amotl commented Jan 20, 2023

amotl commented Jan 20, 2023

mkleen commented Jan 23, 2023 • edited Loading

mkleen commented Jan 23, 2023 • edited Loading

amotl commented Jan 23, 2023 • edited Loading

Footnotes

amotl Jan 24, 2023

Choose a reason for hiding this comment

mkleen Jan 24, 2023

Choose a reason for hiding this comment

seut Jan 24, 2023

Choose a reason for hiding this comment

amotl Jan 24, 2023

Choose a reason for hiding this comment

Footnotes

amotl commented Jan 20, 2023 •

edited

Loading

amotl commented Jan 20, 2023 •

edited

Loading

mkleen commented Jan 20, 2023 •

edited

Loading

amotl commented Jan 20, 2023 •

edited

Loading

mkleen commented Jan 20, 2023 •

edited

Loading

amotl commented Jan 20, 2023 •

edited

Loading

amotl commented Jan 20, 2023 •

edited

Loading

proddata commented Jan 20, 2023 •

edited

Loading

mkleen commented Jan 23, 2023 •

edited

Loading

mkleen commented Jan 23, 2023 •

edited

Loading

amotl commented Jan 23, 2023 •

edited

Loading