Add support to remote kafka plugin #316

ktaborski · 2021-04-20T06:24:11Z

Add support to remote Kafka plugin.
I've added similar code part as for JNLP connection.

Resolves: #315

Make sure you are opening from a topic/feature/bugfix branch (right side) and not your master branch!
Ensure that the pull request title represents the desired changelog entry
Please describe what you did
Link to relevant issues in GitHub or Jira
Link to relevant pull requests, esp. upstream and downstream changes
Ensure you have provided tests - that demonstrates feature works or fixes the issue

olivergondza · 2021-04-20T07:39:00Z

Thanks for the contribution!

I have to admit I am uneasy about the kafka-remoting plugin. Total 80 installations worldwide, no other Jenkins plugins supporting this, fairly rich dependencies, no release in past 2 years, etc. Also, the scarce documentation mentioning that "There are also issues with traffic prioritization and multi-agent communications, which impact Jenkins stability and scalability.". All in all it seems to me as an experiment that did not quite gained popularity, and there is no investment into it anymore.

@ktaborski, do you have positive experience using kafka-remoting? How about implementing this as a standalone plugin depending on both kafka-remoting and openstack-cloud that will add support for openstack-cloud to use kafka nodes? I would be happy to help you on crafting one...

ktaborski · 2021-04-20T09:25:50Z

do you have positive experience using kafka-remoting?
I have quite negative experience with JNLP connections (they are not stable - at least in my environment) and based on documentation kafka-remoting seems to be promising alternative.

Now I am in PoC/testing phase, so hard to say about positive experience. Normally I am using openstack plugin to launch slaves and I met lack of kafka-remoting support.

How about implementing this as a standalone plugin depending on both kafka-remoting and openstack-cloud that will add support for openstack-cloud to use kafka nodes?
I have no experience in plugin development and adaptations in current plugin seems to be easiest solution for me.

ktaborski · 2021-04-20T10:23:35Z

Also, the scarce documentation mentioning that "There are also issues with traffic prioritization and multi-agent communications, which impact Jenkins stability and scalability."
This part of documentation is about TCP-based connection.

In "Benefits to the community section":
Help to resolve traffic prioritization and multi-agent communications issue in Jenkins.

olivergondza · 2021-04-23T07:43:37Z

Also, the scarce documentation mentioning that "There are also issues with traffic prioritization and multi-agent communications, which impact Jenkins stability and scalability."

This part of documentation is about TCP-based connection.

You are right, my bad.

I have no experience in plugin development and adaptations in current plugin seems to be easiest solution for me.

Let me help you with that. I will do the code review here to keep you going. Once the new plugin is ready, the code can be moved there easily.

olivergondza · 2021-04-23T07:44:49Z

plugin/src/main/java/jenkins/plugins/openstack/compute/JCloudsCloud.java

@@ -164,6 +164,8 @@ private Object readResolve() {
        LauncherFactory lf = null;
        if ("JNLP".equals(slaveOptions.slaveType)) {
            lf = LauncherFactory.JNLP.JNLP;
+        } else if ("KAFKA".equals(slaveOptions.slaveType)) {


There is no need to add these into readResolve methods. They are only used to migrate old config to new one. And since kafka is just being added, it does not exist in user configs.

olivergondza · 2021-04-23T07:45:00Z

plugin/src/main/java/jenkins/plugins/openstack/compute/JCloudsSlave.java

+                lf = new LauncherFactory.SSH(credentialsId);
+            } else if ("JNLP".equals(slaveType)) {
+                lf = LauncherFactory.JNLP.JNLP;
+            } else if ("KAFKA".equals(slaveType)) {


There is no need to add these into readResolve methods. They are only used to migrate old config to new one. And since kafka is just being added, it does not exist in user configs.

olivergondza · 2021-04-23T07:45:11Z

plugin/src/main/java/jenkins/plugins/openstack/compute/JCloudsSlaveTemplate.java

@@ -122,6 +122,8 @@ private Object readResolve() {
                lf = new LauncherFactory.SSH(credentialsId);
            } else if("JNLP".equals(slaveType)) {
                lf = LauncherFactory.JNLP.JNLP;
+            } else if("KAFKA".equals(slaveType)) {


There is no need to add these into readResolve methods. They are only used to migrate old config to new one. And since kafka is just being added, it does not exist in user configs.

olivergondza · 2021-04-23T07:45:19Z

plugin/src/main/java/jenkins/plugins/openstack/compute/JCloudsSlaveTemplate.java

@@ -153,6 +155,9 @@ private Object readResolve() {
            } else if ("SSH".equals(slaveOptions.slaveType)) {
                lf = new LauncherFactory.SSH(slaveOptions.credentialsId);
                slaveOptions.slaveType = slaveOptions.credentialsId = null;
+            } else if ("KAFKA".equals(slaveOptions.slaveType)) {


There is no need to add these into readResolve methods. They are only used to migrate old config to new one. And since kafka is just being added, it does not exist in user configs.

olivergondza · 2021-04-23T07:46:16Z

plugin/src/main/java/jenkins/plugins/openstack/compute/UserDataVariableResolver.java

@@ -77,6 +78,16 @@
                "Labels of the node.",
                r -> r.labelString
        );
+        stub(
+                "SLAVE_KAFKA_SECRET",


Please refrain from using the word slave and prefer agent. https://www.jenkins.io/blog/2020/06/18/terminology-update/

I just want to be compliant with SLAVE_JNLP_SECRET variable.

Yeah, we would have to rename that one as well.

olivergondza · 2021-04-23T07:46:22Z

plugin/src/main/java/jenkins/plugins/openstack/compute/UserDataVariableResolver.java

+                r ->  Util.fixNull(KafkaSecretManager.getConnectionSecret(r.serverName))
+        );
+        stub(
+                "SLAVE_NAME",


Please refrain from using the word slave and prefer agent. https://www.jenkins.io/blog/2020/06/18/terminology-update/

olivergondza · 2021-04-23T07:48:19Z

...ain/resources/jenkins/plugins/openstack/compute/slaveopts/LauncherFactory/KAFKA/config.jelly

+<j:jelly xmlns:j="jelly:core" xmlns:f="/lib/form">
+    <f:section title="KAFKA remote agent launch">
+        <f:entry>
+            <p>Jenkins will wait for the Kafka connection to be initiated from the slave itself. Note this plugin does not initiate


Please refrain from using the word slave and prefer agent. https://www.jenkins.io/blog/2020/06/18/terminology-update/

olivergondza · 2021-04-23T07:51:21Z

...ain/resources/jenkins/plugins/openstack/compute/slaveopts/LauncherFactory/KAFKA/config.jelly

+    <f:section title="KAFKA remote agent launch">
+        <f:entry>
+            <p>Jenkins will wait for the Kafka connection to be initiated from the slave itself. Note this plugin does not initiate
+            that. See help for <i>User Data</i> field for more information on the available options.</p>


We might need to massage this a bit, by ideally moving the details here. For now, JNLP was the only inbound connection supported so the user-data config mentioned that only. Kafka seems to require specific launching instructions, that needs to be documented here (or at least link to plugin documentation). Also, there will be no way for the other plugin to contribute documentation for User Data, so this might be the only place to document it.

ktaborski marked this pull request as ready for review April 20, 2021 06:24

ktaborski marked this pull request as draft April 20, 2021 06:25

ktaborski force-pushed the remote-kafka branch 5 times, most recently from 87cbd15 to 41cbafe Compare April 20, 2021 07:02

ktaborski force-pushed the remote-kafka branch 4 times, most recently from 977e5ab to 4de0c41 Compare April 20, 2021 08:40

ktaborski force-pushed the remote-kafka branch 3 times, most recently from 62cd279 to 1078233 Compare April 23, 2021 06:00

olivergondza reviewed Apr 23, 2021

View reviewed changes

Add support to remote kafka plugin

6fe6e03

ktaborski force-pushed the remote-kafka branch from c902d46 to 6fe6e03 Compare April 23, 2021 10:10

ktaborski closed this Feb 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support to remote kafka plugin #316

Add support to remote kafka plugin #316

ktaborski commented Apr 20, 2021

olivergondza commented Apr 20, 2021

ktaborski commented Apr 20, 2021

ktaborski commented Apr 20, 2021

olivergondza commented Apr 23, 2021

olivergondza Apr 23, 2021

olivergondza Apr 23, 2021

olivergondza Apr 23, 2021

olivergondza Apr 23, 2021

olivergondza Apr 23, 2021

ktaborski Apr 23, 2021

olivergondza Apr 23, 2021

olivergondza Apr 23, 2021

olivergondza Apr 23, 2021 •

edited

Loading

olivergondza Apr 23, 2021

Add support to remote kafka plugin #316

Add support to remote kafka plugin #316

Conversation

ktaborski commented Apr 20, 2021

olivergondza commented Apr 20, 2021

ktaborski commented Apr 20, 2021

ktaborski commented Apr 20, 2021

olivergondza commented Apr 23, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olivergondza Apr 23, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olivergondza Apr 23, 2021 •

edited

Loading