Stream pipeline console output

[das7pad]

• Fixes Viewing large logs runs out of memory #704
• Uses incremental version of Implement non-buffered streaming for step logs workflow-api-plugin#438 and Stream HTML console log jenkins#11177

Instead of reading the entire step log into memory before emitting it in the JSON response, this PR is using the streaming based reading for step logs from stapler/Jenkins core.

I've kept the old consoleLog endpoint as it is part of the API docs. What is the policy for removing endpoints?

The new streaming mode ships logs as is, without stopping at the line end. So we need to glue lines back together, see added test incremental log fetching > when gluing lines together.

For later: use ?searchNewLineUntil when fetching more logs to avoid displaying partial lines.

Testing done

• Build locally (Using the incrementals locally does not seem to work for me)
• View large logs (Viewing large logs runs out of memory #704), see no notable memory increase on the server-side. 🎉
• View logs of completed steps
• View logs of pending steps (initial tail request, then with incrementing ?start: Implement incremental tailing of logs #976)
• View annotations (e.g. timestamper and prompts/input)

Submitter checklist

• Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• Ensure that the pull request title represents the desired changelog entry
• Please describe what you did
• Link to relevant issues in GitHub or Jira
• Link to relevant pull requests, esp. upstream and downstream changes
• Ensure you have provided tests that demonstrate the feature works or the issue is fixed

[timja]

Build locally (Using the incrementals locally does not seem to work for me)

If you're using IntelliJ you need to enable the consume-incrementals profile. There's a bug in IntelliJ where it doesn't read profiles from .mvn/maven.config

(I can't find the bug report right now but I'm pretty sure its reported: https://youtrack.jetbrains.com/issues?q=%7Bmaven.config%7D%20%7Bprofiles%7D%20state:%20Open&sort=state)

[timja]

I think its unlikely people are using it, the more likely APIs are the ones for stages and steps.

I would prefer to remove it if its no longer in use for the plugin. If people are using it we can restore it and mark as deprecated, removing after some time.

[timja]

You say part of #704

Isn't this PR the last in the chain and it will fix the issue?

[das7pad]

You say part of #704

Isn't this PR the last in the chain and it will fix the issue?

Indeed, I've updated the link to "Fixes".

[das7pad]

The tests should pass now. I used the wrong incremental for Jenkins core (jenkinsci/jenkins#11071 -> jenkinsci/jenkins#11177).

[das7pad]

https://github.com/jenkinsci/workflow-api-plugin/releases/tag/1398.v67030756d3fb_

Thanks for creating the release, @timja ! What's the preferred method for adopting it? Bump the version in the pom.xml here or pull in the upgrade through BOM after the next dependabot update cycle?

https://github.com/jenkinsci/bom/blob/645d8bdc5d54d123155182426ffba1fe6b241460/bom-weekly/pom.xml#L34

[timja]

Either is fine

[das7pad]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.jenkins-ci.tools:maven-hpi-plugin:3.1762.vd3ff902a_5b_c4:validate-hpi (default-validate-hpi) on project pipeline-graph-view: Dependency org.jenkins-ci.plugins.workflow:workflow-api:jar:1398.v67030756d3fb_ requires Jenkins 2.532 or higher.

🕵️ I'm currently on 2.532-rc37610.379ef8546550, which should satisfy >=2.532, no? I'll try to get a new incremental for Jenkins with 2.533.

pipeline-graph-view-plugin/pom.xml

Line 37 in 2f9403a

<jenkins.version>2.532-rc37610.379ef8546550</jenkins.version>