Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Job is not resubmited because of post condition failure #439

Open
domingo2000 opened this issue Apr 25, 2024 · 0 comments
Open

Job is not resubmited because of post condition failure #439

domingo2000 opened this issue Apr 25, 2024 · 0 comments
Labels

Comments

@domingo2000
Copy link

Issue Details

Currently Im running a Pipeline Job that is something as follow:

pipeline {
    agent any
    
    stages {
        stage('Parallel Stages') {
            parallel {
                stage('1') {
                    steps {
                        echo '1'
                    }
                }
                stage('2') {
                    steps {
                        echo '2'
                    }
                    post {
                        failure {
                            echo '2 failure post condition'
                        }
                        always {
                            echo '2 always post condition'
                        }
                    }
                }
            }
        }
        stage('3') {
            steps{
                echo '3'
            }
        }
    }
}

The pipeline have multiple steps, some of them are parallel, and some of them have post conditions that run on failure and always.

The problem is that the jobs are not getting re-submited by the EC2 Fleet plugin when spot instances goes down. When i watch my Pipeline console i see that the steps of the stages are getting aborted, but the post condition fails because the agent is already disconnected.

The logs got this at the aborted step

[60] Cannot contact i-0cf5f2bc3900a0408: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@2a50c887:i-0cf5f2bc3900a0408": Remote call on i-0cf5f2bc3900a0408 failed. The channel is closing down or has closed down
[61] Could not connect to i-0cf5f2bc3900a0408 to send interrupt signal to process

But then in the post condition fails with

Found unhandled org.jenkinsci.plugins.workflow.support.steps.AgentOfflineException exception:
Unable to create live FilePath for i-0cf5f2bc3900a0408; agent-a i-0cf5f2bc3900a0408 was marked offline: Connection was broken

So the build is marked as failed because the post condition fail becaue the agent is offline (spot terminated)

But the plugin is not sending the rebuild of the job, neither im seeing any logs from the plugin resubmiting as seen in #45.

¿Is there a way to make the plugin resubmit even if a post condition is failes?
¿Is this really the problem or could be another thing?
If the problem is really in the post condition. ¿How could i rewrite the pipeline to make the plugin actually resubmit the job?

To Reproduce

  1. Make a Jenkins Pipeline Syntax with parallel stages and post conditions
  2. Wait unit a spot instance terminate
  3. See that the post condition failed
  4. The Job is not resubmitted automatically.

** Logs **
Tip: See this guide to configure a logger in Jenkins UI. Please attach fine logs if you think they are relevant.

Environment Details

Spot Fleet or ASG?
Spot Fleet

Linux or Windows?
Linux

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant