Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] CI Test should always terminate after 1 hour #14680

Open
1 task done
lupyuen opened this issue Nov 7, 2024 · 6 comments
Open
1 task done

[FEATURE] CI Test should always terminate after 1 hour #14680

lupyuen opened this issue Nov 7, 2024 · 6 comments
Labels
Type: Enhancement New feature or request

Comments

@lupyuen
Copy link
Member

lupyuen commented Nov 7, 2024

Is your feature request related to a problem? Please describe.

CI Test will sometimes run for 6 hours (before getting killed by GitHub):

This is not so great because:

  1. It will increase our usage of GitHub Runners. Which may overrun the GitHub Actions Budget allocated by ASF.
  2. Suppose right after CI Test there's another build. If CI Test runs for all 6 hours, then the build after CI Test will never run.
  3. We are now running our own Ubuntu PCs as a NuttX Build Farm. The PCs will hang forever until we restart the Build Jobs.

Describe the solution you'd like

CI Test should complete within 1 hour. It should gracefully terminate itself (and report an error) if the runtime exceeds 1 hour.

Describe alternatives you've considered

Right now I'm manually killing all CI Jobs that run over 3 hours. And restarting the Ubuntu PCs in our NuttX Build Farm.

Verification

  • I have verified before submitting the report.
@lupyuen lupyuen added the Type: Enhancement New feature or request label Nov 7, 2024
@simbit18
Copy link
Contributor

simbit18 commented Nov 7, 2024

@lupyuen maybe we also need to put a maximum number of minutes for a job to run.

GitHub Actions timeout
https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes

@lupyuen
Copy link
Member Author

lupyuen commented Nov 7, 2024

@simbit18 Yep right now it quits after 6 hours: https://github.com/NuttX/nuttx/actions/runs/11714861244

@simbit18
Copy link
Contributor

simbit18 commented Nov 7, 2024

@lupyuen so we should put

jobs:
  build:
    runs-on: ubuntu-latest
    timeout-minutes: 180 #  (3 hours) Decrease this timeout value as needed

@lupyuen
Copy link
Member Author

lupyuen commented Nov 7, 2024

@simbit18 Hmmm suppose right after CI Test there's another build. If CI Test runs for all 3 hours, then the build after CI Test will never run. So actually I prefer if CI Test could terminate itself (after 1 hour) and let other builds run.

Unless we always park CI Test at the end of the job?

@simbit18
Copy link
Contributor

simbit18 commented Nov 7, 2024

@simbit18 Hmmm suppose right after CI Test there's another build. If CI Test runs for all 3 hours, then the build after CI Test will never run. So actually I prefer if CI Test could terminate itself (after 1 hour) and let other builds run.
right !!!

Describe the solution you'd like
CI Test should complete within 1 hour. It should gracefully terminate itself (and report an error) if the runtime exceeds 1 hour.

This in my opinion is the right solution

The GitHub Actions timeout is only for safety and not to fall back into the tunnel #14376

@lupyuen
Copy link
Member Author

lupyuen commented Nov 15, 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants