Pushing a ruby application fails with DNS server misbehaving Ubuntu Noble stemcell

[davewalter]

Current behavior

The noble-stemcell-validation pipeline is failing to stage an application with the Ruby buildpack when running the CF Acceptance tests:

  [2025-01-20 09:24:59.76 (UTC)]> cf push CATS-3-BRKR-031938ce78aeb3aa -b ruby_buildpack -m 256M -p assets/service_broker --health-check-type http --endpoint /v2/catalog 
  Pushing app CATS-3-BRKR-031938ce78aeb3aa to org CATS-3-ORG-26d97ea0e18920fe / space CATS-3-SPACE-fd8ffc94232fef3a as CATS-3-USER-d8b3e9e08c300515...
  Packaging files to upload...
  Uploading files...
 269.15 KiB / 269.15 KiB [==============================================================================================================================================================================================================================================================================================================================================================================================================================================================================] 100.00% 1s

  Waiting for API to complete processing files...

  Staging app and tracing logs...
     Downloading ruby_buildpack...
     Downloaded ruby_buildpack
     Cell a6736615-7775-4211-9041-f79ce23c5668 creating container for instance 8d5cf9a5-cded-4773-a925-74363d5b2e30
     Security group rules were updated
     Cell a6736615-7775-4211-9041-f79ce23c5668 successfully created container for instance 8d5cf9a5-cded-4773-a925-74363d5b2e30
     Downloading app package...
     Downloaded app package (956.7K)
     -----> Ruby Buildpack version 1.10.19
     -----> Bootstrapping Ruby
     -----> Installing ruby 3.2.5
     Download [https://buildpacks.cloudfoundry.org/dependencies/ruby/ruby_3.2.5_linux_x64_cflinuxfs4_7cb2e65f.tgz]
     error: Get "https://buildpacks.cloudfoundry.org/dependencies/ruby/ruby_3.2.5_linux_x64_cflinuxfs4_7cb2e65f.tgz": dial tcp: lookup buildpacks.cloudfoundry.org on 169.254.0.2:53: server misbehaving, retrying in 1.333232717s...
     error: Get "https://buildpacks.cloudfoundry.org/dependencies/ruby/ruby_3.2.5_linux_x64_cflinuxfs4_7cb2e65f.tgz": dial tcp: lookup buildpacks.cloudfoundry.org on 169.254.0.2:53: server misbehaving, retrying in 1.540439299s...
     error: Get "https://buildpacks.cloudfoundry.org/dependencies/ruby/ruby_3.2.5_linux_x64_cflinuxfs4_7cb2e65f.tgz": dial tcp: lookup buildpacks.cloudfoundry.org on 169.254.0.2:53: server misbehaving, retrying in 1.51748035s...
     error: Get "https://buildpacks.cloudfoundry.org/dependencies/ruby/ruby_3.2.5_linux_x64_cflinuxfs4_7cb2e65f.tgz": dial tcp: lookup buildpacks.cloudfoundry.org on 169.254.0.2:53: server misbehaving, retrying in 2.376751591s...
     error: Get "https://buildpacks.cloudfoundry.org/dependencies/ruby/ruby_3.2.5_linux_x64_cflinuxfs4_7cb2e65f.tgz": dial tcp: lookup buildpacks.cloudfoundry.org on 169.254.0.2:53: server misbehaving, retrying in 3.506204711s...
     error: Get "https://buildpacks.cloudfoundry.org/dependencies/ruby/ruby_3.2.5_linux_x64_cflinuxfs4_7cb2e65f.tgz": dial tcp: lookup buildpacks.cloudfoundry.org on 169.254.0.2:53: server misbehaving, retrying in 8.971668577s...
     error: Get "https://buildpacks.cloudfoundry.org/dependencies/ruby/ruby_3.2.5_linux_x64_cflinuxfs4_7cb2e65f.tgz": dial tcp: lookup buildpacks.cloudfoundry.org on 169.254.0.2:53: server misbehaving, retrying in 9.466592854s...
     error: Get "https://buildpacks.cloudfoundry.org/dependencies/ruby/ruby_3.2.5_linux_x64_cflinuxfs4_7cb2e65f.tgz": dial tcp: lookup buildpacks.cloudfoundry.org on 169.254.0.2:53: server misbehaving, retrying in 13.170218427s...
     error: Get "https://buildpacks.cloudfoundry.org/dependencies/ruby/ruby_3.2.5_linux_x64_cflinuxfs4_7cb2e65f.tgz": dial tcp: lookup buildpacks.cloudfoundry.org on 169.254.0.2:53: server misbehaving, retrying in 16.675823105s...
  BuildpackCompileFailed - App staging failed in the buildpack compile phase
  FAILED

Desired behavior

The application should successfully stage and start.

Affected Version

2.112.0

[MarcPaquette]

Hi @davewalter

What indication do you have this is a Diego issue at this time?

Is there any place where we can access the Diego cell logs?

Thanks!

[dimivel]

hi @MarcPaquette,

In order to access Diego cell logs you need to jump on stable-bellatrix BBL environment.
This is how you can do it:

• go to /relint-envs/environments/test/bellatrix

• and execute
  docker run -it -v $PWD:/home/bbl cloudfoundry/cf-deployment-concourse-tasks
  cd /home/bbl/bbl-state
  eval "$(bbl print-env)"

• after that navigate to bbl-state directory
  cd /home/bbl/bbl-state
  eval "$(bbl print-env)"

• list bosh vms and ssh on one of the diego-cell vms
  sudo su -
  bosh vms
  bosh -d cf ssh diego-cell/79a1de3b-e814-4c0a-b431-a5f2c2985970

• under /var/vcap/sys/log/ you can see logs for all jobs
  diego-cell/79a1de3b-e814-4c0a-b431-a5f2c2985970:/var/vcap/sys/log/

Let me know if you have issues to access the environment.

Also you can have a look at run-cats concourse job here:
https://concourse.wg-ard.ci.cloudfoundry.org/teams/main/pipelines/noble-stemcell-validation/jobs/run-cats/builds/1

Thanks!
Dimitar

[dimivel]

hi @MarcPaquette,

Did you manage to access the Diego cell logs?
Do you have any findings and feedback related to this issue?

Thanks and regards,
Dimitar