PilotSyncAgent fails to sync pilot files to web server

[ernstpijper]

Hi,

I have 7.3.17 py3 dirac server installation setup, spread over 3 VMs: web, proxy and wms(everything else). All looks good but the syncing of the pilot files from the wms to web server is failing. In the logs I'm seeing the following.

On the wms server: /opt/dirac/runit/WorkloadManagement/PilotSyncAgent/log/current:

022-03-23 13:05:43 UTC WorkloadManagement/PilotSyncAgent INFO: Moving pilot files to /opt/dirac/webRoot/www/pilot
2022-03-23 13:05:43 UTC WorkloadManagement/PilotSyncAgent INFO: Attempting to upload to https://nl-dirac-web.grid.surfsara.nl/pilot
2022-03-23 13:05:43 UTC WorkloadManagement/PilotSyncAgent ERROR: Could not upload to https://nl-dirac-web.grid.surfsara.nl/pilot: status 404
2022-03-23 13:05:43 UTC WorkloadManagement/PilotSyncAgent ERROR: Could not upload to https://nl-dirac-web.grid.surfsara.nl/pilot: status 404
2022-03-23 13:05:43 UTC WorkloadManagement/PilotSyncAgent ERROR: Could not upload to https://nl-dirac-web.grid.surfsara.nl/pilot: status 404
2022-03-23 13:05:43 UTC WorkloadManagement/PilotSyncAgent ERROR: Could not upload to https://nl-dirac-web.grid.surfsara.nl/pilot: status 404
2022-03-23 13:05:43 UTC WorkloadManagement/PilotSyncAgent ERROR: Could not upload to https://nl-dirac-web.grid.surfsara.nl/pilot: status 404
2022-03-23 13:05:43 UTC WorkloadManagement/PilotSyncAgent ERROR: Could not upload to https://nl-dirac-web.grid.surfsara.nl/pilot: status 404
2022-03-23 13:05:43 UTC WorkloadManagement/PilotSyncAgent ERROR: Could not upload to https://nl-dirac-web.grid.surfsara.nl/pilot: status 404
2022-03-23 13:05:43 UTC WorkloadManagement/PilotSyncAgent ERROR: Could not upload to https://nl-dirac-web.grid.surfsara.nl/pilot: status 404
2022-03-23 13:05:43 UTC WorkloadManagement/PilotSyncAgent ERROR: Could not upload to https://nl-dirac-web.grid.surfsara.nl/pilot: status 404
2022-03-23 13:05:43 UTC WorkloadManagement/PilotSyncAgent ERROR: Could not upload to https://nl-dirac-web.grid.surfsara.nl/pilot: status 404
2022-03-23 13:05:43 UTC WorkloadManagement/PilotSyncAgent ERROR: Could not upload to https://nl-dirac-web.grid.surfsara.nl/pilot: status 404

On the web server, at the same time: /opt/dirac/runit/Web/WebApp/log/current:

2022-03-23 13:05:43 UTC WebApp/Web WARN: 404 PUT /pilot (145.38.219.85) 0.39ms
2022-03-23 13:05:43 UTC WebApp/Web WARN: 404 PUT /pilot (145.38.219.85) 0.59ms
2022-03-23 13:05:43 UTC WebApp/Web WARN: 404 PUT /pilot (145.38.219.85) 0.29ms
2022-03-23 13:05:43 UTC WebApp/Web WARN: 404 PUT /pilot (145.38.219.85) 0.32ms
2022-03-23 13:05:43 UTC WebApp/Web WARN: 404 PUT /pilot (145.38.219.85) 0.28ms
2022-03-23 13:05:43 UTC WebApp/Web WARN: 404 PUT /pilot (145.38.219.85) 0.27ms
2022-03-23 13:05:43 UTC WebApp/Web WARN: 404 PUT /pilot (145.38.219.85) 0.28ms
2022-03-23 13:05:43 UTC WebApp/Web WARN: 404 PUT /pilot (145.38.219.85) 0.39ms
2022-03-23 13:05:43 UTC WebApp/Web WARN: 404 PUT /pilot (145.38.219.85) 0.28ms
2022-03-23 13:05:43 UTC WebApp/Web WARN: 404 PUT /pilot (145.38.219.85) 0.27ms
2022-03-23 13:05:43 UTC WebApp/Web WARN: 404 PUT /pilot (145.38.219.85) 0.27ms
2022-03-23 13:05:46 UTC WebApp WARN: Not creating symlinks to static resources Path /WebApp/StaticResourceLinkDir does not exist or it's not an option
2022-03-23 13:05:46 UTC WebApp NOTICE: Set next path(s):  WebApp.handler
2022-03-23 13:05:46 UTC WebApp/Web ALWAYS:

Some relevant sections from the dirac configuration:

  Hosts
  {
    nl-dirac-wms.grid.surfsara.nl
    {
      DN = /DC=org/DC=terena/DC=tcs/C=NL/ST=Utrecht/O=SURF B.V./CN=nl-dirac-wms.grid.surfsara.nl
      Properties = TrustedHost
      Properties += CSAdministrator
      Properties += JobAdministrator
      Properties += FullDelegation
      Properties += ProxyManagement
      Properties += Operator
    }
    nl-dirac-proxy.grid.surfsara.nl
    { 
      DN = /DC=org/DC=terena/DC=tcs/C=NL/ST=Utrecht/O=SURF B.V./CN=nl-dirac-proxy.grid.surfsara.nl
      Properties = TrustedHost
      Properties += CSAdministrator
      Properties += ProxyManagement
    }
    nl-dirac-web.grid.surfsara.nl
    { 
      DN = /DC=org/DC=terena/DC=tcs/C=NL/ST=Utrecht/O=SURF B.V./CN=nl-dirac-web.grid.surfsara.nl
      Properties = TrustedHost
      Properties += CSAdministrator
    }
  }

  DIRAC-Production
  {
    Pilot
    {
      Version = 7.3.17
      checkVersion = False
      GenericPilotGroup = dirac_pilot
      GenericPilotUser = r2d2
      pilotFileServer = nl-dirac-web.grid.surfsara.nl
      Commands
      {
        Test = GetPilotVersion
        Test += CheckWorkerNode
        Test += InstallDIRAC
        Test += ConfigureBasics
        Test += ConfigureCPURequirements
        Test += ConfigureArchitecture
        Test += CheckCECapabilities
        Test += LaunchAgent
      }
    }
  }

WebApp
{
  StaticDirs = pilot
  Access
  {
    upload = TrustedHost
  }
}

        PilotSyncAgent
        {
          PollingTime = 60
          #Directory where the files can be moved. If running on the WebApp, use /opt/dirac/webRoot/www/pilot
          SaveDirectory = /opt/dirac/webRoot/www/pilot
          #List of locations where to upload the pilot files. Can be https://some.where, or DIRAC SE names.
          UploadLocations = https://nl-dirac-web.grid.surfsara.nl/pilot
          #Set to False (or No, or N) to exclude the master CS from the list of CS servers
          IncludeMasterCS = True
        }

Reading on the other end from https://nl-dirac-web.grid.surfsara.nl/pilot works fine. If I copy by hand all the pilot files in /opt/dirac/webRoot/www/pilot on the web server, a the pilot wrapper script is able to download all the pilot files and run a job.

Together with my colleague, we've tried many things but can get it to work. We feel this is maybe be some kind of permission problem.

It did work on the py2 3vm installation but not sure what I did different there.

Hope you can help,

Ernst

[atsareg]

Hi Ernst,
Setting up the web server for files uploading is tricky, but I do not think you need it. Just run the PilotSyncAgent on the nl-dirac-web.grid.surfsara.nl host. In its configuration, leave the UploadLocations blank, e.g.

    PilotSyncAgent
            {
              PollingTime = 60
              #Directory where the files can be moved. If running on the WebApp, use /opt/dirac/webRoot/www/pilot
              SaveDirectory = /opt/dirac/webRoot/www/pilot
              #List of locations where to upload the pilot files. Can be https://some.where, or DIRAC SE names.
              UploadLocations = 
              #Set to False (or No, or N) to exclude the master CS from the list of CS servers
              IncludeMasterCS = True
            }

Check that /opt/dirac/webRoot/www/pilot is indeed the directory from which the web server serves pilot files.

Andrei

[ernstpijper]

Hi Andrei,

That does indeed sound like a great solution but I have no idea how to configure PilotSyncAgent on nl-dirac-web. PilotSyncAgent is configured in DIRAC-Production.cfg which is located on the configuration master (nl-dirac-wms).

Do I configure PilotSyncAgent on nl-dirac-web? Or should I configure it in DIRAC-Production.cfg on nl-dirac-wms such that PilotSyncAgent runs on nl-dirac-web. But then again: I don't know how.

[atsareg]

Hi Ernst,
You configure the PilotSyncAgent in the global configuration which is served by the Configuration service. You just do it in the web Configuration Manager. This configuration is indeed kept in the DIRAC-Production.cfg file but you should not edit it by hand unless you know what you are doing.
The PilotSyncAgent configuration does not define on which host it runs, only the running parameters. You should stop/uninstall the PilotSyncAgent on the host where it runs now and start it on the host where the web portal is running. You can do it via the SysAdmin console. For example:

> dirac-proxy-init -g dirac_admin
> dirac-admin-sysadmin-cli --host nl-dirac-wms.grid.surfsara.nl
    Pinging nl-dirac-wms.grid.surfsara.nl...
[nl-dirac-wms.grid.surfsara.nl]> show status
[nl-dirac-wms.grid.surfsara.nl]> uninstall WorkloadManagement PilotSyncAgent
[nl-dirac-wms.grid.surfsara.nl]> set host nl-dirac-web.grid.surfsara.nl
   Pinging nl-dirac-web.grid.surfsara.nl...
[nl-dirac-web.grid.surfsara.nl]> install agent WorkloadManagement PilotSyncAgent
[nl-dirac-web.grid.surfsara.nl]> show status

The console has a help command.
Please try this out and if still problems, let's get in contact offline to sort this out.

Andrei