How to improve performance of TransformationManager

[zhangxiaomei]

When some transformation tasks with >200,000 inputdata are submitted, it seemed as if the response of TransformationManager becomes very slow although it is still working:
(1) Transformation monitoring page get stuck with the error "timed out:SSLTimeoutError('timed out',)"
(2) The related agents such as TransformationAgent and TransformationCleaningAgent responds with the timeout problems from time to time:
"Issue getting socket: <DIRAC.Core.DISET.private.Transports.M2SSLTransport.SSLTransport object at 0x7f2478419c10> : ('dips', 'prod-dirac.ihep.ac.cn', 9131, 'Transformation/TransformationManager') : timed out:SSLTimeoutError('timed out',)"
How can I improve this situation? All the services are in one server. The load of DIRAC server is not high (2~5) and The memory used by TransformationManager is 1.1GB.

[fstagni]

In LHCb we regularly run transformations with more than 200000 input files without issues. Your issues might not depend specifically from these transformations, rather from running out of sockets on the machine. The CPU load might not be high, but your networking might be.

You can try a few things, simplest are:

• run services on more than one machine
• increase the number of sockets (e.g. with sysctl) of your servers

[zhangxiaomei]

This could be one of the reasons. At least I saw the maximum open files limit is too low. I will have a try.

[arrabito]

Hi @zhangxiaomei,

We have also similar issues.

I would like to ask you if you are using the TSCatalog interface as described here:

https://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/Transformation/index.html#id12

or the InputDataQuery Agent.

In the case of TSCatalog interface, unfortunately the TS does not support running multiple services, but It's in my to-do list to enable this feature.

[fstagni]

In the case of TSCatalog interface, unfortunately the TS does not support running multiple services, but It's in my to-do list to enable this feature.

I would like to know why -- and maybe this could be transferred as separate issue.

[arrabito]

The reason is that the service depends on the value of a variable local to a given instance:

self.filterQueries

which is the list of the inputMetaQueries.

So if the there are 2 instances of the TSManager (TSManager-1 and TSManager-2) and a transformation with inputDataQuery is added to the system connecting to TSManager-1, the self.filterQueries list of the TSManager-1 is updated but not the one of the TSManager-2. So if then a file is added to the TSCatalog through the addFile method and connecting to TSManager-2, a filter is applied against the self.filterQueries list of TSManager-2 which has not been updated.

See:

https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/TransformationSystem/DB/TransformationDB.py#L211

and:

https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/TransformationSystem/DB/TransformationDB.py#L1602

I guess the solution would be make the list of self.filterQueries global to every TSManager instance.

Any suggestion on how to achieve this?

[fstagni]

I don't see any other solution but effectively storing self.filterQueries in a DB.

[zhangxiaomei]

@arrabito I haven't tried TS catalog, just use the InputDataQuery Agent. If TS catalog can help, I would like to have a try.

[arrabito]

I don't think it will help in performances of the TS. It might reduce some load on the DFC but it implies more calls to the TSManager.

[arrabito]

@fstagni about your suggestion:

• increase the number of sockets (e.g. with sysctl) of your servers

Do you suggest to increase : net.core.somaxconn ?

Thanks

[fstagni]

This what we set in LHCb:

  # SYSCTL settings
  sysctl { 'net.netfilter.nf_conntrack_tcp_timeout_time_wait': val => '10';
    'net.netfilter.nf_conntrack_tcp_timeout_close_wait': val => '10';
    'net.netfilter.nf_conntrack_tcp_timeout_fin_wait': val => '20';
    'net.ipv4.tcp_max_syn_backlog': val => '32688';
    'net.netfilter.nf_conntrack_max': val => '131072';
    'net.netfilter.nf_conntrack_udp_timeout_stream': val => '45';
    'net.netfilter.nf_conntrack_udp_timeout': val => '10';
    'net.core.somaxconn': val => '32688';
    'net.nf_conntrack_max': val => '131072';
  }

[zhangxiaomei]

thank you for sharing. I will try to tune these values in my server.

[arrabito]

@fstagni thanks a lot for these informations.

In our server, only 2 of the variables you listed are set:
net.core.somaxconn
net.ipv4.tcp_max_syn_backlog

The default values of these are:
net.core.somaxconn = 128
net.ipv4.tcp_max_syn_backlog = 512

I've just tried to just increase:
net.core.somaxconn to 1024 and according to my current tests, it's maybe already enough.

But given your values, I guess I can increase even more if needed.

Do you suggest to eventually also set the other variables?

Then, I've also read that the File descriptors should also be increased.

The values I had originally were:
ulimit -Sn
131072
ulimit -Hn
262144

and in my last test with net.core.somaxconn = 1024 I increased them by a factor 4.

Could you please tell me if it does seem reasonable to you?

[fstagni]

I am no expert on these settings, nor I remember what each of them do. The values I sent are the results of several adaptations, and several trial-and-error done during the years, and nowadays it is what we set in our puppet profile. I am afraid I can't help much more, they also very much dependent on what are your servers.

[arrabito]

OK thank you. I guess we should do the same and tune our settings.
I haven't found yet a working config but it's good to have an idea of which parameters you have tuned.
I take the opportunity of this thread to ask you a related question. In our pilot logs I also observe systematic timeouts for the master instance of the CS:

'Configuration/Server') : timed out:SSLTimeoutError('timed out')
2022-05-03 10:24:14 UTC WorkloadManagement/JobAgent WARN: Non-responding URL temporarily banned dips://dcta-servers02.pic.es:9135/Configuration/Server
2022-05-03 10:24:14 UTC WorkloadManagement/JobAgent INFO: Retry connection : 1 to dips://dcta-servers02.pic.es:9135/Configuration/Server

I have no idea why it's only the Master instance that gives timeouts. Anyway, do you think that it could be a similar network issue and that we should proceed to a similar tuning of the SYSCTL settings?

Thanks.

[fstagni]

Can we consider this thread answered? What is the status now in your installations?
If that's the case then the documentation for installing/configuring DIRAC servers should be updated.

[zhangxiaomei]

I improved the value of "open files" from 1024 to 65536 in "ulimit" settings, and also increased number of sockets in SYSCTL settings:
net.core.somaxconn from 128 to 4096;
net.ipv4.tcp_max_syn_backlog from 2048 to 4096;
This did help with the query in Transformation monitoring web page. It is much better, less stuck.
But for the agents such as TransformationAgent and TransformationCleaningAgent, I still see the timeout problems from time to time
I have also duplicated the TransformationManager service in another machine, but look like not much help.
Do you think if there are other bottlenecks, eg mysql DB connection?

[zhangxiaomei]

I found some errors in mysql log about TransformationDB:
2021-05-24T08:08:47.132131Z 1368011 [Note] Aborted connection 1368011 to db: 'TransformationDB' user: 'Dirac' host: 'localhost' (Got an error reading communication packets)
2021-05-24T08:08:47.132144Z 904136 [Note] Aborted connection 904136 to db: 'TransformationDB' user: 'Dirac' host: 'localhost' (Got an error reading communication packets)
2021-05-24T08:08:47.132133Z 955533 [Note] Aborted connection 955533 to db: 'TransformationDB' user: 'Dirac' host: 'localhost' (Got an error reading communication packets)
2021-05-24T08:08:47.132187Z 1397176 [Note] Aborted connection 1397176 to db: 'TransformationDB' user: 'Dirac' host: 'localhost' (Got an error reading communication packets)
2021-05-24T08:08:47.132201Z 1424619 [Note] Aborted connection 1424619 to db: 'TransformationDB' user: 'Dirac' host: 'localhost' (Got an error reading communication packets)
2021-05-24T08:08:47.132241Z 459438 [Note] Aborted connection 459438 to db: 'TransformationDB' user: 'Dirac' host: 'localhost' (Got an error reading communication packets)

[zhangxiaomei]

I didn't find other suspicious in network, although the SSL timeout WARN appears in the TransformAgent log every 30s:

2022-05-27 07:29:59 UTC Transformation/TransformationAgent WARN: Issue getting socket: <DIRAC.Core.DISET.private.Transports.M2SSLTransport.SSLTransport object at 0x7fdf18487110> : ('dips', 'dirac-web.ihep.ac.cn', 9131, 'Transformation/TransformationManager') : timed out:SSLTimeoutError('timed out',)
2022-05-27 07:30:07 UTC Transformation/TransformationAgent WARN: Issue getting socket: <DIRAC.Core.DISET.private.Transports.M2SSLTransport.SSLTransport object at 0x7fdf327d1d10> : ('dips', 'dirac-web.ihep.ac.cn', 9131, 'Transformation/TransformationManager') : timed out:SSLTimeoutError('timed out',)
2022-05-27 07:33:59 UTC Transformation/TransformationAgent WARN: Issue getting socket: <DIRAC.Core.DISET.private.Transports.M2SSLTransport.SSLTransport object at 0x7fdf3049a250> : ('dips', 'prod-dirac.ihep.ac.cn', 9131, 'Transformation/TransformationManager') : timed out:SSLTimeoutError('timed out',)
2022-05-27 07:34:30 UTC Transformation/TransformationAgent WARN: Issue getting socket: <DIRAC.Core.DISET.private.Transports.M2SSLTransport.SSLTransport object at 0x7fdf3049a0d0> : ('dips', 'prod-dirac.ihep.ac.cn', 9131, 'Transformation/TransformationManager') : timed out:SSLTimeoutError('timed out',)
2022-05-27 07:38:00 UTC Transformation/TransformationAgent WARN: Issue getting socket: <DIRAC.Core.DISET.private.Transports.M2SSLTransport.SSLTransport object at 0x7fdf1844b050> : ('dips', 'prod-dirac.ihep.ac.cn', 9131, 'Transformation/TransformationManager') : timed out:SSLTimeoutError('timed out',)
2022-05-27 07:38:30 UTC Transformation/TransformationAgent WARN: Issue getting socket: <DIRAC.Core.DISET.private.Transports.M2SSLTransport.SSLTransport object at 0x7fdf1839c610> : ('dips', 'prod-dirac.ihep.ac.cn', 9131, 'Transformation/TransformationManager') : timed out:SSLTimeoutError('timed out',)

[arrabito]

I also tried all the network settings you suggested but unfortunately I don't see any improvement.

[fstagni]

I am closing this discussion as "outdated", as 2+ years went without comments.