forked from DIRACGrid/DIRAC
-
Notifications
You must be signed in to change notification settings - Fork 0
/
release.notes
10245 lines (7426 loc) · 355 KB
/
release.notes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
[v8r0-pre4]
NEW: Basic support for the OAuth/OIDC tokens
*DataManagement
CHANGE: (#5391) The LcgFileCatalogClient has been removed
[v7r3p4]
FIX: fixes from v7r2p28
*WorkloadManagement
CHANGE: (#5429) Use python3 in VMDIRAC instances
[v7r3p3]
*Resources
NEW: (#5396) Add Emies endpoint to the ARC CE
*docs
FIX: (#5424) some minor fixes for user documentation
[v7r3p2]
*WMS
CHANGE: (#5404) use raw.githubusercontent URL for github raw files requests
*TS
CHANGE: (#5395) RequestTaskAgent uses objectLoader for RequestTasks
CHANGE: (#5395) split RequestTasks and WorkflowTasks into distinct module
*docs
NEW: (#5218) Updated Python2 and Python3 server installation instructions
CHANGE: (#5228) Documenting /WebApp/StaticResourceLinkDir option
CHANGE: (#5404) use raw.githubusercontent URL for github raw files requests
FIX: (#5404) use working git links for scripts
[v7r3p1]
FIX: Tag to push the PyPi deployment
[v7r3]
NEW: (#5000) Initial support for Python 3 server installations
NEW: VMDIRAC separate project is merged into the core DIRAC project
CHANGE: (#5035) use registerArgument to register positional arguments for scripts
CHANGE: (#4715) removing env variable DIRAC_USE_NEWTHREADPOOL
FIX: (#5035) use DIRACScript instead of Script
FIX: (#5028) Replaced all the cases of BaseException use by Exception
*Core
NEW: (#5062) provide some docs and add registerArgument method to register arguments in Script
NEW: (#4997) dirac-configure can now be ran without arguments when using Python 3
CHANGE: (#4937) removed dirac-agent, dirac-service, dirac-executor scripts (use '_' counterparts instead)
CHANGE: (#5110) Removed Core.Utilities.Grid.getBdiiCEInfo function use Utilities.Glue2.getGlue2CEInfo.
Also dropped ldapSite, ldapCluster, ldapCE, ldapCEState, ldapCEVOView, ldapService functions
CHANGE: (#4903) Using former RSS State Machine as general State Machine
CHANGE: (#5237) Use generic ObjectLoader.loadObjects() function in the specific Plotting.ObjectLoader class
CHANGE: (#5244) dirac-configure: no upload attempt upon initial proxy generation
CHANGE: (#5246) Drop MySQL._to_string() method
CHANGE: Release constraint on SQLALchemy versions
FIX: (#5210) show extremely small values in pie plots
FIX: (#4997) Handle SIGINT correctly when reading certificate passwords
FIX: (#5269) Use isinstance when checking types in MessageFactory
FIX: (#5272) encode stub before base64
FIX: (#5326) fix bug in getSystemURLs: getOptionsFromCFG can return None
CHANGE: (#5337) removed src/DIRAC/Core/scripts/dirac-install.py
CHANGE: (#5372) RCSID is no longer access for Python 3 based installations
*Configuration
NEW: (#5062) add registerCmdArg to register arguments and group argument to group returned arguments
NEW: (#5286) test new PathFinder methods
NEW: (#5286) PathFinder - add checkServiceURL that check URL port and path
NEW: (#5286) PathFinder - add getServiceURLs that return list type result
NEW: (#5286) PathFinder - add getSystemURLs that return all services URLs for system
CHANGE: (#5110) Dropped Glue2Only and Glue2URLS options from Bdii2CSAgent, Glue2 is now the only way
CHANGE: (#5110) dirac-admin-add-resources: drop -g -G options, Glue2 is now the only way
CHANGE: (#5110) Configuration.Client.Utilities functions getGridCEs, getSiteUpdates no longer take glue2 parameter
CHANGE: (#5000) Deprecate CSGlobals.getInstalledExtensions, DIRAC.Core.Utilities.Extensions.extensionsByPriority
should be used instead
CHANGE: (#5000) Deprecate getCSExtensions, DIRAC.Core.Utilities.Extensions.extensionsByPriority should be used instead
CHANGE: (#5286) PathFinder - getSystemSection do not use serviceTuple, add docs, use system and service instead
of componentTuple
FIX: (#5286) PathFinder - getGatewayURLs pass list to randomize when serviceName is empty
FIX: (#5298) fix getServiceFailoverURL for multi url case
FIX: (#5375) Handle S_ERROR result in CSAPI, docs
*DMS
CHANGE: (#5206) prints why dirac-dms-add-files failed
*Framework:
CHANGE: (#4303) Removed completely the SystemLogging
CHANGE: (#5164) Remove HashTag table from Framework/UserProfile
CHANGE: (#5164) Remove dataTypeRE argument when retrieving values from Framework/UserProfileClient
FIX: (#5340) Fix using the System Administrator to install Python 3 releases of DIRAC from Python 2
*Interfaces
CHANGE: (#5110) DiracAdmin: dropped getBDII* functions, which were looking for Glue1 information
CHANGE: (#5110) removed dirac-admin-bdii-info command
*Resources
CHANGE: (#5314) Remove GlobusComputingElement
*RMS
CHANGE: (#5291) ReqDB don't cast datetime to strings
*RSS
NEW: (#5042) Multi-VO mode of operation support
FIX: (#5335) VO parameter is optional. When omitted commands will act on all VOs.
*WMS
NEW: (#4903) Added WMS (Jobs) State Machine
NEW: (#5214) Added a HTTPs JobManager service
NEW: (#5214) Added a HTTPs JobMonitor service
NEW: (#5214) Added a HTTPs JobStateUpdate service
NEW: (#5314) Add a PilotStatus module to clearly define the pilot status in the code
CHANGE: (#5214) Removed need for ThreadScheduler in JobManagerHandler
CHANGE: (#5289) SiteDirector: do not send dirac-install if it's py3 pilot
CHANGE: (#4884) JobDB: compress JDLs by default (no need for flag anymore)
CHANGE: (#4937) removed StatesMonitoringAgent (use StatesAccountingAgent agent instead)
CHANGE: (#5141) add an Aborted_HOUR column to the pilot efficiency table needed by the WebApp
CHANGE: (#5178) JobMonitoring does not need to look into TaskQueue
CHANGE: (#5246) remove pilots before start Test_PilotsClient test
CHANGE: (#5318) Move SiteDirector.getQueues in QueueUtilities module
FIX: (#5246) mock _escapeString in Test_Agent_PilotStatusAgent to pass test
FIX: (#5246) escape values in PilotAgentsDB
FIX: (#5246) decode values using decode() method in JobDB
NEW: (#5314) Add a PilotStatus module to clearly define the pilot status in the code
FIX: (#5325) Fix-up minor errors introduced during VMDIRAC merge
NEW: (#5325) Tests for VirtualMachine DB & Service
FIX: (#5342) Allow Platform parameter to be missing from the queue configuration
*TS
NEW: (#5327) Added TransformationFilesStatus module
*tests
NEW: (#5062) add Test_LocalConfiguration
CHANGE: (#5046) don't use mail in the self generated certificates
CHANGE: (#5301) added RALPP as a multi-VO test site and added env dump to job.log to ease debugging
NEW: (#5321) added integration test for AccountingDB
CHANGE: (#5362) Add proxy correct setup within wms-script.sh
*docs
CHANGE: (#5313) replace the theme by the sphinx_rtd_theme
[v7r2p28]
*Core
FIX: (#5416) unless explicitly set, do not skip the CA checks
*WorkloadManagement
FIX: (#5431) Ensure invalid tasks don't get rescheduled by ExecutorDispatcher
NEW: (#5430) add support for execution of pilot wrappers when the python executable is not set
*RMS
FIX: (#5432) ReqClient uses correct credentials to talk to ReqProxies
FIX: (#5432) ReqClient uses proper WMS client insteand of RPCClient
*Resources
CHANGE: (#5428) asctime now includes microsecond precision when using the MessageQueueBackend for logging
*DMS
FIX: (#5427) adapt csv use to py3
*tests
FIX: (#5427) adapt csv use to py3
[v7r2p27]
*TS
CHANGE: (#5413) TransformationDB: moved GroupSize from INTEGER to FLOAT
*Resources
FIX: (#5421) fix Slurm getJobStatus() method
[v7r2p26]
*Core
FIX: (#5390) Handle S_ERROR in Service._executeAction
NEW: (#5145) Add returnValueOrRaise and convertToReturnValue to simplify writing code with exceptions
CHANGE: (#5394) Requests are now dropped if the backlog is too large
FIX: (#5411) RPCClient and TornadoClient support proxyChain
*Framework
FIX: (#5410) ProxyManagerClient - upload proxy without embedded DIRAC group
*WMS
FIX: (#5414) JobCleaningAgent uses OwnerDN to put the sandbox removal request
*DMS
CHANGE: (#5284) FTS DB change (see https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-v7r2#new-status-for-fts3-files-and-jobs)
NEW: (#5284) enable FTS archive monitoring
NEW: (#5284) allow to use FTS activity based on FTS3Plugin
FIX: (#5408) making use of freesize in WLCGAccountingJson if exists, instead of calculating
as Total - usedsize
*RMS
CHANGE: (#5284) ReplicateAndRegister prefers disk replicas
*Resources
NEW: (#5402) DIRAC_GFAL_GRIDFTP_ENABLE_IPV6 to control gridftp behavior
*TS
FIX: (#5405) Transformation.setBody correctly checks all the operation tuples
*tests
NEW: (#5403) force commit format
[v7r2p25]
FIX: (#5373) Fix un-returned use of S_ERROR and S_OK
*tests
CHANGE: (#5380) Update DIRACCAProxyProvider test to use pytest
FIX: (#5381) Adapt Test_FilePlugin.py for https://bugs.python.org/issue43219
*WMS
FIX: (#5380) Fix bad merge in JobMonitoringHandler.py when producing v7r2p24
[v7r2p24]
FIX: fixes from v7r1p46
*Resources
FIX: (#5267) return error for StompMQConnector
[v7r2p23]
*WMS
CHANGE: (#5348) PilotWrapper: added file:/cvmfs/dirac.egi.eu/pilot/ as location
CHANGE: (#5356) SiteDirector - continue queues initialization if one of them fails
*Resources
FIX: (#5356) executeBatch - fallback to urllib if six module is not available
FIX: (#5356) Host.py - return Message in case of non-zero status in getCEStatus
FIX: (#5356) SSHBatchComputingElement - added forgotten instantiation of the BatchSystem plugin
[v7r2p22]
*Workflow
FIX: (#5346) UnicodeEncodeError when application log contains non-ascii characters
*Resources
FIX: (#5345) Pass --cfg /tmp/pilot.cfg when reusing host DIRAC installation in SingularityComputingElement
[v7r2p21]
*Core
FIX: (#5334) Use selectors(2) instead of select.select to avoid issues with file descriptors > 1024, fixes #5232
FIX: (#5336) ElasticSearchDB: use ca certs file if requested
FIX: (#5328) ElasticSearchDB: work on a copy of the documents in generateDocs()
*WorkloadManagement
FIX: (#5332) Fix unassigning sandboxes when removing jobs from the transformation system
*Resources
NEW: (#5329) Added PBSResourceUsage unit test
[v7r2p20]
*Resources
NEW: (#5316) Support Python 3 pilots in SingularityCE
CHANGE: (#5310) changes some log levels in StorageElement
CHANGE: (#5310) Default to using CERNVM4 in SingularityCE
FIX: (#5310) Propagate the DIRAC and DIRACOS variables when not re-installing DIRAC in SingularityCE
FIX: (#5315) Parallel Library Srun: remove chmod of executableFile
*WMS
FIX: (#5323) fix python3 incompatibility in InputDataByProtocol
[v7r2p19]
*Configuration
FIX: VOMS2CSSynchronizer: if a user is in QuarantineGroup, remove it from all other VOMS groups
[v7r2p18]
*Framework
FIX: ComponentSupervisionAgent: consistently use the actual "Setup", and not the "instance" to look up options. Note: you might need to change your config from "Production" to "DIRAC-Production" depending on you config.
FIX: fix ProxyManagerClient docs
*Resources
NEW: removal throttle for Echo
CHANGE: private methods in GFAL2_StorageBase are now protected
*Transformation
FIX: Python 3 serialisation error when calling JobMonitoringClient.getJobsStatus in WorkflowTasks.getSubmittedFileStatus
FIX: DataRecoveryAgent: fix exception when there is exactly 1 done or failed job for a transformation
[v7r2p17]
*Framework
FIX: (#5288) fix use of --excludeHosts in dirac-admin-update-instance
NEW: (#5284) Introduce Local SubLogger
*WMS
CHANGE: (#5284) attempt to use Local subLogger
NEW: (#5279) Added option for rescheduling Stalled Jobs, for certain sites
*Resources
FIX: (#5281) SSHComputingElement - treat properly the case of the pilots submitted to SSHBatch CE
FIX: (#5281) SSHBatchComputingElement - use special SSHBATCH execution context
FIX: (#5281) Host.py - put the actual execution host name/IP into the SSH_NODE_HOST environment
[v7r2p16]
*Configuration
FIX: (#5266) fix RefresherBase log
*Resources
FIX: (#5271) only evaluate number of GPUs if necessary
*WMS
FIX: (#5271) Limiter: no error if the section does not exist
*TS
NEW: (#5268) add currentStatus parameter to TransformationClient.setTransformationParameter
[v7r2p15]
*WorkloadManagement
FIX: (#5270) SandboxStoreClient tarfile mode
[v7r2p14]
NEW: Patch release is broken, use v7r2p15 instead
*Core
FIX: (#5265) Avoid pinging the database every time MySQL.__escapeString is called
NEW: (#5263) Support datetime.date objects in JEncode
*Framework
FIX: (#5254) ComponentSupervisionAgent - use socket.getfqdn() instead of gethostname()
*WorkloadManagement
FIX: (#5264) TypeError when running dirac-wms-pilot-job-info
CHANGE: (#5248) When removing Deleted jobs, they could not be removed if they had still an attached
request. With this change, if the request is in final state, it will be removed and the job as well.
*Transformation
NEW: (#5265) Add columns keyword argument to TransformationClient.getTransformations
NEW: (#5265) Add index on (TransformationID,Status) in TransformationFiles table
NEW: (#5265) Add index on (TransformationID,ExternalStatus) in TransformationTasks table
NEW: (#5265) Add index on (TransformationFamily) in Transformations table
*Resources
NEW: (#5256) SLURM can set gpus-per-task job requirement
NEW: (#5256) added getNumberOfGPUs method to JobParameters utility
FIX: (#5242) HTCondorCE: corrected ClassAd for requesting MP slots
[v7r2p13]
*Core
CHANGE: (#5233) protect client doc generation against new importlib_resources
*WorkloadManagement
FIX: (#5245) Bytes confusion when using SandboxStoreClient with Python 3
*Resources
FIX: (#5245) Bytes confusion when using SingularityComputingElement with Python 3
[v7r2p12]
CHANGE: (#5219) Service Handlers discover DBs in Extensions
*Core
FIX: (#5226) DIRAC_USE_TORNADO_IOLOOP is set in the runsv file rather than the python starting script (#5223 )
FIX: (#5234) GOC: taking hostname from URL, if present
*WMS
NEW: (#5212) SiteDirector: added flag for pilots to install python3 DIRAC clients
FIX: (#5220) JobPath executor: fix bug for per-job list of optimizers
FIX: (#5229) JobAgent - protect against trying to send a None operation in a request
CHANGE: (#5229) CHANGE: JobCleaningAgent - fixed some printouts
FIX: (#5229) dirac-admin-ban-se - make it work with option --Site
*Resources
FIX: (#5231) fix path of the executable file when using a parallel library within SSHCE
[v7r2p11]
*Core
FIX: (#5197) Fix handling Subprocess calls which produce invalid output
*Framework
FIX: (#5195) Avoid TypeError in ColorGenerator
FIX: (#5198) ComponentSupervisionAgent: Pick up PollingTime from a base agent config
*Resources
FIX: (#5199) remove CPUScalingFactor calls
*WorkloadManagementSystem
FIX: (#5199) remove CPUScalingFactor calls
FIX: (#5200) Matcher: use custom exception for Pilot Version Error
[v7r2p10]
*Python 3
FIX: (#5175) Improve "git describe" command used by setuptools-scm to ignore v6e7p27
*Core
NEW: (#5180) Add callStack keyword argument to S_ERROR to allow object to be created much faster
FIX: (#5182) Optimise ClassAd.asJDL to remove quadratic runtime dependence on the length of the output
*Framework
FIX: (#5187) ComponentSupervisionAgent: add necessary SystemAdministrator Authorization
Settings in the docstring
*Configuration
CHANGE: (#5180) No longer include a full CallStack in the S_ERROR object returned by
ConfigurationClient.getOption
NEW: (#5185) Add DIRAC_FEWER_CFG_LOCKS environment variable to significantly improve multithreading
performance in CS heavy workloads
*WMS
FIX: (#5184) JobDB: Getting InputDataModule from Operations section
FIX: (#5184) JobCleaningAgent: get jobIDs old and recent, and combine
*Resources
FIX: (#5179) Handle the case where inputs is None in SSHComputingElement.submitJobToHost()
CHANGE: (#5189) PoolComputingElement use concurrent.futures.ProcessPoolExecutor
CHANGE: (#5189) PoolComputingElement: added a shutdown (called by JobAgent)
*tests
CHANGE: (#5186) AgentOptionTests: re-work ConfigTemplate discovery, should be
transparent for the users, but work better for extensions
[v7r2p9]
*FrameworkSystem
NEW: (#5173) Add option to exclude list of hosts from dirac-admin-instace-update
CHANGE: (#5169) Deprecate checkTypeRe and HashTags in UserProfileClient
FIX: (#5169) Python 3 compatibility in UserProfileClient
[v7r2p8]
*Core
FIX: (#5158) Workaround for OSError when copying vomses directory with too many extended attributes
*Framework
NEW: (#5155) ComponentSupervisionAgent to Monitor running components. See Agent Documentation for details.
*WorkloadManagement
NEW: (#5122) multi-node allocation support via the ParallelLibraries and srun
[v7r2p7]
*Core
CHANGE: (#5142) Move MySQL.ConnectionPool class to the module level
CHANGE: (#5142) Move RequestHandler.ConnectionError to the module level
FIX: (#5133) The RPC Client class should now be threadsafe
FIX: (#5133) Correctly add RPC attributes to client objects in Python 3
FIX: (#5130) Avoid race when importing modules which can cause classes to be redefined
*Transformation
FIX: (#5135) TransformationBody update is too long for parameters
[v7r2p6]
*Core
FIX: (#5000) Display thousands separators in PrettyScalarFormatter again
[v7r2p5]
FIX: fixes from v7r0p56, v7r1p39
*WMS
CHANGE: (#5102) JobCleaningAgent will first DELETE and only then REMOVE jobs
[v7r2p4]
*Core
NEW: (#5094) DIRAC.Core.Utilities.Extensions module for interacting with extensions in a Python 3 compatible way
*tests
NEW: (#5094) ./integration_tests.py helper script for running the integration tests locally
[v7r2p3]
FIX: Fixes from v7r054 and v7r1p37
FIX: (#5088) Fix various Python 3.9 deprecations
*Core
FIX: (#5095) Avoid KeyError when calling parseCommandLine
*Framework
CHANGE: (#5092) Use String(64) for DIRACVersion column in InstalledComponentsDB.HostLogging
CHANGE: (#5098) in the run files created by ComponentInstaller, execute directly the deployed scripts
*WMS
NEW: (#5073) JobManager service now exposes a "RemoveJob" RPC call
NEW: (#5073) implementing bulk indexing for jobs parameters
CHANGE: (#5073) JobCleaningAgent will not remove those jobs that, while marked "DELETED", have still Operations to do in RMS
*RequestManagement
FIX: (#5091) Ensure BLOB columns are always passed bytes data
*Resources
CHANGE: (#5050) StompMQConnector uses a single connection
*tests
NEW: (#5086) Integration tests can be re-ran locally without re-installing the entire setup
[v7r2p2]
FIX: Fixes from v7r053 and v7r1p36
[v7r2p1]
NEW: (#5041) Add Python 3 install docs to README
FIX: (#5065) Explicitly convert variables assigned to range() into lists
*Core
CHANGE: (#5052) Added IS NULL mySql condition
*WMS
FIX: (#5039) dirac_wms_cpu_normalization: fix for python3 client, and stop looking for MJF
CHANGE: (#5070) Using ThreadPool in StalledJobAgent
*DMS
NEW: (#5057) Allows to retrieve the non recursive size of a directory from the DFC
CHANGE: (#5057) Factorize TornadoFileCatalogHandler and FileCatalogHander
[v7r2]
FIX: python 2-3 compatibility fixes
NEW: (#4209) Extends the MonitoringSystem to support RMS and DMS/Agents/RequestOperations.
CHANGE: (#4586) Use __future__ imports to get Python 3 like behaviour for imports, division and printing
CHANGE: (#4796, #4797) removed obsoleted codes
CHANGE: (#4823) Service handlers use class variables instead of global
NEW: Moved to a src/ repository layout
NEW: (#4906) Deploy releases to PyPI
FIX: (#4921) use six.moves for urllib
CHANGE: (#4958) use __doc__ for script help message, remove filecfg as argument from usage description
FIX: (#5015) Add prompt-toolkit to setuptools metadata
*Core
FIX: (#4283) Added protection against empty certificates directory in dirac-install
CHANGE: (#4582) Use concurrent.future.ThreadPoolExecutor by default.
FIX: (#4606) Handle expected HTTP status codes in dirac-install (#4562)
CHANGE: (#4630) Create symlinks to bashrc when using versions directory structure
with dirac-install.py. Fixes #4565
NEW: (#4497) add histogram support to the Graph package.
NEW: (#4497) adapt Graph package to python3
CHANGE: (#4581) Replace DIRAC.Core.Utilities.CFG with independent diraccfg module
CHANGE: (#4671) Add an example for DIRAC_NO_CFG to bashrc.
FIX: (#4736) dirac-install: use system tar command for extracting tarballs, fixes #4246
NEW: (#4677) X509Chain returns DN entry
NEW: (#4677) introduction of Tornado based services
NEW: (#4732) Support installing Python 3 clients using by passing --pythonVersion=3 to ./dirac-install.py
CHANGE: (#4791) dirac-install with diracos: remove TERMINFO, RRD_DEFAULT_FONT, GFAL and ARC paths from bashrc
FIX: (#4791) Late import of MonitoringReport
NEW: (#4823) flipped default value of DIRAC_USE_JSON_DECODE env variable
FIX: (#4843) X509Chain: fix downloading X509 certificates
FIX: (#4870) ElasticSearchDB: fix exception handling when checking existance of indices
FIX: (#4887) fix exception in Services when trying to use uninitialized gMonitor client while
targeting ES monitoring
NEW: (#4898) added a base DB class for all DB types
NEW: (#4865) Fully support installation with setuptools
CHANGE: (#4932) Rename DIRAC_VOMSES to X509_VOMSES in dirac-install.py
CHANGE: (#4958) when import Script, a help message is formed from __doc__, not need to use Script.setUsageMessage(__doc__)
FIX: (#4956) use logging.warning in favor of deprecated logging.warn
CHANGE: (#4997) Use prompt-toolkit for password input when generating proxies
CHANGE: (#4994) dirac-install.py should now be taken from management
CHANGE: (#5010) Use ConfigurationClient in CSCLI and CSShellCLI
CHANGE: (#5026) Remove DIRAC.Core.Utilities.ExitCallback
*Configuration
CHANGE: (#4866) split standard and tornado refresher in two files to avoid tornado dependency on the client
CHANGE: (#4958) setUsageMessage in LocalConfiguration will parse received str value for building help message
FIX: (#5008) HTTPS: allow to run master CS in parallel of other tornado services
FIX: (#5016) harmonize https and dips version of ServiceHandler, and make doc consistent with reality
*Framework
CHANGE: (#4787) expose deleteProxyBundle call to ProxyManagerClient
FIX: (#4850) MQ logging sends also the message,asctime and levelname field
FIX: fix link to DIRAC install instruction and comments for WebApp
FIX: (#4977) SystemAdministratorClientCLI - fix log message about responding hosts in sysadmin cli
CHANGE: (#4956) renames of column names to avoid reserved keywords
*Interfaces
CHANGE: (#4799) remove runLocalAgent method from DIRAC API
*Accounting
NEW: (#4537) Add histogram of CPU usage to Job accounting
NEW: (#4537) Implement the method used to retrieve data for creating histograms
*WMS
NEW: (#4367) Added option "CompressJDLs" to JobDB, which compresses JDLs before they are
added to the JobJDLs table
CHANGE: (#4428) removed the watchdog for mac, simplified linux
CHANGE: (#4577) Removed the Pilot2
NEW: (#4799) JobWrapperTemplate kills the JobWrapper in case of Exception during Execution phase
CHANGE: (#4799) Remove MultiProcessorSiteDirector (use standard SiteDirector)
CHANGE: (#4799) stop setting JDL requirements MaxCPUTime, SubmitPools, GridRequiredCEs, Origin, JobMode
CHANGE: (#4823) using JobReport and delayed send for JobStateUpdate
FIX: (#4829) PilotCStoJSONSynchronizer - upload method uses WebDav
NEW: (#4874) new agent PilotsSyncAgent for syncing pilot files.
NEW: (#4874) new script dirac-admin-sync-pilots for syncing pilot files.
CHANGE: (#4874) largely simplified PilotCStoJSONSynchronizer
CHANGE: (#4874) don't call PilotCStoJSONSynchronizer from the CS
NEW: (#4905) moving towards a single agent for WMS history
CHANGE: (#4925) SiteDirector - removed using of SubmitPools requirement by default
CHANGE: (#4925) SiteDirector, JobAgent - no more using ProxyManager tokens while user proxy retrieval
NEW: (#4950) JobAgent adds GridCE and queue to Job Parameters
CHANGE: (#4966) added flag to exclude the master CS from the list of servers synchronized by PilotCS2JSON
FIX: (#5017) Fix uploading jobDescription.xml in upload sandboxes with Python 3
*DataManagement
CHANGE: (#5019) Remove IRODSStorageElementHandler
*Monitoring
FIX: (#4768) Adapting RMSMonitoring type to ES7
FIX: (#4844) Correct the rounding of integers when computing buckets in MonitoringDB
NEW: (#4898) added possibility to configure the period and so the indexing strategy
CHANGE: (#4976) removed DynamicMonitoring (replaced by ActivityMonitoring)
*ResourceStatusSystem
CHANGE: (#4235) Removes locals() way of passing parameters
NEW: (#4900) Remove old entries from history
CHANGE: (#5002) RSS handlers: removed global objects
*RMS
CHANGE: (#4887) use dedicated ES monitoring flag RMSMonitoring
*docs
FIX: (#4740) make dirac-docs-get-release-notes.py python3 compatible
CHANGE: (#4823) added some clarifications on starting the pilots
CHANGE: (#4829) updated configuration in terms of security, added description of webdav
FIX: (#4905) added config and instructions for Monitoring DBs (ES) configuration
FIX: (#4973) fix link to DIRAC install instruction and comments for WebApp
*tests
NEW: (#4179) Set up Gitlab CI pipeline using Docker containers
FIX: (#4677) Ignore tornado services and test DBs
NEW: (#4677) new integration tests for Tornado based services
NEW: (#4910) --runslow option on unit tests to allow faster local tests
NEW: (#4938) added a helloworld test for the (yet to be implemented) cloud testing in certification
CHANGE: (#4968) Change the defaults for tests (to MySQL 8 and ES 7)
[v7r1p47]
*Accounting
FIX: (#5412) do not attempt to write buckets if there's nothing to write
NEW: (#5392) Add AccountingMonitor permission
FIX: (#5257) GROUP BY statement syntax to be compatible with more recent SQL reference
(e.g. as used in MariaDB 10.5). This fix is backward compatible.
*WMS
FIX: (#5382) Fix the job matching algorithm used by the dirac-wms-match command
*RequestManagement
FIX: (#5260) Fix the GROUP BY syntax in the RequestSummaryWeb to be compliant with MariaDB 10.5.
The fix is backward compatible with MySQL 5.7
[v7r1p46]
*WorkloadManagement
FIX: (#5364) convert Owner attribute into OwnerDN in PilotManagerHandler.getPilotStatistics
FIX: (#5368) JobMonitoring - fix "Unknown column 'PilotJobReference'" error
[v7r1p45]
Data Management
FIX: (#5290) Type error when calling FTS3ManagerHandler.getActiveJobs
[v7r1p44]
*WorkloadManagement
CHANGE: (#5205) Make error message from SandboxStoreClient clearer when the sandbox is not found
*DataManagement
CHANGE: (#5211) FileCatalogClientCLI register File now takes a fixed number of arguments (6)
including Checksum
*Resources
NEW: GFAL2_XROOTStorage - set XrdSecGSIDELEGPROXY=1 to enable third party copy
*Interfaces
NEW: (#5201) Implemented a function that allows programmatic listing of directories in the DFC.
Meant to replace the never fully deployed listCatalog function.
[v7r1p43]
CHANGE: new features and fixes from v7r0p57
*WorkloadManagement
FIX: (#5177) Look for dirac-jobexec instead of $DIRACROOT to decide when to copy the pilot.cfg
to improve compatibility with v7r2
[v7r1p42]
*Framework
FIX: (#5168) Avoid minor caching bug which causes constraints to be violated when rapidly
writing to UserProfileDB
[v7r1p41]
*Accounting
FIX: (#5152) Bug in PilotSubmission efficiency plot
*Framework
NEW: (#5159) Add UserProfileClient.listStatesForWeb endpoint to make the Public State Manager faster
CHANGE: (#5153) change SQLITE connection option for speed improvement
*DataManagement
FIX: (#5163) DatasetManager - Fix the syntax of the query to get dataset files
FIX: (#5162) Improve listing error if file info is missing from DB
*WorkloadManagement
CHANGE: (#5161) SiteDirector - Combine CE and Queue name in PilotSubmissionAccounting
to avoid duplicated queue name
[v7r1p40]
*Core
CHANGE: (#5123) Read X509_CERT_DIR from central place
*WorkloadManagement
CHANGE: (#5143) Clarify that "tarfile failed with message" can be normal
FIX: (#5146) SiteDirector._getPilotsWeMayWantToSubmit: fix return type in case of error
*DataManagement
CHANGE: (#5144) FileManagerPS has bulk method for getReplicas
*Resources
CHANGE: (#5137) HTCondorCE submits jobs with -spool option when a local schedd is used to
spool the pilot wrappers that can be deleted afterwards
[v7r1p39]
*WMS
CHANGE: (#5121) for HTCondor, the SiteDirectory write the executable in the globally defined working directory
[v7r1p38]
FIX: fixes from v7r0p55
*WMS
FIX: (#5108) delete the local output sandbox tarfile in case it's still there
FIX: (#5106) SiteDirector - use | operand for the union of tests
CHANGE: (#5104) JobDB.setJobAttributes: use _update instead of _transaction (as not needed)
[v7r1p37]
*DataManagement
NEW: (#5096) FTS3 plugins to alter TPC and source SE preferences
[v7r1p36]
*DMS
FIX: (#5080) SQLAlchemy 1.4 support for FTS3DB
FIX: (#5054) Submit FTS3 jobs with lowercase checksum
*RMS
FIX: (#5080) SQLAlchemy 1.4 support for ReqDB
*Resources
FIX: (#5077) Host.py - Fix for the case where no subprocess32 module found
[v7r1p35]
FIX: Fixes from v7r0p52
*Core
NEW: (#5047) DIRAC_M2CRYPTO_DEBUG env variable for SSL debugging
*WorkloadManagement
FIX: (#5062) Move the CPU work left computation at the beginning of the JobAgent execution
CHANGE: (#5053) Remove margins from the TimeLeft utility
CHANGE: (#5051) if the HeartBeatTime of a job is not set, it cannot be seen as Stalled.
Hence set the HeartBeatTime (if not set) when setting the StartExecTime
CHANGE (#5051) (StalledJobAgent): don't try and get pilot status if "Unknown"
*Interfaces
CHANGE: (#5051) print the jobID in dirac-wms-job-logging-info
*DMS
CHANGE: (#5051) lower logging level for individual LFNs (from INFO to VERBOSE) to decrease
amount of logged information. Replace with a summary
[v7r1p34]
FIX: Fixes from v7r0p51 patch release
[v7r1p33]
*DMS
FIX: (#4960) FTS3 explicitly use verify_checksum
NEW: (#4960) add --FTSOnly and --ExcludeSE options to dirac-dms-protocol-matrix
*RMS
CHANGE: (#4960) ReqClient prints the FTS3 job url directly
[v7r1p32]
*Core
FIX: (#5021) Glue2: fix error when too many execution environments were looked for
FIX: (#5021) Subprocess: when the systemCall failed, S_ERROR now contains a string
in 'Message' instead of an exception object
[v7r1p31]
*Resources
FIX: (#4996) Getting access to $_CONDOR_JOB_AD in HTCondorResourceUsage
*WMS
CHANGE: (#5001) JobAgent: if we don't match a job, independently from the reason, we wait a
bit longer before trying again
CHANGE: (#5001) JobAgent: moved check of availability of CE before doing anything else
NEW: (#4998) JobWrapper - find executables from PATH if not using an absolute filename
FIX: (#5009) As the job status is set AFTER the request is uploaded, it may happen that the request
is executed BEFORE the status is set... Add thus a delay of 2 minutes for execution
[v7r1p30]
*Configuration
FIX: (#4879) The loop in the siteCE mapping was exiting as soon as a mis-configured site was found.
Fixed to continue the loop, while adding an error message.
FIX: (#4970) Utilities - fixed bug in addToChangeSet(): use tuple instead 4 arguments
FIX: (#4974) Utilities - fix setting of LocalCEType
*WorkloadManagementSystem
FIX: (#4972) fix the cpu work left computation in JobAgent when batch system information cannot be processed
CHANGE: (#4978) Improved logging of the Matcher: adding pilotReference
*Resources
FIX: (#4955) SingularityComputingElement - better error message when the Singularity CE not found
CHANGE: (#4972) add a condition in TimeLeft module to know whether the batch system relies on wallclock and/or cpu time
NEW: (#4972) HTCondorResourceUsage module to get the time left in CERN-specific HTCondor resources.
CHANGE: (#4935) HTCondorCE: add a periodic remove in case we are not using the local schedd
[v7r1p29]
FIX: Fixes from v7r0p49 patch release
*Core
CHANGE: (#4954) using argparse for a few scripts instead of getopt
NEW: (#4957) ObjectLoader: do not fail if one module fails importing in recursive loading
*Framework
CHANGE: (#4964) dirac-proxy-init always uploads the proxy unless --no-upload is specified
FIX: (#4964) ProxyManagerClient checks for group and groupless proxies in its cache
*WMS
FIX: (#4954) PilotCStoJSONSynchronizer - bug fixed in getting CE information
CHANGE: (#4956) JobAgent - resorting to calculate time left with info in possession
*Workflow
FIX: (#4953) ModuleBase - remove deprecated function that does not do anything
[v7r1p28]
CHANGE: Revert #4919 - absolute imports in several modules
*Configuration
FIX: (#4945) CS.Client.Utilities: change exception caught to IndexError
*WMS
FIX: (#4947) SiteDirector: fix for case of empty tag
*DMS
CHANGE: (#4891) split staging and access protocols
NEW: (#4891) FTS3 can submit transfers with staging protocols different from transfer protocols
[v7r1p27]
FIX: Fixes from v7r0p47
FIX: (#4919) Added from future import absolute_import when having conflicts when importing standard modules.
Fixes issue #2207
*Core
FIX: (#4918) Glue2: fix problem for CEs with multiple Queues, only one queue was discovered
FIX: (#4926) ElasticSearchDB: fix exception handling when checking existence of indices
*Configuration
CHANGE: (#4926) Adding Pool as LocalCEType for MultiProcessor queues (invoked by BDII2CSAgent)
*Framework
CHANGE: (#4928) initialized, processMessage and flushAllMessages functions of Logging are deprecated
*WMS
FIX: (#4923) The StalledJobAgent's behaviour changed (for bad) as only jobs stalled for more than 24.5 hours were
checked before being set Failed... Restore v7r0 behaviour with immediate check
CHANGE: (#4926) JobAgent: getting CEType only from LocalSite/LocalCE config
CHANGE: (#4926) JobAgent: flipped the default for fillingMode
CHANGE: (#4926) SiteDirector: The max number of jobs in filling mode is not set anymore by the SD. The option
MaxJobsInFillMode was removed
*Resources
FIX: (#4931) InProcess/SingularityComputingElement - Fix calling _monitorProxy as a periodic task
*Monitoring
NEW: (#4924) added possibility to login to ES via certificates.
*docs
NEW: (#4853) user management, ProxyManager
CHANGE: (#4853) move Registry from ConfReference to dirac.cfg
CHANGE: (#4853) move CommandReference content to scripts docs
[v7r1p26]
FIX: Fixes from v7r0p46
*Resources
FIX: (#4909) use grid CA to download the srr file
FIX: (#4904) ARCComputingElement: fixed a few places where unicode literals have leaked: wrapped in str
*WMS
FIX: (#4904) JobAgent: using internal function for rescheduling jobs
[v7r1p25]
FIX: Fixes from v7r0p45
*Core
CHANGE: (#4890) logging failing MySQL commands
*WMS
FIX: (#4889) fix for retrieving the pilot reference when it's an ES JobParameter
FIX: (#4889) fix for using the ES JobParameters for static data of heartbeats (app std out)
FIX: (#4897) added timestamp to pilot.json
[v7r1p24]
FIX: fixes from v7r0p44
*TransformationSystem
FIX: (#4883) fix to allow to pass a list of sourceSE to dirac-transformation-replication
[v7r1p23]
FIX: Fixes from v7r0p43
*Configuration
FIX: (#4869) Bdii2CSAgent: remove constant update of NumberOfProcessors parameter.
Comparison between int and str lead to "new" value every run.
*WMS
FIX: (#4868) JobLoggingDB: fix for backward compatibility
*Monitoring
CHANGE: (#4872) better, faster aggregations. Always using term queries for keyword fields,
and other less important changes
CHANGE: (#4872) do not return error if trying to delete non-existing index
*tests
NEW: (#4872) added integration test for MonitoringDB
[v7r1p22]
*WMS
NEW: (#4852) new module JobMinorStatus.py to put some standard minor statuses
CHANGE: (#4852) JobStateUpdateHandler - set startTime of job (in attributes) as from when the
(#4852) job is Running (was when the application started). As agreed in a mail discussion...
CHANGE: (#4852) JobWrapper - send failover requests and accounting information after setting the final status
[v7r1p21]
*ConfigurationSystem
FIX: (#4854) Fix typo in CERNLDAPSyncPlugin
*Resources
FIX: (#4848) Fix SingularityComputingElement for non-server-style installations
[v7r1p20]
*Core
CHANGE: (#4842) TLS: read proxy flag from M2Crypto.X509
*Framework
CHANGE: (#4832) NotificationHandler: sendMail: the avoidSpam parameter is deprecated.
All emails (same subject, address, body) are now only sent once per 24h
*WorkloadManagementSystem
FIX: (#4839) There was a bug handling the bulk status updates that was not updating
correctly the job status although the job logging information was correct).
This PR fixes this bug. It was tested in certification as a hotfix
CHANGE: (#4839) internally always call the setJobStatusBulk() method in order to be consistent
CHANGE: (#4839) in JobReport, only send statuses that are not empty, and remove handling of
ApplicationCounter that is never sent nor used anywhere
CHANGE: (#4834) JobStateUpdateHandler: use class variables instead of global
*Resources
FIX: (#4841) Fix using SingularityComputingElement with the host's DIRAC installation
*Interfaces
FIX: (#4834) Job.py: use --cfg option for JobConfigArgs
*RSS
FIX: (#4834) Added one field for retrieving the corrected order of columns
[v7r1p19]
FIX: Fixes from v7r0p40
*Configuration
FIX: (#4821) Bdii2CSAgent: Updating CE information wasn't working because the list of
CEs wasn't picked up from the right place (Site/CE option, instead of Site/CES/<CE_Sections>)
FIX: (#4821) Glue2: fix association of CEs to sites if the CE associated sitename in the BDII differs
from the Name given in the CS