v0.71.0-qe.01

This is a "quality engineering" release to help track what was tested.

v0.69.9 (agent-only)

This is an agent-only release, fixing a major performance bug in the post-processing of benchmark results, and delivering a fix to restore sos report collections.

In addition, the CI infrastructure that depended on the now-unavailable Travis CI, was moved to the Jenkins infrastructure that we now use for development. A few behavioral (and other) bugs have been fixed as well. There are some server fixes too, but we are not quite ready to make a new release of the server yet.

Changelog

Here is the list of commits since the v0.69.8 agent release:

• a12a9aa Fix stand-alone use of pbench-make-result-tb (#2404)
• 2d9fb23 Tell RPM pbench-agent.cfg file is a config file (#2403)
• 0aa01a9 Fio (et al.) postprocessing performance improvements
• 97d8021 Finialize the bit-rot detection script behaviors
• 28454b8 Quick-n-Dirty bit-rot detection for tar balls
• d49f2ab Use common config for unit test audit server runs
• 10e5e9c Remove mapping of /dev/log
• fa3d91e Exclude .venv from Black's attention
• cc35c09 Update the Jenkins container to F33 and add more packages Also, don't install the Black RPM -- it's the wrong version.
• bb0f1e2 Add Jenkins support to the b0.69 branch
• e5f2eaf Backport detox from main, and other useful tox.ini updates
• 3a93045 Clean up linting requirements for b0.69
• 9bb68a0 pbench-agent.cfg is no longer a ghost (#2389)
• 5cae0ad Drop Fedora 32, add Fedora 34 to builds
• 65133ac Backport "Fixes for sos-report >= 4.0 versions" to 0.69 (#2373)
• 4eb8a7a Fix the util-scripts unittests
• d259c53 Rename pbench-agent.cfg.example file
• 8a8e8c0 Add a helpful success message for audits
• 948f58f Various fixes to get pbench-server running on RHEL 8

v0.69.8 (agent only)

This release is a pbench-agent only release.

There are no changes in installation: please follow the instructions in the Installation Guide.

Bug fixes

There were several areas where bug fixes were applied. The following summary might be useful. The details can be found in the Changelog section.

• pbench-trafficgen was the recipient of a bunch of fixes, but there is still work remaining to be done.

• pbench-fio was updated to require the 3.21 of fio, had 99.5 percentile support added, and the localhost latency profiles are now calculated properly.

• pbench-move-results now takes a --controller option, backported from the latest version. Also, a bug in the ssh copy loop was fixed.

• A locale issue on Centos 8 was fixed.

• Problems with sosreport handling were fixed (fixes backported from latest).

• HammerDB compatibility support was added (backported from latest).

• Bug fixes to pbench-list-tools and pbench-register-tool-trigger.

Changelog

The list of commits since version 0.69.4 (the previous publicly available version of pbench-agent) is as follows:

• 90314ee Fix pbench-register-tool-trigger (#2319)
• c0c911b Fix pbench-list-tools --name (#2301)
• 509bd97 Backport pbench-move-results --controller
• a00739a Bump pbench-agent version to 0.69.7
• cadf50c proc-iter-samp: compare ref value to empty string
• bd18858 pbench-trafficgen: consistent syntax ${tool_group}
• 4821ffb trafficgen-postprocess: clean up uid label logic
• 208b7c9 Fix mock'd binary-search.py logic
• e2d593c process-iteration-samples: indent 2nd line of errors
• a129b13 pbench-trafficgen - use if-then-elif-else-fi
• 6d58b23 Validate tool-period option
• fa5eb34 Emit pbench-trafficgen usage message on stderr
• 5b7540b Use perl's exists() function where appropriate
• 31a95d1 Remove unused mean() function
• f012fdc Fix more whitespace formatting issues
• fffaf03 Remove use of Dumper that is unused
• 31b6f96 Fix the trafficgen mock
• e1376fb Enhance unit tests to capture generated commands
• cd0a48e Refactor to fix trafficgen UIDs (back-port)
• eee43a3 Only emit the benchmark exit code on failure
• 0a3e24c Make proc-iter-samp loop param names consistent
• 639dc08 Address whitespace issues in proc-iter-samp
• a06f969 Refactor config result directory identifier
• 31bd901 Move normalization of active devices before use
• 44d1d8f Make use of default variables for cmd line args
• 40172f2 Drop pbench_cmd in favor of orig_cmd
• bb4c895 Sort trafficgen default variables
• 8e72d54 Separate pbench defaults from trafficgen defaults
• 961ad19 Fix up spelling in help message output
• fa43d75 Emit usage on stderr where appropriate
• 8274e63 Fix case statement indentation to stand out
• d76faf6 Don't generate any files in post-process only mode
• fae58e1 Only use inner most loop arguments
• 4ee7872 DRY out the repeated messages
• bfcdb62 Move loop control variable immediately before loop
• 92376c7 Consolidate benchmark command file construction and fix pushd
• 416382d Consistently apply variable references building iteration options
• 9031c25 Use double bracket if conditional to avoid obscure syntax
• d0d3e90 Consolidate logic for iteration options construction
• c960ad2 Unindent now that we have refactored the inner loop
• ed60b0a Pre-increment inner loop count
• c665c83 Move inner_loop count control to top
• ac6be05 Backport HammerDB compatibility support
• fb97518 Backport sosreport handling fixes
• c8f89b2 Backport gold file updates for Perl 5.30+
• 2c295a7 Address Python3 syntax issue, "is" instead of "=="
• 3d33e81 Backport update to fio 3.21
• e5d334e Change version target to 0.69.6
• c6209a1 Add 99.5 percentile support
• 7768b08 pbench-fio: compute localhost latency profiles
• ac1a476 Fix tabs vs spaces for option processing
• 42a1d2e Correct locale issues with CentOS 8
• f12e022 Fedora 33
• 975bc23 Fix move results ssh loop
• a0ebc0c Add missing "g" to commit id field in the server config file
• 5e3781c Correct the use of commit_id
• 22ed51e Make the backup cron job run every minute
• 778e899 Backport PR #2035 to 0.69
• 9d5b091 Avoid recursive restorecon
• c02634d Fix SELinux labels on the pbench directory structure
• 017291c Fix spec file and ansible roles
• dd96370 More restorecon-related fixes
• a01f1b4 Backport server RPM making to b0.69
• bac3e95 Fix the SELinux context of the tar ball's MD5 file
• d5769ed Restore the SELinux context of the tarball after the move
• d7326a5 Backport container build enhancements
• beab972 Fix the README to restore working URLs

v0.69.7 (agent only)

[N.B. This was never released officially. The latest available version of pbench-agent is 0.69.4 and the next release will be 0.69.8.]

This is a pbench-agent only release.

It includes the following changes:

• Fixes to pbench_fio:
  • Latency profiles for localhost were being skipped - they are now
    calculated properly.
  • Add 99.5 percentile support.
  • Update fio version to 3.21.
• Fixes for sosreport handling:
  • Record the command that was used.
  • Fix option handling bugs.
• Fix locale issue on CentOS 8.
• Fixes to pbench-trafficgen: there are multiple problems with
  pbench-trafficgen and the set of fixes in this release, although
  extensive, is still incomplete.

Installation

There are ansible playbooks to install the pbench-agent and the pieces needed
(key and config files) to be able to send results to a server.

There are no other installation changes in this release: see the Getting Started
Guide for how to install or update.

After installation or update, you should have version 0.69.4-1gaa13bda5d of the
pbench-agent RPM installed.

Agent

This is an agent-only release, fixing the issues described above.

Server

This is an agent-only release. There is one change to server code, however we
are not ready to make a release yet.

Web Server

There are no changes and no new web-server RPMs have been produced.

Pbench Dashboard

The development of the dashboard is not tracked in these release notes. The
dashboard has been moved into its own git repo.

ChangeLog

This is the list of visible commits since v0.69.5-1 of the agent:

• a00739a Bump pbench-agent version to 0.69.7
• cadf50c proc-iter-samp: compare ref value to empty string
• bd18858 pbench-trafficgen: consistent syntax ${tool_group}
• 4821ffb trafficgen-postprocess: clean up uid label logic
• 208b7c9 Fix mock'd binary-search.py logic
• e2d593c process-iteration-samples: indent 2nd line of errors
• a129b13 pbench-trafficgen - use if-then-elif-else-fi
• 6d58b23 Validate tool-period option
• fa5eb34 Emit pbench-trafficgen usage message on stderr
• 5b7540b Use perl's exists() function where appropriate
• 31a95d1 Remove unused mean() function
• f012fdc Fix more whitespace formatting issues
• fffaf03 Remove use of Dumper that is unused
• 31b6f96 Fix the trafficgen mock
• e1376fb Enhance unit tests to capture generated commands
• cd0a48e Refactor to fix trafficgen UIDs (back-port)
• eee43a3 Only emit the benchmark exit code on failure
• 0a3e24c Make proc-iter-samp loop param names consistent
• 639dc08 Address whitespace issues in proc-iter-samp
• a06f969 Refactor config result directory identifier
• 31bd901 Move normalization of active devices before use
• 44d1d8f Make use of default variables for cmd line args
• 40172f2 Drop pbench_cmd in favor of orig_cmd
• bb4c895 Sort trafficgen default variables
• 8e72d54 Separate pbench defaults from trafficgen defaults
• 961ad19 Fix up spelling in help message output
• fa43d75 Emit usage on stderr where appropriate
• 8274e63 Fix case statement indentation to stand out
• d76faf6 Don't generate any files in post-process only mode
• fae58e1 Only use inner most loop arguments
• 4ee7872 DRY out the repeated messages
• bfcdb62 Move loop control variable immediately before loop
• 92376c7 Consolidate benchmark command file construction and fix pushd
• 416382d Consistently apply variable references building iteration options
• 9031c25 Use double bracket if conditional to avoid obscure syntax
• d0d3e90 Consolidate logic for iteration options construction
• c960ad2 Unindent now that we have refactored the inner loop
• ed60b0a Pre-increment inner loop count
• c665c83 Move inner_loop count control to top
• ac6be05 Backport HammerDB compatibility support
• fb97518 Backport sosreport handling fixes
• c8f89b2 Backport gold file updates for Perl 5.30+
• 2c295a7 Address Python3 syntax issue, "is" instead of "=="
• 3d33e81 Backport update to fio 3.21
• e5d334e Change version target to 0.69.6
• c6209a1 Add 99.5 percentile support
• 7768b08 pbench-fio: compute localhost latency profiles
• ac1a476 Fix tabs vs spaces for option processing
• 42a1d2e Correct locale issues with CentOS 8
• f12e022 Fedora 33
• 975bc23 Fix move results ssh loop
• a0ebc0c Add missing "g" to commit id field in the server config file
• 5e3781c Correct the use of `commit_id`

v0.69.5-1 (server release only)

Installed on production 2020/12/16 17:00 UTC

v0.70.0-beta

v0.70.0 (Beta) Release Notes

This is a very significant "minor" release of the pbench code base, agent and server.

The "Tool Meister" functionality (PR #1248) is the major piece of functionality delivered with this release. This is a significant change, where the pbench-agent first orchestrates the instantiation of a "Tool Meister" process on all hosts registered with tools, using a Redis instance to coordinate their operation, and the new "Tool Data Sink" handles the collection of data into the pbench run directory hierarchy. We effectively eliminate all remote SSH operations for tools except one per host to orchestrate the creation of the Tool Meister instance.

The Tool Meister work has enabled us to integrate with Prometheus for tool data collection. Two new pbench "tools" have been added, node-exporter and dcgm. If one registers either or both of these new tools (e.g. via pbench-register-tools --name=node-exporter), then the Tool Meister sub-system will run the node_exporter code on the registered hosts, and a local instance of Prometheus to collect the data. The collected Prometheus data is stored in the pbench result directory as a tar ball at: ${pbench_run}/<script>_<config>_YYYY.MM.DDTHH.mm.ss/tools-<group>/prometheus. For the duration of the run, the Prometheus instance is available on localhost:9090 if one desires to review the metrics being collected live.

NOTE WELL: like all the other "tools" the pbench-agent supports, the node-exporter and dcgm tools themselves need to be installed separately.

Along with the delivery of the "Tool Meister" work, the notion of tool registration has changed significantly, where tools are now recorded as registered only on the local host were pbench-register-tool and pbench-register-tool-set are invoked. As a result of this change, the following behavioral changes follow:

• The process of registering tools on local or remote hosts no longer validates that those tools are available during tool registration
• All tools registered prior to installing v0.70.0-beta must be re-registered; tools registered locally or remotely on a controller with a version of the pbench-agent prior to v0.70.0 will be ignored until they are re-registered

For both the Pbench Agent and the Server we have removed the use of the Software Collections Library (SCL) in order to provide a Python 3 environment for RHEL 7 & CentOS 7 distributions. We now rely on the Python 3 provided with RHEL 7.7 and later.

We did not bump the "major" release version number with these changes because we still don't consider all the necessary functionality in place for such a major version bump.

Installation

There are no other installation changes in this release: see the Getting Started Guide for how to install or update.

After installation or update, you should have version 0.70.0-9ga7d1f0d6 of the pbench-agent RPM installed.

RPMs are available from Fedora COPR, covering Fedora 31, 32, EPEL 7 & 8.

There are Ansible playbooks available via Ansible Galaxy to install the pbench-agent, and the pieces needed (key and configuration files) to be able to send results to a server. To use the RPMs provided above via COPR with the playbooks, an inventory file needs to include the fedoraproject_username variable set to portante, for example:

...

[servers:vars]
fedoraproject_username: portante

...

Alternatively, one can specify fedoraproject_username on the command line, rather than having it specified in the inventory file:

ansible-playbook -i <inventory> <playbook> -e '{fedoraproject_username: portante}'

NOTE WELL: If the inventory file also has a definition for pbench_repo_url_prefix (which was standard practice before fedoraproject_username was introduced), it needs to be deleted, otherwise it will override the default repo URL and the fedoraproject_username change is not going to take effect.

While we don't include installation instructions for the new node-exporter and dcgm tools in the published documentation, you can find a manual installation procedure for the Prometheus "node_exporter" and references to the Nvidia "DCGM" documentation in the agent/tool-scripts/README.

Container images built using the above RPMs are available in the Pbench organization in the Quay.io container image repository using tags beta, v0.70.0-9, and a7d1f0d6.

Agent

In addition to the major changes described above for this release, the following significant changes for the agent are also worth calling out specifically:

• The pbench-fio bench script now requires fio-3.21 or later; see 5048a14

  • You can find a Fedora COPR build of fio-3.21-6.pbench for EPEL 7 & 8

• [DEPRECATED] The pbench-cleanup utility command is deprecated, and will be removed in a subsequent release (see PR #1828)

• The release takes in the latest code from stockpile, including support for GPU data collection via the Nvidia SMI role

• The new dcgm tool requires Python 2, an Nvidia based install which might conflict with the Pbench Agent's Python 3 operational requirement in some cases

Server

Initial support for moving results to the pbench-server via HTTP PUT operations.

Web Server

There are no changes and no new web-server RPMs have been produced.

Pbench Dashboard

The development of the dashboard is not tracked in these release notes. The dashboard has been moved into its own git repo.

ChangeLog

This is the list of visible commits since the v0.69.3-agent release:

a7d1f0d Add 'v0.70.0-beta' release notes
46aa2ad Update the development image we use for 'b0.70'
8ae1367 Add missing 'net-tools' req agent RPM
8b7424a Remove colorlog; add Makefile for pbench-devel
e702457 Rework pass-thru API implementation a bit
6dc6e4f Jenkins integration using Fedora 32 container
e927f02 Fix the flaky util-scripts test-51 & test-52
dd1c52d Address common logging between agent and server
a37b40c Fix agent side 'test_move_results'
54b6522 Use '_pbench_' prefixed env var for host names
03c14b4 Make sure we explicitly ask for the full hostname
50c301d Refactor base to extract unit test overrides
1261d6d Address some undesirable tox behaviors
af520f2 Fix misaligned deps in tox.ini
f0f6217 Direct 'black' to ignore '.git' subtree
589caa6 Cleanup 'datalog/prometheus-metrics-datalog'
ea58135 Rejig functional unit tests
aaef93a Fix warning while running unit tests
1496c7c Remove colorlog
1609912 Only do git submodule init at the top of the tree
98f42b7 Remove wayward py3-functional unit test
cb93ab3 Add ability to tag agent images with beta and alpha
f28c7a3 Require pyesbulk 1.0.0 for now
0c05fb8 Remove SCL from the agent side
4eb6d19 Remove SCL reference in the server trampoline code
98d50ce Fix the README to restore working URLs
e3e893f Enhance agent container builds
7dd58f0 Add required package name for vmstat
a431636 Remove use of 'screen' from Tool Meister
5671ea5 Stop invoking screen directly in unit tests
1a1dfb7 Source agent 'base' for remote tool meisters
b37ae97 Record the pip3 command output to a log file
6977291 Add warning for left-over 'id_rsa' file
ad6e9de Encapsulate building the final requirements files
8c7cadf Don't '%ghost' 'id_rsa'; only 'rm' in '%postun'
afa583a Move to rh-python38 for RHEL 7 & use python RPMs
9574bf6 Add redis and python3-redis agent RPM requires
3241acb Add a "common" unit test environment
6d44341 Move md5sum() method to common utils
fa6b25d Don't remove 'id_rsa' and 'pbench-agent.cfg' when uninstalling
e1ca5f3 Remove blanket executable file permissions
b722ecc Use the %config for the agent config file
3612c00 Ensure proper use of rh-python36 only for RHEL 7
c4d6848 Remove pbench-agent RPM dependencies for tools
d72f829 Update minimum supported Fedora to 31
92d0383 Update pbench-agent spec file summary & desc
8458a45 Add use of 'rpmlint' on '.spec' files
5b425b4 DRY out run-unittests, adding one for the server
f908712 Create the notion of a utils directory
d438663 First pass at a container image for the pbench-server
fdda138 Making a server RPM (without SCL)
720e6c3 Initial pbench-agent container image layering
20a307e RPM making for pbench-agent
3aa69d6 General method of fetching git commit ID
59d1989 update agent cli to reflect server upload PUT changes
3526a76 Additional changes to break up testing requirements
80c4b7e Break up agent & server requirements
c29f7ca Resolves logging error in pbench-server-prep-shim
ed39d86 Fix missing space in sosreport --quiet option
f32e0fd write md5sum of a tarfile on disk on server and add upload POST to upload PUT
7d122d2 Fix pbench-clear-tools based on new unit tests
736bd27 Fix handling of expected exit status in unit tests
b03cddd Remove , add periods in pbench-clear-tools
2b8df4d give pbench user access to /run/pbench-server directory
2021484 Deprecate pbench-cleanup & add tests for clear-results
3d0d1f4 Switch legacy unit tests to use C.UTF-8 locale
79086b4 `Updated benchmark...

v0.69.4 (agent release only)

This is a pbench-agent only release.

Minor update to deliver a fix for improperly handling tool arguments, and a fix for cases where the perf tool would not run.

The primary reason for this release is to fix the issue described in #1751 (via PR #1812), where if multiple arguments to a tool are given when registering that tool, only the first argument was being used when the tool was invoked. This was introduced in the v0.68 release.

In addition, we also provide a fix for the perf tool which would fail to run if the debugfs file system was not present (issue #1811, fix via PR #1815). This bug was also introduced in v0.68 release. This release changes the behavior so that a warning is issued for the user to know that kernel debug information may not be available in the perf output.

Installation

There are ansible playbooks to install the pbench-agent and the pieces needed (key and config files) to be able to send results to a server.

There are no other installation changes in this release: see the Getting Started Guide for how to install or update.

After installation or update, you should have version 0.69.4-1gaa13bda5d of the pbench-agent RPM installed.

Agent

This is an agent-only release, fixing issues #1751 and #1811 as described above.

Server

This is an agent-only release. There are several changes to server code, however we are not ready to make a release yet.

Web Server

There are no changes and no new web-server RPMs have been produced.

Pbench Dashboard

The development of the dashboard is not tracked in these release notes. The dashboard has been moved into its own git repo.

ChangeLog

This is the list of visible commits since v0.69.3-agent:

aa13bda RPM making for pbench-agent
3071aa6 Initial pbench-agent container image layering
798fca4 Back-port agent ansible work
94c14ae Sort modules in requirements.txt
ea3416f Only emit a warning message on missing debugfs
88e9878 Fix tool argument handling
2f0f114 Ensure tool option order is preserved
4e1e520 Stabilize server test-27
e741de4 Stablize the cpuacct-datalog test
23ab59f Backport black and flake8 version lock

v0.69.3-agent

This has a fix for uperf and some ansible fixes.

N.B. The version bump to 0.69.2 was never made into a release.

Release v0.69.1-agent

Includes some last minute fixes to ansible playbooks since v0.69-agent. Although this is going to be the official name of the release, the names of the RPMs will change but the contents will not.

v0.69-agent

The most visible change is that the RPMs for fio and uperf are now called (drum roll...) fio and uperf.