Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add helper binary for bulk certificate processing. #15

Merged
merged 1 commit into from
Apr 16, 2018

Conversation

busykai
Copy link
Contributor

@busykai busykai commented Apr 6, 2018

Instead of launching 'openssl x509' for each file, use small binary to
process files in bulk. This optimization improves performance more than
7 times (measured on store generation). Performance of 'clrtrust
generate' is important for first-time booting.

Before:

$ time sudo clrtrust generate 
Trust store generated at /var/cache/ca-certs

real	0m2.736s
user	0m2.368s
sys	0m0.390s

After:

$ time sudo ./clrtrust generate 
Trust store generated at /var/cache/ca-certs

real	0m0.352s
user	0m0.280s
sys	0m0.086s

@busykai busykai force-pushed the kai/perf-helper-for-bulk-ops branch from 95371d9 to a9f59fa Compare April 6, 2018 20:00
@busykai
Copy link
Contributor Author

busykai commented Apr 6, 2018

@icarus-sparry would you fancy reviewing this change?

@busykai busykai force-pushed the kai/perf-helper-for-bulk-ops branch from a9f59fa to bfa9134 Compare April 6, 2018 20:27
@@ -0,0 +1,148 @@

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs copyright.

Copy link

@icarus-sparry icarus-sparry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some optional changes.

:
done 2>/dev/null
done
)
if [ $? -ne 0 ]; then

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless you want to explicitly want to return just 0/1, you could loose lines 196-199, as the function by default will return the value of the last command (in this case the subshell) by default.

clrtrust.in Outdated
finger=${finger#SHA1 Fingerprint=}
printf "%s\t%s\n" "${f}" "${finger}"
done
find $dir -maxdepth 1 -type f | "${CLR_HELPER_CMD}" -f

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any sanity checking somewhere that filenames do not include embedded newlines and tabs?

If not, then can I suggest using

find "$dir" -maxdepth 1 -type f -exec "${CLR_HELPER_CMD}" -f {} +

and making the helper program loop over its arguments? Another popular approach is to add a -z flag to the helper to accept NUL terminated filenames rather than newline terminated ones.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since there are few other places already which would fail anyways, I documented this as an issue #16. I will address it at some point, but not as part of this change.

clrtrust.in Outdated
# configured location.
LIBEXEC_DIR=LIBEXEC_CONFIG_VALUE
CLR_HELPER_NAME=clrtrust-helper
if [ -n "${CLR_LIBEXEC_DIR}" ]; then # environment override

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine. If I was writing it I would do it more like

for d in ${CLR_LIBEXEC_DIR+"$CLR_LIB_EXEC_DIR"} "$(dirname "$0")" "${LIBEXEC_DIR}"
do if [ -x "$d/${CLR_HELPER_NAME}" ] ; then
    CLR_HELPER_CMD="$d/${CLR_HELPER_NAME}"
    break
    fi
 done

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I started with it, just like the lock file location few lines below, but after few modifications got confused. This is way cleaner, thanks!


if (fname) free(fname);

return 0;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps a global return code to indicate if there were any errors (e.g. missing files) rather than success except for "somehow I got to line 136" and "missing or conflicting -s -f switches"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do. Actually, the errors are not handled correctly in the caller script either, so this must be taken care of.

print_help();
return 1;
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest below allowing the filenames to be specified on the command line.

If you don't then perhaps a check that nothing extra has been passed on the command line?

Maybe a '-z' mode to read the names NUL deliminated rather than newline deliminated?

@busykai
Copy link
Contributor Author

busykai commented Apr 14, 2018

thank you for the review, @icarus-sparry! it's very helpful.

@busykai busykai force-pushed the kai/perf-helper-for-bulk-ops branch from bfa9134 to 257bab0 Compare April 16, 2018 03:38
Instead of launching 'openssl x509' for each file, use small binary to
process files in bulk. This optimization improves performance more than
7 times (measured on store generation). Performance of 'clrtrust
generate' is important for first-time booting.
@busykai busykai force-pushed the kai/perf-helper-for-bulk-ops branch from 257bab0 to 33ccc14 Compare April 16, 2018 04:33
@busykai busykai merged commit 7fe4c17 into master Apr 16, 2018
@busykai busykai deleted the kai/perf-helper-for-bulk-ops branch April 16, 2018 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants