ippfuzz: Fuzz test generator for ipptool

[michaelrsweet]

Copied from ippsample issue 72

As part of our discussions during the February 2018 IPP F2F meeting, we discussed ways to introduce fuzz-testing into ipptool.

Two approaches were considered:

1. Adding support to ipptool to randomly generate requests, attributes, values, etc.
2. Adding a program to generate random ipptool test files that can be fed into ipptool.

Since the first approach would still require ipptool to be able to write out a failed test for later diagnosis (what the second approach will do as its primary function), it seems like creating a test file generator is the simpler approach.

The generator should provide options to limit the amount of randomness, for example "always use Get-Printer-Attributes", "always include the following attributes and values", etc. so that "mostly valid" requests can be generated to test specific areas.

The generator should also provide options for the expected responses, for example "expect the following status codes" or "expect the random attributes to be in the unsupported attributes group of the response."

Ideally ippfuzz should be able to automate running ipptool and iterate N times, storing the results in a directory and providing a summary of the test results on stdout.

---

Also, when automating the tool should stop after a certain number of repeated failures (default 3?) - at that point the Printer has probably crashed...

---

When writing the results to a test directory, rename tests and results, e.g. "pass-nnnnnnnn.ext" for passed tests and "fail-nnnnnnnn.ext" for failed tests? Be sure to include the .test file and the XML results plist file.

---

Maybe focus on supporting fuzzing templates which look like ipptool test files? Expand values like:

• "$[ascii(100)]": Random string of exactly 100 characters
• "$[ascii(0,32767)]": Random string of 0 to 32767 characters
• "$[utf8(1000)]": Random Unicode string of 1000 characters
• "$[bytes(10,5000)]": 10 to 5000 random bytes
• "$[integer(10,20)]": Random integer from 10 to 20
• "$[boolean]": Random boolean value
• "$[value1|value2|value3]": randomly choose between the three values
• "$[repeat(10000) ... ]": expand the "..." text 10000 times.

"$[...]" might not be the best choice, consider other markup options.

Need to see what other fuzzing tools do.

---

Resources:

• The Fuzzing Project
• American fuzzy lop
• Scapy

[attah]

Inspired by this, i added fuzzing to my IPP implementation.
I went with LibFuzzer since it was so quick to get running: attah/ppm2pwg@e71a6e7

The people that worked on it has since moved on and via centipede, the efforts were merged into google/fuzztest.
In doing so it has gained functionality for property based testing, which should be good for the guided fuzzing mentioned above. Though i must admit i'm a bit apprehensive about guiding fuzzing too much, as i'd probably be blind to several problematic areas and might guide the fuzzer away from them. Coverage guided fuzzing definitely seems like a good approach, and with fuzztest it sounds like you can combine the approaches as you wish.

So how did LibFuzzer do on my code? It managed to smoke out one pretty bad undefined behavior that could be hit already without a corpus. The decoder itself has now held for something like 8 thread-hours. Not sure what that says though; either i did a good job, or the fuzzer is not very effective. ¯\(ツ)/¯
Fort sure, combining with the other sanitization options will turn up all sorts of things.

[fish98]

Existing OpenPrinting's integration with Oss-Fuzz currently supports fuzzing for libcups (specificlaly for IPP). While it does not yet provide full coverage, ongoing efforts are being made to improve.

[michaelrsweet]

@fish98 This issue is tracking efforts to automate fuzzing of IPP servers and not just the low-level IPP message parsing code.

[fish98]

@fish98 This issue is tracking efforts to automate fuzzing of IPP servers and not just the low-level IPP message parsing code.

It is true that existing OSS-Fuzz harnesses for OpenPrinting focus primarily on function-level testing, given the integration cost considerations across all OpenPrinting projects. However, fuzzing IPP servers via OSS-Fuzz is theoretically feasible, while it would require more effort (similar to fuzzers in projects such as OpenSSL).

IMHO, although current OSS-Fuzz building has integrated different fuzzing engines (including AFL and Honggfuzz), I believe that end-to-end testing, e.g., directly applying AFL to fuzz the IPP server, would be more effective in discovering exploitable bugs than function-level fuzz testings. That said, I have not found end-to-end fuzzing integration scripts within OpenPrinting projects. Personally, I run these fuzzing engines on my own local machine to test the IPP server, which is far less continous and scalable comparing to online testing services such as OSS-Fuzz. Integrating exising end-to-end fuzzers into the existing codebase would likely require further organization to ensure efficiency and sustainable testing.

[michaelrsweet]

@fish98 The goal here is to develop a generic fuzzing tool for IPP requests sent to an IPP server, not just the ISTO-PWG ippsample project, CUPS ippeveprinter, CUPS cupsd/cupslocald/cupssharingd, and PAPPL-based printer applications, but real printers.

[fish98]

@fish98 The goal here is to develop a generic fuzzing tool for IPP requests sent to an IPP server, not just the ISTO-PWG ippsample project, CUPS ippeveprinter, CUPS cupsd/cupslocald/cupssharingd, and PAPPL-based printer applications, but real printers.

Noted with thanks 😃. Please correct me if I'm wrong, but from my perspective, as long as fuzz testing is applied to the printing service, there is no significant difference between testing the ipp projects and real printers in terms of the design of fuzzing techniques (e.g., drivers, input mutations, and detection oracles). The only difference lies in the deployment target, not the fuzzer itself.

[michaelrsweet]

@fish98 Agreed, you'd employ the same techniques to generate the fuzzed requests, and in fact you could also employ the same technique for detecting if a service crashes during or after sending the test request.

ippfuzz is intended for sending otherwise valid IPP request messages to test the edges and limits of the implementation - how does it handle values that are too long, or unexpected attributes, syntax, etc. vs. sending randomized content that may or may not be a valid request (which is also a valid thing to test but I find it easier to simply test the IPP message parser like we do with the current IPP fuzzing...)