Test framework incorrectly passes arguments with spaces

[mwichmann]

This was encountered while investigating #4585, which escaped detection as there was no test for the specific case.

Somewhat surprisingly, in trying to add such a test, an end-to-end test gave different results than doing a similar thing by directly invoking SCons on the command line. For example, the reproducer for #4585 offers this simple command: scons TEST="a b"'. When running that, the quoted substring arrives correctly as an un-split string into the Variables code, but putting the same thing into a testcase with self.run(arguments='TEST="a b"') it does not, it hasbeen broken apart.

Turns out TestCmd.command_args needs to protect spaces in quoted substrings of argument, and also needs to not strip the quotes from quoted substrings. The former is for this problem; the latter is because some SCons usage, particularly on Windows, requires quoted strings to be emitted when launching tests.

Two tests are proposed to add to the framework unit test TestCmdTests to show this:

• r = test.command_args('prog', 'python', 'arg1 arg2="quoted"')
• r = test.command_args('prog', 'python', 'arg1 arg2="quoted with space"')

With the existing code, which uses arguments.split(), we fail to keep the quoted+spaced arg together

AssertionError: Lists differ: ['pyt[27 chars]85617.ijff6m2m/prog', 'arg1', 'arg2="quoted with space"'] != ['pyt[27 chars]85617.ijff6m2m/prog', 'arg1', 'arg2="quoted', 'with', 'space"']

First differing element 3:
'arg2="quoted with space"'
'arg2="quoted'

Second list contains 2 additional elements.
First extra element 4:
'with'

  ['python',
   '/tmp/scons/testcmd.485617.ijff6m2m/prog',
   'arg1',
-  'arg2="quoted with space"']
+  'arg2="quoted',
+  'with',
+  'space"']

Switching to shlex.split(arguments) solves the broken-apart probem, but loses the quote marks:

AssertionError: Lists differ: ['pyt[21 chars]tcmd.482013.mlrpvblb/prog', 'arg1', 'arg2="quoted with space"'] != ['pyt[21 chars]tcmd.482013.mlrpvblb/prog', 'arg1', 'arg2=quoted with space']

First differing element 3:
'arg2="quoted with space"'
'arg2=quoted with space'

  ['python',
   '/tmp/scons/testcmd.482013.mlrpvblb/prog',
   'arg1',
-  'arg2="quoted with space"']
?        -                 -

+  'arg2=quoted with space']

A regular expression can be used to do this work since neither routine quite suits, but there's a side effect here: when called through the framework, SCons' Variable code may receive arguments with quote marks included, which causes string matching not to work. That is, '"quoted with space"' != 'quoted with space'