Append: tuple and other questions

[mwichmann]

This discussion is to nail down the intended behavior in the Append family of functions for a few edge cases that are still a bit fuzzy.

The implementation works hard to be clever, with a stated intent that you can just go ahead and Append without worrying what the type of any existing value is. This can lead to a bit of ambiguity, and is certainly not universally true in any case, but, arguably, (nearly?) all of the common cases work on this basis.

You can consider there to be two categories of object that are likely to go into construction variables: scalar and collection. Scalars are single-valued things (strings, integers, callables, and class instances), while collections are as Python describes them (dicts and sequences, not including strings which are a weird hybrid of scalar and iterable).

For a number of types, if the orig and new value are the same type, you can concatenate them:

XXX='abc' ; Append(XXX='def') -> 'abcdef'
XXX=['a', 'b'] ; Append(XXX=['c', 'd']) -> ['a', 'b', 'c', 'd']
XXX=('a', 'b') ; Append(XXX=('c', 'd')) -> ('a', 'b', 'c', 'd')

That's because Python lets you do that via the add operator. Very convenient, though possibly not always the right choice (more on tuple combining a moment)

If the types don't match (or are not otherwise addable), either a different approach is chosen, or some form of conversion tends to take place.

If orig is a mutable sequence (list, UserList, deqeue, etc.), SCons tries to append() new:

XXX=['a', 'b'] ; Append(XXX='c')  -> ['a', 'b', 'c']

If orig didn't have an .append, SCons tries to insert orig to the front of new, in the hopes new might be a mutable sequence, and you can still get a good result:

XXX='a' ; Append(XXX=['b', 'c'])  -> ['a', 'b', 'c']

Here a kind of type conversion has happened implicitly: XXX was a string, now it's a list of strings.

There's also special handing for a mutable mapping type (dict and relatives).

However, there are some questions.

What should happen for scalar types? It seems that except for strings, where we document the behavior (will be combined), some should perhaps not be appended to, but maybe some should? If we have a construction variable that provides the function to call in a certain situation (consider ESCAPE), what does it mean to change that value by "appending to it"? Unless that has specific meaning (usually indicated by the object implementing an __add__ and/or __append__ method), it should probably fail. But there might be cases where that's not right - LDMODULE_EMITTER is a list of emitter functions. The container for construction variables is just a simple dictionary, so can't carry extra information about intent - there's no way to express "intent" in some kind of metadata carried along with the variable, so it's probably best to just leave those cases alone as failing.

What should happen with tuples? An orig tuple will fail the add and append tests - Python sees an immutable which can neither add nor append. Except - if new is another tuple; Python lets you do that, you get a new (flattened) tuple. So these three examples show an irritating bit of inconsistency - tuple + tuple flattens, tuple + list and list + tuple do not.

XXX=('a', 'b') ; Append(XXX=['c', 'd']) -> [('a', 'b'), 'c', 'd']
XXX=('a', 'b') ; Append(XXX=('c', 'd')) -> ('a', 'b', 'c', 'd')
XXX=['a', 'b'] ; Append(XXX=('c', 'd')) -> ['a', 'b', ('c', 'd')]

Also a bit inconsistent is what you can add to a tuple: a tuple and a list work, but a string does not:

XXX=('a', 'b') ; Append(XXX=('c', 'd')) -> ('a', 'b', 'c', 'd')
XXX=('a', 'b') : Append(XXX='c') -> FAIL 'str' object has no attribute 'insert'
XXX=('a', 'b') ; Append(XXX=['c', 'd']) -> [('a', 'b'), 'c', 'd']

Should tuples flatten? On the one hand, SCons has a tradition of "if you don't want the contents messed with, enclose it in a list" - e.g. to keep spaces from stripping from strings, and on that basis you could say tuples will be flattened, to avoid that, enclose in a list. On the other hand, there's some benefit to the idea of considering tuples as a kind of scalar type - you don't look inside, so they're not flattened, just added as-is. In discussion on Discord we had comments in support of both positions. What we shouldn't have is inconsistency - flattens in one case, not in another.

What should happen if you Append to an empty list? This is a pretty minor question, but there's a quick bailout available if you say "if orig is empty, just replace it with new". However, based on the current implementation, a few unit tests fail - specifically the ones where a string is added to an empty list.

TODO: there are some possible questions about unique appending, and about dictionary types.