apr_thread_pool: don't detach worker threads (and always join them).

Detached threads are out of control and don't give the user a way to
synchronize when the program ends (or when the thread_pool is destroyed).

Rework the synchronization logic so that dead threads are always joined. This
is done by adding dead_thds ring and a join_dead_threads() function called from
several places: on termination when the thread_pool is destroyed, on resizing
with apr_thread_pool_{thread,idle}_max_set() when stopping off limits threads,
on maintenance when a task is added/cancelled.

Since join_dead_threads() waits for threads already dead (not the idle/busy
ones asked to die) the operation is not blocking indefinitely, so there is no
indefinite (nor much) overhead added to these places due to waiting for dead
threads.

The thread_pool_func() worker function is reworked to have a single point of
exit, and to always update the rings and counters accurately according to the
thread state, which allows for simpler maintenance and/or termination from the
other functions. The threads now put themselves into the (new) dead_thds ring
before exit, and the apr_thread_pool_{thread,idle}_max_set() functions don't
modify the idle or busy rings concurrently with the thread_pool_func() workers
anymore. They only ask the threads to stop and put themselves in dead_thds.

To join all the threads in thread_pool_cleanup(), cancel all the tasks and wait
for the busy threads handling them to die/idle (thanks to the new "work_done"
condition variable triggered by the thread after its task), then stop all the
remaining threads by calling apr_thread_pool_thread_max_set(0), and finally
wait for the last thread to exit (thanks to the new "all_done" condition var
triggered by the last thread exiting, replacing the spin wait).

So destroying the thread_pool (or a parent pool) may wait (indefinitely) for
all the active tasks to complete, which is the purpose of this commit: avoid
undefined behaviour in this case.


git-svn-id: https://svn.apache.org/repos/asf/apr/apr/trunk@1884098 13f79535-47bb-0310-9956-ffa450edef68