- One could argue that without this change the system is functioning correctly and that previous behaviour
was a bug. On the other hand, people have come to rely on the previous behaviour.
- The issue really centeres around state changes in the blocked tasks. Since blocking on upstream/downstream
relies on checking the building projects and the queued (excluding blocked) tasks we need any change in
the blocked task list to be visible immediately (i.e. update the snapshot)
- I was able to reliably reproduce this behaviour with a convoluted set of manually configured projects
but turning this into a test case has not proved quite as easy. Manual testing confirms that the issue is
fixed for my manual test case
- I have also added a sorting of the blocked list when probing for tasks to unblock. This should prioritise
tasks as intended by the QueueSorter
(cherry picked from commit de87736795)
More fun here:
- All this originates from Executor extending Thread.
- There is funky logic in the lock handling code of the JVM that makes assumptions
about how it might proceed with the lock when the thread holding the lock has its
interrupt flag set.
- Really it would be better if Executor did not extend Thread as that way we wouldn't
have to deal with some of that complexity. But OTOH we are where we are and backwards
compatibility may make such a change not possible without a lot of breakage.
- Fixing the issue at hand, firstly requires that interrupting a Computer happens with the
Queue lock held (to speed up tests we have Jenkins.cleanup get the lock for all Computers)
That prevents the Queue maintain thread from getting caught
- Secondly, when removing an executor from a computer we process the removal while
holding the Queue lock, but we move the removal itself to a separate thread if we cannot
get the Queue lock in order to avoid deadlock.
- Also add helper methods to wrap tasks to be performed while holding the lock
and a helper method for Runnables that exposes the tryLock functionality
(cherry picked from commit 6f343dc7c2)
- Rather fun one here. The Lock code relies on assuming that Thread.interrupted() is clear on entry
- If it then sees Thread.interrupted() set, it will interrupt the current thread in order to set the
flag again.
- Executor is a thread that does funky things with an overridden interrupt method
- Executor.abortResult() is used to track a build be interrupted or aborted in some other way
- As a result the abortResult can cause a deadlockif there is a genuine interruption
- This fix clears the interrupt flag in abortResult() and uses the write lock in order to ensure:
- The same lock as used in interrupt() is helf
- The interrupt flag is clear
- Clearing the interrupt flag should be safe as the only time it is called is immediately after
an interruption and the resulting exception is caught and rethrown/logged anyway
(cherry picked from commit ddb0a472ad)