Fixed the throughput problem between master/slave communication.
This fix contains two independent problems.
One was in the remoting. During a large sustained data transfer
(such as artifact archiving and large test reports), the way we
were doing flow control and ACK-ing were penalizing us badly.
I improved the flow control algorithm in remoting 1.23, and also
increased advertised window size so that the transfer can saturate
available bandwidth even when a latency is large. (And unless
the reader side is excessivesly slow, this shouldn't increase
any memory consumption.)
The other fix was in trilead-ssh2, which is our SSH client
implementation used by ssh-slaves plugin. The buffer size for flow
control was too small. I improved the way buffering is done to reduce
the memory footprint when the reader closely follows the writer, then I
increased the advertised window size. Again, this shouldn't increase
memory consumption (in fact it'll likely actually reduce them) unless
the reader end gets abandoned.
On my simulated latency-injected network, the sustained transfer rate is
now on par with scp. We win for smaller files because of the TCP slow
start penality that scp would incur, and we lose a bit as files get
larger due to additional framing overhead.
If you have manually extracted slave.jar and placed them on slaves, you
need to update them to 2.23 to see the performance benefits.
See 010b47d7307a11f0750fc01a00d99b96bbab9234 in stapler/stapler for the
actual fix.
I first discovered this on my local desktop yesterday as I was verifying
the fix for json-lib 2.4. From what I can tell, this has been broken all
along and it was working only because Facets are discovered in the lucky
order; JellyFacet got a chance to get at it before GroovyFacet did.
When I rebuilt from clean the problem stopped appearing, so I assumed
that there are some factors that cause JellyFacet to get loaded before
GroovyFacet (maybe the order in which Maven iterates dependencies or
something like that.)
But I'm no longer so sure now that I know J-on-J saw this problem at
about the same time. That's just too many coincidences.
But let's see if this would make J-on-J happy.
There are several incompatibilities in XStream 1.4 that resulted in our
changing the core, but according to the text search on all the public
plugins, none of these changes affect the 600+ plugins we have today.
However, it is still possible that there are some parts of XStream 1.3
that plugins depend on but the core doesn't depend on --- we won't be
able to tell those problems at this point.