spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cody Koeninger <c...@koeninger.org>
Subject Re: Jenkins networking / port contention
Date Fri, 01 Jul 2016 16:49:49 GMT
Makes sense.  I'll submit a fix for kafka 0.8 and do a scan through of
other tests to see if I can find similar issues.

On Fri, Jul 1, 2016 at 11:45 AM, Reynold Xin <rxin@databricks.com> wrote:
> Multiple instances of test runs are usually running in parallel, so they
> would need to bind to different ports.
>
>
> On Friday, July 1, 2016, Cody Koeninger <cody@koeninger.org> wrote:
>>
>> Thanks for the response.  I'm talking about test code that starts up
>> embedded network services for integration testing.
>>
>> KafkaTestUtils in particular always attempts to start a kafka broker
>> on the standard port, 9092.  Util.startServiceInPort is intended to
>> pick a higher port if the starting one has a bind collision... but in
>> my local testing multiple KafkaTestUtils instances running at the same
>> time on the same machine don't actually behave correctly.
>>
>> I already updated the kafka 0.10 consumer tests to use a random port,
>> and can do the same for the 0.8 consumer tests, but wanted to make
>> sure I understood what was happening in the Jenkins environment.
>>
>> On Fri, Jul 1, 2016 at 11:18 AM, shane knapp <sknapp@berkeley.edu> wrote:
>> > i assume you're talking about zinc ports?
>> >
>> > the tests are designed to run one at a time on randomized ports -- no
>> > containerization.  we're on bare metal.
>> >
>> > the test launch code executes this for each build:
>> > # Generate random point for Zinc
>> > export ZINC_PORT
>> > ZINC_PORT=$(python -S -c "import random; print
>> > random.randrange(3030,4030)")
>> >
>> > On Fri, Jul 1, 2016 at 6:02 AM, Cody Koeninger <cody@koeninger.org>
>> > wrote:
>> >> Can someone familiar with amplab's jenkins setup clarify whether all
>> >> tests
>> >> running at a given time are competing for network ports, or whether
>> >> there's
>> >> some sort of containerization being done?
>> >>
>> >> Based on the use of Utils.startServiceOnPort in the tests, I'd assume
>> >> the
>> >> former.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message