flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Piotr Nowojski (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-10603) Reduce kafka test duration
Date Mon, 29 Oct 2018 14:29:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-10603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667287#comment-16667287
] 

Piotr Nowojski commented on FLINK-10603:
----------------------------------------

I think the single largest overhead comes from This run time overhead comes from this code:
{code:java}
				// this means that this is either:
				// (1) the first execution of this application
				// (2) previous execution has failed before first checkpoint completed
				//
				// in case of (2) we have to abort all previous transactions
				abortTransactions(transactionalIdsGenerator.generateIdsToAbort());
{code}

in {{org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011#initializeState}} 

Commenting this line out single handedly speeds up the tests by 30-40%. Of course we can not
do that on the production, but maybe we could introduce private/internal switch {{assumeFirstExecution}}
or {{forceNotAbortOldTransactions}} that we could use in all of the tests except of the one
test that is checking for this exact feature {{FlinkKafkaProducer011ITCase#testScaleDownBeforeFirstCheckpoint}}.

But this wouldn't be a healthy "hack"...

Besides that one would have to time profile the most time consuming tests more and try to
figure out if we can speed them up.

> Reduce kafka test duration
> --------------------------
>
>                 Key: FLINK-10603
>                 URL: https://issues.apache.org/jira/browse/FLINK-10603
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Kafka Connector, Tests
>    Affects Versions: 1.7.0
>            Reporter: Chesnay Schepler
>            Assignee: vinoyang
>            Priority: Major
>              Labels: pull-request-available
>
> The tests for the modern kafka connector take more than 10 minutes which is simply unacceptable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message