spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-27002) Latest kafka delegation token not always picked up
Date Wed, 27 Feb 2019 18:08:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-27002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Marcelo Vanzin reassigned SPARK-27002:
--------------------------------------

    Assignee: Gabor Somogyi

> Latest kafka delegation token not always picked up
> --------------------------------------------------
>
>                 Key: SPARK-27002
>                 URL: https://issues.apache.org/jira/browse/SPARK-27002
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 3.0.0
>            Reporter: Gabor Somogyi
>            Assignee: Gabor Somogyi
>            Priority: Critical
>
> Lately KAFKA-8004 has been discovered and this triggered me to do further tests with
Spark.
> Namely I've created long running Kafka to Kafka tests on 4 node cluster with randomly
thrown artificial exceptions.
> Test scenario:
> * 4 node cluster
> * Yarn
> * Kafka broker version 2.1.0
> * security.protocol = SASL_SSL
> * sasl.mechanism = SCRAM-SHA-512
> Kafka broker settings:
> * delegation.token.expiry.time.ms=600000 (10 min)
> * delegation.token.max.lifetime.ms=1200000 (20 min)
> * delegation.token.expiry.check.interval.ms=300000 (5 min)
> After each 7.5 minutes new delegation token obtained from Kafka broker (10 min * 0.75).
> But when token expired after 10 minutes (Spark obtains new one and doesn't renew the
old), the brokers expiring thread comes after each 5 minutes (invalidates expired tokens)
and artificial exception has been thrown inside the Spark application (such case Spark closes
connection), then the latest delegation token not always picked up.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message