spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Faisal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-14737) Kafka Brokers are down - spark stream should retry
Date Tue, 10 May 2016 17:55:13 GMT

    [ https://issues.apache.org/jira/browse/SPARK-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15278560#comment-15278560
] 

Faisal commented on SPARK-14737:
--------------------------------

I am not sure if i am following you correctly. You mean to resubmit the failed application
through command prompt? 
{code}
spark-submit \
--master yarn-cluster \
--files /home/siddiquf/spark/log4j-spark.xml
--conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j-spark.xml" \
--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j-spark.xml" \
--class com.example.MyDataStreamProcessor \
myapp.jar
{code}

or is there any event we can capture that driver is going down because kafka is unavailable?

*Here is my test setup*

{code}

I have 5 nodes of kafka cluster. Topic is created with 5 partition and 5 replication.

Topic:ETL PartitionCount:5 ReplicationFactor:5 Configs:

Topic: ETL        Partition: 0    Leader: 1       Replicas: 1,5,2,3,4     Isr: 5,1,2,3,4

Topic: ETL        Partition: 1    Leader: 2       Replicas: 2,1,3,4,5     Isr: 5,1,2,3,4

Topic: ETL        Partition: 2    Leader: 3       Replicas: 3,2,4,5,1     Isr: 5,1,2,3,4

Topic: ETL        Partition: 3    Leader: 4       Replicas: 4,3,5,1,2     Isr: 5,1,2,3,4

Topic: ETL        Partition: 4    Leader: 5       Replicas: 5,4,1,2,3     Isr: 5,1,2,3,4
{code}


Test 1#

- 1 node is shutdown in kafka cluster. 
- Processed messages 
- Messages streamed and spark works fine

Test 2#

- 2 nodes are shutdown in kafka cluster. 
- Processed messages.
- Messages streamed and spark works fine

Test 3#

- 3 nodes are shutdown in kafka cluster. 
- Notice Spark job failed. 
- Tried to start again but it failed with following error message
{code}
org.apache.spark.SparkException:java.nio.channels.UnresolvedAddressException java.nio.channels.UnresolvedAddressException
java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused
java.nio.channels.UnresolvedAddressException at org.apache.spark.streaming.kafka.DirectKafkaInputDStream$DirectKafkaInputDStreamCheckpointData$$anonfun$2.apply(DirectKafkaInputDStream.scala:149)
at org.apache.spark.streaming.kafka.DirectKafkaInputDStream$DirectKafkaInputDStreamCheckpointData$$anonfun$2.apply(DirectKafkaInputDStream.scala:149)
Is it correct that kafka requires at-least 3 brokers up for successfull replication? 1 leader
and 2 followers? Note: no zookeeper is involved here.
{code}
I restarted the 3rd node so that 3 nodes are up. 
like test#2 and spark should start working. 
But unfortunately spark job continue failing with same error messages unless i start the 4th
node as well.

Note: Kafka cluster is with default server configuration.


> Kafka Brokers are down - spark stream should retry
> --------------------------------------------------
>
>                 Key: SPARK-14737
>                 URL: https://issues.apache.org/jira/browse/SPARK-14737
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>    Affects Versions: 1.3.0
>         Environment: Suse Linux, Cloudera Enterprise 5.4.8 (#7 built by jenkins on 20151023-1205
git: d7dbdf29ac1d57ae9fb19958502d50dcf4e4fffd), kafka_2.10-0.8.2.2
>            Reporter: Faisal
>
> I have spark streaming application that uses direct streaming - listening to KAFKA topic.
> {code}
> HashMap<String, String> kafkaParams = new HashMap<String, String>();
>     kafkaParams.put("metadata.broker.list", "broker1,broker2,broker3");
>     kafkaParams.put("auto.offset.reset", "largest");
>     HashSet<String> topicsSet = new HashSet<String>();
>     topicsSet.add("Topic1");
>     JavaPairInputDStream<String, String> messages = KafkaUtils.createDirectStream(
>             jssc, 
>             String.class, 
>             String.class,
>             StringDecoder.class, 
>             StringDecoder.class, 
>             kafkaParams, 
>             topicsSet
>     );
> {code}
> I notice when i stop/shutdown kafka brokers, my spark application also shutdown.
> Here is the spark execution script
> {code}
> spark-submit \
> --master yarn-cluster \
> --files /home/siddiquf/spark/log4j-spark.xml
> --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j-spark.xml" \
> --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j-spark.xml" \
> --class com.example.MyDataStreamProcessor \
> myapp.jar 
> {code}
> Spark job submitted successfully and i can track the application driver and worker/executor
nodes.
> Everything works fine but only concern if kafka borkers are offline or restarted my application
controlled by yarn should not shutdown? but it does.
> If this is expected behavior then how to handle such situation with least maintenance?
Keeping in mind Kafka cluster is not in hadoop cluster and managed by different team that
is why requires our application to be resilient enough.
> Thanks



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message