spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "萝卜丝炒饭" <1427357...@qq.com>
Subject Re: Failed to connect to master ...
Date Fri, 10 Mar 2017 07:32:27 GMT
if you want debug app with remote cluster, you should submit the jar in cmd with java debug
option and the start the ideal to connect to the cluster.

---Original---
From: "Shixiong(Ryan) Zhu"<shixiong@databricks.com>
Date: 2017/3/8 15:38:35
To: "Mina Aslani"<aslanimina@gmail.com>;
Cc: "user@spark.apache.org"<user@spark.apache.org>;"ayan guha"<guha.ayan@gmail.com>;
Subject: Re: Failed to connect to master ...


The Spark master may bind to a different address. Take a look at this page to find the correct
URL: http://VM_IPAddress:8080/


On Tue, Mar 7, 2017 at 10:13 PM, Mina Aslani <aslanimina@gmail.com> wrote:
Master and worker processes are running!

On Wed, Mar 8, 2017 at 12:38 AM, ayan guha <guha.ayan@gmail.com> wrote:
You need to start Master and worker processes before connecting to them.&#xA0;

On Wed, Mar 8, 2017 at 3:33 PM, Mina Aslani <aslanimina@gmail.com> wrote:
Hi,

I am writing a spark Transformer in intelliJ in Java and trying to connect to the spark in
a VM using setMaster. I get "Failed to connect to master ..."


I get 17/03/07 16:20:55 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master&#xA0;VM_IPAddress:7077

org.apache.spark.SparkException: Exception thrown in awaitResult
at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)



SparkSession spark = SparkSession
      .builder()
      .appName("Java Spark SQL")
      //.master("local[1]")
      .master("spark://VM_IPAddress:7077")
      .getOrCreate();
Dataset<String> lines = spark       .readStream()       .format("kafka")       .option("kafka.bootstrap.servers",
brokers)       .option("subscribe", topic)       .load()       .selectExpr("CAST(value AS
STRING)")       .as(Encoders.STRING());



I get same error when I try&#xA0;master("spark://spark-master:7077").

However, .master("local[1]") no exception is thrown.
 My Kafka is in the same VM and being new to SPARK still trying to understand: - Why I get
above exception and how I can fix it (connect to SPARK in VM and read form KAfKA in VM)?-
Why using "local[1]" no exception is thrown and how to setup to read from kafka in VM?- How
to stream from Kafka (data in the topic is in json format)?  Your input is appreciated! Best
regards, Mina




 






-- 
Best Regards,
Ayan Guha
Mime
View raw message