spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: Unable to execute query on SAPHANA using SPARK
Date Tue, 29 Mar 2016 15:52:09 GMT
Sounds like Drive issue as it can prepare the DF but cannot collect it.

In general as long you have the correct JDBC drivers it should work.

Have you modified spark-defaults.conf and added the driver list there?

For example here I have both Oracle and Sybase IQ drivers

spark.driver.extraClassPath
/home/hduser/jars/jconn4.jar:/home/hduser/jars/ojdbc6.jar

You also need to refer to these jar files in your spark-shell

spark-shell --master spark://<IP_ADDR> --jars
/home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar,/home/hduser/jars/junit-4.12.jar

HTH

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com



On 29 March 2016 at 16:15, Ted Yu <yuzhihong@gmail.com> wrote:

> As the error said, com.sap.db.jdbc.topology.Host is not serializable.
>
> Maybe post question on Sap Hana mailing list (if any) ?
>
> On Tue, Mar 29, 2016 at 7:54 AM, reena upadhyay <
> reena.upadhyay@impetus.co.in> wrote:
>
>> I am trying to execute query using spark sql on SAP HANA  from spark
>> shell. I
>> am able to create the data frame object. On calling any action on the data
>> frame object, I am getting* java.io.NotSerializableException.*
>>
>> Steps I followed after adding saphana driver jar in spark class path.
>>
>> 1. Start spark-shell
>> 2. val df = sqlContext.load("jdbc", Map("url" ->
>> "jdbc:sap://
>> 172.26.52.54:30015/?databaseName=system&user=SYSTEM&password=Saphana123",
>> "dbtable" -> "SYSTEM.TEST1"));
>> 3. df.show();
>>
>> *I get below exception on calling any action on dataframe object.*
>>
>> *org.apache.spark.SparkException: Job aborted due to stage failure: Task
>> not
>> serializable: java.io.NotSerializableException:
>> com.sap.db.jdbc.topology.Host*
>> Serialization stack:
>>         - object not serializable (class: com.sap.db.jdbc.topology.Host,
>> value:
>> 172.26.52.54:30015)
>>         - writeObject data (class: java.util.ArrayList)
>>         - object (class java.util.ArrayList, [172.26.52.54:30015])
>>         - writeObject data (class: java.util.Hashtable)
>>         - object (class java.util.Properties, {dburl=jdbc:sap://
>> 172.26.52.54:30015,
>> user=SYSTEM, password=Saphana123,
>> url=jdbc:sap://172.26.52.54:30015/?system&user=SYSTEM&password=Saphana123
>> ,
>> dbtable=SYSTEM.TEST1, hostlist=[172.26.52.54:30015]})
>>
>>
>> Caused by: java.io.NotSerializableException: com.sap.db.jdbc.topology.Host
>> Serialization stack:
>>         - object not serializable (class: com.sap.db.jdbc.topology.Host,
>> value:
>> 172.26.52.54:30015)
>>         - writeObject data (class: java.util.ArrayList)
>>         - object (class java.util.ArrayList, [172.26.52.54:30015])
>>         - writeObject data (class: java.util.Hashtable)
>>         - object (class java.util.Properties, {dburl=jdbc:sap://
>> 172.26.52.54:30015,
>> user=SYSTEM, password=Saphana123,
>> url=jdbc:sap://172.26.52.54:30015/?system&user=SYSTEM&password=Saphana123
>> ,
>> dbtable=SYSTEM.TEST1, hostlist=[172.26.52.54:30015]})
>>
>>
>> Appreciate help on this.
>> Thank you
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-execute-query-on-SAPHANA-using-SPARK-tp26628.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>

Mime
View raw message