spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: Unable to execute query on SAPHANA using SPARK
Date Tue, 29 Mar 2016 22:54:55 GMT
I concur with Gourav on this. Both SAP HANNA and Oracle exalytics push
serial scans into HW with HANA, it is by pushing the bitmaps into the L2
cache on the chip whilst Oracle has special processors on SPARC T5 called
D???? <something> that offloads the column bit scan off the cpu and onto
separate specialized HW.    As a result, both rely on massive
parallelization.  HANA is a true column store and does not end up
duplicating the data as Oracle does. Now going back to using Spark as
middleware accessing SAP HANA , it may make sense if the objective is to
extract data from SAP HANA and save it into HDFS. However, I am pretty sure
HANA already has the capability to do so. I am more familiar with SAP
Sybase IQ and I have tested its drivers and they work with Spark.

HTH

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com



On 29 March 2016 at 18:42, Gourav Sengupta <gourav.sengupta@gmail.com>
wrote:

> Hi Reena,
>
> Why would you want to run a SPARK off data in SAP HANA? Is not SAP HANA
> already an in memory, columnar storage, SAP bells-and-whistles, super-duper
> expensive way of doing what poor people do in SPARK sans SAP ERP
> integration layers?
>
> I am just trying to understand the used case here.
>
> Regards,
> Gourav
>
> On Tue, Mar 29, 2016 at 3:54 PM, reena upadhyay <
> reena.upadhyay@impetus.co.in> wrote:
>
>> I am trying to execute query using spark sql on SAP HANA  from spark
>> shell. I
>> am able to create the data frame object. On calling any action on the data
>> frame object, I am getting* java.io.NotSerializableException.*
>>
>> Steps I followed after adding saphana driver jar in spark class path.
>>
>> 1. Start spark-shell
>> 2. val df = sqlContext.load("jdbc", Map("url" ->
>> "jdbc:sap://
>> 172.26.52.54:30015/?databaseName=system&user=SYSTEM&password=Saphana123",
>> "dbtable" -> "SYSTEM.TEST1"));
>> 3. df.show();
>>
>> *I get below exception on calling any action on dataframe object.*
>>
>> *org.apache.spark.SparkException: Job aborted due to stage failure: Task
>> not
>> serializable: java.io.NotSerializableException:
>> com.sap.db.jdbc.topology.Host*
>> Serialization stack:
>>         - object not serializable (class: com.sap.db.jdbc.topology.Host,
>> value:
>> 172.26.52.54:30015)
>>         - writeObject data (class: java.util.ArrayList)
>>         - object (class java.util.ArrayList, [172.26.52.54:30015])
>>         - writeObject data (class: java.util.Hashtable)
>>         - object (class java.util.Properties, {dburl=jdbc:sap://
>> 172.26.52.54:30015,
>> user=SYSTEM, password=Saphana123,
>> url=jdbc:sap://172.26.52.54:30015/?system&user=SYSTEM&password=Saphana123
>> ,
>> dbtable=SYSTEM.TEST1, hostlist=[172.26.52.54:30015]})
>>
>>
>> Caused by: java.io.NotSerializableException: com.sap.db.jdbc.topology.Host
>> Serialization stack:
>>         - object not serializable (class: com.sap.db.jdbc.topology.Host,
>> value:
>> 172.26.52.54:30015)
>>         - writeObject data (class: java.util.ArrayList)
>>         - object (class java.util.ArrayList, [172.26.52.54:30015])
>>         - writeObject data (class: java.util.Hashtable)
>>         - object (class java.util.Properties, {dburl=jdbc:sap://
>> 172.26.52.54:30015,
>> user=SYSTEM, password=Saphana123,
>> url=jdbc:sap://172.26.52.54:30015/?system&user=SYSTEM&password=Saphana123
>> ,
>> dbtable=SYSTEM.TEST1, hostlist=[172.26.52.54:30015]})
>>
>>
>> Appreciate help on this.
>> Thank you
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-execute-query-on-SAPHANA-using-SPARK-tp26628.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>

Mime
View raw message