spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: Has anyone managed to connect to Oracle via JDBC from Spark CDH 5.5.2
Date Thu, 22 Dec 2016 21:03:51 GMT
Why not upgrade to ojdbc7 - this one is for java 7+8? Keep in mind that the jdbc driver is
updated constantly (although simply called ojdbc7). I would be surprised if this does not
work with cloudera as it runs on the oracle big data appliance.

> On 22 Dec 2016, at 21:44, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:
> 
> Thanks all sorted. The admin had updated these lines incorrectly in $SPARK_HOME/conf/spark-defaults.conf
by updating one of the parameters only for Oracle ojdbc6.jar and the other one for Sybase
jconn4.jar!
> 
>  
> spark.driver.extraClassPath      /home/hduser/jars/ojdbc6.jar
> spark.executor.extraClassPath    /home/hduser/jars/jconn4.jar
> 
> Also noticed that although we could see Oracle table metadata once connected no action
such as collecting data could be done. It was crashing.
> 
> So decided to downgrade ojdbc6.jar to ojdbc5.jar and that worked with Spark. Interestingly
Spoop has no problem using ojdbc6.jar. FYI the Oracle database version accessed is 11g, R2
> 
>  Also it is a challenge in a multi-talented cluster to maintain multiple versions of
jars for the same database type through $SPARK_HOME/conf/spark-defaults.conf!
> 
> HTH
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> http://talebzadehmich.wordpress.com
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage
or destruction of data or any other property which may arise from relying on this email's
technical content is explicitly disclaimed. The author will in no case be liable for any monetary
damages arising from such loss, damage or destruction.
>  
> 
>> On 22 December 2016 at 10:44, Alexander Kapustin <kpavn@hotmail.com> wrote:
>> Hello,
>> 
>>  
>> 
>> For Spark 1.5 (and for 1.6) we use Oracle jdbc via spark-submit.sh --jars /path/to/ojdbc6.jar
>> 
>> Also we use additional oracle driver properties via --driver-java-options
>> 
>>  
>> 
>> Sent from Mail for Windows 10
>> 
>>  
>> 
>> From: Mich Talebzadeh
>> Sent: 22 декабря 2016 г. 0:40
>> To: user @spark
>> Subject: Has anyone managed to connect to Oracle via JDBC from Spark CDH 5.5.2
>> 
>>  
>> 
>> This works with Spark 2 with Oracle jar file added to
>> 
>> $SPARK_HOME/conf/ spark-defaults.conf 
>>  
>>  
>> spark.driver.extraClassPath      /home/hduser/jars/ojdbc6.jar
>> spark.executor.extraClassPath    /home/hduser/jars/ojdbc6.jar
>>  
>> and you get
>>  cala> val s = HiveContext.read.format("jdbc").options(
>>      | Map("url" -> _ORACLEserver,
>>      | "dbtable" -> "(SELECT to_char(ID) AS ID, to_char(CLUSTERED) AS CLUSTERED,
to_char(SCATTERED) AS SCATTERED, to_char(RANDOMISED) AS RANDOMISED, RANDOM_STRING, SMALL_VC,
PADDING FROM scratchpad.dummy)",
>>      | "partitionColumn" -> "ID",
>>      | "lowerBound" -> "1",
>>      | "upperBound" -> "100000000",
>>      | "numPartitions" -> "10",
>>      | "user" -> _username,
>>      | "password" -> _password)).load
>> s: org.apache.spark.sql.DataFrame = [ID: string, CLUSTERED: string ... 5 more fields]
>> 
>> that works.
>> However, with CDH 5.5.2 (Spark 1.5) it fails with error
>> 
>> java.sql.SQLException: No suitable driver
>>   at java.sql.DriverManager.getDriver(DriverManager.java:315)
>>   at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$2.apply(JdbcUtils.scala:54)
>>   at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$2.apply(JdbcUtils.scala:54)
>>   at scala.Option.getOrElse(Option.scala:121)
>>   at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.createConnectionFactory(JdbcUtils.scala:53)
>>   at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:123)
>>   at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.<init>(JDBCRelation.scala:117)
>>   at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:53)
>>   at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:315)
>>   at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:149)
>>   at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:122)
>> 
>> 
>> Any ideas?
>> 
>> Thanks
>> 
>> 
>> 
>> 
>> Dr Mich Talebzadeh
>>  
>> LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>  
>> http://talebzadehmich.wordpress.com
>> 
>> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage
or destruction of data or any other property which may arise from relying on this email's
technical content is explicitly disclaimed. The author will in no case be liable for any monetary
damages arising from such loss, damage or destruction.
>>  
> 

Mime
View raw message