spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Javier Domingo Cansino <javier.domi...@fon.com>
Subject Re: dse spark-submit multiple jars issue
Date Thu, 13 Aug 2015 10:22:29 GMT
Please notice that 'jars: null'

I don't know why you put ///..... but I would propose you just put normal
absolute paths.

dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
--jars /home/missingmerch/postgresql-9.4-1201.jdbc41.jar
/home/missingmerch/dse.jar
/home/missingmerch/spark-cassandra-connector-java_2.10-1.1.1.jar
/home/missingmerch/etl-0.0.1-SNAPSHOT.jar

Hope this is helpful!

[image: Fon] <http://www.fon.com/>Javier Domingo CansinoResearch &
Development Engineer+34 946545847Skype: javier.domingo.fonAll information
in this email is confidential <http://corp.fon.com/legal/email-disclaimer>

On Tue, Aug 11, 2015 at 3:42 PM, satish chandra j <jsatishchandra@gmail.com>
wrote:

> HI,
>
> Please find the log details below:
>
>
> dse spark-submit --verbose --master local --class HelloWorld
> etl-0.0.1-SNAPSHOT.jar --jars
> file:/home/missingmerch/postgresql-9.4-1201.jdbc41.jar
> file:/home/missingmerch/dse.jar
> file:/home/missingmerch/postgresql-9.4-1201.jdbc41.jar
>
> Using properties file: /etc/dse/spark/spark-defaults.conf
>
> Adding default property:
> spark.cassandra.connection.factory=com.datastax.bdp.spark.DseCassandraConnectionFactory
>
> Adding default property: spark.ssl.keyStore=.keystore
>
> Adding default property: spark.ssl.enabled=false
>
> Adding default property: spark.ssl.trustStore=.truststore
>
> Adding default property:
> spark.cassandra.auth.conf.factory=com.datastax.bdp.spark.DseAuthConfFactory
>
> Adding default property: spark.ssl.keyPassword=cassandra
>
> Adding default property: spark.ssl.keyStorePassword=cassandra
>
> Adding default property: spark.ssl.protocol=TLS
>
> Adding default property: spark.ssl.useNodeLocalConf=true
>
> Adding default property: spark.ssl.trustStorePassword=cassandra
>
> Adding default property:
> spark.ssl.enabledAlgorithms=TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA
>
> Parsed arguments:
>
>   master                  local
>
>   deployMode              null
>
>   executorMemory          null
>
>   executorCores           null
>
>   totalExecutorCores      null
>
>   propertiesFile          /etc/dse/spark/spark-defaults.conf
>
>   driverMemory            512M
>
>   driverCores             null
>
>   driverExtraClassPath    null
>
>   driverExtraLibraryPath  null
>
>   driverExtraJavaOptions  -Dcassandra.username=missingmerch
> -Dcassandra.password=STMbrjrlb -XX:MaxPermSize=256M
>
>   supervise               false
>
>   queue                   null
>
>   numExecutors            null
>
>   files                   null
>
>   pyFiles                 null
>
>   archives                null
>
>   mainClass               HelloWorld
>
>   primaryResource         file:/home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>
>   name                    HelloWorld
>
>   childArgs               [--jars
> file:/home/missingmerch/postgresql-9.4-1201.jdbc41.jar
> file:/home/missingmerch/dse.jar
> file:/home/missingmerch/postgresql-9.4-1201.jdbc41.jar]
>
>   jars                    null
>
>   verbose                 true
>
>
>
> Spark properties used, including those specified through
>
> --conf and those from the properties file
> /etc/dse/spark/spark-defaults.conf:
>
>   spark.cassandra.connection.factory ->
> com.datastax.bdp.spark.DseCassandraConnectionFactory
>
>   spark.ssl.useNodeLocalConf -> true
>
>   spark.ssl.enabled -> false
>
>   spark.executor.extraJavaOptions -> -XX:MaxPermSize=256M
>
>   spark.ssl.keyStore -> .keystore
>
>   spark.ssl.trustStore -> .truststore
>
>   spark.ssl.trustStorePassword -> cassandra
>
>   spark.ssl.enabledAlgorithms ->
> TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA
>
>   spark.cassandra.auth.conf.factory ->
> com.datastax.bdp.spark.DseAuthConfFactory
>
>   spark.ssl.protocol -> TLS
>
>   spark.ssl.keyPassword -> cassandra
>
>   spark.ssl.keyStorePassword -> cassandra
>
>
>
>
>
> Main class:
>
> HelloWorld
>
> Arguments:
>
> --jars
>
> file:/home/missingmerch/postgresql-9.4-1201.jdbc41.jar
>
> file:/home/missingmerch/dse.jar
>
> file:/home/missingmerch/postgresql-9.4-1201.jdbc41.jar
>
> System properties:
>
> spark.cassandra.connection.factory ->
> com.datastax.bdp.spark.DseCassandraConnectionFactory
>
> spark.driver.memory -> 512M
>
> spark.ssl.useNodeLocalConf -> true
>
> spark.ssl.enabled -> false
>
> SPARK_SUBMIT -> true
>
> spark.executor.extraJavaOptions -> -XX:MaxPermSize=256M
>
> spark.app.name -> HelloWorld
>
> spark.ssl.enabledAlgorithms ->
> TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA
>
> spark.ssl.trustStorePassword -> cassandra
>
> spark.driver.extraJavaOptions -> -Dcassandra.username=missingmerch
> -Dcassandra.password=STMbrjrlb -XX:MaxPermSize=256M
>
> spark.ssl.keyStore -> .keystore
>
> spark.ssl.trustStore -> .truststore
>
> spark.jars -> file:/home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>
> spark.cassandra.auth.conf.factory ->
> com.datastax.bdp.spark.DseAuthConfFactory
>
> spark.master -> local
>
> spark.ssl.protocol -> TLS
>
> spark.ssl.keyPassword -> cassandra
>
> spark.ssl.keyStorePassword -> cassandra
>
> Classpath elements:
>
> file:/home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>
>
>
>
>
> WARN  2015-08-11 08:23:25 org.apache.spark.util.Utils: Service 'SparkUI'
> could not bind on port 4040. Attempting port 4041.
>
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.postgresql.Driver
>
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
>         at java.lang.Class.forName0(Native Method)
>
>         at java.lang.Class.forName(Class.java:190)
>
>         at HelloWorld$.main(HelloWorld.scala:26)
>
>         at HelloWorld.main(HelloWorld.scala)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>         at java.lang.reflect.Method.invoke(Method.java:606)
>
>         at
> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
>
>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
> Regards,
>
> Satish Chandra J
>
> On Tue, Aug 11, 2015 at 6:15 PM, Javier Domingo Cansino <
> javier.domingo@fon.com> wrote:
>
>> use --verbose, it might give you some insights on what0s happening,....
>>
>> [image: Fon] <http://www.fon.com/>Javier Domingo CansinoResearch &
>> Development Engineer+34 946545847Skype: javier.domingo.fonAll
>> information in this email is confidential
>> <http://corp.fon.com/legal/email-disclaimer>
>>
>> On Tue, Aug 11, 2015 at 2:44 PM, satish chandra j <
>> jsatishchandra@gmail.com> wrote:
>>
>>> HI ,
>>> I have used --jars option as well, please find the command below
>>>
>>> *Command:*
>>>
>>> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
>>> *--jars* ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar
>>> ///home/missingmerch/dse.jar
>>> ///home/missingmerch/spark-cassandra-connector-java_2.10-1.1.1.jar
>>> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>>>
>>> Regards,
>>> Satish
>>>
>>> On Tue, Aug 11, 2015 at 4:08 PM, Javier Domingo Cansino <
>>> javier.domingo@fon.com> wrote:
>>>
>>>> I have no real idea (not java user), but have you tried with the --jars
>>>> option?
>>>>
>>>>
>>>> http://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management
>>>>
>>>> AFAIK, you are currently submitting the jar names as arguments to the
>>>> called Class instead of the jars themselves
>>>>
>>>> [image: Fon] <http://www.fon.com/>Javier Domingo CansinoResearch &
>>>> Development Engineer+34 946545847Skype: javier.domingo.fonAll
>>>> information in this email is confidential
>>>> <http://corp.fon.com/legal/email-disclaimer>
>>>>
>>>> On Tue, Aug 11, 2015 at 12:29 PM, satish chandra j <
>>>> jsatishchandra@gmail.com> wrote:
>>>>
>>>>>
>>>>> *HI,*
>>>>>
>>>>> Please let me know if i am missing anything in the command below
>>>>>
>>>>> *Command:*
>>>>>
>>>>> dse spark-submit --master spark://10.246.43.15:7077 --class
>>>>> HelloWorld --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar
>>>>> ///home/missingmerch/dse.jar
>>>>> ///home/missingmerch/spark-cassandra-connector-java_2.10-1.1.1.jar
>>>>> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>>>>>
>>>>>
>>>>> *Error:*
>>>>>
>>>>> *java.lang.ClassNotFoundException: HelloWorld*
>>>>>
>>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>>>>
>>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>>
>>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>>
>>>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>>
>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>>>
>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>>>
>>>>>         at java.lang.Class.forName0(Native Method)
>>>>>
>>>>>         at java.lang.Class.forName(Class.java:270)
>>>>>
>>>>>         at
>>>>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:342)
>>>>>
>>>>>         at
>>>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>>>>>
>>>>>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>>
>>>>>
>>>>> I understand the way I am giving multiple jar file paths in the
>>>>> command is an issue, please provide an appropriate format for providing
>>>>> multiple jars in the command
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks for support
>>>>>
>>>>>
>>>>> Satish Chandra
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message