spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Gummelt <mgumm...@mesosphere.io>
Subject Re: How to make Mesos Cluster Dispatcher of Spark 1.6.1 load my config files?
Date Wed, 19 Oct 2016 17:14:59 GMT
See https://issues.apache.org/jira/browse/SPARK-13258 for an explanation
and workaround.

On Wed, Oct 19, 2016 at 1:35 AM, Chanh Le <giaosudau@gmail.com> wrote:

> Thank you Daniel,
> Actually I tried this before but this way is still not flexible way if you
> are running multiple jobs at the time and may different dependencies
> between each job configuration so I gave up.
>
> Another simple solution is set the command bellow as a service and I am
> using it.
>
> /build/analytics/spark-1.6.1-bin-hadoop2.6/bin/spark-submit \
>>
>>
>>
>>
>> *--files /build/analytics/kafkajobs/prod.conf \--conf
>> 'spark.executor.extraJavaOptions=-Dconfig.fuction.conf' \--conf
>> 'spark.driver.extraJavaOptions=-Dconfig.file=/build/analytics/kafkajobs/prod.conf'
>> \--conf
>> 'spark.driver.extraClassPath=/build/analytics/spark-1.6.1-bin-hadoop2.6/lib/postgresql-9.3-1102.jdbc41.jar'
>> \--conf
>> 'spark.executor.extraClassPath=/build/analytics/spark-1.6.1-bin-hadoop2.6/lib/postgresql-9.3-1102.jdbc41.jar'
>> \*
>> --class com.ants.util.kafka.PersistenceData \
>>
>> *--master mesos://10.199.0.19:5050 \*--executor-memory 5G \
>> --driver-memory 2G \
>> --total-executor-cores 4 \
>> --jars /build/analytics/kafkajobs/spark-streaming-kafka_2.10-1.6.2.jar \
>> /build/analytics/kafkajobs/kafkajobs-prod.jar
>>
>
> [Unit]
> Description=Mesos Cluster Dispatcher
>
> [Service]
> ExecStart=/build/analytics/kafkajobs/persist-job.sh
> PIDFile=/var/run/spark-persist.pid
> [Install]
> WantedBy=multi-user.target
>
>
> Regards,
> Chanh
>
> On Oct 19, 2016, at 2:15 PM, Daniel Carroza <dcarroza@stratio.com> wrote:
>
> Hi Chanh,
>
> I found a workaround that works to me:
> http://stackoverflow.com/questions/29552799/spark-
> unable-to-find-jdbc-driver/40114125#40114125
>
> Regards,
> Daniel
>
> El jue., 6 oct. 2016 a las 6:26, Chanh Le (<giaosudau@gmail.com>)
> escribió:
>
>> Hi everyone,
>> I have the same config in both mode and I really want to change config
>> whenever I run so I created a config file and run my application with it.
>> My problem is:
>> It’s works with these config without using Mesos Cluster Dispatcher.
>>
>> /build/analytics/spark-1.6.1-bin-hadoop2.6/bin/spark-submit \
>>
>>
>>
>>
>> *--files /build/analytics/kafkajobs/prod.conf \--conf
>> 'spark.executor.extraJavaOptions=-Dconfig.fuction.conf' \--conf
>> 'spark.driver.extraJavaOptions=-Dconfig.file=/build/analytics/kafkajobs/prod.conf'
>> \--conf
>> 'spark.driver.extraClassPath=/build/analytics/spark-1.6.1-bin-hadoop2.6/lib/postgresql-9.3-1102.jdbc41.jar'
>> \--conf
>> 'spark.executor.extraClassPath=/build/analytics/spark-1.6.1-bin-hadoop2.6/lib/postgresql-9.3-1102.jdbc41.jar'
>> \*
>> --class com.ants.util.kafka.PersistenceData \
>>
>> *--master mesos://10.199.0.19:5050 \*--executor-memory 5G \
>> --driver-memory 2G \
>> --total-executor-cores 4 \
>> --jars /build/analytics/kafkajobs/spark-streaming-kafka_2.10-1.6.2.jar \
>> /build/analytics/kafkajobs/kafkajobs-prod.jar
>>
>>
>> And it’s didn't work with these:
>>
>> /build/analytics/spark-1.6.1-bin-hadoop2.6/bin/spark-submit \
>>
>>
>>
>>
>> *--files /build/analytics/kafkajobs/prod.conf \--conf
>> 'spark.executor.extraJavaOptions=-Dconfig.fuction.conf' \--conf
>> 'spark.driver.extraJavaOptions=-Dconfig.file=/build/analytics/kafkajobs/prod.conf'
>> \--conf
>> 'spark.driver.extraClassPath=/build/analytics/spark-1.6.1-bin-hadoop2.6/lib/postgresql-9.3-1102.jdbc41.jar'
>> \--conf
>> 'spark.executor.extraClassPath=/build/analytics/spark-1.6.1-bin-hadoop2.6/lib/postgresql-9.3-1102.jdbc41.jar'
>> \*
>> --class com.ants.util.kafka.PersistenceData \
>>
>>
>> *--master mesos://10.199.0.19:7077 \--deploy-mode cluster \--supervise \*
>> --executor-memory 5G \
>> --driver-memory 2G \
>> --total-executor-cores 4 \
>> --jars /build/analytics/kafkajobs/spark-streaming-kafka_2.10-1.6.2.jar \
>> /build/analytics/kafkajobs/kafkajobs-prod.jar
>>
>> It threw me an error: *Exception in thread "main" java.sql.SQLException:
>> No suitable driver found for jdbc:postgresql://psqlhost:5432/kafkajobs*
>> which means my —conf didn’t work and those config I put in */build/analytics/kafkajobs/prod.conf
>> *wasn’t loaded. It only loaded thing I put in application.conf (default
>> config).
>>
>> How to make MCD load my config?
>>
>> Regards,
>> Chanh
>>
>> --
> Daniel Carroza Santana
> Vía de las Dos Castillas, 33, Ática 4, 3ª Planta.
> 28224 Pozuelo de Alarcón. Madrid.
> Tel: +34 91 828 64 73 // *@stratiobd <https://twitter.com/StratioBD>*
>
>
>


-- 
Michael Gummelt
Software Engineer
Mesosphere

Mime
View raw message