spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Teoh <chris.t...@gmail.com>
Subject Re: Control Sqoop job from Spark job
Date Mon, 02 Sep 2019 21:43:48 GMT
Hey Chetan,

How many database connections are you anticipating in this job? Is this for
every row in the dataframe?

Kind regards
Chris


On Mon., 2 Sep. 2019, 9:11 pm Chetan Khatri, <chetan.opensource@gmail.com>
wrote:

> Hi Chris, Thanks for the email. You're right. but it's like Sqoop job gets
> launched based on dataframe values in spark job. Certainly it can be
> isolated and broken.
>
> On Sat, Aug 31, 2019 at 8:07 AM Chris Teoh <chris.teoh@gmail.com> wrote:
>
>> I'd say this is an uncommon approach, could you use a workflow/scheduling
>> system to call Sqoop outside of Spark? Spark is usually multiprocess
>> distributed so putting in this Sqoop job in the Spark code seems to imply
>> you want to run Sqoop first, then Spark. If you're really insistent on
>> this, call it from the driver using Sqoop Java APIs.
>>
>> On Fri, 30 Aug 2019 at 06:02, Chetan Khatri <chetan.opensource@gmail.com>
>> wrote:
>>
>>> Sorry,
>>> I call sqoop job from above function. Can you help me to resolve this.
>>>
>>> Thanks
>>>
>>> On Fri, Aug 30, 2019 at 1:31 AM Chetan Khatri <
>>> chetan.opensource@gmail.com> wrote:
>>>
>>>> Hi Users,
>>>> I am launching a Sqoop job from Spark job and would like to FAIL Spark
>>>> job if Sqoop job fails.
>>>>
>>>> def executeSqoopOriginal(serverName: String, schemaName: String, username:
String, password: String,
>>>>                  query: String, splitBy: String, fetchSize: Int, numMappers:
Int, targetDir: String, jobName: String, dateColumns: String) = {
>>>>
>>>>   val connectionString = "jdbc:sqlserver://" + serverName + ";" + "databaseName="
+ schemaName
>>>>   var parameters = Array("import")
>>>>   parameters = parameters :+ "-Dmapreduce.job.user.classpath.first=true"
>>>>   parameters = parameters :+ "--connect"
>>>>   parameters = parameters :+ connectionString
>>>>   parameters = parameters :+ "--mapreduce-job-name"
>>>>   parameters = parameters :+ jobName
>>>>   parameters = parameters :+ "--username"
>>>>   parameters = parameters :+ username
>>>>   parameters = parameters :+ "--password"
>>>>   parameters = parameters :+ password
>>>>   parameters = parameters :+ "--hadoop-mapred-home"
>>>>   parameters = parameters :+ "/usr/hdp/2.6.5.0-292/hadoop-mapreduce/"
>>>>   parameters = parameters :+ "--hadoop-home"
>>>>   parameters = parameters :+ "/usr/hdp/2.6.5.0-292/hadoop/"
>>>>   parameters = parameters :+ "--query"
>>>>   parameters = parameters :+ query
>>>>   parameters = parameters :+ "--split-by"
>>>>   parameters = parameters :+ splitBy
>>>>   parameters = parameters :+ "--fetch-size"
>>>>   parameters = parameters :+ fetchSize.toString
>>>>   parameters = parameters :+ "--num-mappers"
>>>>   parameters = parameters :+ numMappers.toString
>>>>   if (dateColumns.length() > 0) {
>>>>     parameters = parameters :+ "--map-column-java"
>>>>     parameters = parameters :+ dateColumns
>>>>   }
>>>>   parameters = parameters :+ "--target-dir"
>>>>   parameters = parameters :+ targetDir
>>>>   parameters = parameters :+ "--delete-target-dir"
>>>>   parameters = parameters :+ "--as-avrodatafile"
>>>>
>>>> }
>>>>
>>>>
>>
>> --
>> Chris
>>
>

Mime
View raw message