spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ZHANG Wei <wezh...@outlook.com>
Subject Re: How to pass a constant value to a partitioned hive table in spark
Date Thu, 16 Apr 2020 11:26:15 GMT
> scala>   spark.sql($sqltext)
> <console>:41: error: not found: value $sqltext
>          spark.sql($sqltext)
                     ^
                     +-- should be Scala language

Try this:

scala> spark.sql(sqltext)

-- 
Cheers,
-z

On Thu, 16 Apr 2020 08:49:40 +0100
Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:

> I have a variable to be passed to a column of partition as shown below
> 
> *val broadcastValue = "123456789" * // I assume this will be sent as a
> constant for the batch
> // Create a DF on top of XML
> 
> df.createOrReplaceTempView("tmp")
> // Need to create and populate target Parquet table
> michtest.BroadcastStaging
> //
> HiveContext.sql("""DROP TABLE IF EXISTS michtest.BroadcastStaging""")
> 
>   var sqltext = """
>   CREATE TABLE IF NOT EXISTS michtest.BroadcastStaging (
>      partyId STRING
>    , phoneNumber STRING
>   )
>   PARTITIONED BY (
>      broadcastId STRING
>    , brand STRING)
>   STORED AS PARQUET
>   """
>   HiveContext.sql(sqltext)
> 
> // Now insert the data from temp table
>   //
>   // Put data in Hive table
>   //
>      // Dynamic partitioning is disabled by default. We turn it on
>      spark.sql("SET hive.exec.dynamic.partition = true")
>      spark.sql("SET hive.exec.dynamic.partition.mode = nonstrict ")
> 
>   sqltext = """
> 
> *  $INSERT INTO TABLE michtest.BroadcastStaging PARTITION (broadcastId =
> $broadcastValue, brand = "dummy")*  SELECT
>           ocis_party_id AS partyId
>         , target_mobile_no AS phoneNumber
>   FROM tmp
>   """
>   spark.sql($sqltext)
> 
> 
> However, this does not work!
> 
> 
> scala>   sqltext = """
>      |   $INSERT INTO TABLE michtest.BroadcastStaging PARTITION
> (broadcastId = $broadcastValue, brand = "dummy")
>      |   SELECT
>      |           ocis_party_id AS partyId
>      |         , target_mobile_no AS phoneNumber
>      |   FROM tmp
>      |   """
> sqltext: String =
>   $INSERT INTO TABLE michtest.BroadcastStaging PARTITION (broadcastId =
> $broadcastValue, brand = "dummy")
>   SELECT
>           ocis_party_id AS partyId
>         , target_mobile_no AS phoneNumber
>   FROM tmp
> 
> 
> scala>   spark.sql($sqltext)
> <console>:41: error: not found: value $sqltext
>          spark.sql($sqltext)
> 
> 
> Any ideas?
> 
> 
> Thanks
> 
> 
> Dr Mich Talebzadeh
> 
> 
> 
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
> 
> 
> 
> http://talebzadehmich.wordpress.com
> 
> 
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message