spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From naresh Goud <nareshgoud.du...@gmail.com>
Subject Re: New Spark Datasource for Hive ACID tables
Date Sat, 27 Jul 2019 23:11:35 GMT
It looks there is some internal dependency missing.

libraryDependencies ++= Seq(
"com.qubole" %% "spark-acid-shaded-dependencies" % "0.1"
)

How do we get it?


Thank you,
Naresh




Thanks,
Naresh
www.linkedin.com/in/naresh-dulam
http://hadoopandspark.blogspot.com/



On Sat, Jul 27, 2019 at 5:34 PM naresh Goud <nareshgoud.dulam@gmail.com>
wrote:

> Hi Abhishek,
>
>
> We are not able to build jar using git hub code with below error?
>
> Any others able to build jars? Is there anything else missing?
>
>
>
> Note: Unresolved dependencies path:
> [warn]          com.qubole:spark-acid-shaded-dependencies_2.11:0.1
> (C:\Data\Hadoop\spark-acid-master\build.sbt#L51-54)
> [warn]            +- com.qubole:spark-acid_2.11:0.4.0
> sbt.ResolveException: unresolved dependency:
> com.qubole#spark-acid-shaded-dependencies_2.11;0.1: not found
>         at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:313)
>         at
> sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191)
>         at
> sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:168)
>         at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
>         at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
>         at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:133)
>         at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:57)
>         at sbt.IvySbt$$anon$4.call(Ivy.scala:65)
>         at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
>         at
> xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
>         at
> xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
>         at xsbt.boot.Using$.withResource(Using.scala:10)
>         at xsbt.boot.Using$.apply(Using.scala:9)
>         at
> xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
>         at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
>         at xsbt.boot.Locks$.apply0(Locks.scala:31)
>         at xsbt.boot.Locks$.apply(Locks.scala:28)
>         at sbt.IvySbt.withDefaultLogger(Ivy.scala:65)
>         at sbt.IvySbt.withIvy(Ivy.scala:128)
>         at sbt.IvySbt.withIvy(Ivy.scala:125)
>         at sbt.IvySbt$Module.withModule(Ivy.scala:156)
>         at sbt.IvyActions$.updateEither(IvyActions.scala:168)
>         at
> sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1541)
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Thanks,
> Naresh
> www.linkedin.com/in/naresh-dulam
> http://hadoopandspark.blogspot.com/
>
>
>
> On Sat, Jul 27, 2019 at 3:25 PM Nicolas Paris <nicolas.paris@riseup.net>
> wrote:
>
>> Congrats
>>
>> The read/write feature with hive3 is highly interesting
>>
>> On Fri, Jul 26, 2019 at 06:07:55PM +0530, Abhishek Somani wrote:
>> > Hi All,
>> >
>> > We at Qubole have open sourced a datasource that will enable users to
>> work on
>> > their Hive ACID Transactional Tables using Spark.
>> >
>> > Github: https://github.com/qubole/spark-acid
>> >
>> > Hive ACID tables allow users to work on their data transactionally, and
>> also
>> > gives them the ability to Delete, Update and Merge data efficiently
>> without
>> > having to rewrite all of their data in a table, partition or file. We
>> believe
>> > that being able to work on these tables from Spark is a much desired
>> value add,
>> > as is also apparent in
>> https://issues.apache.org/jira/browse/SPARK-15348 and
>> > https://issues.apache.org/jira/browse/SPARK-16996 with multiple people
>> looking
>> > for it. Currently the datasource supports reading from these ACID
>> tables only,
>> > and we are working on adding the ability to write into these tables via
>> Spark
>> > as well.
>> >
>> > The datasource is also available as a spark package, and instructions
>> on how to
>> > use it are available on the Github page.
>> >
>> > We welcome your feedback and suggestions.
>> >
>> > Thanks,
>> > Abhishek Somani
>>
>> --
>> nicolas
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>>

Mime
View raw message