spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From naresh Goud <nareshgoud.du...@gmail.com>
Subject Re: New Spark Datasource for Hive ACID tables
Date Sun, 28 Jul 2019 02:22:53 GMT
Thanks Abhishek.
I will check it out.

Thank you,
Naresh

On Sat, Jul 27, 2019 at 9:21 PM Abhishek Somani <abhisheksomani88@gmail.com>
wrote:

> Hey Naresh,
>
> There is a `shaded-dependecies` project inside the root directory. You
> need to go into that and build and publish that to local first.
>
> cd shaded-dependencies
>> sbt clean publishLocal
>>
>
> After that, come back out to the root directory and build that project.
> The spark-acid-shaded-dependencies jar will now be found:
>
>> cd ..
>> sbt assembly
>
>
> This will create the jar which you can use.
>
> On another note, unless you are making changes in the code, you don't need
> to build yourself as the jar is published in
> https://spark-packages.org/package/qubole/spark-acid. So you can just use
> it as:
>
> spark-shell --packages qubole:spark-acid:0.4.0-s_2.11
>
>
> ...and it will be automatically fetched and used.
>
> Thanks,
> Abhishek
>
>
> On Sun, Jul 28, 2019 at 4:42 AM naresh Goud <nareshgoud.dulam@gmail.com>
> wrote:
>
>> It looks there is some internal dependency missing.
>>
>> libraryDependencies ++= Seq(
>> "com.qubole" %% "spark-acid-shaded-dependencies" % "0.1"
>> )
>>
>> How do we get it?
>>
>>
>> Thank you,
>> Naresh
>>
>>
>>
>>
>> Thanks,
>> Naresh
>> www.linkedin.com/in/naresh-dulam
>> http://hadoopandspark.blogspot.com/
>>
>>
>>
>> On Sat, Jul 27, 2019 at 5:34 PM naresh Goud <nareshgoud.dulam@gmail.com>
>> wrote:
>>
>>> Hi Abhishek,
>>>
>>>
>>> We are not able to build jar using git hub code with below error?
>>>
>>> Any others able to build jars? Is there anything else missing?
>>>
>>>
>>>
>>> Note: Unresolved dependencies path:
>>> [warn]          com.qubole:spark-acid-shaded-dependencies_2.11:0.1
>>> (C:\Data\Hadoop\spark-acid-master\build.sbt#L51-54)
>>> [warn]            +- com.qubole:spark-acid_2.11:0.4.0
>>> sbt.ResolveException: unresolved dependency:
>>> com.qubole#spark-acid-shaded-dependencies_2.11;0.1: not found
>>>         at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:313)
>>>         at
>>> sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191)
>>>         at
>>> sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:168)
>>>         at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
>>>         at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
>>>         at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:133)
>>>         at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:57)
>>>         at sbt.IvySbt$$anon$4.call(Ivy.scala:65)
>>>         at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
>>>         at
>>> xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
>>>         at
>>> xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
>>>         at xsbt.boot.Using$.withResource(Using.scala:10)
>>>         at xsbt.boot.Using$.apply(Using.scala:9)
>>>         at
>>> xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
>>>         at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
>>>         at xsbt.boot.Locks$.apply0(Locks.scala:31)
>>>         at xsbt.boot.Locks$.apply(Locks.scala:28)
>>>         at sbt.IvySbt.withDefaultLogger(Ivy.scala:65)
>>>         at sbt.IvySbt.withIvy(Ivy.scala:128)
>>>         at sbt.IvySbt.withIvy(Ivy.scala:125)
>>>         at sbt.IvySbt$Module.withModule(Ivy.scala:156)
>>>         at sbt.IvyActions$.updateEither(IvyActions.scala:168)
>>>         at
>>> sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1541)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Thanks,
>>> Naresh
>>> www.linkedin.com/in/naresh-dulam
>>> http://hadoopandspark.blogspot.com/
>>>
>>>
>>>
>>> On Sat, Jul 27, 2019 at 3:25 PM Nicolas Paris <nicolas.paris@riseup.net>
>>> wrote:
>>>
>>>> Congrats
>>>>
>>>> The read/write feature with hive3 is highly interesting
>>>>
>>>> On Fri, Jul 26, 2019 at 06:07:55PM +0530, Abhishek Somani wrote:
>>>> > Hi All,
>>>> >
>>>> > We at Qubole have open sourced a datasource that will enable users to
>>>> work on
>>>> > their Hive ACID Transactional Tables using Spark.
>>>> >
>>>> > Github: https://github.com/qubole/spark-acid
>>>> >
>>>> > Hive ACID tables allow users to work on their data transactionally,
>>>> and also
>>>> > gives them the ability to Delete, Update and Merge data efficiently
>>>> without
>>>> > having to rewrite all of their data in a table, partition or file. We
>>>> believe
>>>> > that being able to work on these tables from Spark is a much desired
>>>> value add,
>>>> > as is also apparent in
>>>> https://issues.apache.org/jira/browse/SPARK-15348 and
>>>> > https://issues.apache.org/jira/browse/SPARK-16996 with multiple
>>>> people looking
>>>> > for it. Currently the datasource supports reading from these ACID
>>>> tables only,
>>>> > and we are working on adding the ability to write into these tables
>>>> via Spark
>>>> > as well.
>>>> >
>>>> > The datasource is also available as a spark package, and instructions
>>>> on how to
>>>> > use it are available on the Github page.
>>>> >
>>>> > We welcome your feedback and suggestions.
>>>> >
>>>> > Thanks,
>>>> > Abhishek Somani
>>>>
>>>> --
>>>> nicolas
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>
>>>> --
Thanks,
Naresh
www.linkedin.com/in/naresh-dulam
http://hadoopandspark.blogspot.com/

Mime
View raw message