spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhishek Somani <abhisheksoman...@gmail.com>
Subject Re: New Spark Datasource for Hive ACID tables
Date Sun, 28 Jul 2019 03:14:35 GMT
I realised that the build instructions in the README.md were not very clear
due to some recent changes. I have updated those now.

Thanks,
Abhishek Somani

On Sun, Jul 28, 2019 at 7:53 AM naresh Goud <nareshgoud.dulam@gmail.com>
wrote:

> Thanks Abhishek.
> I will check it out.
>
> Thank you,
> Naresh
>
> On Sat, Jul 27, 2019 at 9:21 PM Abhishek Somani <
> abhisheksomani88@gmail.com> wrote:
>
>> Hey Naresh,
>>
>> There is a `shaded-dependecies` project inside the root directory. You
>> need to go into that and build and publish that to local first.
>>
>> cd shaded-dependencies
>>> sbt clean publishLocal
>>>
>>
>> After that, come back out to the root directory and build that project.
>> The spark-acid-shaded-dependencies jar will now be found:
>>
>>> cd ..
>>> sbt assembly
>>
>>
>> This will create the jar which you can use.
>>
>> On another note, unless you are making changes in the code, you don't
>> need to build yourself as the jar is published in
>> https://spark-packages.org/package/qubole/spark-acid. So you can just
>> use it as:
>>
>> spark-shell --packages qubole:spark-acid:0.4.0-s_2.11
>>
>>
>> ...and it will be automatically fetched and used.
>>
>> Thanks,
>> Abhishek
>>
>>
>> On Sun, Jul 28, 2019 at 4:42 AM naresh Goud <nareshgoud.dulam@gmail.com>
>> wrote:
>>
>>> It looks there is some internal dependency missing.
>>>
>>> libraryDependencies ++= Seq(
>>> "com.qubole" %% "spark-acid-shaded-dependencies" % "0.1"
>>> )
>>>
>>> How do we get it?
>>>
>>>
>>> Thank you,
>>> Naresh
>>>
>>>
>>>
>>>
>>> Thanks,
>>> Naresh
>>> www.linkedin.com/in/naresh-dulam
>>> http://hadoopandspark.blogspot.com/
>>>
>>>
>>>
>>> On Sat, Jul 27, 2019 at 5:34 PM naresh Goud <nareshgoud.dulam@gmail.com>
>>> wrote:
>>>
>>>> Hi Abhishek,
>>>>
>>>>
>>>> We are not able to build jar using git hub code with below error?
>>>>
>>>> Any others able to build jars? Is there anything else missing?
>>>>
>>>>
>>>>
>>>> Note: Unresolved dependencies path:
>>>> [warn]          com.qubole:spark-acid-shaded-dependencies_2.11:0.1
>>>> (C:\Data\Hadoop\spark-acid-master\build.sbt#L51-54)
>>>> [warn]            +- com.qubole:spark-acid_2.11:0.4.0
>>>> sbt.ResolveException: unresolved dependency:
>>>> com.qubole#spark-acid-shaded-dependencies_2.11;0.1: not found
>>>>         at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:313)
>>>>         at
>>>> sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191)
>>>>         at
>>>> sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:168)
>>>>         at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
>>>>         at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
>>>>         at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:133)
>>>>         at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:57)
>>>>         at sbt.IvySbt$$anon$4.call(Ivy.scala:65)
>>>>         at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
>>>>         at
>>>> xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
>>>>         at
>>>> xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
>>>>         at xsbt.boot.Using$.withResource(Using.scala:10)
>>>>         at xsbt.boot.Using$.apply(Using.scala:9)
>>>>         at
>>>> xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
>>>>         at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
>>>>         at xsbt.boot.Locks$.apply0(Locks.scala:31)
>>>>         at xsbt.boot.Locks$.apply(Locks.scala:28)
>>>>         at sbt.IvySbt.withDefaultLogger(Ivy.scala:65)
>>>>         at sbt.IvySbt.withIvy(Ivy.scala:128)
>>>>         at sbt.IvySbt.withIvy(Ivy.scala:125)
>>>>         at sbt.IvySbt$Module.withModule(Ivy.scala:156)
>>>>         at sbt.IvyActions$.updateEither(IvyActions.scala:168)
>>>>         at
>>>> sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1541)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Naresh
>>>> www.linkedin.com/in/naresh-dulam
>>>> http://hadoopandspark.blogspot.com/
>>>>
>>>>
>>>>
>>>> On Sat, Jul 27, 2019 at 3:25 PM Nicolas Paris <nicolas.paris@riseup.net>
>>>> wrote:
>>>>
>>>>> Congrats
>>>>>
>>>>> The read/write feature with hive3 is highly interesting
>>>>>
>>>>> On Fri, Jul 26, 2019 at 06:07:55PM +0530, Abhishek Somani wrote:
>>>>> > Hi All,
>>>>> >
>>>>> > We at Qubole have open sourced a datasource that will enable users
>>>>> to work on
>>>>> > their Hive ACID Transactional Tables using Spark.
>>>>> >
>>>>> > Github: https://github.com/qubole/spark-acid
>>>>> >
>>>>> > Hive ACID tables allow users to work on their data transactionally,
>>>>> and also
>>>>> > gives them the ability to Delete, Update and Merge data efficiently
>>>>> without
>>>>> > having to rewrite all of their data in a table, partition or file.
>>>>> We believe
>>>>> > that being able to work on these tables from Spark is a much desired
>>>>> value add,
>>>>> > as is also apparent in
>>>>> https://issues.apache.org/jira/browse/SPARK-15348 and
>>>>> > https://issues.apache.org/jira/browse/SPARK-16996 with multiple
>>>>> people looking
>>>>> > for it. Currently the datasource supports reading from these ACID
>>>>> tables only,
>>>>> > and we are working on adding the ability to write into these tables
>>>>> via Spark
>>>>> > as well.
>>>>> >
>>>>> > The datasource is also available as a spark package, and
>>>>> instructions on how to
>>>>> > use it are available on the Github page.
>>>>> >
>>>>> > We welcome your feedback and suggestions.
>>>>> >
>>>>> > Thanks,
>>>>> > Abhishek Somani
>>>>>
>>>>> --
>>>>> nicolas
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>>
>>>>> --
> Thanks,
> Naresh
> www.linkedin.com/in/naresh-dulam
> http://hadoopandspark.blogspot.com/
>
>

Mime
View raw message