spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: Pros and Cons
Date Wed, 25 May 2016 16:52:15 GMT

Hive has a little bit more emphasis on the case that your data that is queried is much bigger
than available memory or when you need to query many different small data subsets or recently
interactively queries (llap  etc.). 

Spark is more for machine learning working iteravely over the whole same dataset in memory.
Additionally it has streaming and graph processing capabilities that can be used together.


Besides this depending on your needs other ecosystem components are relevant. For instance,
both are less good with lookups of single entries in a dataset. They are not so good for text
analytics.

Said that both develop rapidly and this may change. Additionally both have replacements ,
such as Flink for Spark etc



Sent from my iPhone
> On 25 May 2016, at 18:11, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:
> 
> Can you be a bit more specific how are you going to use Spark. For example as a powerful
query tool, Analytics, Data migration.
> 
> Spark SQL and Spark-shell provide a subset of Hive SQL (depending on which version of
Hive and Spark you have in mind).
> 
> As a query tool Spark is very powerful as it uses DAG and In-memory computation, provides
Scala (and others) as the language. You can create your own uber JAR fie for distribution
etc
> You can of course use Spark as an execution engine for Hive as opposed to map-reduce
to take advantage of Spark processing
> 
> etc etc
> 
> HTH
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> http://talebzadehmich.wordpress.com
>  
> 
>> On 25 May 2016 at 16:34, Aakash Basu <raj2cool16@gmail.com> wrote:
>> Hi,
>> 
>>  
>> 
>> I’m new to the Spark Ecosystem, need to understand the Pros and Cons of fetching
data using SparkSQL vs Hive in Spark vs Spark API.
>> 
>>  
>> 
>> PLEASE HELP!
>> 
>>  
>> 
>> Thanks,
>> 
>> Aakash Basu.
>> 
> 

Mime
View raw message