spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arush Kharbanda <ar...@sigmoidanalytics.com>
Subject Re: Union in Spark
Date Mon, 02 Feb 2015 07:48:44 GMT
Hi Deep,

What is your configuration and what is the size of the 2 data sets?

Thanks
Arush

On Mon, Feb 2, 2015 at 11:56 AM, Deep Pradhan <pradhandeep1991@gmail.com>
wrote:

> I did not check the console because once the job starts I cannot run
> anything else and have to force shutdown the system. I commented parts of
> codes and I tested. I doubt it is because of union. So, I want to change it
> to something else and see if the problem persists.
>
> Thank you
>
> On Mon, Feb 2, 2015 at 11:53 AM, Jerry Lam <chilinglam@gmail.com> wrote:
>
>> Hi Deep,
>>
>> How do you know the cluster is not responsive because of "Union"?
>> Did you check the spark web console?
>>
>> Best Regards,
>>
>> Jerry
>>
>>
>> On Mon, Feb 2, 2015 at 1:21 AM, Deep Pradhan <pradhandeep1991@gmail.com>
>> wrote:
>>
>>> The cluster hangs.
>>>
>>> On Mon, Feb 2, 2015 at 11:25 AM, Jerry Lam <chilinglam@gmail.com> wrote:
>>>
>>>> Hi Deep,
>>>>
>>>> what do you mean by stuck?
>>>>
>>>> Jerry
>>>>
>>>>
>>>> On Mon, Feb 2, 2015 at 12:44 AM, Deep Pradhan <
>>>> pradhandeep1991@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> Is there any better operation than Union. I am using union and the
>>>>> cluster is getting stuck with a large data set.
>>>>>
>>>>> Thank you
>>>>>
>>>>
>>>>
>>>
>>
>


-- 

[image: Sigmoid Analytics] <http://htmlsig.com/www.sigmoidanalytics.com>

*Arush Kharbanda* || Technical Teamlead

arush@sigmoidanalytics.com || www.sigmoidanalytics.com

Mime
View raw message