whirr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Alves <davidral...@gmail.com>
Subject Re: launching a one job cluster
Date Thu, 22 Sep 2011 22:24:33 GMT
As I said the thing is I'm NOT using hadoop :)
I'm just running generic scripts/ssh commands.

-david

On Sep 22, 2011, at 5:20 PM, Andrei Savu wrote:

> I don't know that much about how to manage jobs in Hadoop using the
> API. Maybe Tom can provide a good answer to this. I completely
> understand the elegance part :)
> 
> -- Andrei Savu
> 
> On Thu, Sep 22, 2011 at 3:17 PM, David Alves <davidralves@gmail.com> wrote:
>> First there is the question of accuracy, as I said I am collecting metrics that I'd
like to be as accurate as possible.
>> Second there is the matter of elegance. I always like to avoid polls whenever possible.
>> 
>> That being said, I don't wan't to embark in some odyssey just to avoid poll, so if
it really is too much trouble I am ok with letting it go.
>> Anyhow even with poll is there something already implemented that enables it in generic
cases?
>> 
>> thanks
>> -david
>> 
>> On Sep 22, 2011, at 5:09 PM, Andrei Savu wrote:
>> 
>>> Why is so important to avoid having a poll? The cost is low and almost
>>> any job is running at least for a few minutes.
>>> 
>>> -- Andrei
>>> 
>>> On Thu, Sep 22, 2011 at 3:07 PM, David Alves <davidralves@gmail.com> wrote:
>>>> Hi Andrei
>>>> 
>>>>        I know…
>>>>        The thing is that code used the Hadoop JobClient class's runJob()
method that actually polls for progress.
>>>>        I am not using hadoop (in hindsight using the word "job" might have
been a mistake) and I was wondering if there is already a way to do that for generic cases
(e.g., scripts or java programs).
>>>>        In particular as I'm collecting accurate metrics I'd like a non poll
based technique.
>>>>        Even if there is none I can always try and code it, so all ideas are
welcome.
>>>> 
>>>> thanks
>>>> david
>>>> 
>>>> 
>>>> On Sep 22, 2011, at 4:52 PM, Andrei Savu wrote:
>>>> 
>>>>> This is exactly what the example code is doing (and the hadoop
>>>>> integration test). The job running code is blocking while the job is
>>>>> executing.
>>>>> 
>>>>> -- Andrei Savu / andreisavu.ro
>>>>> 
>>>>> On Thu, Sep 22, 2011 at 2:03 PM, David Alves <davidralves@gmail.com>
wrote:
>>>>>> Hi All
>>>>>> 
>>>>>>        I need to launch a cluster run a job and terminate the cluster
as the job is finished (as soon as possible).
>>>>>>        Is there any "nice" way to do this, or do you have any suggestions?
>>>>>>        On the top of my head I can imagine some quick and dirty solutions
(like creating a file whenever the task is completed and polling for its existence from the
whirr handler) but I'd like to do it without polling if possible. Any ideas?
>>>>>> 
>>>>>> thanks
>>>>>> -david
>>>>>> 
>>>> 
>>>> 
>> 
>> 


Mime
View raw message