whirr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Alves <davidral...@gmail.com>
Subject Re: launching a one job cluster
Date Thu, 22 Sep 2011 22:17:58 GMT
First there is the question of accuracy, as I said I am collecting metrics that I'd like to
be as accurate as possible.
Second there is the matter of elegance. I always like to avoid polls whenever possible.

That being said, I don't wan't to embark in some odyssey just to avoid poll, so if it really
is too much trouble I am ok with letting it go.
Anyhow even with poll is there something already implemented that enables it in generic cases?

thanks
-david

On Sep 22, 2011, at 5:09 PM, Andrei Savu wrote:

> Why is so important to avoid having a poll? The cost is low and almost
> any job is running at least for a few minutes.
> 
> -- Andrei
> 
> On Thu, Sep 22, 2011 at 3:07 PM, David Alves <davidralves@gmail.com> wrote:
>> Hi Andrei
>> 
>>        I know…
>>        The thing is that code used the Hadoop JobClient class's runJob() method that
actually polls for progress.
>>        I am not using hadoop (in hindsight using the word "job" might have been a
mistake) and I was wondering if there is already a way to do that for generic cases (e.g.,
scripts or java programs).
>>        In particular as I'm collecting accurate metrics I'd like a non poll based
technique.
>>        Even if there is none I can always try and code it, so all ideas are welcome.
>> 
>> thanks
>> david
>> 
>> 
>> On Sep 22, 2011, at 4:52 PM, Andrei Savu wrote:
>> 
>>> This is exactly what the example code is doing (and the hadoop
>>> integration test). The job running code is blocking while the job is
>>> executing.
>>> 
>>> -- Andrei Savu / andreisavu.ro
>>> 
>>> On Thu, Sep 22, 2011 at 2:03 PM, David Alves <davidralves@gmail.com> wrote:
>>>> Hi All
>>>> 
>>>>        I need to launch a cluster run a job and terminate the cluster as
the job is finished (as soon as possible).
>>>>        Is there any "nice" way to do this, or do you have any suggestions?
>>>>        On the top of my head I can imagine some quick and dirty solutions
(like creating a file whenever the task is completed and polling for its existence from the
whirr handler) but I'd like to do it without polling if possible. Any ideas?
>>>> 
>>>> thanks
>>>> -david
>>>> 
>> 
>> 


Mime
View raw message