flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: Question about jobmanager.web.upload.dir
Date Wed, 17 May 2017 14:49:57 GMT
I don't know why we delete it either, but my guess is that at one point 
this was a temporary directory that we properly cleaned up, and later 
allowed it to be configurable.

Currently, this directory must be in the local file system. We could 
change it to also allow non-local paths, which should be fairly easy to 
do. Add a condition that the directory is only cleaned up when it wasn't 
explicitly configured and we should've covered everything.

In regards to the HistoryServer, i mean of course could we include the 
user-jar, but I'm wondering about the user-cases for it.
It's not like you could "just download it and give it to a JobManager"; 
you would be missing any additional libraries put under /lib, as well as 
the original configuration of the cluster. It would also require extra 
steps (and tools) to identify which dependencies are required.

And there are downsides to this, like a significant increase in the size 
of the job archives (along with a duplication of jars for every job 
submission), and the archiving also becomes more complex.

On 17.05.2017 15:52, Timo Walther wrote:
> Hey Mauro,
> I'm not aware of any reason for that. I loop in Chesnay, maybe he 
> knows why. @Chesnay wouldn't it be helpful to also archive the jars 
> using the HistoryServer?
> Timo
> Am 17.05.17 um 12:31 schrieb Mauro Cortellazzi:
>> Hi Flink comunity,
>> is there a particular reason to delete the jobmanager.web.upload.dir 
>> content when the JobManager restart? Could it be interesting have the 
>> jar previously uploaded when JM restart? Could the jars be saved to 
>> HDFS for HA mode?
>> Thank you for your support.
>> Mauro

View raw message