metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto Fowler <ottobackwa...@gmail.com>
Subject Re: Question about SourceHandler and HdfsWriter
Date Mon, 04 Sep 2017 16:19:27 GMT
!ret.isUsable

On September 4, 2017 at 11:23:09, Otto Fowler (ottobackwards@gmail.com)
wrote:

> It looks like if the SourceHandler has a problem with it’s output stream,
> it will never recover.
> The handler will be in the map and continue to be used, but it will
> continue to throw exceptions.
>
> Is there a reason why we don’t try to recover and recreate the
> SourceHandler, such as:
>
> synchronized SourceHandler getSourceHandler(String sourceType, String stellarResult,
WriterConfiguration config) throws IOException {
>   SourceHandlerKey key = new SourceHandlerKey(sourceType, stellarResult);
>   SourceHandler ret = sourceHandlerMap.get(key);
>
>   // BEGIN FAKE CODE PART
>   if(ret != null && !ret.isUsableOrIsOpenOrSomething()) {
>     try {
>       ret.close();
>     } catch(Exception e){
>       // NO LOGGER IN THIS CLASS
>     }
>     sourceHandlerMap.remove(key);
>     ret = null;
>   }
>   // END FAKE CODE PART
>
>   if(ret == null) {
>     if(sourceHandlerMap.size() >= maxOpenFiles) {
>       throw new IllegalStateException("Too many HDFS files open!");
>     }
>     ret = new SourceHandler(rotationActions,
>                             rotationPolicy,
>                             syncPolicyCreator.create(sourceType, config),
>                             new PathExtensionFileNameFormat(key.getStellarResult(), fileNameFormat),
>                             new SourceHandlerCallback(sourceHandlerMap, key));
>     sourceHandlerMap.put(key, ret);
>   }
>   return ret;
> }
>
>
>
> It seems strange for something that is supposed to be a long long running
> process, writing to a distributed network store.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message