metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto Fowler <ottobackwa...@gmail.com>
Subject Question about SourceHandler and HdfsWriter
Date Mon, 04 Sep 2017 15:23:09 GMT
It looks like if the SourceHandler has a problem with it’s output stream,
it will never recover.
The handler will be in the map and continue to be used, but it will
continue to throw exceptions.

Is there a reason why we don’t try to recover and recreate the SourceHandler,
such as:

synchronized SourceHandler getSourceHandler(String sourceType, String
stellarResult, WriterConfiguration config) throws IOException {
  SourceHandlerKey key = new SourceHandlerKey(sourceType, stellarResult);
  SourceHandler ret = sourceHandlerMap.get(key);

  // BEGIN FAKE CODE PART
  if(ret != null && ret.isUsableOrIsOpenOrSomething()) {
    try {
      ret.close();
    } catch(Exception e){
      // NO LOGGER IN THIS CLASS
    }
    sourceHandlerMap.remove(key);
    ret = null;
  }
  // END FAKE CODE PART

  if(ret == null) {
    if(sourceHandlerMap.size() >= maxOpenFiles) {
      throw new IllegalStateException("Too many HDFS files open!");
    }
    ret = new SourceHandler(rotationActions,
                            rotationPolicy,
                            syncPolicyCreator.create(sourceType, config),
                            new
PathExtensionFileNameFormat(key.getStellarResult(), fileNameFormat),
                            new SourceHandlerCallback(sourceHandlerMap, key));
    sourceHandlerMap.put(key, ret);
  }
  return ret;
}



It seems strange for something that is supposed to be a long long running
process, writing to a distributed network store.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message