metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Elliston Ball <si...@simonellistonball.com>
Subject Re: [HELP!]Writing to HDFS from Storm
Date Wed, 12 Apr 2017 12:30:25 GMT
Makes sense, however, would it make sense to unpack to local file system, which would then
avoid further HDFS ops?


> On 12 Apr 2017, at 13:24, Otto Fowler <ottobackwards@gmail.com> wrote:
> 
> "The parsers are packaged as ‘bundles’ ( our version of NIFI NARS ) and are deployed
> if required.  So the writes for HDFS are the unpacking the bundles ( with the unpacked
jars being loaded into a classloader ).
> 
> If the unbundled is the same as the bundle, there is no op.  In this case, it is first
run.
> 
> So this is the parser bolt using the HDFS backed extension loader.”
> 
> 
> This is part of the parser side loading, also feeds into the stellar load from hdfs stuff.
> 
> * metron has extensions
> * extensions are packaged with their configurations and their ‘bundle’ ( the nar
part )
> * extension bundles are deployed to hdfs *lib directory
> * parts of the system that use extensions discover and load them, with class loaders
setup for the bundle
> * part of that, the bundles are unpackaged into a ‘working’ area if required ( so
if you drop a new version into the lib, the next app will unpack the new version )
> * thus, the bundle system needs to be able to write to the working areas in hdfs
> * ???
> * profit
> 
> 
> On April 12, 2017 at 08:11:28, Simon Elliston Ball (simon@simonellistonball.com <mailto:simon@simonellistonball.com>)
wrote:
> 
>> I’m curious: Otto, what’s your use case for writing to HDFS from a parsers? 
>> 
>> Simon 
>> 
>> > On 12 Apr 2017, at 13:04, Justin Leet <justinjleet@gmail.com <mailto:justinjleet@gmail.com>>
wrote: 
>> >  
>> > Chown it to metron:hadoop and it'll work. Storm is in the hadoop group and 
>> > with 775 will be able to write. 
>> >  
>> > On Wed, Apr 12, 2017 at 7:56 AM, David Lyle <dlyle65535@gmail.com <mailto:dlyle65535@gmail.com>>
wrote: 
>> >  
>> >> It's curious to me that you're writing directly from parsing, but I suspect

>> >> that your parsing topology is running as the storm user and it can't write

>> >> to those directories. 
>> >>  
>> >> -D... 
>> >>  
>> >> On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <ottobackwards@gmail.com
<mailto:ottobackwards@gmail.com>> 
>> >> wrote: 
>> >>  
>> >>> The indexing dir is created: 
>> >>>  
>> >>> self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,

>> >>> type="directory", 
>> >>> action="create_on_execute", 
>> >>> owner=self.__params.metron_user, 
>> >>> group=self.__params.metron_group, 
>> >>> mode=0775, 
>> >>> ) 
>> >>>  
>> >>>  
>> >>>  
>> >>>  
>> >>> On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwards@gmail.com
<mailto:ottobackwards@gmail.com>) 
>> >>> wrote: 
>> >>>  
>> >>>  
>> >>> I am trying to write to HDFS from ParserBolt, but I’m getting the

>> >> following 
>> >>> exception: 
>> >>>  
>> >>> Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:

>> >>> user=storm, access=WRITE, 
>> >>> inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x

>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. 
>> >>> check(FSPermissionChecker.java:319) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. 
>> >>> check(FSPermissionChecker.java:292) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. 
>> >>> checkPermission(FSPermissionChecker.java:213) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. 
>> >>> checkPermission(FSPermissionChecker.java:190) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory. 
>> >>> checkPermission(FSDirectory.java:1827) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory. 
>> >>> checkPermission(FSDirectory.java:1811) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(

>> >>> FSDirectory.java:1794) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs( 
>> >>> FSDirMkdirOp.java:71) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs( 
>> >>> FSNamesystem.java:4011) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer. 
>> >>> mkdirs(NameNodeRpcServer.java:1102) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi 
>> >>> deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi 
>> >>> deTranslatorPB.java:630) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$

>> >>> ClientNamenodeProtocol$2.callBlockingMethod( 
>> >> ClientNamenodeProtocolProtos. 
>> >>> java) 
>> >>> at 
>> >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(

>> >>> ProtobufRpcEngine.java:640) 
>> >>>  
>> >>>  
>> >>> The HDFS directory is created as such: 
>> >>>  
>> >>> self.__params.HdfsResource(self.__params.hdfs_metron_ 
>> >>> apps_extensions_working, 
>> >>> type="directory", 
>> >>> action="create_on_execute", 
>> >>> owner=self.__params.metron_user, 
>> >>> mode=0775) 
>> >>>  
>> >>>  
>> >>> As the hdfs write handlers I am logging in as such: 
>> >>>  
>> >>> HdfsSecurityUtil.login(stormConfig, fsConf); 
>> >>> FileSystem fileSystem = FileSystem.get(fsConf); 
>> >>>  
>> >>> I am not sure what is different from the indexing hdfs writer setup
here, 
>> >>> but what I’m doing obviously is not working. 
>> >>>  
>> >>> Any ideas? 
>> >>>  
>> >>>  
>> >>> - the branch: 
>> >>> https://github.com/ottobackwards/incubator-metron/tree/parser_deploy
<https://github.com/ottobackwards/incubator-metron/tree/parser_deploy> 
>> >>>  
>> >>> I am not up to date with master. 
>> >>>  
>> >> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message