metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto Fowler <ottobackwa...@gmail.com>
Subject Re: [HELP!]Writing to HDFS from Storm
Date Wed, 12 Apr 2017 17:28:27 GMT
Thanks Justin,
I’m working on rebasing right now, then I’ll be able to try it out.


On April 12, 2017 at 08:26:08, Justin Leet (justinjleet@gmail.com) wrote:

We changed the HDFS owner of /apps/metron/indexing/indexed a bit ago to be
metron:hadoop, specifically because Storm wasn't writing to HDFS (and had
perms issues). If you look into the indexing output directories, my
expectation is that you aren't getting any data out (and the Storm indexing
topology is throwing permissions errors).

PR for that change: https://github.com/apache/incubator-metron/pull/488

Your branch has the old code:
https://github.com/ottobackwards/incubator-metron/blob/parser_deploy/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py#L98
New code:
https://github.com/apache/incubator-metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py

To set up ownership on startup with the hadoop group after you pull in
master (or at least make the changes from that PR)

self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
type="directory",
action="create_on_execute",
owner=self.__params.metron_user,
group=self.__params.hadoop_group,
mode=0775,
)

Justin

On Wed, Apr 12, 2017 at 8:24 AM, Otto Fowler <ottobackwards@gmail.com>
wrote:

> "The parsers are packaged as ‘bundles’ ( our version of NIFI NARS ) and
are
> deployed
> if required. So the writes for HDFS are the unpacking the bundles ( with
> the unpacked jars being loaded into a classloader ).
>
> If the unbundled is the same as the bundle, there is no op. In this case,
> it is first run.
>
> So this is the parser bolt using the HDFS backed extension loader.”
>
>
> This is part of the parser side loading, also feeds into the stellar load
> from hdfs stuff.
>
> * metron has extensions
> * extensions are packaged with their configurations and their ‘bundle’ (
> the nar part )
> * extension bundles are deployed to hdfs *lib directory
> * parts of the system that use extensions discover and load them, with
> class loaders setup for the bundle
> * part of that, the bundles are unpackaged into a ‘working’ area if
> required ( so if you drop a new version into the lib, the next app will
> unpack the new version )
> * thus, the bundle system needs to be able to write to the working areas
in
> hdfs
> * ???
> * profit
>
>
> On April 12, 2017 at 08:11:28, Simon Elliston Ball (
> simon@simonellistonball.com) wrote:
>
> I’m curious: Otto, what’s your use case for writing to HDFS from a
parsers?
>
> Simon
>
> > On 12 Apr 2017, at 13:04, Justin Leet <justinjleet@gmail.com> wrote:
> >
> > Chown it to metron:hadoop and it'll work. Storm is in the hadoop group
> and
> > with 775 will be able to write.
> >
> > On Wed, Apr 12, 2017 at 7:56 AM, David Lyle <dlyle65535@gmail.com>
> wrote:
> >
> >> It's curious to me that you're writing directly from parsing, but I
> suspect
> >> that your parsing topology is running as the storm user and it can't
> write
> >> to those directories.
> >>
> >> -D...
> >>
> >> On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <ottobackwards@gmail.com>
> >> wrote:
> >>
> >>> The indexing dir is created:
> >>>
> >>>
self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
> >>> type="directory",
> >>> action="create_on_execute",
> >>> owner=self.__params.metron_user,
> >>> group=self.__params.metron_group,
> >>> mode=0775,
> >>> )
> >>>
> >>>
> >>>
> >>>
> >>> On April 12, 2017 at 07:49:16, Otto Fowler (ottobackwards@gmail.com)
> >>> wrote:
> >>>
> >>>
> >>> I am trying to write to HDFS from ParserBolt, but I’m getting the
> >> following
> >>> exception:
> >>>
> >>> Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied:
> >>> user=storm, access=WRITE,
> >>> inode="/apps/metron/extension_working/framework":metron:
> hdfs:drwxrwxr-x
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> >>> check(FSPermissionChecker.java:319)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> >>> check(FSPermissionChecker.java:292)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> >>> checkPermission(FSPermissionChecker.java:213)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.
> >>> checkPermission(FSPermissionChecker.java:190)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> >>> checkPermission(FSDirectory.java:1827)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> >>> checkPermission(FSDirectory.java:1811)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.
> checkAncestorAccess(
> >>> FSDirectory.java:1794)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(
> >>> FSDirMkdirOp.java:71)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(
> >>> FSNamesystem.java:4011)
> >>> at
> >>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
> >>> mkdirs(NameNodeRpcServer.java:1102)
> >>> at
> >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi
> >>> deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi
> >>> deTranslatorPB.java:630)
> >>> at
> >>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$
> >>> ClientNamenodeProtocol$2.callBlockingMethod(
> >> ClientNamenodeProtocolProtos.
> >>> java)
> >>> at
> >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
> ProtoBufRpcInvoker.call(
> >>> ProtobufRpcEngine.java:640)
> >>>
> >>>
> >>> The HDFS directory is created as such:
> >>>
> >>> self.__params.HdfsResource(self.__params.hdfs_metron_
> >>> apps_extensions_working,
> >>> type="directory",
> >>> action="create_on_execute",
> >>> owner=self.__params.metron_user,
> >>> mode=0775)
> >>>
> >>>
> >>> As the hdfs write handlers I am logging in as such:
> >>>
> >>> HdfsSecurityUtil.login(stormConfig, fsConf);
> >>> FileSystem fileSystem = FileSystem.get(fsConf);
> >>>
> >>> I am not sure what is different from the indexing hdfs writer setup
> here,
> >>> but what I’m doing obviously is not working.
> >>>
> >>> Any ideas?
> >>>
> >>>
> >>> - the branch:
> >>> https://github.com/ottobackwards/incubator-metron/tree/parser_deploy
> >>>
> >>> I am not up to date with master.
> >>>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message