metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dima Kovalyov <Dima.Koval...@sstech.us>
Subject Re: Confluence write access to a space
Date Tue, 03 Jan 2017 07:01:36 GMT
Thank you Matt,

I haven't seen 634 before. Should I merge my tickets as sub-tasks
addressing some of the points your bring up there?

Also, I have a note about elasticsearch_config_path which is set in
metron-deployment/roles/metron_streaming/defaults/main.yml. It seems
like it is not used anywhere in the code base.

Please let me know how I should proceed with two tickets I have created
that are relevant to yours.

- Dima

On 01/01/2017 04:43 AM, Matt Foley wrote:
> Hi Dima,
> Great to have the how-to doc in the wiki where it belongs.  Now we have a doc to edit
as we improve the install process :-)
>
> Did you look at https://issues.apache.org/jira/browse/METRON-634 before opening METRON-642?
 Please see my comments in the Jira for METRON-642.
> Thanks,
> --Matt
>
>
>
> On 12/30/16, 11:44 PM, "Dima Kovalyov" <Dima.Kovalyov@sstech.us> wrote:
>
>     Hey,
>     
>     I wanted to finish what I've started with document for Metron with HDP
>     2.5, so I have migrated document (with minor text fixes and
>     clarifications) to here:
>     https://cwiki.apache.org/confluence/display/METRON/Metron+with+HDP+2.5+bare-metal+install
>     Old google doc was replaced with the link to this article.
>     
>     I also, created number of pull requests to fix minor bugs here and there
>     and created these two tickets: METRON-641 and METRON-642.
>     Please let me know if I did something out of proper procedure.
>     
>     Also, I agree that we should eventually strip HDP related steps from the
>     document, so in the end it will be like:
>     1. Build Mpack
>     2. Add to Ambari
>     3. Assigned Masters and Slave
>     4. PROFIT
>     But since we are where we are, let's leave it like that and fix all the
>     bugs first.
>     
>     p.s. have a happy holidays everyone
>     
>     - Dima
>     
>     On 12/16/2016 04:21 AM, Matt Foley wrote:
>     > I seem to have found the difficulty.  It will NOT show up on any system that
has /bin/java defined, which may account for why other folks with Centos7 test systems aren’t
seeing the behavior.
>     >
>     > On my Centos7 test system, it so happens that /bin/java is not defined, even
though $JAVA_HOME is correctly defined, and “$JAVA_HOME/bin” is in the PATH.  In Centos7,
when services launch through the (new in 7) systemctl process, it drops all inherited environment
variables and starts over fresh.  Although the systemd launch script /usr/lib/systemd/system/elasticsearch.service
does read in the /etc/sysconfig/elasticsearch as an “EnvironmentFile”, it does not include
JAVA_HOME.
>     >
>     > When, eventually, the user-level launcher script at /usr/share/elasticsearch/bin/elasticsearch
gets invoked, JAVA_HOME is still undefined.  But it looks for $JAVA_HOME/bin/java, so if “/bin/java”
is linked in the file system, then it’s good!  But if not, the launcher script dies.  Regrettably
that launcher script, even though it is fairly complex, does not write to any log file, and
its stdout was closed long ago by the service-level launcher.  So I had to hack it to see
what it was doing.
>     >
>     > The solution is to simply write JAVA_HOME={{java64_home}} into the elastic-sysconfig
template.
>     >
>     > BTW, while munging thru code I reached the conclusion that elastic-env.sh is
basically orphaned.  Does anyone know of scripts that source it? (Of course elastic-env.xml
is still important, I’m only asking about the elastic-env.sh file templated from it.)
>     >
>     > Thanks,
>     > --Matt
>     >
>     >
>     > On 12/14/16, 2:41 PM, "Matt Foley" <mfoley@hortonworks.com> wrote:
>     >
>     > No, node.data and node.master are both correctly set to true (with Ambari’s
agreement/participation) in the elasticsearch.yml file in CONF_DIR, and this is being correctly
picked up by ES when launched interactively.  I really think this is in the service management
stuff in /etc/init.d/elasticsearch and /usr/share/elasticsearch/bin/elasticsearch .  Remains
to be proven, of course…
>     >
>     > The reason I think ES isn’t even being successfully launched by systemd, is
there is zero logging anywhere, except in ambari where it shows nothing but a successful service
launch.  No files created in /var/log/elasticsearch, which all scripts agree is the value
of LOG_DIR, despite permissions set to “drwxr-xr-x. elasticsearch elasticsearch”
>     >
>     > Thanks,
>     > --Matt
>     >
>     > On 12/14/16, 2:07 PM, "David Lyle" <dlyle65535@gmail.com> wrote:
>     >
>     >     Aha! There's your problem. :)
>     >     
>     >     Kidding aside, that is weird. I would expect the ES instance to come up
and
>     >     go status red right away, not up and die.
>     >     
>     >     I did have a horrible, horrible hack that made all that work, it involved
>     >     modifying the stored es templates to both have node.master and node.data
>     >     set to true in
>     >     /var/lib/ambari-server/resources/common-services/ELASTICSEARCH (from
>     >     memory, path may be a bit off). It occurred to me that an easy incremental
>     >     step toward to full METRON-608 would be to simply expose the templates to
>     >     the config pages...
>     >     
>     >     -D...
>     >     
>     >     
>     >     On Wed, Dec 14, 2016 at 4:59 PM, Matt Foley <mfoley@hortonworks.com>
wrote:
>     >     
>     >     > David,
>     >     > No, it’s in my METRON-608 single-node deployment  :-)
>     >     >
>     >     > On 12/14/16, 12:46 PM, "David Lyle" <dlyle65535@gmail.com> wrote:
>     >     >
>     >     >     Hi Matt and Jon,
>     >     >
>     >     >     FWIW, Metron with the MPack has been tested extensively on CentOS
7.
>     >     > Works
>     >     >     like a champ. The issue right now is that sensor install is a CentOS
>     >     >     6/Ansible proposition.
>     >     >
>     >     >     I haven't seen the issue with ES that you're experiencing, Matt.
Is
>     >     > that in
>     >     >     a 1 Master 3 Data node ES config or something else?
>     >     >
>     >     >     -D...
>     >     >
>     >     >
>     >     >     On Wed, Dec 14, 2016 at 3:29 PM, Matt Foley <mfoley@hortonworks.com>
>     >     > wrote:
>     >     >
>     >     >     > I hope we will try to support Centos7. Many of my company’s
>     >     > customers are
>     >     >     > requiring new installs to use RHEL 7 or Centos 7 rather than
6.  In
>     >     >     > addition, we have the benefit that Centos 6 RPMs basically
always
>     >     > run fine
>     >     >     > in Centos7 (barring heavy-duty low-level system manipulations
that
>     >     > Metron,
>     >     >     > Kibana, and ES just don’t use), ES doesn’t distinguish
between them,
>     >     > and
>     >     >     > Kibana install support for Centos7 is already in place in
the
>     >     > Mpack.  And
>     >     >     > of course the Hadoop Stack runs fine in 7.
>     >     >     >
>     >     >     > For what it’s worth, I’ve been working with the Mpack
installation on
>     >     >     > Centos7 for the last few weeks, and it works fine except for
a
>     >     > persistent
>     >     >     > issue where the Elasticsearch service works fine if launched
>     >     > interactively
>     >     >     > but terminates immediately if launched with same arguments
as a
>     >     > service.
>     >     >     > Hope to find the cause in the next couple days.  (If anyone
knows
>     >     > why,
>     >     >     > would love to hear.  The daemonized launch scripts come from
ES, not
>     >     > from
>     >     >     > Metron, so “should just work”.)
>     >     >     >
>     >     >     > Thanks,
>     >     >     > --Matt
>     >     >     >
>     >     >     > On 12/13/16, 6:46 PM, "Zeolla@GMail.com" <zeolla@gmail.com>
wrote:
>     >     >     >
>     >     >     >     For now, I'm not sure what the solution is, but I would
think
>     >     > choosing
>     >     >     > one
>     >     >     >     specific list of required/supported software (including
OS) and
>     >     >     > documenting
>     >     >     >     that thoroughly is the right start.  Something like what
>     >     > currently
>     >     >     > exists
>     >     >     >     for the vagrant side of things.
>     >     >     >
>     >     >     >     That said, long term I would love to see broader support
and
>     >     >     > automatically
>     >     >     >     generating documentation derived from the code.  Isn't
CentOS 6
>     >     > still
>     >     >     >     preferred over 7?  That's what I've been working on solely
for
>     >     > that
>     >     >     >     reason.
>     >     >     >
>     >     >     >     Jon
>     >     >     >
>     >     >     >     On Tue, Dec 13, 2016, 21:07 Matt Foley <mattf@apache.org>
wrote:
>     >     >     >
>     >     >     >     > In my work on METRON-608, I’ve found a lot of small
but
>     >     > significant
>     >     >     > bugs
>     >     >     >     > in the existing Mpack (version 0.3.0).  These bugs
integrate
>     >     > with a
>     >     >     > lot of
>     >     >     >     > the oddball tweaks specified in the draft Install
docs I saw.
>     >     > I
>     >     >     > would like
>     >     >     >     > to submit a PR for these bugs, but they must be accompanied
by
>     >     >     > changes in
>     >     >     >     > the Install doc.
>     >     >     >     >
>     >     >     >     > How do we want to manage this?  Should we version
the Install
>     >     > doc?
>     >     >     > Or
>     >     >     >     > have multiple sections for the different versions?
 Or have
>     >     >     > footnotes or
>     >     >     >     > sidebar comments about which version certain paragraphs
do and
>     >     > do
>     >     >     > not apply
>     >     >     >     > to?
>     >     >     >     >
>     >     >     >     > I’m inclined to use sidebars within the document,
because the
>     >     >     > dependencies
>     >     >     >     > aren’t just on which versions of HDP, Ambari, and
Metron you
>     >     > use.
>     >     >     > They
>     >     >     >     > also depend on Python 2.6 vs 2.7, and Centos 6 vs
7.
>     >     >     >     >
>     >     >     >     > Thanks,
>     >     >     >     > --Matt
>     >     >     >     >
>     >     >     >     >
>     >     >     >     > On 12/13/16, 4:13 PM, "Zeolla@GMail.com" <zeolla@gmail.com>
>     >     > wrote:
>     >     >     >     >
>     >     >     >     >     Sorry about the delay.  To be honest I delayed
once I saw
>     >     > the
>     >     >     >     > management UI
>     >     >     >     >     PR because I was going to wait for that to be
merged into
>     >     > master
>     >     >     >     > before I
>     >     >     >     >     did my mpack install.  My initial thought is
that there
>     >     > may need
>     >     >     > to be
>     >     >     >     > some
>     >     >     >     >     sort of a merger between what you provided and
the
>     >     > Hortonworks
>     >     >     > blog
>     >     >     >     > post,
>     >     >     >     >     as I would prefer a single, comprehensive post
over
>     >     >     > fragmentation.
>     >     >     >     >
>     >     >     >     >     Jon
>     >     >     >     >
>     >     >     >     >     On Tue, Dec 13, 2016, 18:29 Dima Kovalyov <
>     >     >     > Dima.Kovalyov@sstech.us>
>     >     >     >     > wrote:
>     >     >     >     >
>     >     >     >     >     > Thank you Jon,
>     >     >     >     >     >
>     >     >     >     >     > Just wondering if you have finished polishing
the
>     >     > document? I
>     >     >     > will
>     >     >     >     > have
>     >     >     >     >     > some time this week to polish and publish
it if there is
>     >     >     > anything
>     >     >     >     > left
>     >     >     >     >     > to format.
>     >     >     >     >     >
>     >     >     >     >     > - Dima
>     >     >     >     >     >
>     >     >     >     >     > On 12/01/2016 04:41 PM, Zeolla@GMail.com
wrote:
>     >     >     >     >     > > While I can't delegate access to the
wiki, I am
>     >     > working on
>     >     >     >     > migrating the
>     >     >     >     >     > > documentation over.  I have been modifying
the
>     >     > document for
>     >     >     >     > clarity and
>     >     >     >     >     > > updating up some of the formatting,
so please give me
>     >     > some
>     >     >     > time to
>     >     >     >     > do so.
>     >     >     >     >     > >
>     >     >     >     >     > > Jon
>     >     >     >     >     > >
>     >     >     >     >     > > On Thu, Dec 1, 2016 at 9:33 AM Dima
Kovalyov <
>     >     >     >     > Dima.Kovalyov@sstech.us>
>     >     >     >     >     > > wrote:
>     >     >     >     >     > >
>     >     >     >     >     > >> Hello,
>     >     >     >     >     > >>
>     >     >     >     >     > >> Who can provide me with an access
to add new page in
>     >     > this
>     >     >     >     > Installation
>     >     >     >     >     > >> space below?
>     >     >     >     >     > >> https://cwiki.apache.org/confluence/display/METRON/
>     >     >     > Installation
>     >     >     >     >     > >> I want to add up-to-date article
on how-to install
>     >     > Metron
>     >     >     > using
>     >     >     >     > Ambari
>     >     >     >     >     > >> mpack.
>     >     >     >     >     > >> Please advise.
>     >     >     >     >     > >>
>     >     >     >     >     > >> My username is "dima" registered
with electronic mail
>     >     >     > dimdroll
>     >     >     >     > hosted by
>     >     >     >     >     > >> gmail.
>     >     >     >     >     > >>
>     >     >     >     >     > >> Right now I have it posted here:
>     >     >     >     >     > >> https://goo.gl/HWGwpj
>     >     >     >     >     > >> But instead of maintaining it there
it will be
>     >     > convenient
>     >     >     > for me
>     >     >     >     > to have
>     >     >     >     >     > >> it in confluence instead of gdocs
as I am using
>     >     > confluence
>     >     >     > in our
>     >     >     >     >     > internal
>     >     >     >     >     > >> project as well, so I can basically
copy formatted doc
>     >     >     > between
>     >     >     >     > each
>     >     >     >     >     > other.
>     >     >     >     >     > >>
>     >     >     >     >     > >> Thank you.
>     >     >     >     >     > >>
>     >     >     >     >     > >> - Dima
>     >     >     >     >     > >>
>     >     >     >     >     > >> On 11/30/2016 04:57 PM, Zeolla@GMail.com<mailto:
>     >     > Zeolla
>     >     >     > @GMail.com>
>     >     >     >     >     > wrote:
>     >     >     >     >     > >> Probably the best long term place
for this would be
>     >     > here<
>     >     >     >     >     > >> https://cwiki.apache.org/confluence/display/METRON/
>     >     >     > Installation>,
>     >     >     >     >     > >> potentially with a new mpack article
(to
>     >     > differentiate from
>     >     >     > the
>     >     >     >     > ambari
>     >     >     >     >     > >> install which is already there,
but dated).  I can
>     >     > help with
>     >     >     >     > migrating
>     >     >     >     >     > it
>     >     >     >     >     > >> to the wiki - I will run through
your documentation
>     >     > later
>     >     >     > today
>     >     >     >     > and
>     >     >     >     >     > request
>     >     >     >     >     > >> edit permissions if I find anything
worth updating.
>     >     >     >     >     > >>
>     >     >     >     >     > >> Would you be OK if I put your doc
in the wiki at some
>     >     > point?
>     >     >     >     >     > >>
>     >     >     >     >     > >> Jon
>     >     >     >     >     > >>
>     >     >     >     >     > >> On Wed, Nov 30, 2016 at 9:51 AM
Dima Kovalyov <
>     >     >     >     > Dima.Kovalyov@sstech.us
>     >     >     >     >     > >> <mailto:Dima.Kovalyov@sstech.us>>
wrote:
>     >     >     >     >     > >> As multiple users asked I have
published document for
>     >     >     > installing
>     >     >     >     > Metron
>     >     >     >     >     > on
>     >     >     >     >     > >> Centos 6 using Ambari 2.4 with
HDP 2.5:
>     >     >     >     >     > >> https://goo.gl/HWGwpj
>     >     >     >     >     > >> It may look ugly as this is direct
export from
>     >     > internal
>     >     >     >     > confluence with
>     >     >     >     >     > >> cosmetic changes and our internal
details removed.
>     >     > Let me
>     >     >     > know if
>     >     >     >     > there
>     >     >     >     >     > is
>     >     >     >     >     > >> a better place to publish it.
>     >     >     >     >     > >>
>     >     >     >     >     > >> Feel free to request edit permissions,
I would like
>     >     > the
>     >     >     > document
>     >     >     >     > to be
>     >     >     >     >     > >> up-to-date, simplified and corrected
if there are any
>     >     >     > errors. The
>     >     >     >     > goal
>     >     >     >     >     > for
>     >     >     >     >     > >> this document is to provide installation
guidance in a
>     >     >     > manner
>     >     >     >     > that Linux
>     >     >     >     >     > >> administrator with basic knowledge
could perform
>     >     >     > installation
>     >     >     >     > without
>     >     >     >     >     > >> opening any other links or searching
for commands,
>     >     > step by
>     >     >     > step.
>     >     >     >     >     > >> Thank you.
>     >     >     >     >     > >>
>     >     >     >     >     > >> p.s. feel free to let me know if
you will encounter
>     >     > any
>     >     >     > errors as
>     >     >     >     > I've
>     >     >     >     >     > >> seen them bunch.
>     >     >     >     >     > >>
>     >     >     >     >     > >> - Dima
>     >     >     >     >     > >>
>     >     >     >     >     > >>
>     >     >     >     >     > >>
>     >     >     >     >     > >> On 11/30/2016 04:45 PM, Zeolla@GMail.com<mailto:
>     >     > Zeolla
>     >     >     > @GMail.com>
>     >     >     >     >     > wrote:
>     >     >     >     >     > >> I also have written up a method
to retrieve and setup
>     >     > the
>     >     >     > required
>     >     >     >     >     > >> packages in my script<
>     >     >     >     >     > >>
>     >     >     >     >     >
>     >     >     >     > https://github.com/JonZeolla/Development/blob/master/Bash/
>     >     >     > setupMetron.sh>
>     >     >     >     >     > >> which could probably be cleaned
up and incorporated.
>     >     >     >     >     > >>
>     >     >     >     >     > >> Jon
>     >     >     >     >     > >>
>     >     >     >     >     > >> On Wed, Nov 30, 2016 at 9:24 AM
Nick Allen <
>     >     >     > nick@nickallen.org
>     >     >     >     > <mailto:
>     >     >     >     >     > >> nick@nickallen.org>> wrote:
>     >     >     >     >     > >> In regards to your comment about
a "full package
>     >     > version
>     >     >     > dump"
>     >     >     >     > there is
>     >     >     >     >     > at
>     >     >     >     >     > >> least a start at `metron-deployment/scripts/
>     >     >     > platform-info.sh`.
>     >     >     >     > If we
>     >     >     >     >     > can
>     >     >     >     >     > >> enhance that, let's do it.
>     >     >     >     >     > >>
>     >     >     >     >     > >> On Wed, Nov 30, 2016 at 4:04 AM,
shoggi <
>     >     > shoggi@gmail.com
>     >     >     > <mailto:
>     >     >     >     >     > >> shoggi@gmail.com>> wrote:
>     >     >     >     >     > >> Hi
>     >     >     >     >     > >>
>     >     >     >     >     > >> Just to wrap this up. I've tried
to deploy it on HDP
>     >     > 2.5
>     >     >     > but I
>     >     >     >     > got the
>     >     >     >     >     > >> exact same problem. What I did
was removing all
>     >     > components
>     >     >     > for
>     >     >     >     > 2.4.3.0
>     >     >     >     >     > and
>     >     >     >     >     > >> then created a new cluster based
on 2.5. I left it at
>     >     > that
>     >     >     > as the
>     >     >     >     > most
>     >     >     >     >     > >> likely issue could be with my base
install. May that
>     >     > be
>     >     >     > wrong
>     >     >     >     > version of
>     >     >     >     >     > >> something or even a missing dev
package.
>     >     >     >     >     > >>
>     >     >     >     >     > >> As my work also coincided with
the announcement by
>     >     > Casey I
>     >     >     > went
>     >     >     >     > ahead
>     >     >     >     >     > and
>     >     >     >     >     > >> installed 0.3.0 as a dev installation
instead. That
>     >     > worked
>     >     >     > great
>     >     >     >     > as the
>     >     >     >     >     > >> dev-full-platform guide is very
complete.
>     >     >     >     >     > >>
>     >     >     >     >     > >> As a general feedback, in order
to get more people
>     >     >     > interested and
>     >     >     >     > not
>     >     >     >     >     > >> having noobs like me come up with
redundant questions
>     >     > - At
>     >     >     > this
>     >     >     >     > stage
>     >     >     >     >     > also
>     >     >     >     >     > >> sorry for not providing a meaningful
subject, I
>     >     > simply was
>     >     >     > too
>     >     >     >     > excited
>     >     >     >     >     > :)
>     >     >     >     >     > >>
>     >     >     >     >     > >> There is much to win with a small
extension of the
>     >     >     > installation
>     >     >     >     > guide. I
>     >     >     >     >     > >> don't think it needs much more
prose around but a full
>     >     >     > package
>     >     >     >     > version
>     >     >     >     >     > dump
>     >     >     >     >     > >> with every new release would help
greatly to get
>     >     > things
>     >     >     > going.
>     >     >     >     > For the
>     >     >     >     >     > >> things which need to be compiled
in, the versioning
>     >     >     > information
>     >     >     >     > should
>     >     >     >     >     > be
>     >     >     >     >     > >> enough. I am saying this, as everyone
starts with a
>     >     >     > different base
>     >     >     >     >     > system.
>     >     >     >     >     > >> Some have the chance to start from
scratch, some have
>     >     > a
>     >     >     > system
>     >     >     >     > running
>     >     >     >     >     > and
>     >     >     >     >     > >> build something on top and then
there are the
>     >     > experienced
>     >     >     > Metron
>     >     >     >     > users,
>     >     >     >     >     > who
>     >     >     >     >     > >> have many base requirements.. they
tend to forget.
>     >     > And I
>     >     >     > believe
>     >     >     >     > at the
>     >     >     >     >     > end
>     >     >     >     >     > >> it always fails just because of
some very tiny small
>     >     >     > difference.
>     >     >     >     >     > >>
>     >     >     >     >     > >> If there is a space somewhere available,
this could
>     >     > even be
>     >     >     > a
>     >     >     >     > spot,
>     >     >     >     >     > where
>     >     >     >     >     > >> people can say, "hey, I got this
running on XYZ and
>     >     > here is
>     >     >     > the
>     >     >     >     > package
>     >     >     >     >     > >> version dump". A simple table with
'Metron version',
>     >     > 'OS
>     >     >     > release',
>     >     >     >     >     > 'Package
>     >     >     >     >     > >> list file', 'Version Details of
every other needed
>     >     >     > to-be-compiled
>     >     >     >     >     > package',
>     >     >     >     >     > >> is already a big help. This really
could supplement
>     >     > the
>     >     >     > existing
>     >     >     >     > guide
>     >     >     >     >     > >> found here:
>     >     >     >     >     > >>
>     >     >     >     >     >
>     >     >     >     > https://cwiki.apache.org/confluence/display/METRON/
>     >     >     > Metron+Installation+on+an+Ambari-Managed+Cluster
>     >     >     >     >     > >> .
>     >     >     >     >     > >>
>     >     >     >     >     > >> But then, I could be alone and
others did not
>     >     > experience
>     >     >     > the same
>     >     >     >     > and
>     >     >     >     >     > >> simply got it to run, I don't know.
Anyway, I would be
>     >     >     > happy to
>     >     >     >     > help if
>     >     >     >     >     > >> help is needed.
>     >     >     >     >     > >>
>     >     >     >     >     > >> Cheers
>     >     >     >     >     > >> Shoggi
>     >     >     >     >     > >>
>     >     >     >     >     > >> On Sun, Nov 27, 2016 at 3:00 PM,
shoggi <
>     >     > shoggi@gmail.com
>     >     >     > <mailto:
>     >     >     >     >     > >> shoggi@gmail.com>> wrote:
>     >     >     >     >     > >> Hi Simon
>     >     >     >     >     > >>
>     >     >     >     >     > >> Thank you for your very quick response.
I installed
>     >     > it on
>     >     >     > HDP
>     >     >     >     > 2.4.3.0.
>     >     >     >     >     > >>
>     >     >     >     >     > >> I had a 0.2.0 instance running
and was very conscious
>     >     > about
>     >     >     >     > versioning
>     >     >     >     >     > the
>     >     >     >     >     > >> various components. Somehow I had
the understanding
>     >     > of not
>     >     >     >     > touching 2.5
>     >     >     >     >     > as
>     >     >     >     >     > >> of yet. I give that a try.
>     >     >     >     >     > >>
>     >     >     >     >     > >> Thanks heaps !
>     >     >     >     >     > >>
>     >     >     >     >     > >> Shoggi
>     >     >     >     >     > >>
>     >     >     >     >     > >> On Sun, Nov 27, 2016 at 2:52 PM,
Simon Elliston Ball <
>     >     >     >     >     > >> simon@simonellistonball.com<mailto:simon@
>     >     >     > simonellistonball.com>>
>     >     >     >     > wrote:
>     >     >     >     >     > >> Hi Shoggi,
>     >     >     >     >     > >>
>     >     >     >     >     > >> Can I ask which version of HDP
you used as a base. The
>     >     >     > article is
>     >     >     >     > a
>     >     >     >     >     > little
>     >     >     >     >     > >> out of date in that is calls for
2.4, which the 0.3.0
>     >     >     > release is
>     >     >     >     > based
>     >     >     >     >     > on
>     >     >     >     >     > >> 2.5. There was a major version
difference in storm
>     >     > between
>     >     >     > the
>     >     >     >     > two,
>     >     >     >     >     > which
>     >     >     >     >     > >> seems likely to have caused the
problem you're seeing.
>     >     >     >     >     > >>
>     >     >     >     >     > >> Simon
>     >     >     >     >     > >>
>     >     >     >     >     > >> Sent from my iPad
>     >     >     >     >     > >>
>     >     >     >     >     > >> On 27 Nov 2016, at 13:48, shoggi
<shoggi@gmail.com
>     >     > <mailto:
>     >     >     >     >     > shoggi@gmail.com>>
>     >     >     >     >     > >> wrote:
>     >     >     >     >     > >>
>     >     >     >     >     > >> Hi all
>     >     >     >     >     > >>
>     >     >     >     >     > >> I followed the installation guide
to deploy Metron via
>     >     >     > Ambari
>     >     >     >     >     > >>
>     >     >     >     >     >
>     >     >     >     > https://community.hortonworks.com/articles/60805/deploying-
>     >     >     > a-fresh-metron-cluster-using-ambari-serv.html
>     >     >     >     >     > >> .
>     >     >     >     >     > >>
>     >     >     >     >     > >> I ended up with all but the Metron
Components
>     >     > (Enrichment,
>     >     >     >     > Parsing,
>     >     >     >     >     > >> Parsers) to start up. The closest
to pinpoint the
>     >     > issue is,
>     >     >     > when
>     >     >     >     > I try
>     >     >     >     >     > to
>     >     >     >     >     > >> start any new parser topology such
as with bro:
>     >     >     >     >     > >>
>     >     >     >     >     > >> /usr/metron/0.3.0/bin/start_parser_topology.sh
-k
>     >     >     >     > x.localdomain:6667 -z
>     >     >     >     >     > >> x.localdomain:2181 -s bro
>     >     >     >     >     > >>
>     >     >     >     >     > >> The error I get is:
>     >     >     >     >     > >>
>     >     >     >     >     > >> Exception in thread "main" java.lang.
>     >     > BootstrapMethodError:
>     >     >     >     >     > >> java.lang.NoClassDefFoundError:
>     >     > org/apache/storm/Config
>     >     >     >     >     > >>
>     >     >     >     >     > >> Or in more details:
>     >     >     >     >     > >>
>     >     >     >     >     > >> Running: /usr/jdk64/jdk1.8.0_77/bin/java
-client
>     >     >     > -Ddaemon.name=
>     >     >     >     >     > >> -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.3.0-
>     >     > 227/storm
>     >     >     >     >     > >> -Dstorm.log.dir=/var/log/storm
>     >     >     >     >     > >>
>     >     >     >     >     >
>     >     >     >     > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/
>     >     >     > lib:/usr/hdp/current/storm-client/lib
>     >     >     >     >     > >> -Dstorm.conf.file= -cp
>     >     >     >     >     > >>
>     >     >     >     >     >
>     >     >     >     > /usr/hdp/2.4.3.0-227/storm/lib/asm-4.0.jar:/usr/hdp/2.4.
>     >     >     > 3.0-227/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.3.0-227/
>     >     >     > storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.3.0-227/
>     >     >     > storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/
>     >     >     > clojure-1.6.0.jar:/usr/hdp/
>     >     >     >     >     > >> 2.4.3.<http://2.4.3
>     >     >     >     >     > >>
>     >     >     >     >     >
>     >     >     >     > .>0-227/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.3.0-227/
>     >     >     > storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.3.0-227/
>     >     >     > storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.3.0-227/storm/
>     >     >     > lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/
>     >     >     > disruptor-2.10.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/
>     >     >     > gmetric4j-1.0.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/hadoop-
>     >     >     > auth-2.7.1.2.4.3.0-227.jar:/usr/hdp/
>     >     >     >     >     > >> 2.4.3.<http://2.4.3
>     >     >     >     >     > >>
>     >     >     >     >     >
>     >     >     >     > .>0-227/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.3.0-227/
>     >     >     > storm/lib/jackson-core-2.3.1.jar:/usr/hdp/
>     >     >     >     >     > >> 2.4.3.<http://2.4.3
>     >     >     >     >     > >>
>     >     >     >     >     >
>     >     >     >     > .>0-227/storm/lib/jackson-dataformat-smile-2.3.1.jar:/
>     >     >     > usr/hdp/2.4.3.0-227/storm/lib/java.classpath-0.2.2.jar:/usr/
>     >     >     > hdp/2.4.3.0-227/storm/lib/javax.servlet-2.5.0.
>     >     >     > v201103041518.jar:/usr/hdp/2.4.3.0-227/storm/lib/jline-0.9.
>     >     >     > 94.jar:/usr/hdp/2.4.3.0-227/storm/lib/kryo-2.21.jar:/usr/hdp/
>     >     >     >     >     > >> 2.4.3.<http://2.4.3
>     >     >     >     >     > >>
>     >     >     >     >     >
>     >     >     >     > .>0-227/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.3.0-227/
>     >     >     > storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/
>     >     >     > log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.3.0-227/storm/lib/
>     >     >     > log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/
>     >     >     > minlog-1.2.jar:/usr/hdp/
>     >     >     >     >     > >> 2.4.3.<http://2.4.3
>     >     >     >     >     >
>     >     >     >     > .>0-227/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.3.0-
>     >     >     > 227/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.3.0-227/
>     >     >     > storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.3.0-
>     >     >     > 227/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.3.0-227/
>     >     >     > storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.3.0-227/
>     >     >     > storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.3.0-
>     >     >     > 227/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.3.0-227/
>     >     >     > storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.3.0-227/
>     >     >     > storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.3.0-227/storm/
>     >     >     > lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/
>     >     >     > storm-core-0.10.0.2.4.3.0-227.jar:/usr/hdp/2.4.3.0-227/
>     >     >     > storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/
>     >     >     > tools.logging-0.2.3.jar:/usr/hdp/2.4.3.0-227/storm/lib/
>     >     >     > tools.namespace-0.2.4.jar:/usr/hdp/2.4.3.0-227/storm/lib/
>     >     >     > zookeeper.jar:/usr/metron/0.3.0/lib/metron-parsers-0.3.0-
>     >     >     > uber.jar:/usr/hdp/current/storm-supervisor/conf:/usr/
>     >     >     > hdp/2.4.3.0-227/storm/bin
>     >     >     >     >     > >> -Dstorm.jar=/usr/metron/0.3.0/
>     >     >     > lib/metron-parsers-0.3.0-uber.jar
>     >     >     >     >     > >> org.apache.metron.parsers.topology.ParserTopologyCLI
>     >     > -k
>     >     >     >     >     > x.localdomain:6667
>     >     >     >     >     > >> -z x.localdomain:2181 -s bro
>     >     >     >     >     > >> Exception in thread "main" java.lang.
>     >     > BootstrapMethodError:
>     >     >     >     >     > >> java.lang.NoClassDefFoundError:
>     >     > org/apache/storm/Config
>     >     >     >     >     > >> at
>     >     >     >     >     > >>
>     >     >     >     >     >
>     >     >     >     > org.apache.metron.parsers.topology.ParserTopologyCLI$
>     >     >     > ParserOptions.<init>(ParserTopologyCLI.java:199)
>     >     >     >     >     > >> at
>     >     >     >     >     > >>
>     >     >     >     >     >
>     >     >     >     > org.apache.metron.parsers.topology.ParserTopologyCLI$
>     >     >     > ParserOptions.<clinit>(ParserTopologyCLI.java:47)
>     >     >     >     >     > >> at
>     >     >     >     >     > >>
>     >     >     >     >     >
>     >     >     >     > org.apache.metron.parsers.topology.ParserTopologyCLI.
>     >     >     > main(ParserTopologyCLI.java:266)
>     >     >     >     >     > >> Caused by: java.lang.NoClassDefFoundError:
>     >     >     > org/apache/storm/Config
>     >     >     >     >     > >> ... 3 more
>     >     >     >     >     > >> Caused by: java.lang.ClassNotFoundException:
>     >     >     >     > org.apache.storm.Config
>     >     >     >     >     > >> at java.net.URLClassLoader.
>     >     > findClass(URLClassLoader.java:
>     >     >     > 381)
>     >     >     >     >     > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:
>     >     > 424)
>     >     >     >     >     > >> at sun.misc.Launcher$AppClassLoader.loadClass(
>     >     >     > Launcher.java:331)
>     >     >     >     >     > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:
>     >     > 357)
>     >     >     >     >     > >> ... 3 more
>     >     >     >     >     > >>
>     >     >     >     >     > >> Must be something wrong with my
config, possibly I
>     >     > have
>     >     >     > overlooked
>     >     >     >     >     > >> something but can't put my finger
on it. Any help is
>     >     >     > appreciated.
>     >     >     >     >     > >>
>     >     >     >     >     > >> Many thanks
>     >     >     >     >     > >> Shoggi
>     >     >     >     >     > >>
>     >     >     >     >     > >>
>     >     >     >     >     > >>
>     >     >     >     >     > >>
>     >     >     >     >     > >>
>     >     >     >     >     > >> --
>     >     >     >     >     > >> Nick Allen <nick@nickallen.org<mailto:nic
>     >     > k@nickallen.org>>
>     >     >     >     >     > >> --
>     >     >     >     >     > >>
>     >     >     >     >     > >> Jon
>     >     >     >     >     > >>
>     >     >     >     >     > >> Sent from my mobile device
>     >     >     >     >     > >>
>     >     >     >     >     > >> --
>     >     >     >     >     > >>
>     >     >     >     >     > >> Jon
>     >     >     >     >     > >>
>     >     >     >     >     > >> Sent from my mobile device
>     >     >     >     >     > >>
>     >     >     >     >     > >> --
>     >     >     >     >     > > Jon
>     >     >     >     >     > >
>     >     >     >     >     > > Sent from my mobile device
>     >     >     >     >     > >
>     >     >     >     >     >
>     >     >     >     >     > --
>     >     >     >     >
>     >     >     >     >     Jon
>     >     >     >     >
>     >     >     >     >     Sent from my mobile device
>     >     >     >     >
>     >     >     >     >
>     >     >     >     >
>     >     >     >     > --
>     >     >     >
>     >     >     >     Jon
>     >     >     >
>     >     >     >     Sent from my mobile device
>     >     >     >
>     >     >     >
>     >     >     >
>     >     >
>     >     >
>     >     >
>     >     
>     >
>     >
>     >
>     >
>     >
>     
>     
>     
>
>
>



Mime
View raw message