metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dima Kovalyov <Dima.Koval...@sstech.us>
Subject Confluence write access to a space
Date Thu, 01 Dec 2016 14:33:22 GMT
Hello,

Who can provide me with an access to add new page in this Installation space below?
https://cwiki.apache.org/confluence/display/METRON/Installation
I want to add up-to-date article on how-to install Metron using Ambari mpack.
Please advise.

My username is "dima" registered with electronic mail dimdroll hosted by gmail.

Right now I have it posted here:
https://goo.gl/HWGwpj
But instead of maintaining it there it will be convenient for me to have it in confluence
instead of gdocs as I am using confluence in our internal project as well, so I can basically
copy formatted doc between each other.

Thank you.

- Dima

On 11/30/2016 04:57 PM, Zeolla@GMail.com<mailto:Zeolla@GMail.com> wrote:
Probably the best long term place for this would be here<https://cwiki.apache.org/confluence/display/METRON/Installation>,
potentially with a new mpack article (to differentiate from the ambari install which is already
there, but dated).  I can help with migrating it to the wiki - I will run through your documentation
later today and request edit permissions if I find anything worth updating.

Would you be OK if I put your doc in the wiki at some point?

Jon

On Wed, Nov 30, 2016 at 9:51 AM Dima Kovalyov <Dima.Kovalyov@sstech.us<mailto:Dima.Kovalyov@sstech.us>>
wrote:
As multiple users asked I have published document for installing Metron on Centos 6 using
Ambari 2.4 with HDP 2.5:
https://goo.gl/HWGwpj
It may look ugly as this is direct export from internal confluence with cosmetic changes and
our internal details removed. Let me know if there is a better place to publish it.

Feel free to request edit permissions, I would like the document to be up-to-date, simplified
and corrected if there are any errors. The goal for this document is to provide installation
guidance in a manner that Linux administrator with basic knowledge could perform installation
without opening any other links or searching for commands, step by step.
Thank you.

p.s. feel free to let me know if you will encounter any errors as I've seen them bunch.

- Dima



On 11/30/2016 04:45 PM, Zeolla@GMail.com<mailto:Zeolla@GMail.com> wrote:
I also have written up a method to retrieve and setup the required packages in my script<https://github.com/JonZeolla/Development/blob/master/Bash/setupMetron.sh>
which could probably be cleaned up and incorporated.

Jon

On Wed, Nov 30, 2016 at 9:24 AM Nick Allen <nick@nickallen.org<mailto:nick@nickallen.org>>
wrote:
In regards to your comment about a "full package version dump" there is at least a start at
`metron-deployment/scripts/platform-info.sh`.  If we can enhance that, let's do it.

On Wed, Nov 30, 2016 at 4:04 AM, shoggi <shoggi@gmail.com<mailto:shoggi@gmail.com>>
wrote:
Hi

Just to wrap this up. I've tried to deploy it on HDP 2.5 but I got the exact same problem.
What I did was removing all components for 2.4.3.0 and then created a new cluster based on
2.5. I left it at that as the most likely issue could be with my base install. May that be
wrong version of something or even a missing dev package.

As my work also coincided with the announcement by Casey I went ahead and installed 0.3.0
as a dev installation instead. That worked great as the dev-full-platform guide is very complete.

As a general feedback, in order to get more people interested and not having noobs like me
come up with redundant questions - At this stage also sorry for not providing a meaningful
subject, I simply was too excited :)

There is much to win with a small extension of the installation guide. I don't think it needs
much more prose around but a full package version dump with every new release would help greatly
to get things going. For the things which need to be compiled in, the versioning information
should be enough. I am saying this, as everyone starts with a different base system. Some
have the chance to start from scratch, some have a system running and build something on top
and then there are the experienced Metron users, who have many base requirements.. they tend
to forget. And I believe at the end
it always fails just because of some very tiny small difference.

If there is a space somewhere available, this could even be a spot, where people can say,
"hey, I got this running on XYZ and here is the package version dump". A simple table with
'Metron version', 'OS release', 'Package list file', 'Version Details of every other needed
to-be-compiled package', is already a big help. This really could supplement the existing
guide found here: https://cwiki.apache.org/confluence/display/METRON/Metron+Installation+on+an+Ambari-Managed+Cluster.

But then, I could be alone and others did not experience the same and simply got it to run,
I don't know. Anyway, I would be happy to help if help is needed.

Cheers
Shoggi

On Sun, Nov 27, 2016 at 3:00 PM, shoggi <shoggi@gmail.com<mailto:shoggi@gmail.com>>
wrote:
Hi Simon

Thank you for your very quick response. I installed it on HDP 2.4.3.0.

I had a 0.2.0 instance running and was very conscious about versioning the various components.
Somehow I had the understanding of not touching 2.5 as of yet. I give that a try.

Thanks heaps !

Shoggi

On Sun, Nov 27, 2016 at 2:52 PM, Simon Elliston Ball <simon@simonellistonball.com<mailto:simon@simonellistonball.com>>
wrote:
Hi Shoggi,

Can I ask which version of HDP you used as a base. The article is a little out of date in
that is calls for 2.4, which the 0.3.0 release is based on 2.5. There was a major version
difference in storm between the two, which seems likely to have caused the problem you're
seeing.

Simon

Sent from my iPad

On 27 Nov 2016, at 13:48, shoggi <shoggi@gmail.com<mailto:shoggi@gmail.com>> wrote:

Hi all

I followed the installation guide to deploy Metron via Ambari https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html.

I ended up with all but the Metron Components (Enrichment, Parsing, Parsers) to start up.
The closest to pinpoint the issue is, when I try to start any new parser topology such as
with bro:

/usr/metron/0.3.0/bin/start_parser_topology.sh -k x.localdomain:6667 -z x.localdomain:2181
-s bro

The error I get is:

Exception in thread "main" java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError:
org/apache/storm/Config

Or in more details:

Running: /usr/jdk64/jdk1.8.0_77/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.3.0-227/storm
-Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib
-Dstorm.conf.file= -cp /usr/hdp/2.4.3.0-227/storm/lib/asm-4.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.3.<http://2.4.3.>0-227/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.3.0-227/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.3.0-227/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/hadoop-auth-2.7.1.2.4.3.0-227.jar:/usr/hdp/2.4.3.<http://2.4.3.>0-227/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.3.0-227/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.3.<http://2.4.3.>0-227/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.3.0-227/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.3.0-227/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.3.0-227/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.3.<http://2.4.3.>0-227/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.3.0-227/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.3.<http://2.4.3.>0-227/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.3.0-227/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.3.0-227/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.3.0-227/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.3.0-227/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/storm-core-0.10.0.2.4.3.0-227.jar:/usr/hdp/2.4.3.0-227/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.3.0-227/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.3.0-227/storm/lib/zookeeper.jar:/usr/metron/0.3.0/lib/metron-parsers-0.3.0-uber.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.4.3.0-227/storm/bin
-Dstorm.jar=/usr/metron/0.3.0/lib/metron-parsers-0.3.0-uber.jar org.apache.metron.parsers.topology.ParserTopologyCLI
-k x.localdomain:6667 -z x.localdomain:2181 -s bro
Exception in thread "main" java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError:
org/apache/storm/Config
at org.apache.metron.parsers.topology.ParserTopologyCLI$ParserOptions.<init>(ParserTopologyCLI.java:199)
at org.apache.metron.parsers.topology.ParserTopologyCLI$ParserOptions.<clinit>(ParserTopologyCLI.java:47)
at org.apache.metron.parsers.topology.ParserTopologyCLI.main(ParserTopologyCLI.java:266)
Caused by: java.lang.NoClassDefFoundError: org/apache/storm/Config
... 3 more
Caused by: java.lang.ClassNotFoundException: org.apache.storm.Config
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 3 more

Must be something wrong with my config, possibly I have overlooked something but can't put
my finger on it. Any help is appreciated.

Many thanks
Shoggi





--
Nick Allen <nick@nickallen.org<mailto:nick@nickallen.org>>
--

Jon

Sent from my mobile device

--

Jon

Sent from my mobile device


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message