metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Allen <n...@nickallen.org>
Subject Re: Metron with HDP 2.5
Date Sun, 09 Oct 2016 18:28:25 GMT
Metron is built against Storm 0.10.x but you are trying to run it on Storm
1.0.x.  Unfortunately that will not work.  There is some development and
testing needed to get Metron running on that version of Storm.

On Oct 9, 2016 10:59 AM, "Dima Kovalyov" <Dima.Kovalyov@sstech.us> wrote:

> Thank you Nick,
>
> I've re-build topologies.
>
> Another thing I've noticed after upgrade, my Kafka Broker crashes with
> started topologies after a while. The only relative to the crash message
> I was able to find is this:
> > [2016-10-09 11:30:35,013] ERROR Controller 1001 epoch 14 initiated
> > state change for partition [parser_invalid,0] from OfflinePartition to
> > OnlinePartition failed (state.change.logger)
> I have also attached kafka.tar.gz archive with my kafka logs from the
> moment of the Kafka Broker start till crash (15 minutes past).
>
> When I deploy topologies I receive following message, which I consider
> is just a warning, not error:
> > 20382 [main] WARN  o.a.s.h.DefaultShader - Relocating
> > backtype/storm/security/auth/ICredentialsRenewer to
> > org/apache/storm/security/auth/ICredentialsRenewer in
> > org/apache/storm/hdfs/common/security/AutoHDFS.class. please modify
> > your code to use the new namespace
> Because I have following set in storm.yaml:
> client.jartransformer.class : 'org.apache.storm.hack.
> StormShadeTransformer'
> Which is exactly what you need to set for Storm 1.0+ to work with legacy
> jars according to: https://storm.apache.org/releases/1.0.0/
>
> Last thing, when I start Kafka back again after crash and deploy
> topologies to Storm I have following message displaying in Storm UI:
> > Topology spouts lag error
> > kafkaSpout    KAFKA    Offset lags for kafka not supported for older
> > versions. Please update kafka spout to latest version.
>
> Does anyone know how to fix topology offset lag or how to update kafka
> spout to latest version?
>
> - Dima
>
>
> On 09/28/2016 03:08 PM, Nick Allen wrote:
> > You can re-build and deploy to your 'Quick Dev' environment by doing the
> > following from the root of Metron's source code.
> >
> > mvn clean package -DskipTests
> > cd metron-deployment/vagrant/quick-dev-platform
> > ./run_enrichment_role.sh
> >
> >
> > On Tue, Sep 27, 2016 at 11:57 PM, Dima Kovalyov <Dima.Kovalyov@sstech.us
> >
> > wrote:
> >
> >> Thank you for reply James,
> >>
> >> I have performed Ambari 2.2 -> 2.4 and HDP 2.4 -> 2.5 upgrade on
> >> quick-dev-platform in Vagrant Metron setup. Most of it went just fine,
> >> except few intermittent errors.
> >>
> >> Can you please advise where do you want it to be documented and/or
> >> discussed?
> >>
> >> Right now I am battling issue with Storm. After the upgrade I need to
> >> re-build Storm Topology jars in order to propagate HDP 2.5 files in
> >> them. Can you please advise how I can do that on already running setup?
> >> (TLDR: how to build java jars manually?)
> >>
> >> Thank you!
> >>
> >> - Dima
> >>
> >> On 09/27/2016 06:59 PM, James Sirota wrote:
> >>> We didn't test it enough to know how well it works.  We don't expect it
> >> to work at this time. If you can run it up for us and document all the
> >> issues that go wrong that would be greatly appreciated.
> >>> 26.09.2016, 16:54, "Satish Abburi" <satish.abburi@sstech.us>:
> >>>> All, any one using/tested this combination.
> >>>>
> >>>> Thanks,
> >>>> Satish
> >>> -------------------
> >>> Thank you,
> >>>
> >>> James Sirota
> >>> PPMC- Apache Metron (Incubating)
> >>> jsirota AT apache DOT org
> >>>
> >>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message