metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dima Kovalyov <Dima.Koval...@sstech.us>
Subject Re: Metron with HDP 2.5
Date Sun, 09 Oct 2016 14:59:07 GMT
Thank you Nick,

I've re-build topologies.

Another thing I've noticed after upgrade, my Kafka Broker crashes with
started topologies after a while. The only relative to the crash message
I was able to find is this:
> [2016-10-09 11:30:35,013] ERROR Controller 1001 epoch 14 initiated
> state change for partition [parser_invalid,0] from OfflinePartition to
> OnlinePartition failed (state.change.logger)
I have also attached kafka.tar.gz archive with my kafka logs from the
moment of the Kafka Broker start till crash (15 minutes past).

When I deploy topologies I receive following message, which I consider
is just a warning, not error:
> 20382 [main] WARN  o.a.s.h.DefaultShader - Relocating
> backtype/storm/security/auth/ICredentialsRenewer to
> org/apache/storm/security/auth/ICredentialsRenewer in
> org/apache/storm/hdfs/common/security/AutoHDFS.class. please modify
> your code to use the new namespace
Because I have following set in storm.yaml:
client.jartransformer.class : 'org.apache.storm.hack.StormShadeTransformer'
Which is exactly what you need to set for Storm 1.0+ to work with legacy
jars according to: https://storm.apache.org/releases/1.0.0/

Last thing, when I start Kafka back again after crash and deploy
topologies to Storm I have following message displaying in Storm UI:
> Topology spouts lag error
> kafkaSpout    KAFKA    Offset lags for kafka not supported for older
> versions. Please update kafka spout to latest version.

Does anyone know how to fix topology offset lag or how to update kafka
spout to latest version?

- Dima


On 09/28/2016 03:08 PM, Nick Allen wrote:
> You can re-build and deploy to your 'Quick Dev' environment by doing the
> following from the root of Metron's source code.
>
> mvn clean package -DskipTests
> cd metron-deployment/vagrant/quick-dev-platform
> ./run_enrichment_role.sh
>
>
> On Tue, Sep 27, 2016 at 11:57 PM, Dima Kovalyov <Dima.Kovalyov@sstech.us>
> wrote:
>
>> Thank you for reply James,
>>
>> I have performed Ambari 2.2 -> 2.4 and HDP 2.4 -> 2.5 upgrade on
>> quick-dev-platform in Vagrant Metron setup. Most of it went just fine,
>> except few intermittent errors.
>>
>> Can you please advise where do you want it to be documented and/or
>> discussed?
>>
>> Right now I am battling issue with Storm. After the upgrade I need to
>> re-build Storm Topology jars in order to propagate HDP 2.5 files in
>> them. Can you please advise how I can do that on already running setup?
>> (TLDR: how to build java jars manually?)
>>
>> Thank you!
>>
>> - Dima
>>
>> On 09/27/2016 06:59 PM, James Sirota wrote:
>>> We didn't test it enough to know how well it works.  We don't expect it
>> to work at this time. If you can run it up for us and document all the
>> issues that go wrong that would be greatly appreciated.
>>> 26.09.2016, 16:54, "Satish Abburi" <satish.abburi@sstech.us>:
>>>> All, any one using/tested this combination.
>>>>
>>>> Thanks,
>>>> Satish
>>> -------------------
>>> Thank you,
>>>
>>> James Sirota
>>> PPMC- Apache Metron (Incubating)
>>> jsirota AT apache DOT org
>>>
>>
>


Mime
View raw message