storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From János Háber <janos.ha...@finesolution.hu>
Subject Re: Sample project using Storm and Kafka (using Scala 2.10)
Date Fri, 23 May 2014 08:21:09 GMT
If only the interfaces used from kafka (looked superficially) the provided
scope will be good idea... (both to scala and kafka dependencies)

János Háber
Fine Solution Ltd



On Thu, May 22, 2014 at 6:35 PM, P. Taylor Goetz <ptgoetz@gmail.com> wrote:

> Come to think about it, this is a non-issue. This is just java code that
> happens to depend on a library written in scala. There isn’t anything that
> needs to be cross compiled.
>
> So it comes down to dependency management and documentation. So the scala
> and kafka dependencies would be set to “provided” scope, and it would be up
> to the user to pull in the kafka/scala dependency versions they want.
>
> Correct me if I’m missing anything here.
>
> - Taylor
>
>
>
> On May 22, 2014, at 6:08 AM, Marco <zentropa80@yahoo.co.uk> wrote:
>
> Taylor,
> cross-compatibility is a Scala "thingie" and THEREFORE it reflects onto
> Kafka and THEREFORE onto storm-kafka (or whatever-kafka).
>
> Kafka (if I understand correctly
> https://github.com/apache/kafka/blob/0.8.1/build.gradle#l131) manages
> this with Gradle (which I'm personally not much a fan of).
>
> I don't think it'd be much of an overhead having that in storm-kafka (and
> so effectively have storm-kafka_2_8_0, storm-kafka_2_9_1,
> storm-kafka_2_9_2, storm-kafka_2_10_1).
>
> Maven profiles can do that (as Micheal proved).
>
> Plus Storm seems to have put a lot of interest towards polyglottism
> (mixing Clojure and Java for example).
>
> Also, I agree with all the points János made.
> Having it on Maven Central would be THE perfect starting point for any
> (Scala) developer willing to give it a spin.
>
> ps: Scala 2.11 is just out
>
> -Marco
>   Il Giovedì 22 Maggio 2014 3:43, P. Taylor Goetz <ptgoetz@gmail.com> ha
> scritto:
>
>
> János,
>
> I sense and appreciate your frustration.
>
> The initial goal with bringing the storm-kafka project under the Apache
> Storm project umbrella is to provide a definitive source for the code and
> eliminate fragmentation, which has been an issue in the past. This is a
> good first step for many reasons, and will hopefully improve greatly in
> future iterations.
>
> Just to be clear, Apache releases source code, not binaries. Any binaries
> built against an Apache project's source are provided as a convenience.
>
> The cross-compilation ballet you describe is a feature of scala [1], not
> anything Storm or maven related. Yes, we can and will improve on the build
> and binary release process. But for now the goal is to provide a definitive
> source, and make sure users can build that source as needed -- which I
> think we have done.
>
> -Taylor
>
> [1] a not-so-subtle dig against scala's build process, not the language.
> Hopefully this will get sorted out someday.
>
> On May 21, 2014, at 7:48 PM, János Háber <janos.haber@finesolution.hu>
> wrote:
>
> Dear Taylor, I love your work, but:
> - I don't want to build myself
> - Dependent libraries (like tormenta-kafka) need a cross compiled version
> of storm-kafka, without this they need to clone the project, change the
> group id, handle every changes by hand, and publish to central repository.
> - I need to have own maven repository to store the cross compiled version
> (which need to be public if somebody want to use my application) and
> maintain the changes
> - I think hand made library is the best way to make a project to unstable,
> because if I clone the project I need to clone tormenta-kafka project too
> and handle myself both version changes, solve the compatibility issues,
> etc...
>
> I know how can I compile to 2.10 by hand, but all other project (example
> kafka, which is apache project too) has cross compiled version in CENTRAL
> maven repository, if a project required a cross-compiled scala library - my
> oppinion - the project need to provide cross-compiled version too, no
> exception.
>
> b0c1
>
> János Háber
> Fine Solution Ltd
>
>
>
> On Wed, May 21, 2014 at 11:48 PM, P. Taylor Goetz <ptgoetz@gmail.com>wrote:
>
> If you build yourself, you can do the following:
>
> mvn install -DscalaVersion=2.10.3 -DkafkaArtifact=kafka_2.10
>
> - Taylor
>
>
> On May 21, 2014, at 5:32 PM, János Háber <janos.haber@finesolution.hu>
> wrote:
>
>
> https://github.com/apache/incubator-storm/blob/master/external/storm-kafka/pom.xml#L34
> And 2.10 not backward compatible with 2.9... that's why scala
> applications/libraries is cross compiled with multiple scala version.
> (that's why SBT will handle this natively, but I think in apache sbt is not
> allowed so you need to create multiple maven project (hard way) or switch
> to gradle (like kafka) to produce multiple version)
>
> János Háber
> Fine Solution Ltd
>
>
>
> On Wed, May 21, 2014 at 11:28 PM, János Háber <janos.haber@finesolution.hu
> > wrote:
>
>
> On Wed, May 21, 2014 at 11:27 PM, P. Taylor Goetz <ptgoetz@gmail.com>wrote:
>
> ill include the Kafka spout (for Kafka 0.8.x).
>
>
> Yeah, but the current maven build compile to 2.9 scala not 2.10...
>
>
> János Háber
> Fine Solution Ltd
>
>
>
>
>
>
>
>

Mime
View raw message