storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From János Háber <janos.ha...@finesolution.hu>
Subject Re: Sample project using Storm and Kafka (using Scala 2.10)
Date Fri, 23 May 2014 08:27:39 GMT
I change the issue ( https://issues.apache.org/jira/browse/STORM-325 )...

János Háber
Fine Solution Ltd



On Fri, May 23, 2014 at 10:21 AM, János Háber
<janos.haber@finesolution.hu>wrote:

> If only the interfaces used from kafka (looked superficially) the provided
> scope will be good idea... (both to scala and kafka dependencies)
>
> János Háber
> Fine Solution Ltd
>
>
>
> On Thu, May 22, 2014 at 6:35 PM, P. Taylor Goetz <ptgoetz@gmail.com>wrote:
>
>> Come to think about it, this is a non-issue. This is just java code that
>> happens to depend on a library written in scala. There isn’t anything that
>> needs to be cross compiled.
>>
>> So it comes down to dependency management and documentation. So the scala
>> and kafka dependencies would be set to “provided” scope, and it would be up
>> to the user to pull in the kafka/scala dependency versions they want.
>>
>> Correct me if I’m missing anything here.
>>
>> - Taylor
>>
>>
>>
>> On May 22, 2014, at 6:08 AM, Marco <zentropa80@yahoo.co.uk> wrote:
>>
>> Taylor,
>> cross-compatibility is a Scala "thingie" and THEREFORE it reflects onto
>> Kafka and THEREFORE onto storm-kafka (or whatever-kafka).
>>
>> Kafka (if I understand correctly
>> https://github.com/apache/kafka/blob/0.8.1/build.gradle#l131) manages
>> this with Gradle (which I'm personally not much a fan of).
>>
>> I don't think it'd be much of an overhead having that in storm-kafka (and
>> so effectively have storm-kafka_2_8_0, storm-kafka_2_9_1,
>> storm-kafka_2_9_2, storm-kafka_2_10_1).
>>
>> Maven profiles can do that (as Micheal proved).
>>
>> Plus Storm seems to have put a lot of interest towards polyglottism
>> (mixing Clojure and Java for example).
>>
>> Also, I agree with all the points János made.
>> Having it on Maven Central would be THE perfect starting point for any
>> (Scala) developer willing to give it a spin.
>>
>> ps: Scala 2.11 is just out
>>
>> -Marco
>>   Il Giovedì 22 Maggio 2014 3:43, P. Taylor Goetz <ptgoetz@gmail.com> ha
>> scritto:
>>
>>
>>  János,
>>
>> I sense and appreciate your frustration.
>>
>> The initial goal with bringing the storm-kafka project under the Apache
>> Storm project umbrella is to provide a definitive source for the code and
>> eliminate fragmentation, which has been an issue in the past. This is a
>> good first step for many reasons, and will hopefully improve greatly in
>> future iterations.
>>
>> Just to be clear, Apache releases source code, not binaries. Any binaries
>> built against an Apache project's source are provided as a convenience.
>>
>> The cross-compilation ballet you describe is a feature of scala [1], not
>> anything Storm or maven related. Yes, we can and will improve on the build
>> and binary release process. But for now the goal is to provide a definitive
>> source, and make sure users can build that source as needed -- which I
>> think we have done.
>>
>> -Taylor
>>
>> [1] a not-so-subtle dig against scala's build process, not the language.
>> Hopefully this will get sorted out someday.
>>
>> On May 21, 2014, at 7:48 PM, János Háber <janos.haber@finesolution.hu>
>> wrote:
>>
>> Dear Taylor, I love your work, but:
>> - I don't want to build myself
>> - Dependent libraries (like tormenta-kafka) need a cross compiled version
>> of storm-kafka, without this they need to clone the project, change the
>> group id, handle every changes by hand, and publish to central repository.
>> - I need to have own maven repository to store the cross compiled version
>> (which need to be public if somebody want to use my application) and
>> maintain the changes
>> - I think hand made library is the best way to make a project to
>> unstable, because if I clone the project I need to clone tormenta-kafka
>> project too and handle myself both version changes, solve the compatibility
>> issues, etc...
>>
>> I know how can I compile to 2.10 by hand, but all other project (example
>> kafka, which is apache project too) has cross compiled version in CENTRAL
>> maven repository, if a project required a cross-compiled scala library - my
>> oppinion - the project need to provide cross-compiled version too, no
>> exception.
>>
>> b0c1
>>
>> János Háber
>> Fine Solution Ltd
>>
>>
>>
>> On Wed, May 21, 2014 at 11:48 PM, P. Taylor Goetz <ptgoetz@gmail.com>wrote:
>>
>> If you build yourself, you can do the following:
>>
>> mvn install -DscalaVersion=2.10.3 -DkafkaArtifact=kafka_2.10
>>
>> - Taylor
>>
>>
>> On May 21, 2014, at 5:32 PM, János Háber <janos.haber@finesolution.hu>
>> wrote:
>>
>>
>> https://github.com/apache/incubator-storm/blob/master/external/storm-kafka/pom.xml#L34
>> And 2.10 not backward compatible with 2.9... that's why scala
>> applications/libraries is cross compiled with multiple scala version.
>> (that's why SBT will handle this natively, but I think in apache sbt is not
>> allowed so you need to create multiple maven project (hard way) or switch
>> to gradle (like kafka) to produce multiple version)
>>
>> János Háber
>> Fine Solution Ltd
>>
>>
>>
>> On Wed, May 21, 2014 at 11:28 PM, János Háber <
>> janos.haber@finesolution.hu> wrote:
>>
>>
>> On Wed, May 21, 2014 at 11:27 PM, P. Taylor Goetz <ptgoetz@gmail.com>wrote:
>>
>> ill include the Kafka spout (for Kafka 0.8.x).
>>
>>
>> Yeah, but the current maven build compile to 2.9 scala not 2.10...
>>
>>
>> János Háber
>> Fine Solution Ltd
>>
>>
>>
>>
>>
>>
>>
>>
>

Mime
View raw message