nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Burgess <mattyb...@apache.org>
Subject Re: Will you accept contributions in Scala?
Date Sat, 10 Feb 2018 23:26:04 GMT
I'm fine with a vote, but I'll be voting to keep Java as the single
language for the (non-test) code. I share the same concerns as many of
the other folks as far as accepting other languages, it's mainly the
"slippery slope" argument that I don't want to turn into a
JVM-language flame war.  If Scala, why not Groovy? Certainly the
syntax is closer to Java, and the community has accepted it as a valid
language for writing unit tests, although we stopped short for
allowing it for the deployable NiFi codebase, for the same reasons
IIRC.  If Scala and/or Groovy, why not Kotlin?  The same argument
(albeit more tenuous) goes for Clojure and just about every other JVM
language (although I don't expect a call for LuaJ processors lol).

Whether we decide to support various languages ad-hoc or not, I would
strenuously object to multiple/hybrid build systems for the deployed
artifacts. If I could switch NiFi completely to Gradle I would, but I
realize there are good reasons for not doing so (yet?) in the Apache
NiFi community, and I would never want any hybrid Maven/Gradle build
for the deployable code, likewise for SBT, Leiningen, etc. With a
custom Mojo for Maven NAR builds, and the complexity for hybrid builds
in general, I think this would create a maintenance nightmare.

The language thing is a tough decision though, it's not awesome that
specifying a single language can be a barrier to a more diverse
community, certainly Scala-based bundles would be more than welcome in
the overall NiFi ecosystem, I just think the cons outweigh the pros
for the baseline code. I've written Groovy processors/NARs using
Gradle as the build system, and I'm good with keeping them in my own
repo, especially when the Extension Registry becomes a thing. I can
see the Extension Registry perhaps making this a moot point, but
clearly we need to have this discussion in the meantime.

Regards,
Matt


On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande <aperepel@gmail.com> wrote:
> Wasn't there a warning trigger about the NiFi distro size from Apache
> recently? IMO, before talking alternative languages, solve the modularity
> and NAR distribution problem. I think the implementation of a module won't
> matter much then, the point being not everything has to go in the core,
> base distribution, but can still be easily sourced from a known repo, for
> example.
>
> I have a feeling NiFi 1.6+ can be approaching 2GB distro size soon :)
>
> Andrew
>
> On Sat, Feb 10, 2018, 5:12 PM Joey Frazee <joey.frazee@icloud.com> wrote:
>
>> This probably necessitates a vote, yeah?
>>
>> Frankly, I’m usually happier writing Scala, and I’ve not encountered any
>> problems using processors written in Scala, but I think it’ll be important
>> to tread lightly.
>>
>> There’s a few things that pop into my head:
>>
>> - Maintainability and reviewability. A very very good Java developer need
>> not, by definition, either know how to write or identify good Scala or spot
>> problems and bugs.
>> - Every Scala processor would either end up with a 5MB scala-lang.jar
>> packaged into the .nar or we’d have to start including it in the core
>> somewhere, if it’s not. It’s possible it might have already gotten pulled
>> up from other dependencies.
>> - Style. There’s a tremendous amount of variation in Scala style because
>> of its type system, implicits, macros, and functional nature. There are
>> very good people out there that can write good Scala that isn’t readable by
>> the 99%.
>> - Binary compatibility. Scala tends to be a little more brazen about
>> breaking binary compatibility in major releases and those happen a bit more
>> often than with Java. That’s not a problem for any potential source code in
>> the project, but it could present some dependency issues someday.
>> - Testing. There’s N > 1 test frameworks and testing styles within those,
>> so there’s a lot of options for introducing more variability into the tests.
>> - NiFi uses a lot of statics in setting up properties and relationships
>> and the like, and idiomatic Scala approaches that stuff a bit differently,
>> so it’ll be necessary to impose some style guidelines so there isn’t too
>> much variation.
>>
>> That said, there are some things that won’t be problematic:
>>
>> - As mentioned, processors written in Scala do just work.
>> - The scala-maven-plugin works just fine allowing mixed Java-Scala
>> projects (btw, it’d probably be not super great to do mixed Java-Scala and
>> mixed Maven-SBT though).
>> - A lot of the above concerns could be addressed by having clear style
>> guidelines.
>>
>> Another thing: most of the projects I see deliver separate jars for Scala
>> components are delivering idiomatic APIs wrapping Java (or vice versa). I
>> think publishing   a separate set of jars/nars for stuff written in Scala
>> would be odd since here it’d mostly be processors with new functionality
>> and not functionality for using Scala. I could imagine a lib of implicits,
>> traits, classes that could make the Scala development more enjoyable. That
>> probably would make sense to deliver that way.
>>
>> -joey
>>
>> On Feb 10, 2018, 10:33 AM -0600, Bryan Bende <bbende@gmail.com>, wrote:
>> > I agree more with Andy about sticking with Java. The more varying
>> languages
>> > used, the more challenging it is to maintain. Once the code is part of
>> the
>> > Apache NiFi git repo, it is now the responsibility of the committers and
>> > PMC members to maintain it.
>> >
>> > I’d even say I am somewhat against the groovy/Spock test code that Andy
>> > mentioned. I have frequently spent hours trying to fix a Spock test that
>> > broke from something I was working on. Every committer is familiar with
>> > JUnit, but only a couple know Spock. Just using this as an example that
>> > every committer knows Java, but only a couple probably know Scala,
>> Clojure,
>> > etc.
>> >
>> > On Sat, Feb 10, 2018 at 10:25 AM Jeff <jtswork@gmail.com> wrote:
>> >
>> > > +1 to Joe's response. If you can develop a component in Groovy or Scala
>> > > (or Clojure!) more quickly/comfortably, or if allowing components
>> written
>> > > in other languages would encourage people to contribute more, I'm all
>> for
>> > > it.
>> > >
>> > > On Sat, Feb 10, 2018 at 7:42 AM Joe Witt <joe.witt@gmail.com> wrote:
>> > >
>> > > > i personally would be ok with it for an extension/processor provided
>> it
>> > > > integrates well with the build.
>> > > >
>> > > > i would agree with andys view for core framework stuff but for
>> > > extensions i
>> > > > think we can do it like mikethomsen suggested.
>> > > >
>> > > > others?
>> > > >
>> > > > thanks
>> > > > joe
>> > > >
>> > > > On Feb 10, 2018 7:30 AM, "Mike Thomsen" <mikerthomsen@gmail.com>
>> wrote:
>> > > >
>> > > > > I'm just a community contributor, so take that FWIW, but a
>> compromise
>> > > > might
>> > > > > be to publish the Scala code as separate maven modules to maven
>> central
>> > > > and
>> > > > > then submit a thoroughly tested processor written in Java. As
long
>> as
>> > > you
>> > > > > have enough unit and integration tests to give strong coverage,
I
>> > > > wouldn't
>> > > > > imagine anyone here would have issues reviewing it. If the tests
>> fail
>> > > > > because of code issues in the external dependencies, the obvious
>> answer
>> > > > is
>> > > > > to just hold the PR until the tests pass.
>> > > > >
>> > > > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <
>> > > Adam.Weiss@perkinelmer.com
>> > > > > wrote:
>> > > > >
>> > > > > > Devs,
>> > > > > >
>> > > > > > I have some interns starting with my team and we use Scala
>> internally
>> > > > for
>> > > > > > our work.
>> > > > > > If I wanted to have them work to contribute some new processors,
>> > > would
>> > > > > > they have to be Java to be included with the base distribution
or
>> > > could
>> > > > > > they use Scala as long as it fit within your current build
and
>> test
>> > > > > process?
>> > > > > >
>> > > > > > Thanks,
>> > > > > > -Adam
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > --
>> > Sent from Gmail Mobile
>>

Mime
View raw message