nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Bateman <r...@windofkeltia.com>
Subject Re: Will you accept contributions in Scala?
Date Tue, 13 Feb 2018 17:15:01 GMT
I completely second Mike's sentiment here as well as cautionary 
statements by other contributors over this thread's history (Matt 
Burgess' post in particular). Besides fueling flame wars and religious 
inquisition, you cannot open the door to Scala without opening it to 
other (JVM languages at least) and there's no umbrella that will shield 
you from the chaos doing so will unleash rendering NiFi difficult to 
develop to (and maybe even to deploy). As it is presently, all 
JVM-language developers are able to consume NiFi easily for development 
and to deal with their own deployment issues.

Russ

On 02/13/2018 10:06 AM, Mike Thomsen wrote:
> Milan,
>
> I don't think you can do that without creating a lot of fuel for a flame
> war. I have personally never met a Scala developer who was incapable of
> writing decent Java. The same is true of Groovy. I think anyone who finds
> the requirement to use Java over their language of choice to be a deal
> breaker on contributing is probably someone unlikely to be more help than
> trouble anyway.
>
> Mike
>
> On Tue, Feb 13, 2018 at 11:35 AM, Milan Das <mdas@interset.com> wrote:
>
>> I think we should not add blindly any language but should be open to add
>> couple of language like Scala.
>> In Bigdata world Scala/Python/Java are widely accepted.
>>
>> Thanks,
>> Milan Das
>> Interset
>>
>> On 2/13/18, 10:20 AM, "Weiss, Adam" <Adam.Weiss@PERKINELMER.COM> wrote:
>>
>>      I think it makes the most sense to me for us to publish a separate
>> repo with a module and nar build for now and post when it's available in
>> the users group.
>>
>>      Thanks for the discussion everyone, hopefully we can start making some
>> helpful contributions soon.
>>
>>      -Adam
>>
>>
>>      On 2018/02/10 23:43:31, Tony Kurc <t...@gmail.com<mailto:t...@
>> gmail.com>> wrote:
>>      > It is like Matt read my mind.>
>>      >
>>      > On Sat, Feb 10, 2018 at 6:26 PM, Matt Burgess <ma...@apache.org
>> <mailto:ma...@apache.org>> wrote:>
>>      >
>>      > > I'm fine with a vote, but I'll be voting to keep Java as the
>> single>
>>      > > language for the (non-test) code. I share the same concerns as
>> many of>
>>      > > the other folks as far as accepting other languages, it's mainly
>> the>
>>      > > "slippery slope" argument that I don't want to turn into a>
>>      > > JVM-language flame war.  If Scala, why not Groovy? Certainly the>
>>      > > syntax is closer to Java, and the community has accepted it as a
>> valid>
>>      > > language for writing unit tests, although we stopped short for>
>>      > > allowing it for the deployable NiFi codebase, for the same reasons>
>>      > > IIRC.  If Scala and/or Groovy, why not Kotlin?  The same argument>
>>      > > (albeit more tenuous) goes for Clojure and just about every other
>> JVM>
>>      > > language (although I don't expect a call for LuaJ processors lol).>
>>      > >>
>>      > > Whether we decide to support various languages ad-hoc or not, I
>> would>
>>      > > strenuously object to multiple/hybrid build systems for the
>> deployed>
>>      > > artifacts. If I could switch NiFi completely to Gradle I would,
>> but I>
>>      > > realize there are good reasons for not doing so (yet?) in the
>> Apache>
>>      > > NiFi community, and I would never want any hybrid Maven/Gradle
>> build>
>>      > > for the deployable code, likewise for SBT, Leiningen, etc. With a>
>>      > > custom Mojo for Maven NAR builds, and the complexity for hybrid
>> builds>
>>      > > in general, I think this would create a maintenance nightmare.>
>>      > >>
>>      > > The language thing is a tough decision though, it's not awesome
>> that>
>>      > > specifying a single language can be a barrier to a more diverse>
>>      > > community, certainly Scala-based bundles would be more than
>> welcome in>
>>      > > the overall NiFi ecosystem, I just think the cons outweigh the
>> pros>
>>      > > for the baseline code. I've written Groovy processors/NARs using>
>>      > > Gradle as the build system, and I'm good with keeping them in my
>> own>
>>      > > repo, especially when the Extension Registry becomes a thing. I
>> can>
>>      > > see the Extension Registry perhaps making this a moot point, but>
>>      > > clearly we need to have this discussion in the meantime.>
>>      > >>
>>      > > Regards,>
>>      > > Matt>
>>      > >>
>>      > >>
>>      > > On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande <ap...@gmail.com
>> <mailto:ap...@gmail.com>> wrote:>
>>      > > > Wasn't there a warning trigger about the NiFi distro size from
>> Apache>
>>      > > > recently? IMO, before talking alternative languages, solve the
>> modularity>
>>      > > > and NAR distribution problem. I think the implementation of a
>> module>
>>      > > won't>
>>      > > > matter much then, the point being not everything has to go in
>> the core,>
>>      > > > base distribution, but can still be easily sourced from a known
>> repo, for>
>>      > > > example.>
>>      > > >>
>>      > > > I have a feeling NiFi 1.6+ can be approaching 2GB distro size
>> soon :)>
>>      > > >>
>>      > > > Andrew>
>>      > > >>
>>      > > > On Sat, Feb 10, 2018, 5:12 PM Joey Frazee <jo...@icloud.com
>> <mailto:jo...@icloud.com>>>
>>      > > wrote:>
>>      > > >>
>>      > > >> This probably necessitates a vote, yeah?>
>>      > > >>>
>>      > > >> Frankly, I’m usually happier writing Scala, and I’ve
not
>> encountered any>
>>      > > >> problems using processors written in Scala, but I think it’ll
>> be>
>>      > > important>
>>      > > >> to tread lightly.>
>>      > > >>>
>>      > > >> There’s a few things that pop into my head:>
>>      > > >>>
>>      > > >> - Maintainability and reviewability. A very very good Java
>> developer>
>>      > > need>
>>      > > >> not, by definition, either know how to write or identify
good
>> Scala or>
>>      > > spot>
>>      > > >> problems and bugs.>
>>      > > >> - Every Scala processor would either end up with a 5MB
>> scala-lang.jar>
>>      > > >> packaged into the .nar or we’d have to start including
it in
>> the core>
>>      > > >> somewhere, if it’s not. It’s possible it might have already
>> gotten>
>>      > > pulled>
>>      > > >> up from other dependencies.>
>>      > > >> - Style. There’s a tremendous amount of variation in Scala
>> style because>
>>      > > >> of its type system, implicits, macros, and functional nature.
>> There are>
>>      > > >> very good people out there that can write good Scala that
isn’t>
>>      > > readable by>
>>      > > >> the 99%.>
>>      > > >> - Binary compatibility. Scala tends to be a little more brazen
>> about>
>>      > > >> breaking binary compatibility in major releases and those
>> happen a bit>
>>      > > more>
>>      > > >> often than with Java. That’s not a problem for any potential
>> source>
>>      > > code in>
>>      > > >> the project, but it could present some dependency issues
>> someday.>
>>      > > >> - Testing. There’s N > 1 test frameworks and testing
styles
>> within>
>>      > > those,>
>>      > > >> so there’s a lot of options for introducing more variability
>> into the>
>>      > > tests.>
>>      > > >> - NiFi uses a lot of statics in setting up properties and
>> relationships>
>>      > > >> and the like, and idiomatic Scala approaches that stuff a
bit>
>>      > > differently,>
>>      > > >> so it’ll be necessary to impose some style guidelines so
there
>> isn’t too>
>>      > > >> much variation.>
>>      > > >>>
>>      > > >> That said, there are some things that won’t be problematic:>
>>      > > >>>
>>      > > >> - As mentioned, processors written in Scala do just work.>
>>      > > >> - The scala-maven-plugin works just fine allowing mixed
>> Java-Scala>
>>      > > >> projects (btw, it’d probably be not super great to do mixed
>> Java-Scala>
>>      > > and>
>>      > > >> mixed Maven-SBT though).>
>>      > > >> - A lot of the above concerns could be addressed by having
>> clear style>
>>      > > >> guidelines.>
>>      > > >>>
>>      > > >> Another thing: most of the projects I see deliver separate
jars
>> for>
>>      > > Scala>
>>      > > >> components are delivering idiomatic APIs wrapping Java (or
vice
>> versa).>
>>      > > I>
>>      > > >> think publishing   a separate set of jars/nars for stuff
>> written in>
>>      > > Scala>
>>      > > >> would be odd since here it’d mostly be processors with
new
>> functionality>
>>      > > >> and not functionality for using Scala. I could imagine a
lib of>
>>      > > implicits,>
>>      > > >> traits, classes that could make the Scala development more
>> enjoyable.>
>>      > > That>
>>      > > >> probably would make sense to deliver that way.>
>>      > > >>>
>>      > > >> -joey>
>>      > > >>>
>>      > > >> On Feb 10, 2018, 10:33 AM -0600, Bryan Bende <bb...@gmail.com
>> <mailto:bb...@gmail.com>>, wrote:>
>>      > > >> > I agree more with Andy about sticking with Java. The
more
>> varying>
>>      > > >> languages>
>>      > > >> > used, the more challenging it is to maintain. Once the
code
>> is part of>
>>      > > >> the>
>>      > > >> > Apache NiFi git repo, it is now the responsibility of
the
>> committers>
>>      > > and>
>>      > > >> > PMC members to maintain it.>
>>      > > >> >>
>>      > > >> > I’d even say I am somewhat against the groovy/Spock
test code
>> that>
>>      > > Andy>
>>      > > >> > mentioned. I have frequently spent hours trying to fix
a
>> Spock test>
>>      > > that>
>>      > > >> > broke from something I was working on. Every committer
is
>> familiar>
>>      > > with>
>>      > > >> > JUnit, but only a couple know Spock. Just using this
as an
>> example>
>>      > > that>
>>      > > >> > every committer knows Java, but only a couple probably
know
>> Scala,>
>>      > > >> Clojure,>
>>      > > >> > etc.>
>>      > > >> >>
>>      > > >> > On Sat, Feb 10, 2018 at 10:25 AM Jeff <jt...@gmail.com
>> <mailto:jt...@gmail.com>> wrote:>
>>      > > >> >>
>>      > > >> > > +1 to Joe's response. If you can develop a component
in
>> Groovy or>
>>      > > Scala>
>>      > > >> > > (or Clojure!) more quickly/comfortably, or if allowing
>> components>
>>      > > >> written>
>>      > > >> > > in other languages would encourage people to contribute
>> more, I'm>
>>      > > all>
>>      > > >> for>
>>      > > >> > > it.>
>>      > > >> > >>
>>      > > >> > > On Sat, Feb 10, 2018 at 7:42 AM Joe Witt <jo...@gmail.com
>> <mailto:jo...@gmail.com>>>
>>      > > wrote:>
>>      > > >> > >>
>>      > > >> > > > i personally would be ok with it for an
>> extension/processor>
>>      > > provided>
>>      > > >> it>
>>      > > >> > > > integrates well with the build.>
>>      > > >> > > >>
>>      > > >> > > > i would agree with andys view for core framework
stuff
>> but for>
>>      > > >> > > extensions i>
>>      > > >> > > > think we can do it like mikethomsen suggested.>
>>      > > >> > > >>
>>      > > >> > > > others?>
>>      > > >> > > >>
>>      > > >> > > > thanks>
>>      > > >> > > > joe>
>>      > > >> > > >>
>>      > > >> > > > On Feb 10, 2018 7:30 AM, "Mike Thomsen" <mi...@gmail.com
>> <mailto:mi...@gmail.com>>>
>>      > > >> wrote:>
>>      > > >> > > >>
>>      > > >> > > > > I'm just a community contributor, so
take that FWIW,
>> but a>
>>      > > >> compromise>
>>      > > >> > > > might>
>>      > > >> > > > > be to publish the Scala code as separate
maven modules
>> to maven>
>>      > > >> central>
>>      > > >> > > > and>
>>      > > >> > > > > then submit a thoroughly tested processor
written in
>> Java. As>
>>      > > long>
>>      > > >> as>
>>      > > >> > > you>
>>      > > >> > > > > have enough unit and integration tests
to give strong
>> coverage,>
>>      > > I>
>>      > > >> > > > wouldn't>
>>      > > >> > > > > imagine anyone here would have issues
reviewing it. If
>> the tests>
>>      > > >> fail>
>>      > > >> > > > > because of code issues in the external
dependencies,
>> the obvious>
>>      > > >> answer>
>>      > > >> > > > is>
>>      > > >> > > > > to just hold the PR until the tests pass.>
>>      > > >> > > > >>
>>      > > >> > > > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss,
Adam <>
>>      > > >> > > Adam.Weiss@perkinelmer.com<mailto:Adam.Weiss@perkinelmer.
>> com>>
>>      > > >> > > > > wrote:>
>>      > > >> > > > >>
>>      > > >> > > > > > Devs,>
>>      > > >> > > > > >>
>>      > > >> > > > > > I have some interns starting with
my team and we use
>> Scala>
>>      > > >> internally>
>>      > > >> > > > for>
>>      > > >> > > > > > our work.>
>>      > > >> > > > > > If I wanted to have them work to
contribute some new>
>>      > > processors,>
>>      > > >> > > would>
>>      > > >> > > > > > they have to be Java to be included
with the base>
>>      > > distribution or>
>>      > > >> > > could>
>>      > > >> > > > > > they use Scala as long as it fit
within your current
>> build and>
>>      > > >> test>
>>      > > >> > > > > process?>
>>      > > >> > > > > >>
>>      > > >> > > > > > Thanks,>
>>      > > >> > > > > > -Adam>
>>      > > >> > > > > >>
>>      > > >> > > > >>
>>      > > >> > > >>
>>      > > >> > >>
>>      > > >> > -->
>>      > > >> > Sent from Gmail Mobile>
>>      > > >>>
>>      > >>
>>      >
>>
>>      Adam Weiss | Senior Manager, Digital Solutions R&D
>>      PerkinElmer | Innovating for a Healthier World
>>      adam.weiss@perkinelmer.com<mailto:adam.weiss@perkinelmer.com>
>>      Mobile: +1 716.429.7324
>>      77 Goodell St. Suite 470, Buffalo, NY, 14203
>>      www.perkinelmer.com
>>
>>
>>
>>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message