metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Foley <ma...@apache.org>
Subject Re: [DISCUSS] Next Release (0.3.1) Content
Date Fri, 03 Feb 2017 00:08:11 GMT
Thanks, Casey.  Btw, I’m not far enough along on METRON-322 (batch timeout flush), so it
will have to wait for next cycle.
--Matt

On 2/2/17, 2:03 PM, "Casey Stella" <cestella@gmail.com> wrote:

    Just a quick release update.  As of now, we are waiting on
    
       - METRON-660 to get reviewed and make it in
       - METRON-692 to get our upgrade.md completed for this release
    
    Mike Miklavcic, you sent an email to the legal-discuss about our kraken
    dependency and it looked like we didn't have to change, but could you
    comment on this thread in the dev list so I know if we need to wait for
    METRON-650.
    
    Casey
    
    On Thu, Feb 2, 2017 at 4:58 PM, Casey Stella <cestella@gmail.com> wrote:
    
    > Ok, I've created the upgrading document for 0.3.0 to 0.3.1 and included
    > the things that I know about and the things Jon mentioned here.  Please, if
    > you have knowledge of other breaking/non-compatible changes between the
    > 0.3.0 release and master, comment on this PR (https://github.com/apache/
    > incubator-metron/pull/437) and I will incorporate them.
    >
    > On Fri, Jan 27, 2017 at 10:04 AM, Zeolla@GMail.com <zeolla@gmail.com>
    > wrote:
    >
    >> To start I was mostly concerned with having a per-version list of
    >> non-backwards-compatible changes, so upgrades that may skip a version of
    >> two can look at what all may be impacted.  We should also probably
    >> document
    >> any sort of upgrade flaws as well, such as METRON-447
    >> <https://issues.apache.org/jira/browse/METRON-447>, METRON-448
    >> <https://issues.apache.org/jira/browse/METRON-448>, etc.  I do think that
    >> we should have a more rigorous document, but I wouldn't push that for the
    >> 0.3.1 release.  I see that (along with the Management UI, API, etc.) as
    >> key
    >> (required) components of a 1.0 release.  I'd just like to see the
    >> foundation begin to be laid and iterated on.
    >>
    >> That said, this probably constitutes a mention in the development
    >> guidelines
    >> <https://cwiki.apache.org/confluence/pages/viewpage.action?
    >> pageId=61332235>
    >> once
    >> it's in master.
    >>
    >> Jon
    >>
    >> On Fri, Jan 27, 2017 at 9:05 AM Casey Stella <cestella@gmail.com> wrote:
    >>
    >> > I should add, you may be thinking something more rigorous and
    >> > step-by-step.  If so, you think you might be interested in volunteering
    >> to
    >> > do a first draft as a PR that we can adjust?
    >> >
    >> > On Fri, Jan 27, 2017 at 9:01 AM, Casey Stella <cestella@gmail.com>
    >> wrote:
    >> >
    >> > > So, I agree with the Upgrading.md and I was going to submit a PR at
    >> least
    >> > > to describe the the changes to indexing configurations that I made
    >> during
    >> > > the 3.0.1 release.
    >> > >
    >> > >
    >> > > On Thu, Jan 26, 2017 at 10:50 PM, Zeolla@GMail.com <zeolla@gmail.com>
    >> > > wrote:
    >> > >
    >> > >> I haven't had a chance to look through the unresolved JIRAs but
I did
    >> > want
    >> > >> to mention a few quick things.
    >> > >>
    >> > >> First, when we released 0.3.0 and dropped the BETA flag, one of
the
    >> > things
    >> > >> that was discussed was putting together a method of documenting
    >> upgrades
    >> > >> from one version to the next.  As one of the first steps toward
    >> making
    >> > >> that
    >> > >> a reasonable process, I think we should assemble more detailed
    >> release
    >> > >> notes, especially outlining non-backwards compatible changes. 
In the
    >> > >> "[DISCUSS] Next Release Name" email thread Kyle Richardson suggested
    >> we
    >> > >> use
    >> > >> "UPGRADING.md" to do this, and I still agree with that thought,
but
    >> I'm
    >> > >> open to alternatives.
    >> > >>
    >> > >> Separately, a *nice to have* would be *METRON-660*, which was
    >> discussed
    >> > in
    >> > >> the "[PROPOSAL] up-to-date versioned documentation" thread, to
give
    >> us
    >> > >> some
    >> > >> cleaner documentation using the existing READMEs.  I'd be happy
to
    >> help
    >> > >> with this one, I'm just not sure what the next steps are, aside
from
    >> the
    >> > >> start that Matt has here
    >> > >> <https://github.com/mattf-horton/incubator-metron/tree/METRON-660>.
    >> > >>
    >> > >> My last *nice to have* is *METRON-635*, which I have a *PR open
for
    >> here
    >> > >> <https://github.com/apache/incubator-metron/pull/411>*. 
If I could
    >> get
    >> > >> someone else to reproduce the error that I'm seeing I would be
happy
    >> to
    >> > >> pursue additional testing, troubleshooting, etc.  I've seen others
    >> > report
    >> > >> the same issue
    >> > >> <https://community.hortonworks.com/search.html?f=&type=
    >> > >> question&redirect=search%2Fsearch&sort=relevance&q=scp_if_ssh>
    >> > >> on the HCC boards, so I'm fairly confident that it is not an issue
    >> with
    >> > my
    >> > >> local environment.
    >> > >>
    >> > >> Jon
    >> > >>
    >> > >> On Thu, Jan 26, 2017 at 6:18 PM Casey Stella <cestella@gmail.com>
    >> > wrote:
    >> > >>
    >> > >> I took the liberty of adding the pull requests since the new year
    >> into
    >> > the
    >> > >> in-progress list in the previous email.  If you own one of these
and
    >> do
    >> > >> not
    >> > >> believe that you can complete the review in the next week or so,
let
    >> me
    >> > >> know and we can remove.  The only one of these that I see as
    >> mandatory
    >> > is
    >> > >> METRON-650 because the reliance on the opensoc github repo might
    >> cause
    >> > >> issues with the release being acceptable (see discussion by Mike
    >> > Miklavcic
    >> > >> with the mentors a couple weeks ago).
    >> > >>
    >> > >> On Thu, Jan 26, 2017 at 6:06 PM, Casey Stella <cestella@gmail.com>
    >> > wrote:
    >> > >>
    >> > >> > Hello Everyone!
    >> > >> >
    >> > >> > It's been almost 2 months since the last major release and
I think
    >> it
    >> > >> > might be time to do a minor release of 0.3.1. The purpose
of this
    >> > email
    >> > >> is
    >> > >> > multifold:
    >> > >> >
    >> > >> >    - to take stock of what we have already committed
    >> > >> >    - figure out what (if anything) is missing to make a release
    >> that
    >> > >> >    we're proud of
    >> > >> >    - find volunteers for the missing items
    >> > >> >
    >> > >> > For those who get email alerts on the JIRA changes, it should
be no
    >> > >> > surprise that I have gone through and put the committed items
in
    >> the
    >> > >> 0.3.1
    >> > >> > release bucket.  So, let's take stock of what we have.
    >> > >> >
    >> > >> > *What's made it so far into the next release*
    >> > >> >
    >> > >> >
    >> > >> >
    >> > >> >    - METRON-283 Migrate Geo Enrichment outside of MySQL
    >> (justinleet)
    >> > >> >    closes apache/incubator-metron#421
    >> > >> >    - METRON-668: Remove the "tickUpdate" profile config and
make
    >> the
    >> > >> >    "init" phase not reset variables closes
    >> apache/incubator-metron#420
    >> > >> >    - METRON-672: SolrIndexingIntegrationTest fails intermittently
    >> > closes
    >> > >> >    apache/incubator-metron#424
    >> > >> >    - METRON-664: Make the index configuration per-writer with
    >> > >> >    enabled/disabled closes apache/incubator-metron#419
    >> > >> >    - METRON-600: Fix Metron Website (iraghumitra via cestella)
    >> closes
    >> > >> >    apache/incubator-metron#399
    >> > >> >    - METRON-666 Fix javadoc doclint errors closes
    >> > >> >    apache/incubator-metron#418
    >> > >> >    - METRON-659 Emulate Sensors in Development Environments
    >> > (nickwallen)
    >> > >> >    closes apache/incubator-metron#417
    >> > >> >    - METRON-652: Extract indexing config from enrichment config
    >> closes
    >> > >> >    apache/incubator-metron#415
    >> > >> >    - METRON-654 Create RPM Installer for Profiler (nickwallen)
    >> closes
    >> > >> >    apache/incubator-metron#413
    >> > >> >    - METRON-656: Make Stellar 'in' closer to functioning like
    >> python
    >> > >> >    closes apache/incubator-metron#416
    >> > >> >    - METRON-532 Define Profile Period When Calling PROFILE_GET
    >> closes
    >> > >> >    apache/incubator-metron#414
    >> > >> >    - METRON-624: Updated Comparison/Equality Evaluations in
Stellar
    >> > >> >    closes apache/incubator-metron#404
    >> > >> >    - METRON-644 RPM builds only work with Docker for Mac
    >> > >> (kylerichardson)
    >> > >> >    closes apache/incubator-metron#409
    >> > >> >    - METRON-640: Add a Stellar function to compute shannon
entropy
    >> for
    >> > >> >    strings closes apache/incubator-metron#403
    >> > >> >    - METRON-637: Add a STATS_BIN function to Stellar. closes
    >> > >> >    apache/incubator-metron#401
    >> > >> >    - METRON-622 Create a Metron Docker Compose application
    >> > (merrimanr)
    >> > >> >    closes apache/incubator-metron#393
    >> > >> >    - METRON-643: Stellar function documentation needs to be
updated
    >> > >> >    closes apache/incubator-metron#407
    >> > >> >    - METRON-645 Unable to Start Fastcapa Test Environment
    >> (nickwallen)
    >> > >> >    closes apache/incubator-metron#410
    >> > >> >    - METRON-647 Parser unit test failures due to assumed year
    >> values
    >> > >> >    (justinleet) closes apache/incubator-metron#412
    >> > >> >    - METRON-639: The Network Stellar functions need to have
better
    >> > unit
    >> > >> >    testing closes apache/incubator-metron#402
    >> > >> >    - METRON-631 Broken link on fastcapa README (JonZeolla
via
    >> > >> nickwallen)
    >> > >> >    closes apache/incubator-metron#398
    >> > >> >    - METRON-625: Parser Filters cannot be specified from the
sensor
    >> > >> >    config closes apache/incubator-metron#396
    >> > >> >    - METRON-587 Integration tests should use common processor
    >> > >> >    implementations where possible (ottobackwards) closes
    >> > >> >    apache/incubator-metron#374
    >> > >> >    - METRON-616: Added support for float and long literals
in
    >> Stellar
    >> > >> >    closes apache/incubator-metron#392
    >> > >> >    - METRON-580: Remove hard-coded Metron version from Ambari
MPack
    >> > code
    >> > >> >    (mmiklavc) closes apache/incubator-metron#364
    >> > >> >    - METRON-618 Eliminate Javac Warnings in metron-analytics
    >> > >> (justinleet)
    >> > >> >    closes apache/incubator-metron#391
    >> > >> >    - METRON-364: Preserve Type for Arithmetic Expressions
in
    >> Stellar
    >> > >> >    closes apache/incubator-metron#390
    >> > >> >    - METRON-585 Pcap Replay Fails to Stop (nickwallen) closes
    >> > >> >    apache/incubator-metron#369
    >> > >> >    - METRON-586 STELLAR should have FILL_LEFT and FILL_RIGHT
    >> functions
    >> > >> >    (ottobackwards) closes apache/incubator-metron#370
    >> > >> >    - METRON-612 Clean up Error Prone generated warnings
    >> (justinleet)
    >> > >> >    closes apache/incubator-metron#389
    >> > >> >    - METRON-610: OnlineStatisticsProvider serialization is
broken
    >> at
    >> > >> >    random in the REPL closes apache/incubator-metron#388
    >> > >> >    - METRON-596 Eliminate Maven warnings from build (justinleet)
    >> > closes
    >> > >> >    apache/incubator-metron#378
    >> > >> >    - METRON-604 Mpack installs do not work on clean machines
due to
    >> > >> >    missing Elastic Curator repo (justinleet) closes
    >> > >> apache/incubator-metron#385
    >> > >> >    - METRON-607: Enrichment doc improvement and test cleanup
    >> > (mmiklavc)
    >> > >> >    closes apache/incubator-metron#386
    >> > >> >    - METRON-606 Profiler Overwriting Previously Written Values
    >> > >> >    (nickwallen) closes apache/incubator-metron#387
    >> > >> >    - METRON-591: Make the website in compliance with ASF standards
    >> > >> closes
    >> > >> >    apache/incubator-metron#373
    >> > >> >    - METRON-597 Sporadic Failures of Profiler Integration
Tests
    >> > >> >    (nickwallen) closes apache/incubator-metron#383
    >> > >> >    - METRON-595 Elasticsearch Writer only uses One IP Address
    >> > >> >    (JonathanRider via dlyle65535) closes
    >> apache/incubator-metron#379
    >> > >> >    - METRON-593 Enable an automated static analysis tool in
the
    >> build
    >> > >> >    (justinleet) closes apache/incubator-metron#376
    >> > >> >    - METRON-592 Change popup to Warning message in Assign
Master
    >> page
    >> > >> for
    >> > >> >    Client check (anandsubbu via dlyle65535) closes
    >> > >> apache/incubator-metron#375
    >> > >> >    - METRON-598 Add Kyle Richardson to committers (kylerichardson)
    >> > >> closes
    >> > >> >    apache/incubator-metron#382
    >> > >> >    - METRON-565: apps/metron/enrichment/indexed directory
path does
    >> > not
    >> > >> >    get created for metron cluster deployed via Ambari (mmiklavc)
    >> > closes
    >> > >> >    apache/incubator-metron#365
    >> > >> >    - METRON-588 Remove ASF License Headers from directly copied
    >> Chef
    >> > >> >    Bento Files closes apache/incubator-metron#372
    >> > >> >    - METRON-576 Stellar function resolution takes too long
on
    >> running
    >> > >> >    cluster (nickwallen) closes apache/incubator-metron#366
    >> > >> >    - METRON-570 Add Profiler Link to README (nickwallen) closes
    >> > >> >    apache/incubator-metron#371
    >> > >> >    - METRON-578 Missing error handling bolts for enrichment
and
    >> threat
    >> > >> >    intel closes apache/incubator-metron#363
    >> > >> >    - METRON-522 Need to mandate Client installation on Metron
Host
    >> > >> closes
    >> > >> >    apache/incubator-metron#367
    >> > >> >    - METRON-584 STELLAR should have a TO_LONG Function
    >> (ottobackwards)
    >> > >> >    closes apache/incubator-metron#368
    >> > >> >    - METRON-575 State from different profiles can be co-mingled
    >> > >> >    incorrectly (nickwallen) closes apache/incubator-metron#362
    >> > >> >    - METRON-538 Ensure proper shutdown, exceptions of integration
    >> test
    >> > >> >    components (ottobackwards) closes apache/incubator-metron#344
    >> > >> >    - METRON-567 Usernames as numerics strings attempted to
be
    >> parsed
    >> > and
    >> > >> >    compared as numbers (ottobackwards) closes
    >> > >> apache/incubator-metron#360
    >> > >> >    - METRON-562: Add rudimentary statistical outlier detection
    >> closes
    >> > >> >    apache/incubator-metron#352
    >> > >> >    - METRON-557 Create Stellar Functions for Kafka (nickwallen)
    >> closes
    >> > >> >    apache/incubator-metron#354
    >> > >> >    - METRON-556 Profiler - Refactor 'Group By' Calculation
    >> > (nickwallen)
    >> > >> >    closes apache/incubator-metron#348
    >> > >> >    - METRON-547 MaasIntegrationTests should integrate with
the
    >> Metron
    >> > >> >    Integration classes (ottobackwards) closes
    >> > >> apache/incubator-metron#350
    >> > >> >    - METRON-561 ShellEditor tests hang if vim is set as EDITOR
in
    >> your
    >> > >> >    profile (ottobackwards) closes apache/incubator-metron#351
    >> > >> >
    >> > >> > *What else will make it?*
    >> > >> >
    >> > >> > Ok, so now that we know what's in there already, I need to
ask your
    >> > help
    >> > >> > in determining what you want in the next release that is not
    >> currently
    >> > >> > being worked on.  In order to do that, please help me by
    >> responding to
    >> > >> this
    >> > >> > email and
    >> > >> >
    >> > >> >    - proposing and justifying a JIRA from the list of unresolved
    >> JIRAs
    >> > >> >    <
    >> > >> https://issues.apache.org/jira/browse/METRON-469?jql=project
    >> > >> %20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
    >> > >> 20fixVersion%20is%20EMPTY%20ORDER%20BY%20priority%20DESC>
    >> > >> that
    >> > >> >    are not currently scheduled and let me know which ones
should be
    >> > >> included.
    >> > >> >    Please include whether the JIRA is "Nice to Have" or "Critical
    >> to
    >> > >> Have" and
    >> > >> >    if you volunteer to do it.
    >> > >> >    - proposing and justifying that one of the in-progress
tasks
    >> > >> >    <
    >> > >> https://issues.apache.org/jira/browse/METRON-650?jql=project
    >> > >> %20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
    >> > >> 20fixVersion%20%3D%200.3.1%20ORDER%20BY%20priority%20DESC
    >> > >> >
    >> > >> >     should *not* be included for release
    >> > >> >
    >> > >> > I'd like to aim to get release candidates out as soon as possible,
    >> so
    >> > >> > let's look forward a vigorous discussion and a great release.
    >> > >> >
    >> > >> > Thanks for the contributions and for your continued engagement
in
    >> the
    >> > >> > Metron community.  If you see something that you don't like
in this
    >> > >> > process, please let me know.  Feedback is very welcome.
    >> > >> >
    >> > >> > Best,
    >> > >> >
    >> > >> > Casey
    >> > >> >
    >> > >>
    >> > >> --
    >> > >>
    >> > >> Jon
    >> > >>
    >> > >> Sent from my mobile device
    >> > >>
    >> > >
    >> > >
    >> >
    >> --
    >>
    >> Jon
    >>
    >> Sent from my mobile device
    >>
    >
    >
    




Mime
View raw message