ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: The Spark 2.4 support
Date Mon, 18 Nov 2019 20:47:32 GMT
Alexey, thanks for the details and for reaching out this milestone with the
2.4 support.

Generally, I would advise us to merge the changes to the master only after
we confirm the failing tests are not regressions. We should either remove
them or replace them with some others or just fix.

-
Denis


On Mon, Nov 18, 2019 at 10:06 AM Alexey Zinoviev <zaleslaw.sin@gmail.com>
wrote:

> Right, a few tests from 200 are failed due to known issue and couldnt be
> fixed immediately, related to rare cases. These tests are copies of 2.3
> tests and part of them could have no meaning for 2.4 due to Spark changed
> behaviour.
>
> пн, 18 нояб. 2019 г., 19:42 Denis Magda <dmagda@apache.org>:
>
> > Alexey,
> >
> > Please help to understand what it means that 2.4 integration supports
> "95%
> > of tests of 2.3". Does it mean that 5% of existing tests are failing and,
> > basically, need to be fixed?
> >
> > -
> > Denis
> >
> >
> > On Mon, Nov 18, 2019 at 6:52 AM Alexey Zinoviev <zaleslaw.sin@gmail.com>
> > wrote:
> >
> > > Dear Nikolay Izhikov, I've recreated the PR for 2.4 initial support
> > >
> > > The last commit
> > >
> > >
> >
> https://github.com/apache/ignite/pull/7058/commits/60386802299deedc6ed60bf4736e922201a67fb8
> > > contains
> > > real changes from Spark 2.3
> > >
> > > I suggest to merge to master this initial solution with 95% support of
> > > Spark 2.4 and continue work on known issues listed in JIRA
> > >
> > > This solution supports the new Spark version for all examples and 95%
> of
> > > tests of 2.3.
> > >
> > > вт, 1 окт. 2019 г. в 08:48, Ivan Pavlukhin <vololo100@gmail.com>:
> > >
> > > > Alexey, Nikolay,
> > > >
> > > > Thank you for sharing details!
> > > >
> > > > вт, 1 окт. 2019 г. в 07:42, Alexey Zinoviev <zaleslaw.sin@gmail.com
> >:
> > > > >
> > > > > Great talk and paper, I've learnt it last year
> > > > >
> > > > > пн, 30 сент. 2019 г., 21:42 Nikolay Izhikov <nizhikov@apache.org>:
> > > > >
> > > > > > Yes, I can :)
> > > > > >
> > > > > > В Пн, 30/09/2019 в 11:40 -0700, Denis Magda пишет:
> > > > > > > Nikolay,
> > > > > > >
> > > > > > > Would you be able to review the changes? I'm not sure there
is
> a
> > > > better
> > > > > > candidate for now.
> > > > > > >
> > > > > > > -
> > > > > > > Denis
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Sep 30, 2019 at 11:01 AM Nikolay Izhikov <
> > > > nizhikov@apache.org>
> > > > > > wrote:
> > > > > > > > Hello, Ivan.
> > > > > > > >
> > > > > > > > I had a talk about internals of Spark integration
in Ignite.
> > > > > > > > It answers on question why we should use Spark internals.
> > > > > > > >
> > > > > > > > You can take a look at my meetup talk(in Russian)
[1] or read
> > an
> > > > > > article if you prefer text [2].
> > > > > > > >
> > > > > > > > [1] https://www.youtube.com/watch?v=CzbAweNKEVY
> > > > > > > > [2] https://habr.com/ru/company/sberbank/blog/427297/
> > > > > > > >
> > > > > > > > В Пн, 30/09/2019 в 20:29 +0300, Alexey Zinoviev
пишет:
> > > > > > > > > Yes, as I understand it uses Spark internals
from the first
> > > > commit)))
> > > > > > > > > The reason - we take Spark SQL query execution
plan and try
> > to
> > > > > > execute it
> > > > > > > > > on Ignite cluster
> > > > > > > > > Also we inherit a lot of Developer API related
classes that
> > > > could be
> > > > > > > > > unstable. Spark has no good point for extension
and this
> is a
> > > > reason
> > > > > > why we
> > > > > > > > > should go deeper
> > > > > > > > >
> > > > > > > > > пн, 30 сент. 2019 г. в 20:17, Ivan Pavlukhin
<
> > > > vololo100@gmail.com>:
> > > > > > > > >
> > > > > > > > > > Hi Alexey,
> > > > > > > > > >
> > > > > > > > > > As an external watcher very far from Ignite
Spark
> > > integration I
> > > > > > would
> > > > > > > > > > like to ask a humble question for my understanding.
Why
> > this
> > > > > > > > > > integration uses Spark internals? Is it
a common approach
> > for
> > > > > > > > > > integrating with Spark?
> > > > > > > > > >
> > > > > > > > > > пн, 30 сент. 2019 г. в 16:17, Alexey
Zinoviev <
> > > > > > zaleslaw.sin@gmail.com>:
> > > > > > > > > > >
> > > > > > > > > > > Hi, Igniters
> > > > > > > > > > > I've started the work on the Spark
2.4 support
> > > > > > > > > > >
> > > > > > > > > > > We started the discussion here, in
> > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-12054
> > > > > > > > > > >
> > > > > > > > > > > The Spark internals were totally refactored
between 2.3
> > and
> > > > 2.4
> > > > > > versions,
> > > > > > > > > > > main changes touches
> > > > > > > > > > >
> > > > > > > > > > >    - External catalog and listeners
refactoring
> > > > > > > > > > >    - Changes of HAVING operator semantic
support
> > > > > > > > > > >    - Push-down NULL filters generation
in JOIN plans
> > > > > > > > > > >    - minor changes in Plan Generation
that should be
> > > adopted
> > > > in
> > > > > > our
> > > > > > > > > > >    integration module
> > > > > > > > > > >
> > > > > > > > > > > I propose the initial solution here
via creation of new
> > > > module
> > > > > > spark-2.4
> > > > > > > > > > > here
> https://issues.apache.org/jira/browse/IGNITE-12247
> > > and
> > > > > > addition of
> > > > > > > > > >
> > > > > > > > > > new
> > > > > > > > > > > profile spark-2.4 (to avoid possible
clashes with
> another
> > > > spark
> > > > > > versions)
> > > > > > > > > > >
> > > > > > > > > > > Also I've transformed ticket to an
Umbrella ticket and
> > > > created a
> > > > > > few
> > > > > > > > > > > tickets for muted tests (around 7 from
211 tests are
> > muted
> > > > now)
> > > > > > > > > > >
> > > > > > > > > > > Please, if somebody interested in it,
make an initial
> > > review
> > > > of
> > > > > > modular
> > > > > > > > > > > ignite structure and changes (without
deep diving into
> > > Spark
> > > > > > code).
> > > > > > > > > > >
> > > > > > > > > > > And yes, the proposed code is a copy-paste
of
> > spark-ignite
> > > > > > module with a
> > > > > > > > > > > few fixes
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Best regards,
> > > > > > > > > > Ivan Pavlukhin
> > > > > > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message