spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maciej Szymkiewicz <mszymkiew...@gmail.com>
Subject Re: Python friendly API for Spark 3.0
Date Sat, 15 Sep 2018 14:49:44 GMT
For the reference I raised question of Python 2 support before -
http://apache-spark-developers-list.1001551.n3.nabble.com/Future-of-the-Python-2-support-td20094.html



On Sat, 15 Sep 2018 at 15:14, Alexander Shorin <kxepal@apache.org> wrote:

> What's the release due for Apache Spark 3.0? Will it be tomorrow or
> somewhere at the middle of 2019 year?
>
> I think we shouldn't care much about Python 2.x today, since quite
> soon it support turns into pumpkin. For today's projects I hope nobody
> takes into account support of 2.7 unless there is some legacy still to
> carry on, but do we want to take that baggage into Apache Spark 3.x
> era? The next time you may drop it would be only 4.0 release because
> of breaking change.
>
> --
> ,,,^..^,,,
> On Sat, Sep 15, 2018 at 2:21 PM Maciej Szymkiewicz
> <mszymkiewicz@gmail.com> wrote:
> >
> > There is no need to ditch Python 2. There are basically two options
> >
> > Use stub files and limit yourself to support only Python 3 support.
> Python 3 users benefit from type hints, Python 2 users don't, but no core
> functionality is affected. This is the approach I've used with
> https://github.com/zero323/pyspark-stubs/.
> > Use comment based inline syntax or stub files and don't use backward
> incompatible features (primarily typing module -
> https://docs.python.org/3/library/typing.html). Both Python 2 and 3 is
> supported, but more advanced components are not. Small win for Python 2
> users, moderate loss for Python 3 users.
> >
> >
> >
> > On Sat, 15 Sep 2018 at 02:38, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
> >>
> >> Do we need to ditch Python 2 support to provide type hints? I don’t
> think so.
> >>
> >> Python lets you specify typing stubs that provide the same benefit
> without forcing Python 3.
> >>
> >> 2018년 9월 14일 (금) 오후 8:01, Holden Karau <holden@pigscanfly.ca>님이
작성:
> >>>
> >>>
> >>>
> >>> On Fri, Sep 14, 2018, 3:26 PM Erik Erlandson <eerlands@redhat.com>
> wrote:
> >>>>
> >>>> To be clear, is this about "python-friendly API" or "friendly python
> API" ?
> >>>
> >>> Well what would you consider to be different between those two
> statements? I think it would be good to be a bit more explicit, but I don't
> think we should necessarily limit ourselves.
> >>>>
> >>>>
> >>>> On the python side, it might be nice to take advantage of static
> typing. Requires python 3.6 but with python 2 going EOL, a spark-3.0 might
> be a good opportunity to jump the python-3-only train.
> >>>
> >>> I think we can make types sort of work without ditching 2 (the types
> only would work in 3 but it would still function in 2). Ditching 2 entirely
> would be a big thing to consider, I honestly hadn't been considering that
> but it could be from just spending so much time maintaining a 2/3 code
> base. I'd suggest reaching out to to user@ before making that kind of
> change.
> >>>>
> >>>>
> >>>> On Fri, Sep 14, 2018 at 12:15 PM, Holden Karau <holden@pigscanfly.ca>
> wrote:
> >>>>>
> >>>>> Since we're talking about Spark 3.0 in the near future (and since
> some recent conversation on a proposed change reminded me) I wanted to open
> up the floor and see if folks have any ideas on how we could make a more
> Python friendly API for 3.0? I'm planning on taking some time to look at
> other systems in the solution space and see what we might want to learn
> from them but I'd love to hear what other folks are thinking too.
> >>>>>
> >>>>> --
> >>>>> Twitter: https://twitter.com/holdenkarau
> >>>>> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9
> >>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> >>>>
> >>>>
> >
> >
>

Mime
View raw message