spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Shorin <>
Subject Re: Python friendly API for Spark 3.0
Date Sat, 15 Sep 2018 13:15:19 GMT
What's the release due for Apache Spark 3.0? Will it be tomorrow or
somewhere at the middle of 2019 year?

I think we shouldn't care much about Python 2.x today, since quite
soon it support turns into pumpkin. For today's projects I hope nobody
takes into account support of 2.7 unless there is some legacy still to
carry on, but do we want to take that baggage into Apache Spark 3.x
era? The next time you may drop it would be only 4.0 release because
of breaking change.

On Sat, Sep 15, 2018 at 2:21 PM Maciej Szymkiewicz
<> wrote:
> There is no need to ditch Python 2. There are basically two options
> Use stub files and limit yourself to support only Python 3 support. Python 3 users benefit
from type hints, Python 2 users don't, but no core functionality is affected. This is the
approach I've used with
> Use comment based inline syntax or stub files and don't use backward incompatible features
(primarily typing module - Both Python 2 and
3 is supported, but more advanced components are not. Small win for Python 2 users, moderate
loss for Python 3 users.
> On Sat, 15 Sep 2018 at 02:38, Nicholas Chammas <> wrote:
>> Do we need to ditch Python 2 support to provide type hints? I don’t think so.
>> Python lets you specify typing stubs that provide the same benefit without forcing
Python 3.
>> 2018년 9월 14일 (금) 오후 8:01, Holden Karau <>님이
>>> On Fri, Sep 14, 2018, 3:26 PM Erik Erlandson <> wrote:
>>>> To be clear, is this about "python-friendly API" or "friendly python API"
>>> Well what would you consider to be different between those two statements? I
think it would be good to be a bit more explicit, but I don't think we should necessarily
limit ourselves.
>>>> On the python side, it might be nice to take advantage of static typing.
Requires python 3.6 but with python 2 going EOL, a spark-3.0 might be a good opportunity to
jump the python-3-only train.
>>> I think we can make types sort of work without ditching 2 (the types only would
work in 3 but it would still function in 2). Ditching 2 entirely would be a big thing to consider,
I honestly hadn't been considering that but it could be from just spending so much time maintaining
a 2/3 code base. I'd suggest reaching out to to user@ before making that kind of change.
>>>> On Fri, Sep 14, 2018 at 12:15 PM, Holden Karau <>
>>>>> Since we're talking about Spark 3.0 in the near future (and since some
recent conversation on a proposed change reminded me) I wanted to open up the floor and see
if folks have any ideas on how we could make a more Python friendly API for 3.0? I'm planning
on taking some time to look at other systems in the solution space and see what we might want
to learn from them but I'd love to hear what other folks are thinking too.
>>>>> --
>>>>> Twitter:
>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>> YouTube Live Streams:

To unsubscribe e-mail:

View raw message