spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maciej Szymkiewicz <mszymkiew...@gmail.com>
Subject Re: Python friendly API for Spark 3.0
Date Sat, 15 Sep 2018 11:20:52 GMT
There is no need to ditch Python 2. There are basically two options

   - Use stub files and limit yourself to support only Python 3 support.
   Python 3 users benefit from type hints, Python 2 users don't, but no core
   functionality is affected. This is the approach I've used with
   https://github.com/zero323/pyspark-stubs/.
   - Use comment based inline syntax or stub files and don't use backward
   incompatible features (primarily typing module -
   https://docs.python.org/3/library/typing.html). Both Python 2 and 3 is
   supported, but more advanced components are not. Small win for Python 2
   users, moderate loss for Python 3 users.



On Sat, 15 Sep 2018 at 02:38, Nicholas Chammas <nicholas.chammas@gmail.com>
wrote:

> Do we need to ditch Python 2 support to provide type hints? I don’t think
> so.
>
> Python lets you specify typing stubs that provide the same benefit without
> forcing Python 3.
>
> 2018년 9월 14일 (금) 오후 8:01, Holden Karau <holden@pigscanfly.ca>님이
작성:
>
>>
>>
>> On Fri, Sep 14, 2018, 3:26 PM Erik Erlandson <eerlands@redhat.com> wrote:
>>
>>> To be clear, is this about "python-friendly API" or "friendly python
>>> API" ?
>>>
>> Well what would you consider to be different between those two
>> statements? I think it would be good to be a bit more explicit, but I don't
>> think we should necessarily limit ourselves.
>>
>>>
>>> On the python side, it might be nice to take advantage of static typing.
>>> Requires python 3.6 but with python 2 going EOL, a spark-3.0 might be a
>>> good opportunity to jump the python-3-only train.
>>>
>> I think we can make types sort of work without ditching 2 (the types only
>> would work in 3 but it would still function in 2). Ditching 2 entirely
>> would be a big thing to consider, I honestly hadn't been considering that
>> but it could be from just spending so much time maintaining a 2/3 code
>> base. I'd suggest reaching out to to user@ before making that kind of
>> change.
>>
>>>
>>> On Fri, Sep 14, 2018 at 12:15 PM, Holden Karau <holden@pigscanfly.ca>
>>> wrote:
>>>
>>>> Since we're talking about Spark 3.0 in the near future (and since some
>>>> recent conversation on a proposed change reminded me) I wanted to open up
>>>> the floor and see if folks have any ideas on how we could make a more
>>>> Python friendly API for 3.0? I'm planning on taking some time to look at
>>>> other systems in the solution space and see what we might want to learn
>>>> from them but I'd love to hear what other folks are thinking too.
>>>>
>>>> --
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>
>>>
>>>

Mime
View raw message