spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <hol...@pigscanfly.ca>
Subject Re: Python friendly API for Spark 3.0
Date Sat, 15 Sep 2018 00:01:41 GMT
On Fri, Sep 14, 2018, 3:26 PM Erik Erlandson <eerlands@redhat.com> wrote:

> To be clear, is this about "python-friendly API" or "friendly python API" ?
>
Well what would you consider to be different between those two statements?
I think it would be good to be a bit more explicit, but I don't think we
should necessarily limit ourselves.

>
> On the python side, it might be nice to take advantage of static typing.
> Requires python 3.6 but with python 2 going EOL, a spark-3.0 might be a
> good opportunity to jump the python-3-only train.
>
I think we can make types sort of work without ditching 2 (the types only
would work in 3 but it would still function in 2). Ditching 2 entirely
would be a big thing to consider, I honestly hadn't been considering that
but it could be from just spending so much time maintaining a 2/3 code
base. I'd suggest reaching out to to user@ before making that kind of
change.

>
> On Fri, Sep 14, 2018 at 12:15 PM, Holden Karau <holden@pigscanfly.ca>
> wrote:
>
>> Since we're talking about Spark 3.0 in the near future (and since some
>> recent conversation on a proposed change reminded me) I wanted to open up
>> the floor and see if folks have any ideas on how we could make a more
>> Python friendly API for 3.0? I'm planning on taking some time to look at
>> other systems in the solution space and see what we might want to learn
>> from them but I'd love to hear what other folks are thinking too.
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
>

Mime
View raw message