spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <>
Subject Re: Python friendly API for Spark 3.0
Date Sat, 15 Sep 2018 00:01:41 GMT
On Fri, Sep 14, 2018, 3:26 PM Erik Erlandson <> wrote:

> To be clear, is this about "python-friendly API" or "friendly python API" ?
Well what would you consider to be different between those two statements?
I think it would be good to be a bit more explicit, but I don't think we
should necessarily limit ourselves.

> On the python side, it might be nice to take advantage of static typing.
> Requires python 3.6 but with python 2 going EOL, a spark-3.0 might be a
> good opportunity to jump the python-3-only train.
I think we can make types sort of work without ditching 2 (the types only
would work in 3 but it would still function in 2). Ditching 2 entirely
would be a big thing to consider, I honestly hadn't been considering that
but it could be from just spending so much time maintaining a 2/3 code
base. I'd suggest reaching out to to user@ before making that kind of

> On Fri, Sep 14, 2018 at 12:15 PM, Holden Karau <>
> wrote:
>> Since we're talking about Spark 3.0 in the near future (and since some
>> recent conversation on a proposed change reminded me) I wanted to open up
>> the floor and see if folks have any ideas on how we could make a more
>> Python friendly API for 3.0? I'm planning on taking some time to look at
>> other systems in the solution space and see what we might want to learn
>> from them but I'd love to hear what other folks are thinking too.
>> --
>> Twitter:
>> Books (Learning Spark, High Performance Spark, etc.):
>>  <>
>> YouTube Live Streams:

View raw message