spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bryan Cutler (Jira)" <>
Subject [jira] [Commented] (SPARK-32312) Upgrade Apache Arrow to 1.0.0
Date Fri, 04 Sep 2020 05:35:00 GMT


Bryan Cutler commented on SPARK-32312:

Sorry for the delay, I was holding off for a couple of things. Version 1.0.1 was being released
and wanted to see if there was anything crucial to use that version instead. Also, to see
if ARROW-9528 would be included in the 1.0.1 release, which would have broken our Python tests,
but it will be in v2.0.0. For Arrow Java, there were no real fixes between 1.0.0 and 1.0.1,
so it won't really make a difference, but might as well use 1.0.1. For pyarrow, there were
no fixes that affect our Spark usage, so it should be safe to use a minimum version of 1.0.0.
I will try to put up a PR tomorrow.

> Upgrade Apache Arrow to 1.0.0
> -----------------------------
>                 Key: SPARK-32312
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark, SQL
>    Affects Versions: 3.0.0
>            Reporter: Bryan Cutler
>            Priority: Major
> Apache Arrow will soon release v1.0.0 which provides backward/forward compatibility guarantees
as well as a number of fixes and improvements. This will upgrade the Java artifact and PySpark
API. Although PySpark will not need special changes, it might be a good idea to bump up minimum
supported version and CI testing.
> TBD: list of important improvements and fixes

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message