spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eirik Thorsnes <>
Subject Re: ORC native in Spark 2.3, with zlib, gives java.nio.BufferUnderflowException during read
Date Tue, 03 Apr 2018 17:47:08 GMT
On 28. mars 2018 03:26, Dongjoon Hyun wrote:
> You may hit SPARK-23355 (convertMetastore should not ignore table properties).
> Since it's a known Spark issue for all Hive tables (Parquet/ORC), could you check that
> Bests,
> Dongjoon.


I think you might be right, I can run your example from the other email
OK ( spark.range(10).write.orc("/tmp/zlib_test") +"/tmp/zlib_test").show )

I can also do:


with a corresponding read. Trying to read a more complicated and
partitioned table fails. Could be because of partitioning perhaps?
Looking more into it now.


Eirik Thorsnes

To unsubscribe e-mail:

View raw message