spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eirik Thorsnes <eirik.thors...@uni.no>
Subject Re: ORC native in Spark 2.3, with zlib, gives java.nio.BufferUnderflowException during read
Date Tue, 03 Apr 2018 17:47:08 GMT
On 28. mars 2018 03:26, Dongjoon Hyun wrote:
> You may hit SPARK-23355 (convertMetastore should not ignore table properties).
> 
> Since it's a known Spark issue for all Hive tables (Parquet/ORC), could you check that
too?
> 
> Bests,
> Dongjoon.
> 

Hi,

I think you might be right, I can run your example from the other email
OK ( spark.range(10).write.orc("/tmp/zlib_test") +
spark.read.orc("/tmp/zlib_test").show )

I can also do:

spark.range(10).write.format("orc").option("compression","zlib").saveAsTable("zlib_test3")

with a corresponding read. Trying to read a more complicated and
partitioned table fails. Could be because of partitioning perhaps?
Looking more into it now.

Regards,
Eirik

-- 
Eirik Thorsnes


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message