spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: dremel paper example schema
Date Tue, 30 Oct 2018 07:20:06 GMT
Are you using the same parquet version as Spark uses? Are you using a recent version of Spark?
Why don’t you create the file in Spark?

> Am 30.10.2018 um 07:34 schrieb lchorbadjiev <lubomir.chorbadjiev@gmail.com>:
> 
> Hi Gourav,
> 
> the question in fact is are there any the limitations of Apache Spark
> support for Parquet file format.
> 
> The example schema from the dremel paper is something that is supported in
> Apache Parquet (using Apache Parquet Java API).
> 
> Now I am trying to implement the same schema using Apache Spark SQL types,
> but not very successful. And probably this is not unexpected. 
> 
> What was unexpected is that Apache Spark can't read a parquet file with the
> dremel example schema.
> 
> Probably there are some limitations of what Apache Spark can support from
> Apache Parquet file format, but for me it is not obvious what this
> limitations are.
> 
> Thanks,
> Lubomir Chorbadjiev
> 
> 
> 
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
> 
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> 

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message