spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Renato MarroquĂ­n Mogrovejo <renatoj.marroq...@gmail.com>
Subject Reading parquet files into Spark Streaming
Date Fri, 26 Aug 2016 15:42:00 GMT
Hi all,

I am trying to use parquet files as input for DStream operations, but I
can't find any documentation or example. The only thing I found was [1] but
I also get the same error as in the post (Class
parquet.avro.AvroReadSupport not found).
Ideally I would like to do have something like this:

val oDStream = ssc.fileStream[Void, Order,
ParquetInputFormat[Order]]("data/")

where Order is a case class and the files inside "data" are all parquet
files.
Any hints would be highly appreciated. Thanks!


Best,

Renato M.

[1]
http://stackoverflow.com/questions/35413552/how-do-i-read-in-parquet-files-using-ssc-filestream-and-what-is-the-nature

Mime
View raw message