spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: Implement Dataset reader from SEQ file with protobuf to Dataset
Date Sun, 08 Oct 2017 19:22:22 GMT
spark-avro <https://github.com/databricks/spark-avro> would be a good
example to start with.

On Sun, Oct 8, 2017 at 3:00 AM, Serega Sheypak <serega.sheypak@gmail.com>
wrote:

> Hi, did anyone try to implement Spark SQL dataset reader from SEQ file
> with protobuf inside to Dataset?
>
> Imagine I have protobuf def
> Person
>  - name: String
>  - lastName: String
> - phones: List[String]
>
> and generated scala case class:
> case class Person(name:String, lastName: String, phones: List[String])
>
> I want to write some component that gives me Dataset with types schema.
>
> val personsDataset = spark.read
>   .option("inferSchema", "true")[Person]
>
> Where can I take a look at references?
>

Mime
View raw message