spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Nastetsky <alex.nastet...@vervemobile.com>
Subject Re: spark-avro 2.0.1 generates strange schema (spark-avro 1.0.0 is fine)
Date Wed, 14 Oct 2015 20:49:01 GMT
Here you go: https://github.com/databricks/spark-avro/issues/92

Thanks.

On Wed, Oct 14, 2015 at 4:41 PM, Josh Rosen <rosenville@gmail.com> wrote:

> Can you report this as an issue at
> https://github.com/databricks/spark-avro/issues so that it's easier to
> track? Thanks!
>
> On Wed, Oct 14, 2015 at 1:38 PM, Alex Nastetsky <
> alex.nastetsky@vervemobile.com> wrote:
>
>> I save my dataframe to avro with spark-avro 1.0.0 and it looks like this
>> (using avro-tools tojson):
>>
>> {"field1":"value1","field2":976200}
>> {"field1":"value2","field2":976200}
>> {"field1":"value3","field2":614100}
>>
>> But when I use spark-avro 2.0.1, it looks like this:
>>
>> {"field1":{"string":"value1"},"field2":{"long":976200}}
>> {"field1":{"string":"value2"},"field2":{"long":976200}}
>> {"field1":{"string":"value3"},"field2":{"long":614100}}
>>
>> At this point I'd be happy to use spark-avro 1.0.0, except that it
>> doesn't seem to support specifying a compression codec (I want deflate).
>>
>
>

Mime
View raw message