spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sun, Rui" <>
Subject RE: SparkR Supported Types - Please add "bigint"
Date Fri, 24 Jul 2015 05:30:26 GMT

Reported your issue:

SparkR has support for long(bigint) type in serde. This issue is related to support complex
Scala types in serde.

-----Original Message-----
From: Exie [] 
Sent: Friday, July 24, 2015 10:26 AM
Subject: SparkR Supported Types - Please add "bigint"

Hi Folks,

Using Spark to read in JSON files and detect the schema, it gives me a dataframe with a "bigint"
filed. R then fails to import the dataframe as it cant convert the type.

> head(mydf)
Error in[[i]], optional = TRUE) : 
  cannot coerce class ""jobj"" to a data.frame
> show(mydf)
DataFrame[localEventDtTm:timestamp, asset:string, assetCategory:string, assetType:string,
event:string, extras:array<struct&lt;name:string,value:string>>, ipAddress:string,
memberId:string, system:string, timestamp:bigint, title:string, trackingId:string, version:bigint]

I believe this is related to:

A sample record in raw JSON looks like this:
{"version": 1,"event": "view","timestamp": 1427846422377,"system":
"DCDS","asset": "6404476","assetType": "myType","assetCategory":
"myCategory","extras": [{"name": "videoSource","value": "mySource"},{"name":
"playerType","value": "Article"},{"name": "duration","value":
"202088"}],"trackingId": "155629a0-d802-11e4-13ee-6884e43d6000","ipAddress":
"","title": "myTitle"}

Can someone turn this into a feature request or something for 1.5.0 ?

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail: For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message