spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Khatri <ckhatriman...@gmail.com>
Subject About Error while reading large JSON file in Spark
Date Tue, 18 Oct 2016 07:43:58 GMT
Hello Community members,

I am getting error while reading large JSON file in spark,

*Code:*

val landingVisitor =
sqlContext.read.json("s3n://hist-ngdp/lvisitor/lvisitor-01-aug.json")

*Error:*

16/10/18 07:30:30 ERROR Executor: Exception in task 8.0 in stage 0.0 (TID 8)
java.io.IOException: Too many bytes before newline: 2147483648
at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:249)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
at
org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:135)
at
org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:237)

What would be resolution for the same ?

Thanks in Advance !


-- 
Yours Aye,
Chetan Khatri.

Mime
View raw message