spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sea" <261810...@qq.com>
Subject 回复: Time is ugly in Spark Streaming....
Date Sat, 27 Jun 2015 14:36:10 GMT
Yes , things go well now.  It is a problem of SimpleDateFormat. Thank you all.




------------------ 原始邮件 ------------------
发件人: "Dumas Hwang";<dumas.hwang@gmail.com>;
发送时间: 2015年6月27日(星期六) 晚上8:16
收件人: "Tathagata Das"<tdas@databricks.com>; 
抄送: "Emrehan Tüzün"<emrehan.tuzun@gmail.com>; "Sea"<261810726@qq.com>; "dev"<dev@spark.apache.org>;
"user"<user@spark.apache.org>; 
主题: Re: Time is ugly in Spark Streaming....



Java's SimpleDateFormat is not thread safe.  You can consider using DateTimeFormatter if you
are using Java 8 or Joda-time

On Sat, Jun 27, 2015 at 3:32 AM, Tathagata Das <tdas@databricks.com> wrote:
Could you print the "time" on the driver (that is, in foreachRDD but before RDD.foreachPartition)
and see if it is behaving weird?

TD


On Fri, Jun 26, 2015 at 3:57 PM, Emrehan Tüzün <emrehan.tuzun@gmail.com> wrote:
 





On Fri, Jun 26, 2015 at 12:30 PM, Sea <261810726@qq.com> wrote:

 Hi, all
 

 I find a problem in spark streaming, when I use the time in function foreachRDD... I find
the time is very interesting. 
 val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc,
kafkaParams, topicsSet)
 dataStream.map(x => createGroup(x._2, dimensions)).groupByKey().foreachRDD((rdd, time)
=> {
try {
if (!rdd.partitions.isEmpty) {
      rdd.foreachPartition(partition => {
handlePartition(partition, timeType, time, dimensions, outputTopic, brokerList)
      })
    }
  } catch {
case e: Exception => e.printStackTrace()
  }
})
 

 val dateFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss")
  var date = dateFormat.format(new Date(time.milliseconds)) 
 
 Then I insert the 'date' into Kafka , but I found .....
  

 {"timestamp":"2015-06-00T16:50:02","status":"3","type":"1","waittime":"0","count":17}
 {"timestamp":"2015-06-26T16:51:13","status":"1","type":"1","waittime":"0","count":34}
 {"timestamp":"2015-06-00T16:50:02","status":"4","type":"0","waittime":"0","count":279}
 {"timestamp":"2015-06-26T16:52:00","status":"11","type":"1","waittime":"0","count":9}
 {"timestamp":"0020-06-26T16:50:36","status":"7","type":"0","waittime":"0","count":1722}
 {"timestamp":"2015-06-10T16:51:17","status":"0","type":"0","waittime":"0","count":2958}
 {"timestamp":"2015-06-26T16:52:00","status":"0","type":"1","waittime":"0","count":114}
 {"timestamp":"2015-06-10T16:51:17","status":"11","type":"0","waittime":"0","count":2066}
 {"timestamp":"2015-06-26T16:52:00","status":"1","type":"0","waittime":"0","count":1539}
Mime
View raw message