spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sea" <>
Subject 回复: Time is ugly in Spark Streaming....
Date Fri, 26 Jun 2015 12:59:47 GMT
Yes, I make it.

------------------ 原始邮件 ------------------
发件人: "Gerard Maas";<>;
发送时间: 2015年6月26日(星期五) 下午5:40
收件人: "Sea"<>; 
抄送: "user"<>; "dev"<>; 
主题: Re: Time is ugly in Spark Streaming....

Are you sharing the SimpleDateFormat instance? This looks a lot more like the non-thread-safe
behaviour of SimpleDateFormat (that has claimed many unsuspecting victims over the years),
than any 'ugly' Spark Streaming. Try writing the timestamps in millis to Kafka and compare.

-kr, Gerard.

On Fri, Jun 26, 2015 at 11:06 AM, Sea <> wrote:
Hi, all

I find a problem in spark streaming, when I use the time in function foreachRDD... I find
the time is very interesting.
val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc,
kafkaParams, topicsSet) => createGroup(x._2, dimensions)).groupByKey().foreachRDD((rdd, time)
=> {
  try {
    if (!rdd.partitions.isEmpty) {
      rdd.foreachPartition(partition => {
        handlePartition(partition, timeType, time, dimensions, outputTopic, brokerList)
  } catch {
    case e: Exception => e.printStackTrace()

val dateFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss")
var date = dateFormat.format(new Date(time.milliseconds))

Then I insert the 'date' into Kafka , but I found .....

View raw message