spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sourav Chandra <sourav.chan...@livestream.com>
Subject Re: Change print() in JavaNetworkWordCount
Date Wed, 26 Mar 2014 16:29:29 GMT
This works just fine without any compilation error:

def print() {
    def foreachFunc = (rdd: RDD[T], time: Time) => {

      val total = rdd.collect().toList
      println ("-------------------------------------------")
      println ("Time: " + time)
      println ("-------------------------------------------")

      total.foreach(println)
//      val first11 = rdd.take(11)
//      println ("-------------------------------------------")
//      println ("Time: " + time)
//      println ("-------------------------------------------")
//      first11.take(10).foreach(println)
//      if (first11.size > 10) println("...")
      println()
    }
    new ForEachDStream(this,
context.sparkContext.clean(foreachFunc)).register()
  }


On Wed, Mar 26, 2014 at 8:50 PM, Eduardo Costa Alfaia <
e.costaalfaia@unibs.it> wrote:

>  Thanks Guys for reply, but I have found this piece of code in streaming:
>  def print() {
> 585     def foreachFunc = (rdd: RDD[T], time: Time) => {
> 586       //val first11 = rdd.take(11)
> 587       //val first100 = rdd.take(100)
> 588         val total = rdd.collect()
> 589       println ("-------------------------------------------")
> 590       println ("Time: " + time)
> 591       println ("-------------------------------------------")
> 592       //first100.take(100).foreach(println)
> 593         total.collect().foreach(println)
> 594       //if (first100.size > 99) println("(...)")
> 595       println()
> 596     }
> 597     new ForEachDStream(this,
> context.sparkContext.clean(foreachFunc)).register()
> 598   }
>
> I would like to print whole the RDD, I have seen I should use the
> collect() function instead of take(), I have done but I have gotten a
> compilation error.
> does anyone help me?
>
> BR
>
> Em 3/25/14, 13:48, Sourav Chandra escreveu:
>
> You can extend DStream and override print() method. Then you can create
> your own DSTream extending from this.
>
>
> On Tue, Mar 25, 2014 at 6:07 PM, Eduardo Costa Alfaia <
> e.costaalfaia@unibs.it> wrote:
>
>> Hi Guys,
>> I think that I already did this question, but I don't remember if anyone
>> has answered me. I would like changing in the function print() the quantity
>> of words and the frequency number that are sent to driver's screen. The
>> default value is 10.
>>
>> Anyone could help me with this?
>>
>> Best Regards
>>
>> --
>> Informativa sulla Privacy: http://www.unibs.it/node/8155
>>
>
>
>
>  --
>
> Sourav Chandra
>
> Senior Software Engineer
>
> · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
· · · ·
>
> sourav.chandra@livestream.com
>
> o: +91 80 4121 8723
>
> m: +91 988 699 3746
>
> skype: sourav.chandra
>
> Livestream
>
> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
> Block, Koramangala Industrial Area,
>
> Bangalore 560034
>
> www.livestream.com
>
>
>
> Informativa sulla Privacy: http://www.unibs.it/node/8155
>



-- 

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
· · ·

sourav.chandra@livestream.com

o: +91 80 4121 8723

m: +91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com

Mime
View raw message