spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Debasish Das <debasish.da...@gmail.com>
Subject Re: Actors and sparkcontext actions
Date Tue, 04 Mar 2014 15:02:46 GMT
Hi Ognen,

Any particular reason of choosing scalatra over options like play or spray
?

Is scalatra much better in serving apis or is it due to similarity with
ruby's sinatra ?

Did you try the other options and then pick scalatra ?

Thanks.
Deb



On Tue, Mar 4, 2014 at 4:50 AM, Ognen Duzlevski <ognen@plainvanillagames.com
> wrote:

> Suraj, I posted to this list a link to my blog where I detail how to do a
> simple actor/sparkcontext thing with the added obstacle of it being within
> a Scalatra servlet.
>
> Thanks for the code!
> Ognen
>
>
> On 3/4/14, 3:20 AM, Suraj Satishkumar Sheth wrote:
>
>> Hi Ognen,
>> See if this helps. I was working on this :
>>
>> class MyClass[T](sc : SparkContext, flag1 : Boolean, rdd : RDD[T],
>> hdfsPath : String) extends Actor {
>>
>>    def act(){
>>      if(flag1) this.process()
>>      else this.count
>>    }
>>       private def process(){
>>      println(sc.textFile(hdfsPath).count)
>>      //do the processing
>>    }
>>       private def count(){
>>     println(rdd.count)
>>     //do the counting
>>    }
>>
>> }
>>
>> Thanks and Regards,
>> Suraj Sheth
>>
>>
>> -----Original Message-----
>> From: Ognen Duzlevski [mailto:ognen@nengoiksvelzud.com]
>> Sent: 27 February 2014 01:09
>> To: user@spark.incubator.apache.org
>> Subject: Actors and sparkcontext actions
>>
>> Can someone point me to a simple, short code example of creating a basic
>> Actor that gets a context and runs an operation such as .textFile.count?
>> I am trying to figure out how to create just a basic actor that gets a
>> message like this:
>>
>> case class Msg(filename:String, ctx: SparkContext)
>>
>> and then something like this:
>>
>> class HelloActor extends Actor {
>>       import context.dispatcher
>>
>>       def receive = {
>>           case Msg(fn,ctx) => {
>>               // get the count here!
>>               // cts.textFile(fn).count
>>           }
>>           case _ => println("huh?")
>>       }
>> }
>>
>> Where I would want to do something like:
>>
>> val conf = new
>> SparkConf().setMaster("spark://192.168.10.29:7077").setAppName("Hello").
>> setSparkHome("/Users/maketo/plainvanilla/spark-0.9")
>> val sc = new SparkContext(conf)
>> val system = ActorSystem("mySystem")
>>
>> val helloActor1 = system.actorOf( Props[ HelloActor], name =
>> "helloactor1")
>> helloActor1 ! new Msg("test.json",sc)
>>
>> Thanks,
>> Ognen
>>
>
> --
> Some people, when confronted with a problem, think "I know, I'll use
> regular expressions." Now they have two problems.
> -- Jamie Zawinski
>
>

Mime
View raw message