spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Spark handling parallel requests
Date Mon, 19 Oct 2015 06:34:26 GMT
Yes, there are kafka consumers/producers for almost all the languages, you
can read more over here
https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-PHP
Here's a repo for the php version https://github.com/EVODelavega/phpkafka

Thanks
Best Regards

On Sun, Oct 18, 2015 at 12:58 PM, <tarek.abouzeid91@yahoo.com> wrote:

> hi Akhlis
>
> its a must to push data to a socket as i am using php as a web service to
> push data to socket , then spark catch the data on that socket and process
> it , is there a way to push data from php to kafka directly ?
>
> --  Best Regards, -- Tarek Abouzeid
>
>
>
> On Sunday, October 18, 2015 10:26 AM, "tarek.abouzeid91@yahoo.com" <
> tarek.abouzeid91@yahoo.com> wrote:
>
>
> hi Xiao,
> 1- requests are not similar at all , but they use solr and do commit
> sometimes
> 2- no caching is required
> 3- the throughput must be very high yeah , the requests are tiny but the
> system may receive 100 request/sec ,
> does kafka support listening to a socket ?
>
> --  Best Regards, -- Tarek Abouzeid
>
>
>
> On Monday, October 12, 2015 10:50 AM, Xiao Li <gatorsmile@gmail.com>
> wrote:
>
>
> Hi, Tarek,
>
> It is hard to answer your question. Are these requests similar? Caching
> your results or intermediate results in your applications? Or does that
> mean your throughput requirement is very high? Throttling the number of
> concurrent requests? ...
>
> As Akhil said, Kafka might help in your case. Otherwise, you need to read
> the designs or even source codes of Kafka and Spark Streaming.
>
>  Best wishes,
>
> Xiao Li
>
>
> 2015-10-11 23:19 GMT-07:00 Akhil Das <akhil@sigmoidanalytics.com>:
>
> Instead of pushing your requests to the socket, why don't you push them to
> a Kafka or any other message queue and use spark streaming to process them?
>
> Thanks
> Best Regards
>
> On Mon, Oct 5, 2015 at 6:46 PM, <tarek.abouzeid91@yahoo.com.invalid>
> wrote:
>
> Hi ,
> i am using Scala , doing a socket program to catch multiple requests at
> same time and then call a function which uses spark to handle each process
> , i have a multi-threaded server to handle the multiple requests and pass
> each to spark , but there's a bottleneck as the spark doesn't initialize a
> sub task for the new request , is it even possible to do parallel
> processing using single spark job ?
> Best Regards,
>
> --  Best Regards, -- Tarek Abouzeid
>
>
>
>
>
>
>
>

Mime
View raw message