spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepak Goel <deic...@gmail.com>
Subject Re: spark optimized pagination
Date Sun, 10 Jun 2018 12:10:38 GMT
I think your requirement is that of OLTP system. Spark & Cassandra are more
suitable for batch kind of jobs (They can be used for OLTP but there would
be a performance hit)



Deepak
"The greatness of a nation can be judged by the way its animals are
treated. Please consider stopping the cruelty by becoming a Vegan"

+91 73500 12833
deicool@gmail.com

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

Make In India : http://www.makeinindia.com/home

On Sun, Jun 10, 2018 at 10:42 AM, onmstester onmstester <onmstester@zoho.com
> wrote:

> Hi,
> I'm using spark on top of cassandra as backend CRUD of a Restfull
> Application.
> Most of Rest API's retrieve huge amount of data from cassandra and doing a
> lot of aggregation on them  in spark which take some seconds.
>
> Problem: sometimes the output result would be a big list which make client
> browser throw stop script, so we should paginate the result at the
> server-side,
> but it would be so annoying for user to wait some seconds on each page to
> cassandra-spark processings,
>
> Current Dummy Solution: For now i was thinking about assigning a UUID to
> each request which would be sent back and forth between server-side and
> client-side,
> the first time a rest API invoked, the result would be saved in a
> temptable  and in subsequent similar requests (request for next pages) the
> result would be fetch from
> temptable (instead of common flow of retrieve from cassandra + aggregation
> in spark which would take some time). On memory limit, the old results
> would be deleted.
>
> Is there any built-in clean caching strategy in spark to handle such
> scenarios?
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>

Mime
View raw message