spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srini E (Jira)" <j...@apache.org>
Subject [jira] [Reopened] (SPARK-29337) How to Cache Table and Pin it in Memory and should not Spill to Disk on Thrift Server
Date Tue, 08 Oct 2019 19:00:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-29337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Srini E reopened SPARK-29337:
-----------------------------

Openned and labelled it to Stackoverflow and Question

> How to Cache Table and Pin it in Memory and should not Spill to Disk on Thrift Server

> --------------------------------------------------------------------------------------
>
>                 Key: SPARK-29337
>                 URL: https://issues.apache.org/jira/browse/SPARK-29337
>             Project: Spark
>          Issue Type: Question
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: Srini E
>            Priority: Major
>              Labels: Question, stack-overflow
>         Attachments: Cache+Image.png
>
>
> Hi Team,
> How to pin the table in cache so it would not swap out of memory?
> Situation: We are using Microstrategy BI reporting. Semantic layer is built. We wanted
to Cache highly used tables into CACHE using Spark SQL CACHE Table <table_name>; we
did cache for SPARK context( Thrift server). Please see below snapshot of Cache table, went
to disk over time. Initially it was all in cache , now some in cache and some in disk. That
disk may be local disk relatively more expensive reading than from s3. Queries may take longer
and inconsistent times from user experience perspective. If More queries running using Cache
tables, copies of the cache table images are copied and copies are not staying in memory causing
reports to run longer. so how to pin the table so would not swap to disk. Spark memory management
is dynamic allocation, and how to use those few tables to Pin in memory .



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message