asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Murtadha Hubail <hubail...@gmail.com>
Subject Re: Hyracks Job Requirement Configuration
Date Mon, 29 Jan 2018 01:05:37 GMT
Hi Rana,

The memory used for query processing is automatically calculated as follows:
JVM Max Memory - storage.buffercache.size - storage.memorycomponent.globalbudget

The documentation defaults for these parameters are outdated. The default value for storage.buffercache.size
is (JVM Max Memory / 4) and it's the same for storage.memorycomponent.globalbudget. Since
your dataset is already loaded, you could reduce the budget of storage.memorycomponent.globalbudget.
In addition, if I recall correctly, your dataset size is way smaller than what's allocated
for the buffer cache, so you might want to reduce the buffer cache budget. That should give
you more than enough memory to execute on 39 cores.

Cheers,
Murtadha

´╗┐On 01/29/2018, 3:30 AM, "Mike Carey" <dtabass@gmail.com> wrote:

    + dev
    
    
    On 1/28/18 3:37 PM, Rana Alotaibi wrote:
    > Hi all,
    >
    > I would like to make AsterixDB utilizes all available CPU cores (39) 
    > that I have for the following query:
    >
    > USE mimiciii;
    > SET `compiler.parallelism` "39";
    > SET `compiler.sortmemory` "128MB";
    > SET `compiler.joinmemory` "265MB";
    > SELECT P.SUBJECT_ID
    > FROM   LABITEMS I, PATIENTS P, P.ADMISSIONS A, A.LABEVENTS E
    > WHERE E.ITEMID/*+bcast*/=I.ITEMID AND
    >              E.FLAG = 'abnormal' AND
    >              I.FLUID='Blood' AND
    >              I.LABEL='Haptoglobin'
    >
    >
    > The total memory size that I have is 125GB(57GB for the AsterixDB 
    > buffer cache). By running the above query, I got the following error:
    >
    > "msg": "HYR0009: Job requirement (memory: 10705403904 bytes, CPU 
    > cores: 39) exceeds capacity (memory: 3258744832 bytes, CPU cores: 39)"
    >
    > How can I change this capacity default configuration? I'm looking into 
    > this page : https://asterixdb.apache.org/docs/0.9.2/ncservice.html . 
    > Could you please point me to the appropriate configuration parameter?
    >
    > Thanks
    > -- Rana
    >
    >
    >
    >
    
    



Mime
View raw message