We have been experimenting hbase (version 1.0) and phoenix (version 4.6) for our OLAP workload. In order to precalculate aggregates we have been executing an upsert phoenix query that aggregates raw data (over 10 mil records) to generate an OLAP cube.
While executing the query, one of the region servers in a cluster of 3 RS crashes. I am trying to figure out what could be causing the region server to crash.
The server shows high disk operations before the jvm crashed. Kindly find the disk and other stats attached.
Any suggestions on where could I look into would be helpful.
The upsert query that was executed is
upsert into AGENT_TER_PRO (AGENT_ID,TERRITORY_ID,PRODUCT_ID,SUM_TOTAL_SALES,SUM_TOTAL_EXPENSES,SUM_UNIT_CNT_SOLD,AVG_PRICE_PER_UNIT) select /*+ INDEX(TRANSACTIONS AG_TER_PRO2) */ AGENT_ID,TERRITORY_ID,PRODUCT_ID, sum(TOTAL_SALES) SUM_TOTAL_SALES,sum(TOTAL_EXPENSES) SUM_TOTAL_EXPENSES,sum(UNIT_CNT_SOLD) SUM_UNIT_CNT_SOLD,AVG(PRICE_PER_UNIT) AVG_PRICE_PER_UNIT from TRANSACTIONS group by AGENT_ID,TERRITORY_ID,PRODUCT_ID;