spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhiliang Zhu <zchl.j...@yahoo.com.INVALID>
Subject Re: the spark job is so slow - almost frozen
Date Mon, 18 Jul 2016 10:20:52 GMT
the sql logic in the program is very much complex , so do not describe the detailed codes  
here .  

    On Monday, July 18, 2016 6:04 PM, Zhiliang Zhu <zchl.jump@yahoo.com.INVALID> wrote:
 

 Hi All,  
Here we have one application, it needs to extract different columns from 6 hive tables, and
then does some easy calculation, there is around 100,000 number of rows in each table,finally
need to output another table or file (with format of consistent columns) .
 However, after lots of days trying, the spark hive job is unthinkably slow - sometimes almost
frozen. There is 5 nodes for spark cluster.  Could anyone offer some help, some idea or
clue is also good. 
Thanks in advance~
Zhiliang 

  
Mime
View raw message