spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gordon Benjamin <gordon.benjami...@gmail.com>
Subject Re: Incremental loading data slows performance
Date Thu, 20 Nov 2014 17:45:53 GMT
To follow on:
I asked the developer how we incrementally load data and the response was

no. union only for updated records (every night)
For every minutes export algorithm next:
1. upload file to hadoop.
2. load data inpath... overwrite into table ...._incremental;
3. insert into table ..._cached from ..._incremental

Perhaps this helps understand our issue

On Thursday, November 20, 2014, Gordon Benjamin <gordon.benjamin65@gmail.com>
wrote:

> Hi,
>
> We are seeing bad performance as we incrementally load data. Here is the
> config
>
> Spark standalone cluster
>
> spark01 (spark master, shark, hadoop namenode): 15GB RAM, 4vCPU's
> spark02 (spark worker, hadoop datanode): 15GB RAM, 8vCPU's
> spark03 (spark worker): 15GB RAM, 8vCPU's
> spark04 (spark worker): 15GB RAM, 8vCPU's
>
> spark worker configuration:
> spark.local.dir=/path/to/ssd/disk
> spark.default.parallelism=64
> spark.executor.memory=10g
> spark.serializer=org.apache.spark.serializer.KryoSerializer
>
> shark configuration:
> spark.kryoserializer.buffer.mb=64
> mapred.reduce.tasks=30
> spark.scheduler.mode=FAIR
> spark.serializer=org.apache.spark.serializer.KryoSerializer
> spark.default.parallelism=64
>
> and the performance decreases with more data being loaded into spark
>
> simple query like this:
> select count(*) from customers_cached
> 0.5 second on 12th Nov
> 4.24 seconds now
>
> We have these errors all over the log
>
> 2014-11-20 16:56:42,125 WARN  parse.TypeCheckProcFactory
> (TypeCheckProcFactory.java:convert(180)) - Invalid type entry TOK_INT=null
> 2014-11-20 16:56:51,988 WARN  parse.TypeCheckProcFactory
> (TypeCheckProcFactory.java:convert(180)) - Invalid type entry
> TOK_TABLE_OR_COL=null
>
> Anyone any ideas to help us resolve this? Can post up anything you need
>
>
>
>
>

Mime
View raw message