spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrés Ivaldi <iaiva...@gmail.com>
Subject Spark version performance
Date Wed, 12 Dec 2018 01:57:24 GMT
Hello list,

I'm having a little performance issue, with different Spark versions.

I've a spark embedded application written in scala, Initially I've use
Spark 2.0.2, and works fine, with good speed response, but when I updated
to 2.3.2 , with no any code changes It becomes slower.

Mainly what the application do is to gather information from a source,
apply transformations with filters and performs aggregation over it. It's
source is mainly parquet and no write is done just a serialization from the
result.

Maybe I'm using  deprecated api functions or the order of the operations
are not generating a good plan...

Can someone give me some idea of any change on the versions that could
generate this behavior?

Regards,

-- 
Ing. Ivaldi Andres

Mime
View raw message