spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: log4j Spark-worker performance problem
Date Wed, 30 Sep 2015 06:53:40 GMT
Depends how big the lines are, on a typical HDD you can write at max
10-15MB/s, and on SSDs it can be upto 30-40MB/s.

Thanks
Best Regards

On Mon, Sep 28, 2015 at 3:57 PM, vaibhavrtk <vaibhavrtk@gmail.com> wrote:

> Hello
>
> We need a lot of logging for our application about 1000 lines needed to be
> logged per message we process and we process 1000 msgs/sec. So total lines
> needed to be logged is /1000*1000/sec/. As it is going to be written in a
> file. Will writing so much logs will impact the processing power of spark
> by
> a lot?
> If yes, What can be the alternative?
>
> Note: This much logging is required for the appropriate monitoring of the
> application.
> Let me know if more information is needed.
>
> Thanks
> Vaibhav
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/log4j-Spark-worker-performance-problem-tp24842.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message