spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <>
Subject Re: Loading Files from HDFS Incurs Network Communication
Date Mon, 26 Oct 2015 13:49:22 GMT

On 26 Oct 2015, at 11:21, Sean Owen <<>>

Yeah, are these stats actually reflecting data read locally, like through the loopback interface?
I'm also no expert on the internals here but this may be measuring effectively local reads.
Or are you sure it's not?

HDFS stats are really the general filesystem stats: they measure data through the input and
output streams, not whether they were to/from local or remote systems. Fixable, and metrics
are always good, though as Hadoop (currently) uses Hadoop metrics 2, not the codahale APIs,
it's not seamless to glue it up with the spark context metric registry

On Mon, Oct 26, 2015 at 11:14 AM, Steve Loughran <<>>

> On 26 Oct 2015, at 09:28, Jinfeng Li <<>>
> Replication factor is 3 and we have 18 data nodes. We check HDFS webUI, data is evenly
distributed among 18 machines.

every block in HDFS (usually 64-128-256 MB) is distributed across three machines, meaning
3 machines have it local, 15 have it remote.

for data locality to work properly, you need the executors to be reading in the blocks of
data local to them, and not data from other parts of the files. Spark does try to do locality,
but if there's only a limited set of executors, then more of the workload is remote vs local.

I don't know of an obvious way to get the metrics here of local vs remote; I don't see the
HDFS client library tracking that —though it should be the place to collect stats on local/remote/domain-socket-direct
IO. Does anyone know somewhere in the Spark metrics which tracks placement locality? If not,
both layers could have some more metrics added.

View raw message