hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-11029) LocalFS Statistics performs thread local call per byte written
Date Fri, 29 Aug 2014 20:49:53 GMT
Gopal V created HADOOP-11029:
--------------------------------

             Summary: LocalFS Statistics performs thread local call per byte written
                 Key: HADOOP-11029
                 URL: https://issues.apache.org/jira/browse/HADOOP-11029
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Gopal V
         Attachments: local-fs-locking.png

This code is there in the hot-path of IFile writer via RawLocalFileSystem.

!local-fs-locking.png!

>From a preliminary glance, the lock prefix calls are coming from a threadlocal.get() within
FileSystem.Statistics 

{code}
   /**
     * Get or create the thread-local data associated with the current thread.
     */
    private StatisticsData getThreadData() {
      StatisticsData data = threadData.get();
      if (data == null) {
        data = new StatisticsData(
            new WeakReference<Thread>(Thread.currentThread()));
        threadData.set(data);
        synchronized(this) {
          if (allData == null) {
            allData = new LinkedList<StatisticsData>();
          }
          allData.add(data);
        }
      }
      return data;
    }

    /**
     * Increment the bytes read in the statistics
     * @param newBytes the additional bytes read
     */
    public void incrementBytesRead(long newBytes) {
      getThreadData().bytesRead += newBytes;
    }
{code}

This is incredibly inefficient when used from FSDataOutputStream

{code}
    public void write(int b) throws IOException {
      out.write(b);
      position++;
      if (statistics != null) {
        statistics.incrementBytesWritten(1);
      }
    }
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message