incubator-kato-spec mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carmine Cristallo <>
Subject Native Memory Analisys Tool - some user stories
Date Wed, 21 Jan 2009 13:19:18 GMT
My name is Carmine Cristallo, and I work at IBM in Steve's team.
I would like to start a discussion about the user stories that can be
implemented by a Native Memory Analysis Tool. According to Steve's
definition, NMAT is:

"A program that can retrieve native memory allocations by (or on
behalf of) a Java Runtime and provide trace back to the  Java objects
that hold the allocation reference. The tool will be able to display
what memory allocations exist, the contents of the allocation, and
conservatively identify which entities hold references to that
allocation. Ideally this tool will be able to point to specific fields
within Java objects that hold the references. This tool will
demonstrate the capabilities of the API to find and display native
memory  from a memory manager. Key characteristics will include the
performance of the API in  exhaustively scanning a dump (for memory
allocation handles) and the ability to resolve an arbitrary location
within the dump into a Java object or similar entity."

We had a very interesting discussion last week with our support team,
the outcomes of which are summarised by the following points:

 - most of the native memory problems reported by customers aren't
really imputable to the JVM itself. They're either heap-driven (i.e.
caused by leaking Java objects holding references to native memory),
or JNI-related;

 - there are other possible sources of failing memory allocations
(system running out of file descriptors, too many active threads, too
many loaded classes, etc.) but they're far less frequent;

 - at the moment, the most reliable way of diagnosing such problems
involves monitoring allocations at OS level on live VMs. A tool like
NMAT would be very useful.

I'm trying to put my head around a list of possible user stories for
such a tool.
What I have in mind is some sort of command line tool which would be able to:

 - open a dump;
 - list the address spaces in the dump;
 - for a given address space, list the processes in it;
 - for a given process, list the memory allocations. Now, this is the
tricky one... To implement this scenario without changes in all the
supported VMs (eyecatchers, etc.), we will need to figure out a way of
finding the allocated areas in the dump. Searching for malloc headers
could be a way, although I'm not sure it's feasible for all platforms.
This is the single most important item that will need to be
investigated for the implementation of NMAT;
 - for a given memory allocation, list the Java objects which hold a
reference to it;
 - for a given memory allocation, display how big it is;
 - list all the Java objects holding reference to native memory;
 - list all the memory allocations to which no Java object hold references.

It would also be interesting to provide a link to MAT, for example to
calculate the dominator tree of an object holding a reference to
native memory.

Of course writing NMAT as a command line tool is just an idea. Maybe
it would make more sense to integrate it as part of Adam's Explorers,
but what's important for now is to reach an agreement about the set of
user stories we would like to implement, because that will be the
direct driver for the definition of the API.

I would be delighted to hear from you. The list of user stories is by
no means exhaustive, and I'm sure I'm missing something relevant to
the native memory analysis domain. Any suggestion would be extremely

Thank you! :-)


View raw message