velocity-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Henning P. Schmiedehausen" <...@intermeta.de>
Subject Re: stress application
Date Fri, 28 Oct 2005 12:34:10 GMT
"E.R. van Es" <eric@cdls.nl> writes:

> Hmm didn't test that, but it ran out of memory using 512Mb, it runs
> fine with 800Mb...  I admit there might be a memory problem, as
> there are over 450000 (!) data objects in memory but I don't want
> the application to do disc reads for every data fetch (as it runs
> over data fetching methods over 10000 times).

Do you actually need that number of data objects all the time? I
remember a similar case where adding a single "foo = null" line to the
code brought the memory usage from ~ 1.2 GB down to < 5 MB. If you
drop the references to the data objects as soon as you don't need them
any longer, you can bring your memory usage down dramatically.

Or does Velocity cling onto your objects? That would be an interesting
case to check.

	Best regards
		Henning


>On Thu, 27 Oct 2005 14:14:23 +0100
>Andrew Mason <andrew@assertis.co.uk> wrote:

>> It could just be that it actually DOES need more memory. Out of curiosity at 
>> what amount of memory does it stop being a problem ?
>> 
>> 
>> On Thu October 27 2005 2:09 pm, E.R. van Es wrote:
>> > Hi,
>> >
>> >   I gave the program 800Mb of memory, enabled caching and using a
>> > BufferedWriter, and the program runs with no noticable slow-down at all. 
>> > Maybe I'll stress-test with twice the amount of information for fun :)  Is
>> > it possible to do something like "memory leak detection" using some kind of
>> > debugging on java programs?
>> >
>> > Greets,
>> > Eric
>> >
>> > On Thu, 27 Oct 2005 09:26:00 +0200
>> >
>> > "E.R. van Es" <eric@cdls.nl> wrote:
>> > > The data is read from a plain text file and put in simple 'data' objects
>> > > first.  I need all data from the file to be available before processing
>> > > the templates, so all data objects are saved in an ArrayList, the
>> > > getData() method search through the data objects ArrayList for the key
>> > > matching and returns only those data objects needed in a new ArrayList.
>> > >
>> > > Greetings,
>> > > Eric
>> > >
>> > > On Thu, 27 Oct 2005 08:09:57 +0100
>> > >
>> > > Andrew Mason <andrew@assertis.co.uk> wrote:
>> > > > Just out of curiosity where is it gettng the data from ? database
? If
>> > > > so are you using a connection pool?
>> > > >
>> > > > private
>> > > >
>> > > > On Thu October 27 2005 8:03 am, E.R. van Es wrote:
>> > > > > Hi,
>> > > > >
>> > > > > I'm running a test now with the BufferedWriter, we'll see what
>> > > > > happens. About the getData() method, maybe this actually is the
leak,
>> > > > > it does this:
>> > > > >
>> > > > > *** CODE ***
>> > > > > private ArrayList getData(String key) {
>> > > > > 	ArrayList temp = new ArrayList();
>> > > > > 	... fetch data here and fill temp ...
>> > > > > 	return temp;
>> > > > > }
>> > > > > *** CODE END ***
>> > > > >
>> > > > > I'm not a complete java guru, but is there a new temp ArrayList
>> > > > > allocated in memory every time I call this method?  won't previous
>> > > > > created temp ArrayLists be collected by the garbage collector?
>> > > > >
>> > > > >
>> > > > > Greetings,
>> > > > > Eric
>> > > > >
>> > > > > On Wed, 26 Oct 2005 11:57:08 -0400
>> > > > >
>> > > > > James Kebinger <jkebinger@gmail.com> wrote:
>> > > > > > I would just turn up the amount of memory java can use.
>> > > > > > Does point to a possible memory leak somewhere though. What
does
>> > > > > > getData() do? Note you're also shadowing the data variable
in
>> > > > > > processData().
>> > > > > >
>> > > > > > On 10/26/05, E.R. van Es <eric@cdls.nl> wrote:
>> > > > > > > Hi Andrew,
>> > > > > > >
>> > > > > > > does this really make a memory difference? The generated
files
>> > > > > > > themselves are not that big (up to 50kb each) and the
writer is
>> > > > > > > closed after each file... I didn't thought about it
so I'll try
>> > > > > > > of course :)
>> > > > > > >
>> > > > > > > Greetings,
>> > > > > > > Eric
>> > > > > > >
>> > > > > > > On Wed, 26 Oct 2005 16:12:28 +0100
>> > > > > > >
>> > > > > > > Andrew Mason <andrew@assertis.co.uk> wrote:
>> > > > > > > > Have you thought about using a buffered writer
?
>> > > > > > > >
>> > > > > > > > On Wed October 26 2005 4:08 pm, E.R. van Es wrote:
>> > > > > > > > > Hello Velocity List,
>> > > > > > > > >
>> > > > > > > > > i have to create some big monthly reports
on CD. I use a
>> > > > > > > > > simple java program to generate all reports
in HTML files
>> > > > > > > > > using velocity
>> > > > > > >
>> > > > > > > templates.
>> > > > > > >
>> > > > > > > > > All data is pre-loaded and then the reports
are created by
>> > > > > > > > > iterating
>> > > > > > >
>> > > > > > > over
>> > > > > > >
>> > > > > > > > > the data. However, there are too many files
to create (over
>> > > > > > > > > 9600) and after file 7850 I get a java memory
exception. I
>> > > > > > > > > use a scheme like
>> > > > > > >
>> > > > > > > this:
>> > > > > > > > > *** JAVA CODE ***
>> > > > > > > > > private Context context;
>> > > > > > > > >
>> > > > > > > > > private void processData() {
>> > > > > > > > > ArrayList data = getDataKeys(); // this returns
an array with
>> > > > > > > > > all
>> > > > > > >
>> > > > > > > "data
>> > > > > > >
>> > > > > > > > > keys" Iterator i = data.iterator();
>> > > > > > > > > while (i.hasNext()) {
>> > > > > > > > > String key = (String) i.next();
>> > > > > > > > > ArrayList data = getData(key); // get all
data objects for
>> > > > > > > > > this key. context.put("key", key);
>> > > > > > > > > context.put("data", data);
>> > > > > > > > > processTemplate("template.vm", key + ".html");
>> > > > > > > > > }
>> > > > > > > > > }
>> > > > > > > > >
>> > > > > > > > > private void processTemplate(String template,
String
>> > > > > > > > > outputFile) { FileWriter outputFileWriter
= new
>> > > > > > > > > FileWriter(new File(outputFile));
>> > > > > > > > >
>> > > > > > > > > Template x = Velocity.getTemplate(template);
>> > > > > > > > > x.merge(context, outputFileWriter);
>> > > > > > > > >
>> > > > > > > > > outputFileWriter.close();
>> > > > > > > > > }
>> > > > > > > > > *** JAVA CODE END ***
>> > > > > > > > >
>> > > > > > > > > *** TEMPLATE CODE ***
>> > > > > > > > > <html>
>> > > > > > > > > ... bla bla ...
>> > > > > > > > > #foreach ( $el in $data )
>> > > > > > > > > $data.getX()<br>
>> > > > > > > > > $data.getY()<br>
>> > > > > > > > > and so on and so forth...
>> > > > > > > > > #end
>> > > > > > > > > ... bla bla ...
>> > > > > > > > > </html>
>> > > > > > > > > *** TEMPLATE CODE END ***
>> > > > > > > > >
>> > > > > > > > > As you can see I put every data class in
the context with
>> > > > > > > > > identifier "data", my thought was to overwrite
the previous
>> > > > > > > > > one and NOT get a
>> > > > > > >
>> > > > > > > memory
>> > > > > > >
>> > > > > > > > > problem, however I do get the memory problem.
I can adjust
>> > > > > > > > > the java
>> > > > > > >
>> > > > > > > memory
>> > > > > > >
>> > > > > > > > > allocation to 1Gb but I was wondering if
my approach just
>> > > > > > > > > isn't
>> > > > > > >
>> > > > > > > correct?
>> > > > > > >
>> > > > > > > > > I'm not sure about the processTemplate method,
though it
>> > > > > > > > > workes
>> > > > > > >
>> > > > > > > perfectly
>> > > > > > >
>> > > > > > > > > for the first 7800 files :) By the way: I'm
using
>> > > > > > > > > velocity-1.4.
>> > > > > > > > >
>> > > > > > > > > Thanks in advance,
>> > > > > > > > > Eric
>> > > > > > > > >
>> > > > > > > > > -------------------------------------------------------------
>> > > > > > > > >------ -- To unsubscribe, e-mail:
>> > > > > > > > > velocity-user-unsubscribe@jakarta.apache.org
For additional
>> > > > > > > > > commands, e-mail: velocity-user-help@jakarta.apache.org
>> > > > > > > >
>> > > > > > > > ---------------------------------------------------------------
>> > > > > > > >------ To unsubscribe, e-mail:
>> > > > > > > > velocity-user-unsubscribe@jakarta.apache.org For
additional
>> > > > > > > > commands, e-mail:
>> > > > > > > > velocity-user-help@jakarta.apache.org
>> > > > > > >
>> > > > > > > -----------------------------------------------------------------
>> > > > > > >---- To unsubscribe, e-mail:
>> > > > > > > velocity-user-unsubscribe@jakarta.apache.org For additional
>> > > > > > > commands, e-mail: velocity-user-help@jakarta.apache.org
>> > > > >
>> > > > > ---------------------------------------------------------------------
>> > > > > To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
>> > > > > For additional commands, e-mail:
>> > > > > velocity-user-help@jakarta.apache.org
>> > > >
>> > > > ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
>> > > > For additional commands, e-mail: velocity-user-help@jakarta.apache.org
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
>> > > For additional commands, e-mail: velocity-user-help@jakarta.apache.org
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
>> > For additional commands, e-mail: velocity-user-help@jakarta.apache.org
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: velocity-user-help@jakarta.apache.org
>> 

>---------------------------------------------------------------------
>To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: velocity-user-help@jakarta.apache.org

-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
hps@intermeta.de        +49 9131 50 654 0   http://www.intermeta.de/

RedHat Certified Engineer -- Jakarta Turbine Development  -- hero for hire
   Linux, Java, perl, Solaris -- Consulting, Training, Development

		      4 - 8 - 15 - 16 - 23 - 42

---------------------------------------------------------------------
To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: velocity-user-help@jakarta.apache.org


Mime
View raw message