velocity-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "E.R. van Es" <e...@cdls.nl>
Subject Re: stress application
Date Fri, 28 Oct 2005 12:42:52 GMT
Unfortunatly I think need the data objects all the time, because all reports generated contain
links to references wich are other data objects.  I'll check my code again to see if I can
drop anything, thanks for the hint.  I don't know what you mean with "velocity cling into
your objects"?? (english is not my natural language...)

Regards,
Eric

On Fri, 28 Oct 2005 12:34:10 +0000 (UTC)
"Henning P. Schmiedehausen" <hps@intermeta.de> wrote:

> "E.R. van Es" <eric@cdls.nl> writes:
> 
> > Hmm didn't test that, but it ran out of memory using 512Mb, it runs
> > fine with 800Mb...  I admit there might be a memory problem, as
> > there are over 450000 (!) data objects in memory but I don't want
> > the application to do disc reads for every data fetch (as it runs
> > over data fetching methods over 10000 times).
> 
> Do you actually need that number of data objects all the time? I
> remember a similar case where adding a single "foo = null" line to the
> code brought the memory usage from ~ 1.2 GB down to < 5 MB. If you
> drop the references to the data objects as soon as you don't need them
> any longer, you can bring your memory usage down dramatically.
> 
> Or does Velocity cling onto your objects? That would be an interesting
> case to check.
> 
> 	Best regards
> 		Henning
> 
> 
> >On Thu, 27 Oct 2005 14:14:23 +0100
> >Andrew Mason <andrew@assertis.co.uk> wrote:
> 
> >> It could just be that it actually DOES need more memory. Out of curiosity at

> >> what amount of memory does it stop being a problem ?
> >> 
> >> 
> >> On Thu October 27 2005 2:09 pm, E.R. van Es wrote:
> >> > Hi,
> >> >
> >> >   I gave the program 800Mb of memory, enabled caching and using a
> >> > BufferedWriter, and the program runs with no noticable slow-down at all.

> >> > Maybe I'll stress-test with twice the amount of information for fun :)
 Is
> >> > it possible to do something like "memory leak detection" using some kind
of
> >> > debugging on java programs?
> >> >
> >> > Greets,
> >> > Eric
> >> >
> >> > On Thu, 27 Oct 2005 09:26:00 +0200
> >> >
> >> > "E.R. van Es" <eric@cdls.nl> wrote:
> >> > > The data is read from a plain text file and put in simple 'data' objects
> >> > > first.  I need all data from the file to be available before processing
> >> > > the templates, so all data objects are saved in an ArrayList, the
> >> > > getData() method search through the data objects ArrayList for the
key
> >> > > matching and returns only those data objects needed in a new ArrayList.
> >> > >
> >> > > Greetings,
> >> > > Eric
> >> > >
> >> > > On Thu, 27 Oct 2005 08:09:57 +0100
> >> > >
> >> > > Andrew Mason <andrew@assertis.co.uk> wrote:
> >> > > > Just out of curiosity where is it gettng the data from ? database
? If
> >> > > > so are you using a connection pool?
> >> > > >
> >> > > > private
> >> > > >
> >> > > > On Thu October 27 2005 8:03 am, E.R. van Es wrote:
> >> > > > > Hi,
> >> > > > >
> >> > > > > I'm running a test now with the BufferedWriter, we'll see
what
> >> > > > > happens. About the getData() method, maybe this actually
is the leak,
> >> > > > > it does this:
> >> > > > >
> >> > > > > *** CODE ***
> >> > > > > private ArrayList getData(String key) {
> >> > > > > 	ArrayList temp = new ArrayList();
> >> > > > > 	... fetch data here and fill temp ...
> >> > > > > 	return temp;
> >> > > > > }
> >> > > > > *** CODE END ***
> >> > > > >
> >> > > > > I'm not a complete java guru, but is there a new temp ArrayList
> >> > > > > allocated in memory every time I call this method?  won't
previous
> >> > > > > created temp ArrayLists be collected by the garbage collector?
> >> > > > >
> >> > > > >
> >> > > > > Greetings,
> >> > > > > Eric
> >> > > > >
> >> > > > > On Wed, 26 Oct 2005 11:57:08 -0400
> >> > > > >
> >> > > > > James Kebinger <jkebinger@gmail.com> wrote:
> >> > > > > > I would just turn up the amount of memory java can
use.
> >> > > > > > Does point to a possible memory leak somewhere though.
What does
> >> > > > > > getData() do? Note you're also shadowing the data variable
in
> >> > > > > > processData().
> >> > > > > >
> >> > > > > > On 10/26/05, E.R. van Es <eric@cdls.nl> wrote:
> >> > > > > > > Hi Andrew,
> >> > > > > > >
> >> > > > > > > does this really make a memory difference? The
generated files
> >> > > > > > > themselves are not that big (up to 50kb each)
and the writer is
> >> > > > > > > closed after each file... I didn't thought about
it so I'll try
> >> > > > > > > of course :)
> >> > > > > > >
> >> > > > > > > Greetings,
> >> > > > > > > Eric
> >> > > > > > >
> >> > > > > > > On Wed, 26 Oct 2005 16:12:28 +0100
> >> > > > > > >
> >> > > > > > > Andrew Mason <andrew@assertis.co.uk> wrote:
> >> > > > > > > > Have you thought about using a buffered writer
?
> >> > > > > > > >
> >> > > > > > > > On Wed October 26 2005 4:08 pm, E.R. van
Es wrote:
> >> > > > > > > > > Hello Velocity List,
> >> > > > > > > > >
> >> > > > > > > > > i have to create some big monthly reports
on CD. I use a
> >> > > > > > > > > simple java program to generate all
reports in HTML files
> >> > > > > > > > > using velocity
> >> > > > > > >
> >> > > > > > > templates.
> >> > > > > > >
> >> > > > > > > > > All data is pre-loaded and then the
reports are created by
> >> > > > > > > > > iterating
> >> > > > > > >
> >> > > > > > > over
> >> > > > > > >
> >> > > > > > > > > the data. However, there are too many
files to create (over
> >> > > > > > > > > 9600) and after file 7850 I get a java
memory exception. I
> >> > > > > > > > > use a scheme like
> >> > > > > > >
> >> > > > > > > this:
> >> > > > > > > > > *** JAVA CODE ***
> >> > > > > > > > > private Context context;
> >> > > > > > > > >
> >> > > > > > > > > private void processData() {
> >> > > > > > > > > ArrayList data = getDataKeys(); // this
returns an array with
> >> > > > > > > > > all
> >> > > > > > >
> >> > > > > > > "data
> >> > > > > > >
> >> > > > > > > > > keys" Iterator i = data.iterator();
> >> > > > > > > > > while (i.hasNext()) {
> >> > > > > > > > > String key = (String) i.next();
> >> > > > > > > > > ArrayList data = getData(key); // get
all data objects for
> >> > > > > > > > > this key. context.put("key", key);
> >> > > > > > > > > context.put("data", data);
> >> > > > > > > > > processTemplate("template.vm", key +
".html");
> >> > > > > > > > > }
> >> > > > > > > > > }
> >> > > > > > > > >
> >> > > > > > > > > private void processTemplate(String
template, String
> >> > > > > > > > > outputFile) { FileWriter outputFileWriter
= new
> >> > > > > > > > > FileWriter(new File(outputFile));
> >> > > > > > > > >
> >> > > > > > > > > Template x = Velocity.getTemplate(template);
> >> > > > > > > > > x.merge(context, outputFileWriter);
> >> > > > > > > > >
> >> > > > > > > > > outputFileWriter.close();
> >> > > > > > > > > }
> >> > > > > > > > > *** JAVA CODE END ***
> >> > > > > > > > >
> >> > > > > > > > > *** TEMPLATE CODE ***
> >> > > > > > > > > <html>
> >> > > > > > > > > ... bla bla ...
> >> > > > > > > > > #foreach ( $el in $data )
> >> > > > > > > > > $data.getX()<br>
> >> > > > > > > > > $data.getY()<br>
> >> > > > > > > > > and so on and so forth...
> >> > > > > > > > > #end
> >> > > > > > > > > ... bla bla ...
> >> > > > > > > > > </html>
> >> > > > > > > > > *** TEMPLATE CODE END ***
> >> > > > > > > > >
> >> > > > > > > > > As you can see I put every data class
in the context with
> >> > > > > > > > > identifier "data", my thought was to
overwrite the previous
> >> > > > > > > > > one and NOT get a
> >> > > > > > >
> >> > > > > > > memory
> >> > > > > > >
> >> > > > > > > > > problem, however I do get the memory
problem. I can adjust
> >> > > > > > > > > the java
> >> > > > > > >
> >> > > > > > > memory
> >> > > > > > >
> >> > > > > > > > > allocation to 1Gb but I was wondering
if my approach just
> >> > > > > > > > > isn't
> >> > > > > > >
> >> > > > > > > correct?
> >> > > > > > >
> >> > > > > > > > > I'm not sure about the processTemplate
method, though it
> >> > > > > > > > > workes
> >> > > > > > >
> >> > > > > > > perfectly
> >> > > > > > >
> >> > > > > > > > > for the first 7800 files :) By the way:
I'm using
> >> > > > > > > > > velocity-1.4.
> >> > > > > > > > >
> >> > > > > > > > > Thanks in advance,
> >> > > > > > > > > Eric
> >> > > > > > > > >
> >> > > > > > > > > -------------------------------------------------------------
> >> > > > > > > > >------ -- To unsubscribe, e-mail:
> >> > > > > > > > > velocity-user-unsubscribe@jakarta.apache.org
For additional
> >> > > > > > > > > commands, e-mail: velocity-user-help@jakarta.apache.org
> >> > > > > > > >
> >> > > > > > > > ---------------------------------------------------------------
> >> > > > > > > >------ To unsubscribe, e-mail:
> >> > > > > > > > velocity-user-unsubscribe@jakarta.apache.org
For additional
> >> > > > > > > > commands, e-mail:
> >> > > > > > > > velocity-user-help@jakarta.apache.org
> >> > > > > > >
> >> > > > > > > -----------------------------------------------------------------
> >> > > > > > >---- To unsubscribe, e-mail:
> >> > > > > > > velocity-user-unsubscribe@jakarta.apache.org For
additional
> >> > > > > > > commands, e-mail: velocity-user-help@jakarta.apache.org
> >> > > > >
> >> > > > > ---------------------------------------------------------------------
> >> > > > > To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> >> > > > > For additional commands, e-mail:
> >> > > > > velocity-user-help@jakarta.apache.org
> >> > > >
> >> > > > ---------------------------------------------------------------------
> >> > > > To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> >> > > > For additional commands, e-mail: velocity-user-help@jakarta.apache.org
> >> > >
> >> > > ---------------------------------------------------------------------
> >> > > To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> >> > > For additional commands, e-mail: velocity-user-help@jakarta.apache.org
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> >> > For additional commands, e-mail: velocity-user-help@jakarta.apache.org
> >> 
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> >> For additional commands, e-mail: velocity-user-help@jakarta.apache.org
> >> 
> 
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> >For additional commands, e-mail: velocity-user-help@jakarta.apache.org
> 
> -- 
> Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
> hps@intermeta.de        +49 9131 50 654 0   http://www.intermeta.de/
> 
> RedHat Certified Engineer -- Jakarta Turbine Development  -- hero for hire
>    Linux, Java, perl, Solaris -- Consulting, Training, Development
> 
> 		      4 - 8 - 15 - 16 - 23 - 42
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: velocity-user-help@jakarta.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: velocity-user-help@jakarta.apache.org


Mime
View raw message