velocity-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Mason <and...@assertis.co.uk>
Subject Re: stress application
Date Fri, 28 Oct 2005 13:14:59 GMT
What structure are your objects stored in and how are they related. Can you 
give an example of your object mapping ?

Is it possible that you could create a btree of the files and the file name 
(some sort of index), then use an object to manage these. 

context.put("data_manager", data_manager);

$data_manager.get("xyz")

the .get() method would then search the btree, and open the appropriate file, 
and extract the data. This will take slightly longer, but you would only be 
using the memory you needed at that time + what ever the btree uses.

On Fri October 28 2005 1:42 pm, E.R. van Es wrote:
> Unfortunatly I think need the data objects all the time, because all
> reports generated contain links to references wich are other data objects. 
> I'll check my code again to see if I can drop anything, thanks for the
> hint.  I don't know what you mean with "velocity cling into your objects"??
> (english is not my natural language...)
>
> Regards,
> Eric
>
> On Fri, 28 Oct 2005 12:34:10 +0000 (UTC)
>
> "Henning P. Schmiedehausen" <hps@intermeta.de> wrote:
> > "E.R. van Es" <eric@cdls.nl> writes:
> > > Hmm didn't test that, but it ran out of memory using 512Mb, it runs
> > > fine with 800Mb...  I admit there might be a memory problem, as
> > > there are over 450000 (!) data objects in memory but I don't want
> > > the application to do disc reads for every data fetch (as it runs
> > > over data fetching methods over 10000 times).
> >
> > Do you actually need that number of data objects all the time? I
> > remember a similar case where adding a single "foo = null" line to the
> > code brought the memory usage from ~ 1.2 GB down to < 5 MB. If you
> > drop the references to the data objects as soon as you don't need them
> > any longer, you can bring your memory usage down dramatically.
> >
> > Or does Velocity cling onto your objects? That would be an interesting
> > case to check.
> >
> > 	Best regards
> > 		Henning
> >
> > >On Thu, 27 Oct 2005 14:14:23 +0100
> > >
> > >Andrew Mason <andrew@assertis.co.uk> wrote:
> > >> It could just be that it actually DOES need more memory. Out of
> > >> curiosity at what amount of memory does it stop being a problem ?
> > >>
> > >> On Thu October 27 2005 2:09 pm, E.R. van Es wrote:
> > >> > Hi,
> > >> >
> > >> >   I gave the program 800Mb of memory, enabled caching and using a
> > >> > BufferedWriter, and the program runs with no noticable slow-down at
> > >> > all. Maybe I'll stress-test with twice the amount of information for
> > >> > fun :)  Is it possible to do something like "memory leak detection"
> > >> > using some kind of debugging on java programs?
> > >> >
> > >> > Greets,
> > >> > Eric
> > >> >
> > >> > On Thu, 27 Oct 2005 09:26:00 +0200
> > >> >
> > >> > "E.R. van Es" <eric@cdls.nl> wrote:
> > >> > > The data is read from a plain text file and put in simple 'data'
> > >> > > objects first.  I need all data from the file to be available
> > >> > > before processing the templates, so all data objects are saved
in
> > >> > > an ArrayList, the getData() method search through the data objects
> > >> > > ArrayList for the key matching and returns only those data objects
> > >> > > needed in a new ArrayList.
> > >> > >
> > >> > > Greetings,
> > >> > > Eric
> > >> > >
> > >> > > On Thu, 27 Oct 2005 08:09:57 +0100
> > >> > >
> > >> > > Andrew Mason <andrew@assertis.co.uk> wrote:
> > >> > > > Just out of curiosity where is it gettng the data from ?
> > >> > > > database ? If so are you using a connection pool?
> > >> > > >
> > >> > > > private
> > >> > > >
> > >> > > > On Thu October 27 2005 8:03 am, E.R. van Es wrote:
> > >> > > > > Hi,
> > >> > > > >
> > >> > > > > I'm running a test now with the BufferedWriter, we'll
see what
> > >> > > > > happens. About the getData() method, maybe this actually
is
> > >> > > > > the leak, it does this:
> > >> > > > >
> > >> > > > > *** CODE ***
> > >> > > > > private ArrayList getData(String key) {
> > >> > > > > 	ArrayList temp = new ArrayList();
> > >> > > > > 	... fetch data here and fill temp ...
> > >> > > > > 	return temp;
> > >> > > > > }
> > >> > > > > *** CODE END ***
> > >> > > > >
> > >> > > > > I'm not a complete java guru, but is there a new temp
> > >> > > > > ArrayList allocated in memory every time I call this
method? 
> > >> > > > > won't previous created temp ArrayLists be collected
by the
> > >> > > > > garbage collector?
> > >> > > > >
> > >> > > > >
> > >> > > > > Greetings,
> > >> > > > > Eric
> > >> > > > >
> > >> > > > > On Wed, 26 Oct 2005 11:57:08 -0400
> > >> > > > >
> > >> > > > > James Kebinger <jkebinger@gmail.com> wrote:
> > >> > > > > > I would just turn up the amount of memory java
can use.
> > >> > > > > > Does point to a possible memory leak somewhere
though. What
> > >> > > > > > does getData() do? Note you're also shadowing
the data
> > >> > > > > > variable in processData().
> > >> > > > > >
> > >> > > > > > On 10/26/05, E.R. van Es <eric@cdls.nl>
wrote:
> > >> > > > > > > Hi Andrew,
> > >> > > > > > >
> > >> > > > > > > does this really make a memory difference?
The generated
> > >> > > > > > > files themselves are not that big (up to
50kb each) and
> > >> > > > > > > the writer is closed after each file... I
didn't thought
> > >> > > > > > > about it so I'll try of course :)
> > >> > > > > > >
> > >> > > > > > > Greetings,
> > >> > > > > > > Eric
> > >> > > > > > >
> > >> > > > > > > On Wed, 26 Oct 2005 16:12:28 +0100
> > >> > > > > > >
> > >> > > > > > > Andrew Mason <andrew@assertis.co.uk>
wrote:
> > >> > > > > > > > Have you thought about using a buffered
writer ?
> > >> > > > > > > >
> > >> > > > > > > > On Wed October 26 2005 4:08 pm, E.R.
van Es wrote:
> > >> > > > > > > > > Hello Velocity List,
> > >> > > > > > > > >
> > >> > > > > > > > > i have to create some big monthly
reports on CD. I use
> > >> > > > > > > > > a simple java program to generate
all reports in HTML
> > >> > > > > > > > > files using velocity
> > >> > > > > > >
> > >> > > > > > > templates.
> > >> > > > > > >
> > >> > > > > > > > > All data is pre-loaded and then
the reports are
> > >> > > > > > > > > created by iterating
> > >> > > > > > >
> > >> > > > > > > over
> > >> > > > > > >
> > >> > > > > > > > > the data. However, there are too
many files to create
> > >> > > > > > > > > (over 9600) and after file 7850
I get a java memory
> > >> > > > > > > > > exception. I use a scheme like
> > >> > > > > > >
> > >> > > > > > > this:
> > >> > > > > > > > > *** JAVA CODE ***
> > >> > > > > > > > > private Context context;
> > >> > > > > > > > >
> > >> > > > > > > > > private void processData() {
> > >> > > > > > > > > ArrayList data = getDataKeys();
// this returns an
> > >> > > > > > > > > array with all
> > >> > > > > > >
> > >> > > > > > > "data
> > >> > > > > > >
> > >> > > > > > > > > keys" Iterator i = data.iterator();
> > >> > > > > > > > > while (i.hasNext()) {
> > >> > > > > > > > > String key = (String) i.next();
> > >> > > > > > > > > ArrayList data = getData(key);
// get all data objects
> > >> > > > > > > > > for this key. context.put("key",
key);
> > >> > > > > > > > > context.put("data", data);
> > >> > > > > > > > > processTemplate("template.vm",
key + ".html");
> > >> > > > > > > > > }
> > >> > > > > > > > > }
> > >> > > > > > > > >
> > >> > > > > > > > > private void processTemplate(String
template, String
> > >> > > > > > > > > outputFile) { FileWriter outputFileWriter
= new
> > >> > > > > > > > > FileWriter(new File(outputFile));
> > >> > > > > > > > >
> > >> > > > > > > > > Template x = Velocity.getTemplate(template);
> > >> > > > > > > > > x.merge(context, outputFileWriter);
> > >> > > > > > > > >
> > >> > > > > > > > > outputFileWriter.close();
> > >> > > > > > > > > }
> > >> > > > > > > > > *** JAVA CODE END ***
> > >> > > > > > > > >
> > >> > > > > > > > > *** TEMPLATE CODE ***
> > >> > > > > > > > > <html>
> > >> > > > > > > > > ... bla bla ...
> > >> > > > > > > > > #foreach ( $el in $data )
> > >> > > > > > > > > $data.getX()<br>
> > >> > > > > > > > > $data.getY()<br>
> > >> > > > > > > > > and so on and so forth...
> > >> > > > > > > > > #end
> > >> > > > > > > > > ... bla bla ...
> > >> > > > > > > > > </html>
> > >> > > > > > > > > *** TEMPLATE CODE END ***
> > >> > > > > > > > >
> > >> > > > > > > > > As you can see I put every data
class in the context
> > >> > > > > > > > > with identifier "data", my thought
was to overwrite
> > >> > > > > > > > > the previous one and NOT get a
> > >> > > > > > >
> > >> > > > > > > memory
> > >> > > > > > >
> > >> > > > > > > > > problem, however I do get the memory
problem. I can
> > >> > > > > > > > > adjust the java
> > >> > > > > > >
> > >> > > > > > > memory
> > >> > > > > > >
> > >> > > > > > > > > allocation to 1Gb but I was wondering
if my approach
> > >> > > > > > > > > just isn't
> > >> > > > > > >
> > >> > > > > > > correct?
> > >> > > > > > >
> > >> > > > > > > > > I'm not sure about the processTemplate
method, though
> > >> > > > > > > > > it workes
> > >> > > > > > >
> > >> > > > > > > perfectly
> > >> > > > > > >
> > >> > > > > > > > > for the first 7800 files :) By
the way: I'm using
> > >> > > > > > > > > velocity-1.4.
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks in advance,
> > >> > > > > > > > > Eric
> > >> > > > > > > > >
> > >> > > > > > > > > ------------------------------------------------------
> > >> > > > > > > > >------- ------ -- To unsubscribe,
e-mail:
> > >> > > > > > > > > velocity-user-unsubscribe@jakarta.apache.org
For
> > >> > > > > > > > > additional commands, e-mail:
> > >> > > > > > > > > velocity-user-help@jakarta.apache.org
> > >> > > > > > > >
> > >> > > > > > > > --------------------------------------------------------
> > >> > > > > > > >------- ------ To unsubscribe, e-mail:
> > >> > > > > > > > velocity-user-unsubscribe@jakarta.apache.org
For
> > >> > > > > > > > additional commands, e-mail:
> > >> > > > > > > > velocity-user-help@jakarta.apache.org
> > >> > > > > > >
> > >> > > > > > > ----------------------------------------------------------
> > >> > > > > > >------- ---- To unsubscribe, e-mail:
> > >> > > > > > > velocity-user-unsubscribe@jakarta.apache.org
For
> > >> > > > > > > additional commands, e-mail:
> > >> > > > > > > velocity-user-help@jakarta.apache.org
> > >> > > > >
> > >> > > > > --------------------------------------------------------------
> > >> > > > >------- To unsubscribe, e-mail:
> > >> > > > > velocity-user-unsubscribe@jakarta.apache.org For additional
> > >> > > > > commands, e-mail:
> > >> > > > > velocity-user-help@jakarta.apache.org
> > >> > > >
> > >> > > > ----------------------------------------------------------------
> > >> > > >----- To unsubscribe, e-mail:
> > >> > > > velocity-user-unsubscribe@jakarta.apache.org For additional
> > >> > > > commands, e-mail: velocity-user-help@jakarta.apache.org
> > >> > >
> > >> > > ------------------------------------------------------------------
> > >> > >--- To unsubscribe, e-mail:
> > >> > > velocity-user-unsubscribe@jakarta.apache.org For additional
> > >> > > commands, e-mail: velocity-user-help@jakarta.apache.org
> > >> >
> > >> > --------------------------------------------------------------------
> > >> >- To unsubscribe, e-mail:
> > >> > velocity-user-unsubscribe@jakarta.apache.org For additional
> > >> > commands, e-mail: velocity-user-help@jakarta.apache.org
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> > >> For additional commands, e-mail: velocity-user-help@jakarta.apache.org
> > >
> > >---------------------------------------------------------------------
> > >To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> > >For additional commands, e-mail: velocity-user-help@jakarta.apache.org
> >
> > --
> > Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
> > hps@intermeta.de        +49 9131 50 654 0   http://www.intermeta.de/
> >
> > RedHat Certified Engineer -- Jakarta Turbine Development  -- hero for
> > hire Linux, Java, perl, Solaris -- Consulting, Training, Development
> >
> > 		      4 - 8 - 15 - 16 - 23 - 42
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: velocity-user-help@jakarta.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: velocity-user-help@jakarta.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: velocity-user-help@jakarta.apache.org


Mime
View raw message