velocity-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <lawre...@marchex.com>
Subject Re: OutOfMemoryException Processing Velocity Template
Date Fri, 06 May 2005 22:45:40 GMT
Ok I tried using JProfiler and used a StringWriter, and found that the item that was taking
the most memory (150 MB) was..... char[]!  Ok now I'm completely stumped and clueless.

-L

On Fri, May 06, 2005 at 02:51:15PM -0700, Will Glass-Husain wrote:
> You might consider trying a profiler.  Most of them have trial version 
> available.  This could tell you exactly where the memory is going.
> 
> I just bought JProfiler, and it seems a bit better priced than the others. 
> Here's a user review:
> http://www.jlamp.com/blog/2005/02/01/1107280975000.html
> 
> You might also check out Hyades from the Eclipse project for an open source 
> option.
> 
> WILL
> 
> 
> ----- Original Message ----- 
> From: <lawrence@marchex.com>
> To: "Velocity Users List" <velocity-user@jakarta.apache.org>
> Sent: Friday, May 06, 2005 2:39 PM
> Subject: Re: OutOfMemoryException Processing Velocity Template
> 
> 
> >This is why I'm so confused.   I wouldn't expect it to use so much memory. 
> >What I did to test the memory usage was:
> >
> >System.out.println("Free memory before merge: " + 
> >Runtime.getRuntime().freeMemory());
> >Velocity.mergeTemplate("/myfile.vm", "ISO-8859-1", context, w );
> >System.out.println("Free memory after merge: " + 
> >Runtime.getRuntime().freeMemory());
> >
> >And then setting the JVM to -Xms256M and -Xmx256M.   When I use a String 
> >writer, the free memory is reduced by ~200MB.   When I use an 
> >OutputStreamWriter, it is reduced ~13MB, a huge difference.
> >
> >BTW, I am using 1.4.2_05
> >
> >-L
> >
> >On Fri, May 06, 2005 at 02:22:01PM -0700, Will Glass-Husain wrote:
> >>Hi Lawrence,
> >>
> >>I got interested, and just ran some similar code with JProfiler looking 
> >>at
> >>VM memory usage.  I created a list of 60,000 objects, each with 10 
> >>100char
> >>columns.  Then I used StringWriter to merge as you did.
> >>
> >>The VM used 3MB at the start, 6.5 MB after creating the list, and 25 
> >>after
> >>merging.  (before I closed the writer).  Nowhere close to 200.  So 
> >>perhaps
> >>there's something else going on.  Although if the problem is one with
> >>StringWriter, a lot may depend on the specific JVM you're using.  (I'm
> >>using Sun 1.4.2_04-b05).
> >>
> >>WILL
> >>
> >>----- Original Message ----- 
> >>From: <lawrence@marchex.com>
> >>To: "Velocity Users List" <velocity-user@jakarta.apache.org>
> >>Sent: Friday, May 06, 2005 2:08 PM
> >>Subject: Re: OutOfMemoryException Processing Velocity Template
> >>
> >>
> >>>Ok now I'm confused (suprise, suprise!).   I looked at the source for
> >>>VelocityServlet, and mergeTemplate has:
> >>>
> >>>vw = (VelocityWriter) writerPool.get();
> >>>
> >>>if (vw == null)
> >>>{
> >>>   vw = new VelocityWriter(new OutputStreamWriter(output, encoding),4 *
> >>>1024, true);
> >>>}
> >>>else
> >>>{
> >>>   vw.recycle(new OutputStreamWriter(output, encoding));
> >>>}
> >>>template.merge(context, vw);
> >>>
> >>>===
> >>>
> >>>So the VelocityServlet DOES use OutputStreamWriter already ... so then 
> >>>why
> >>>does it still utilize so much memory?
> >>>
> >>>-L
> >>>
> >>>On Fri, May 06, 2005 at 02:02:07PM -0700, lawrence@marchex.com wrote:
> >>>>Alright, I changed the Writer to be:
> >>>>
> >>>>OutputStreamWriter w = new OutputStreamWriter(new
> >>>>FileOutputStream("test.html"));
> >>>>
> >>>>And this fixes the memory problem.
> >>>>
> >>>>However, my application is running Velocity as a servlet, how should
I
> >>>>extend the VelocityServlet to use an OutputStreamWriter intead of a
> >>>>StringWriter?  Should I post a different email thread about this?
> >>>>
> >>>>-L
> >>>>
> >>>>On Fri, May 06, 2005 at 01:59:19PM -0700, David You wrote:
> >>>>> Well, Open a file, use a FileWriter.
> >>>>>
> >>>>> David
> >>>>>
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: lawrence@marchex.com [mailto:lawrence@marchex.com]
> >>>>> Sent: Friday, May 06, 2005 1:48 PM
> >>>>> To: Velocity Users List
> >>>>> Subject: Re: OutOfMemoryException Processing Velocity Template
> >>>>>
> >>>>> I printed out the available memory before and after I call
> >>>>> "mergeTemplate":
> >>>>>
> >>>>> Free memory before merge: 217393352
> >>>>> Free memory after merge: 16271392
> >>>>>
> >>>>> so I would assume the StringWriter is approximately 200MB after
the
> >>>>> merge.   What should I use/do instead?
> >>>>>
> >>>>> -Lawrence
> >>>>>
> >>>>> On Fri, May 06, 2005 at 01:43:50PM -0700, David You wrote:
> >>>>> > Why do you use StringWriter? That is cached in the memory.
> >>>>> > How big is the StringWriter after the merge?
> >>>>> >
> >>>>> > David
> >>>>> >
> >>>>> >
> >>>>> > -----Original Message-----
> >>>>> > From: lawrence@marchex.com [mailto:lawrence@marchex.com]
> >>>>> > Sent: Friday, May 06, 2005 1:22 PM
> >>>>> > To: Velocity Users List
> >>>>> > Subject: Re: OutOfMemoryException Processing Velocity Template
> >>>>> >
> >>>>> > Alright, I ran this little piece of code:
> >>>>> >
> >>>>> > public static void main( String args[] ) throws Exception
> >>>>> >     {
> >>>>> >         /* first, we init the runtime engine.  Defaults are
fine. 
> >>>>> > */
> >>>>> >
> >>>>> >         Velocity.init();
> >>>>> >
> >>>>> >         /* lets make a Context and put data into it */
> >>>>> >
> >>>>> >         VelocityContext context = new VelocityContext();
> >>>>> >
> >>>>> >         List results = ReportBean.runLargeReport();
> >>>>> >
> >>>>> >         context.put("List", results);
> >>>>> >
> >>>>> >         /* lets render a template */
> >>>>> >
> >>>>> >         StringWriter w = new StringWriter();
> >>>>> >
> >>>>> >         Velocity.mergeTemplate("/myfile.vm", context, w );
> >>>>> >         System.out.println(" template : " + w );
> >>>>> >
> >>>>> >         /* lets make our own string to render */
> >>>>> >     }
> >>>>> >
> >>>>> > results is a List of 66000 objects and requires about 8megs
of
> >>>>> > memory.
> >>>>> > The "mergeTemplate" requires at least 230 MB of memory.   Any
> >>>>> > suggestions?
> >>>>> >
> >>>>> > -L
> >>>>> >
> >>>>> > On Fri, May 06, 2005 at 11:53:30AM -0700, jian chen wrote:
> >>>>> > > Hi,
> >>>>> > >
> >>>>> > > I guess the writer Ilkka talked about is when using Velocity
as a
> >>>>> > > standalone template engine.
> >>>>> > >
> >>>>> > > Right now I don't have a good clue, could you run Velocity
as a
> >>>>> > > standalone template engine and render the page to see
if there is 
> >>>>> > > a
> >>>>> > > huge memory usage?
> >>>>> > >
> >>>>> > > Cheers,
> >>>>> > >
> >>>>> > > Jian
> >>>>> > >
> >>>>> > > On 5/6/05, lawrence@marchex.com <lawrence@marchex.com>
wrote:
> >>>>> > > > Sorry, I'm not sure what you mean by what "kind of
writer" I'm
> >>>>> > using.   Where would I find out?   How would I change this
(to 
> >>>>> > maybe
> >>>>> > a
> >>>>> > StringBuffer writer?)
> >>>>> > > >
> >>>>> > > > Also, each column contains a String that is at most,
200
> >>>>> characters,
> >>>>> > most are closer to 10-20.   There are a total of 10 columns.
> >>>>> > > >
> >>>>> > > > Thanks.
> >>>>> > > >
> >>>>> > > > -Lawrence
> >>>>> > > >
> >>>>> > > > On Fri, May 06, 2005 at 09:11:13PM +0300, Ilkka Priha
wrote:
> >>>>> > > > > What kind of writer are you using and how long
are the items?
> >>>>> > > > > 66
> >>>>> > 000 HTML
> >>>>> > > > > table rows with 5 numeric columns consumes about
32 MB when
> >>>>> > buffered into a
> >>>>> > > > > string writer. If the items are longer, it takes
much more
> >>>>> memory.
> >>>>> > > > >
> >>>>> > > > > -- Ilkka
> >>>>> > > > >
> >>>>> > > > >
> >>>>> > > > > lawrence@marchex.com wrote:
> >>>>> > > > > >I tried putting static HTML in the #foreach
loop, same
> >>>>> > > > > >problem.
> >>>>> > Then I
> >>>>> > > > > >tried setting the large list as a "local"
varible:
> >>>>> > > > > >
> >>>>> > > > > >#set ($localList = $List)
> >>>>> > > > > >#foreach($row in $localList)
> >>>>> > > > > >...
> >>>>> > > > > >
> >>>>> > > > > >but that didnt help either.   I don't see
why velocity 
> >>>>> > > > > >should
> >>>>> use
> >>>>> > 250+ MB
> >>>>> > > > > >of memory iterating over a list ~8MB in
size...
> >>>>> > > > > >-L
> >>>>> > > > > >
> >>>>> > > > > >On Thu, May 05, 2005 at 02:56:28PM -0700,
jian chen wrote:
> >>>>> > > > > >
> >>>>> > > > > >>Well, I don't have a good clue now.
> >>>>> > > > > >>
> >>>>> > > > > >>What about do the debugging step by
step, i.e., use just
> >>>>> static
> >>>>> > html
> >>>>> > > > > >>in your #foreach loop? Will that blow
up the memory?
> >>>>> > > > > >>
> >>>>> > > > > >>Cheers,
> >>>>> > > > > >>
> >>>>> > > > > >>Jian
> >>>>> > > > > >>
> >>>>> > > > > >>
> >>>>> > > > > >>
> >>>>> > > > > >>On 5/5/05, lawrence@marchex.com <lawrence@marchex.com>

> >>>>> > > > > >>wrote:
> >>>>> > > > > >>
> >>>>> > > > > >>>Thanks for the suggestion.   I tried
putting that, along
> >>>>> > > > > >>>with
> >>>>> > > > > >>>
> >>>>> > > > > >>>file.resource.loader.cache = true
> >>>>> > > > > >>>
> >>>>> > > > > >>>in my velocity.properties file.
  Both did not work.   Any
> >>>>> > other
> >>>>> > > > > >>>suggestions?
> >>>>> > > > > >>>
> >>>>> > > > > >>>Thanks!
> >>>>> > > > > >>>
> >>>>> > > > > >>>-L
> >>>>> > > > > >>>
> >>>>> > > > > >>>On Thu, May 05, 2005 at 02:11:28PM
-0700, jian chen wrote:
> >>>>> > > > > >>>
> >>>>> > > > > >>>>Hi,
> >>>>> > > > > >>>>
> >>>>> > > > > >>>>This maybe because you haven't
turned on template 
> >>>>> > > > > >>>>caching.
> >>>>> > > > > >>>>I
> >>>>> > had
> >>>>> > > > > >>>>similar problem before, and
even if I only used
> >>>>> evaluate(...)
> >>>>> > method
> >>>>> > > > > >>>>and load the template using
my own java program, it
> >>>>> > > > > >>>>happened
> >>>>> > when the
> >>>>> > > > > >>>>for loop is lot of iterations.
> >>>>> > > > > >>>>
> >>>>> > > > > >>>>The bottom line for my problem
is I need to turn on
> >>>>> > > > > >>>>template
> >>>>> > caching,
> >>>>> > > > > >>>>so, velocity will not try to
cache the objects it parsed
> >>>>> > during each
> >>>>> > > > > >>>>time of the iteration.
> >>>>> > > > > >>>>
> >>>>> > > > > >>>>I did:
> >>>>> > > > > 
> >>>>>>ri.setProperty(RuntimeConstants.FILE_RESOURCE_LOADER_CACHE,
> >>>>> > "true");
> >>>>> > > > > >>>>
> >>>>> > > > > >>>>where ri is a runtime instance.
> >>>>> > > > > >>>>
> >>>>> > > > > >>>>Will this fix your issue?
> >>>>> > > > > >>>>
> >>>>> > > > > >>>>Cheers,
> >>>>> > > > > >>>>
> >>>>> > > > > >>>>Jian
> >>>>> > > > > >>>>
> >>>>> > > > > >>>>
> >>>>> > > > > >>>>On 5/5/05, lawrence@marchex.com
<lawrence@marchex.com>
> >>>>> wrote:
> >>>>> > > > > >>>>
> >>>>> > > > > >>>>>I am getting an OutOfMemoryException:
> >>>>> > > > > >>>>>
> >>>>> > > > > >>>>>org.apache.velocity.exception.MethodInvocationException:
> >>>>> > Invocation of
> >>>>> > > > > >>>>>method 'get' in  class
> >>>>> > org.apache.velocity.tools.struts.TilesTool
> >>>>> > > > > >>>>>threw exception class java.lang.OutOfMemoryError
: null
> >>>>> > > > > >>>>>
> >>>>> > > > > >>>>>when I am creating a large
velocity file.  The vm file
> >>>>> itself
> >>>>> > is
> >>>>> > > > > >>>>>fairly small, but this piece
of velocity is causing the
> >>>>> > exception:
> >>>>> > > > > >>>>>
> >>>>> > > > > >>>>>#foreach($row in $List)
> >>>>> > > > > >>>>><tr>
> >>>>> > > > > >>>>><td height="20" class="tbltxt">$!row.item1</td>
> >>>>> > > > > >>>>><td class="tbltxt">$!row.item2</td>
> >>>>> > > > > >>>>><td class="tbltxt">$!row.item3</td>
> >>>>> > > > > >>>>><td class="tbltxt">$!row.item4</td>
> >>>>> > > > > >>>>><td class="tbltxt">$!number.format('currency',
> >>>>> > $!row.item5)</td>
> >>>>> > > > > >>>>></tr>
> >>>>> > > > > >>>>>#end
> >>>>> > > > > >>>>>
> >>>>> > > > > >>>>>List is a list of relatively
small objects (5-6 Strings, 
> >>>>> > > > > >>>>>a
> >>>>> > couple of
> >>>>> > > > > >>>>>BigDecimals), but the List
itself contains approximately
> >>>>> > 66,000
> >>>>> > > > > >>>>>entries.  Unfortunately,
unless I set the tomcat JVM to
> >>>>> > > > > >>>>>use
> >>>>> a
> >>>>> > heap
> >>>>> > > > > >>>>>size of at least 256M, this
velocity template throws an
> >>>>> > > > > >>>>>OutOfMemoryException when
it is being processed.
> >>>>> > > > > >>>>>
> >>>>> > > > > >>>>>Any suggestions?
> >>>>> > > > > >>>>>
> >>>>> > > > > >>>>>Thanks.
> >>>>> > > > > >>>>>
> >>>>> > > > > >>>>>-L
> >>>>> > > > > >>>>>
> >>>>> > > > >
> >>>>> >
> >>>>> 
> >>>>>>>-------------------------------------------------------------------
> >>>>> > --
> >>>>> > > > > >>>>>To unsubscribe, e-mail:
> >>>>> > velocity-user-unsubscribe@jakarta.apache.org
> >>>>> > > > > >>>>>For additional commands,
e-mail:
> >>>>> > velocity-user-help@jakarta.apache.org
> >>>>> > > > > >>>>>
> >>>>> > > > > >>>>>
> >>>>> > > > > >>>>
> >>>>> > > > >
> >>>>> >
> >>>>> 
> >>>>>>--------------------------------------------------------------------
> >>>>> > -
> >>>>> > > > > >>>>To unsubscribe, e-mail:
> >>>>> > velocity-user-unsubscribe@jakarta.apache.org
> >>>>> > > > > >>>>For additional commands, e-mail:
> >>>>> > velocity-user-help@jakarta.apache.org
> >>>>> > > > > >>>
> >>>>> > > > >
> >>>>> >
> >>>>> 
> >>>>>---------------------------------------------------------------------
> >>>>> > > > > >>>To unsubscribe, e-mail:
> >>>>> > velocity-user-unsubscribe@jakarta.apache.org
> >>>>> > > > > >>>For additional commands, e-mail:
> >>>>> > velocity-user-help@jakarta.apache.org
> >>>>> > > > > >>>
> >>>>> > > > > >>>
> >>>>> > > > > >>
> >>>>> > > > >
> >>>>> >
> >>>>> 
> >>>>---------------------------------------------------------------------
> >>>>> > > > > >>To unsubscribe, e-mail:
> >>>>> > velocity-user-unsubscribe@jakarta.apache.org
> >>>>> > > > > >>For additional commands, e-mail:
> >>>>> > velocity-user-help@jakarta.apache.org
> >>>>> > > > > >
> >>>>> > > > > >
> >>>>> > > > >
> >>>>> > 
> >>>---------------------------------------------------------------------
> >>>>> > > > > >To unsubscribe, e-mail:
> >>>>> > velocity-user-unsubscribe@jakarta.apache.org
> >>>>> > > > > >For additional commands, e-mail:
> >>>>> > velocity-user-help@jakarta.apache.org
> >>>>> > > > > >
> >>>>> > > > > >.
> >>>>> > > > > >
> >>>>> > > > >
> >>>>> > > > >
> >>>>> > > > >
> >>>>> > 
> >>---------------------------------------------------------------------
> >>>>> > > > > To unsubscribe, e-mail:
> >>>>> > velocity-user-unsubscribe@jakarta.apache.org
> >>>>> > > > > For additional commands, e-mail:
> >>>>> > velocity-user-help@jakarta.apache.org
> >>>>> > > >
> >>>>> > > >
> >>>>> > 
> >>---------------------------------------------------------------------
> >>>>> > > > To unsubscribe, e-mail:
> >>>>> velocity-user-unsubscribe@jakarta.apache.org
> >>>>> > > > For additional commands, e-mail:
> >>>>> > velocity-user-help@jakarta.apache.org
> >>>>> > > >
> >>>>> > > >
> >>>>> > >
> >>>>> > >
> >>>>> ---------------------------------------------------------------------
> >>>>> > > To unsubscribe, e-mail:
> >>>>> > > velocity-user-unsubscribe@jakarta.apache.org
> >>>>> > > For additional commands, e-mail:
> >>>>> velocity-user-help@jakarta.apache.org
> >>>>> >
> >>>>> > 
> >>---------------------------------------------------------------------
> >>>>> > To unsubscribe, e-mail: 
> >>>>> > velocity-user-unsubscribe@jakarta.apache.org
> >>>>> > For additional commands, e-mail:
> >>>>> > velocity-user-help@jakarta.apache.org
> >>>>> >
> >>>>> >
> >>>>> > 
> >>---------------------------------------------------------------------
> >>>>> > To unsubscribe, e-mail: 
> >>>>> > velocity-user-unsubscribe@jakarta.apache.org
> >>>>> > For additional commands, e-mail:
> >>>>> > velocity-user-help@jakarta.apache.org
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> >>>>> For additional commands, e-mail: 
> >>>>> velocity-user-help@jakarta.apache.org
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> >>>>> For additional commands, e-mail: 
> >>>>> velocity-user-help@jakarta.apache.org
> >>>>
> >>>>---------------------------------------------------------------------
> >>>>To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> >>>>For additional commands, e-mail: velocity-user-help@jakarta.apache.org
> >>>
> >>>---------------------------------------------------------------------
> >>>To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> >>>For additional commands, e-mail: velocity-user-help@jakarta.apache.org
> >>>
> >>
> >>
> >>---------------------------------------------------------------------
> >>To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> >>For additional commands, e-mail: velocity-user-help@jakarta.apache.org
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> >For additional commands, e-mail: velocity-user-help@jakarta.apache.org
> >
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: velocity-user-help@jakarta.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: velocity-user-help@jakarta.apache.org


Mime
View raw message