velocity-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Will Glass-Husain" <wgl...@forio.com>
Subject Re: OutOfMemoryException Processing Velocity Template
Date Fri, 06 May 2005 22:51:48 GMT
Yes, but which ones?

The char[] in the StringWriter, the char[] in the Template object, or the
char[] in the strings in your original objects?  That would be useful info.

WILL

----- Original Message ----- 
From: <lawrence@marchex.com>
To: "Velocity Users List" <velocity-user@jakarta.apache.org>
Sent: Friday, May 06, 2005 3:45 PM
Subject: Re: OutOfMemoryException Processing Velocity Template


> Ok I tried using JProfiler and used a StringWriter, and found that the
> item that was taking the most memory (150 MB) was..... char[]!  Ok now I'm
> completely stumped and clueless.
>
> -L
>
> On Fri, May 06, 2005 at 02:51:15PM -0700, Will Glass-Husain wrote:
>> You might consider trying a profiler.  Most of them have trial version
>> available.  This could tell you exactly where the memory is going.
>>
>> I just bought JProfiler, and it seems a bit better priced than the
>> others.
>> Here's a user review:
>> http://www.jlamp.com/blog/2005/02/01/1107280975000.html
>>
>> You might also check out Hyades from the Eclipse project for an open
>> source
>> option.
>>
>> WILL
>>
>>
>> ----- Original Message ----- 
>> From: <lawrence@marchex.com>
>> To: "Velocity Users List" <velocity-user@jakarta.apache.org>
>> Sent: Friday, May 06, 2005 2:39 PM
>> Subject: Re: OutOfMemoryException Processing Velocity Template
>>
>>
>> >This is why I'm so confused.   I wouldn't expect it to use so much
>> >memory.
>> >What I did to test the memory usage was:
>> >
>> >System.out.println("Free memory before merge: " +
>> >Runtime.getRuntime().freeMemory());
>> >Velocity.mergeTemplate("/myfile.vm", "ISO-8859-1", context, w );
>> >System.out.println("Free memory after merge: " +
>> >Runtime.getRuntime().freeMemory());
>> >
>> >And then setting the JVM to -Xms256M and -Xmx256M.   When I use a String
>> >writer, the free memory is reduced by ~200MB.   When I use an
>> >OutputStreamWriter, it is reduced ~13MB, a huge difference.
>> >
>> >BTW, I am using 1.4.2_05
>> >
>> >-L
>> >
>> >On Fri, May 06, 2005 at 02:22:01PM -0700, Will Glass-Husain wrote:
>> >>Hi Lawrence,
>> >>
>> >>I got interested, and just ran some similar code with JProfiler looking
>> >>at
>> >>VM memory usage.  I created a list of 60,000 objects, each with 10
>> >>100char
>> >>columns.  Then I used StringWriter to merge as you did.
>> >>
>> >>The VM used 3MB at the start, 6.5 MB after creating the list, and 25
>> >>after
>> >>merging.  (before I closed the writer).  Nowhere close to 200.  So
>> >>perhaps
>> >>there's something else going on.  Although if the problem is one with
>> >>StringWriter, a lot may depend on the specific JVM you're using.  (I'm
>> >>using Sun 1.4.2_04-b05).
>> >>
>> >>WILL
>> >>
>> >>----- Original Message ----- 
>> >>From: <lawrence@marchex.com>
>> >>To: "Velocity Users List" <velocity-user@jakarta.apache.org>
>> >>Sent: Friday, May 06, 2005 2:08 PM
>> >>Subject: Re: OutOfMemoryException Processing Velocity Template
>> >>
>> >>
>> >>>Ok now I'm confused (suprise, suprise!).   I looked at the source for
>> >>>VelocityServlet, and mergeTemplate has:
>> >>>
>> >>>vw = (VelocityWriter) writerPool.get();
>> >>>
>> >>>if (vw == null)
>> >>>{
>> >>>   vw = new VelocityWriter(new OutputStreamWriter(output, encoding),4
>> >>> *
>> >>>1024, true);
>> >>>}
>> >>>else
>> >>>{
>> >>>   vw.recycle(new OutputStreamWriter(output, encoding));
>> >>>}
>> >>>template.merge(context, vw);
>> >>>
>> >>>===
>> >>>
>> >>>So the VelocityServlet DOES use OutputStreamWriter already ... so then
>> >>>why
>> >>>does it still utilize so much memory?
>> >>>
>> >>>-L
>> >>>
>> >>>On Fri, May 06, 2005 at 02:02:07PM -0700, lawrence@marchex.com wrote:
>> >>>>Alright, I changed the Writer to be:
>> >>>>
>> >>>>OutputStreamWriter w = new OutputStreamWriter(new
>> >>>>FileOutputStream("test.html"));
>> >>>>
>> >>>>And this fixes the memory problem.
>> >>>>
>> >>>>However, my application is running Velocity as a servlet, how should
>> >>>>I
>> >>>>extend the VelocityServlet to use an OutputStreamWriter intead of
a
>> >>>>StringWriter?  Should I post a different email thread about this?
>> >>>>
>> >>>>-L
>> >>>>
>> >>>>On Fri, May 06, 2005 at 01:59:19PM -0700, David You wrote:
>> >>>>> Well, Open a file, use a FileWriter.
>> >>>>>
>> >>>>> David
>> >>>>>
>> >>>>>
>> >>>>> -----Original Message-----
>> >>>>> From: lawrence@marchex.com [mailto:lawrence@marchex.com]
>> >>>>> Sent: Friday, May 06, 2005 1:48 PM
>> >>>>> To: Velocity Users List
>> >>>>> Subject: Re: OutOfMemoryException Processing Velocity Template
>> >>>>>
>> >>>>> I printed out the available memory before and after I call
>> >>>>> "mergeTemplate":
>> >>>>>
>> >>>>> Free memory before merge: 217393352
>> >>>>> Free memory after merge: 16271392
>> >>>>>
>> >>>>> so I would assume the StringWriter is approximately 200MB after
the
>> >>>>> merge.   What should I use/do instead?
>> >>>>>
>> >>>>> -Lawrence
>> >>>>>
>> >>>>> On Fri, May 06, 2005 at 01:43:50PM -0700, David You wrote:
>> >>>>> > Why do you use StringWriter? That is cached in the memory.
>> >>>>> > How big is the StringWriter after the merge?
>> >>>>> >
>> >>>>> > David
>> >>>>> >
>> >>>>> >
>> >>>>> > -----Original Message-----
>> >>>>> > From: lawrence@marchex.com [mailto:lawrence@marchex.com]
>> >>>>> > Sent: Friday, May 06, 2005 1:22 PM
>> >>>>> > To: Velocity Users List
>> >>>>> > Subject: Re: OutOfMemoryException Processing Velocity Template
>> >>>>> >
>> >>>>> > Alright, I ran this little piece of code:
>> >>>>> >
>> >>>>> > public static void main( String args[] ) throws Exception
>> >>>>> >     {
>> >>>>> >         /* first, we init the runtime engine.  Defaults
are fine.
>> >>>>> > */
>> >>>>> >
>> >>>>> >         Velocity.init();
>> >>>>> >
>> >>>>> >         /* lets make a Context and put data into it */
>> >>>>> >
>> >>>>> >         VelocityContext context = new VelocityContext();
>> >>>>> >
>> >>>>> >         List results = ReportBean.runLargeReport();
>> >>>>> >
>> >>>>> >         context.put("List", results);
>> >>>>> >
>> >>>>> >         /* lets render a template */
>> >>>>> >
>> >>>>> >         StringWriter w = new StringWriter();
>> >>>>> >
>> >>>>> >         Velocity.mergeTemplate("/myfile.vm", context, w
);
>> >>>>> >         System.out.println(" template : " + w );
>> >>>>> >
>> >>>>> >         /* lets make our own string to render */
>> >>>>> >     }
>> >>>>> >
>> >>>>> > results is a List of 66000 objects and requires about 8megs
of
>> >>>>> > memory.
>> >>>>> > The "mergeTemplate" requires at least 230 MB of memory.
  Any
>> >>>>> > suggestions?
>> >>>>> >
>> >>>>> > -L
>> >>>>> >
>> >>>>> > On Fri, May 06, 2005 at 11:53:30AM -0700, jian chen wrote:
>> >>>>> > > Hi,
>> >>>>> > >
>> >>>>> > > I guess the writer Ilkka talked about is when using
Velocity as
>> >>>>> > > a
>> >>>>> > > standalone template engine.
>> >>>>> > >
>> >>>>> > > Right now I don't have a good clue, could you run
Velocity as a
>> >>>>> > > standalone template engine and render the page to
see if there
>> >>>>> > > is
>> >>>>> > > a
>> >>>>> > > huge memory usage?
>> >>>>> > >
>> >>>>> > > Cheers,
>> >>>>> > >
>> >>>>> > > Jian
>> >>>>> > >
>> >>>>> > > On 5/6/05, lawrence@marchex.com <lawrence@marchex.com>
wrote:
>> >>>>> > > > Sorry, I'm not sure what you mean by what "kind
of writer"
>> >>>>> > > > I'm
>> >>>>> > using.   Where would I find out?   How would I change this
(to
>> >>>>> > maybe
>> >>>>> > a
>> >>>>> > StringBuffer writer?)
>> >>>>> > > >
>> >>>>> > > > Also, each column contains a String that is at
most, 200
>> >>>>> characters,
>> >>>>> > most are closer to 10-20.   There are a total of 10 columns.
>> >>>>> > > >
>> >>>>> > > > Thanks.
>> >>>>> > > >
>> >>>>> > > > -Lawrence
>> >>>>> > > >
>> >>>>> > > > On Fri, May 06, 2005 at 09:11:13PM +0300, Ilkka
Priha wrote:
>> >>>>> > > > > What kind of writer are you using and how
long are the
>> >>>>> > > > > items?
>> >>>>> > > > > 66
>> >>>>> > 000 HTML
>> >>>>> > > > > table rows with 5 numeric columns consumes
about 32 MB when
>> >>>>> > buffered into a
>> >>>>> > > > > string writer. If the items are longer,
it takes much more
>> >>>>> memory.
>> >>>>> > > > >
>> >>>>> > > > > -- Ilkka
>> >>>>> > > > >
>> >>>>> > > > >
>> >>>>> > > > > lawrence@marchex.com wrote:
>> >>>>> > > > > >I tried putting static HTML in the #foreach
loop, same
>> >>>>> > > > > >problem.
>> >>>>> > Then I
>> >>>>> > > > > >tried setting the large list as a "local"
varible:
>> >>>>> > > > > >
>> >>>>> > > > > >#set ($localList = $List)
>> >>>>> > > > > >#foreach($row in $localList)
>> >>>>> > > > > >...
>> >>>>> > > > > >
>> >>>>> > > > > >but that didnt help either.   I don't
see why velocity
>> >>>>> > > > > >should
>> >>>>> use
>> >>>>> > 250+ MB
>> >>>>> > > > > >of memory iterating over a list ~8MB
in size...
>> >>>>> > > > > >-L
>> >>>>> > > > > >
>> >>>>> > > > > >On Thu, May 05, 2005 at 02:56:28PM -0700,
jian chen wrote:
>> >>>>> > > > > >
>> >>>>> > > > > >>Well, I don't have a good clue now.
>> >>>>> > > > > >>
>> >>>>> > > > > >>What about do the debugging step
by step, i.e., use just
>> >>>>> static
>> >>>>> > html
>> >>>>> > > > > >>in your #foreach loop? Will that
blow up the memory?
>> >>>>> > > > > >>
>> >>>>> > > > > >>Cheers,
>> >>>>> > > > > >>
>> >>>>> > > > > >>Jian
>> >>>>> > > > > >>
>> >>>>> > > > > >>
>> >>>>> > > > > >>
>> >>>>> > > > > >>On 5/5/05, lawrence@marchex.com
<lawrence@marchex.com>
>> >>>>> > > > > >>wrote:
>> >>>>> > > > > >>
>> >>>>> > > > > >>>Thanks for the suggestion. 
 I tried putting that, along
>> >>>>> > > > > >>>with
>> >>>>> > > > > >>>
>> >>>>> > > > > >>>file.resource.loader.cache =
true
>> >>>>> > > > > >>>
>> >>>>> > > > > >>>in my velocity.properties file.
  Both did not work.
>> >>>>> > > > > >>>Any
>> >>>>> > other
>> >>>>> > > > > >>>suggestions?
>> >>>>> > > > > >>>
>> >>>>> > > > > >>>Thanks!
>> >>>>> > > > > >>>
>> >>>>> > > > > >>>-L
>> >>>>> > > > > >>>
>> >>>>> > > > > >>>On Thu, May 05, 2005 at 02:11:28PM
-0700, jian chen
>> >>>>> > > > > >>>wrote:
>> >>>>> > > > > >>>
>> >>>>> > > > > >>>>Hi,
>> >>>>> > > > > >>>>
>> >>>>> > > > > >>>>This maybe because you haven't
turned on template
>> >>>>> > > > > >>>>caching.
>> >>>>> > > > > >>>>I
>> >>>>> > had
>> >>>>> > > > > >>>>similar problem before,
and even if I only used
>> >>>>> evaluate(...)
>> >>>>> > method
>> >>>>> > > > > >>>>and load the template using
my own java program, it
>> >>>>> > > > > >>>>happened
>> >>>>> > when the
>> >>>>> > > > > >>>>for loop is lot of iterations.
>> >>>>> > > > > >>>>
>> >>>>> > > > > >>>>The bottom line for my problem
is I need to turn on
>> >>>>> > > > > >>>>template
>> >>>>> > caching,
>> >>>>> > > > > >>>>so, velocity will not try
to cache the objects it
>> >>>>> > > > > >>>>parsed
>> >>>>> > during each
>> >>>>> > > > > >>>>time of the iteration.
>> >>>>> > > > > >>>>
>> >>>>> > > > > >>>>I did:
>> >>>>> > > > >
>> >>>>>>ri.setProperty(RuntimeConstants.FILE_RESOURCE_LOADER_CACHE,
>> >>>>> > "true");
>> >>>>> > > > > >>>>
>> >>>>> > > > > >>>>where ri is a runtime instance.
>> >>>>> > > > > >>>>
>> >>>>> > > > > >>>>Will this fix your issue?
>> >>>>> > > > > >>>>
>> >>>>> > > > > >>>>Cheers,
>> >>>>> > > > > >>>>
>> >>>>> > > > > >>>>Jian
>> >>>>> > > > > >>>>
>> >>>>> > > > > >>>>
>> >>>>> > > > > >>>>On 5/5/05, lawrence@marchex.com
<lawrence@marchex.com>
>> >>>>> wrote:
>> >>>>> > > > > >>>>
>> >>>>> > > > > >>>>>I am getting an OutOfMemoryException:
>> >>>>> > > > > >>>>>
>> >>>>> > > > > >>>>>org.apache.velocity.exception.MethodInvocationException:
>> >>>>> > Invocation of
>> >>>>> > > > > >>>>>method 'get' in  class
>> >>>>> > org.apache.velocity.tools.struts.TilesTool
>> >>>>> > > > > >>>>>threw exception class
java.lang.OutOfMemoryError :
>> >>>>> > > > > >>>>>null
>> >>>>> > > > > >>>>>
>> >>>>> > > > > >>>>>when I am creating a
large velocity file.  The vm file
>> >>>>> itself
>> >>>>> > is
>> >>>>> > > > > >>>>>fairly small, but this
piece of velocity is causing
>> >>>>> > > > > >>>>>the
>> >>>>> > exception:
>> >>>>> > > > > >>>>>
>> >>>>> > > > > >>>>>#foreach($row in $List)
>> >>>>> > > > > >>>>><tr>
>> >>>>> > > > > >>>>><td height="20" class="tbltxt">$!row.item1</td>
>> >>>>> > > > > >>>>><td class="tbltxt">$!row.item2</td>
>> >>>>> > > > > >>>>><td class="tbltxt">$!row.item3</td>
>> >>>>> > > > > >>>>><td class="tbltxt">$!row.item4</td>
>> >>>>> > > > > >>>>><td class="tbltxt">$!number.format('currency',
>> >>>>> > $!row.item5)</td>
>> >>>>> > > > > >>>>></tr>
>> >>>>> > > > > >>>>>#end
>> >>>>> > > > > >>>>>
>> >>>>> > > > > >>>>>List is a list of relatively
small objects (5-6
>> >>>>> > > > > >>>>>Strings,
>> >>>>> > > > > >>>>>a
>> >>>>> > couple of
>> >>>>> > > > > >>>>>BigDecimals), but the
List itself contains
>> >>>>> > > > > >>>>>approximately
>> >>>>> > 66,000
>> >>>>> > > > > >>>>>entries.  Unfortunately,
unless I set the tomcat JVM
>> >>>>> > > > > >>>>>to
>> >>>>> > > > > >>>>>use
>> >>>>> a
>> >>>>> > heap
>> >>>>> > > > > >>>>>size of at least 256M,
this velocity template throws
>> >>>>> > > > > >>>>>an
>> >>>>> > > > > >>>>>OutOfMemoryException
when it is being processed.
>> >>>>> > > > > >>>>>
>> >>>>> > > > > >>>>>Any suggestions?
>> >>>>> > > > > >>>>>
>> >>>>> > > > > >>>>>Thanks.
>> >>>>> > > > > >>>>>
>> >>>>> > > > > >>>>>-L
>> >>>>> > > > > >>>>>
>> >>>>> > > > >
>> >>>>> >
>> >>>>>
>> >>>>>>>-------------------------------------------------------------------
>> >>>>> > --
>> >>>>> > > > > >>>>>To unsubscribe, e-mail:
>> >>>>> > velocity-user-unsubscribe@jakarta.apache.org
>> >>>>> > > > > >>>>>For additional commands,
e-mail:
>> >>>>> > velocity-user-help@jakarta.apache.org
>> >>>>> > > > > >>>>>
>> >>>>> > > > > >>>>>
>> >>>>> > > > > >>>>
>> >>>>> > > > >
>> >>>>> >
>> >>>>>
>> >>>>>>--------------------------------------------------------------------
>> >>>>> > -
>> >>>>> > > > > >>>>To unsubscribe, e-mail:
>> >>>>> > velocity-user-unsubscribe@jakarta.apache.org
>> >>>>> > > > > >>>>For additional commands,
e-mail:
>> >>>>> > velocity-user-help@jakarta.apache.org
>> >>>>> > > > > >>>
>> >>>>> > > > >
>> >>>>> >
>> >>>>>
>> >>>>>---------------------------------------------------------------------
>> >>>>> > > > > >>>To unsubscribe, e-mail:
>> >>>>> > velocity-user-unsubscribe@jakarta.apache.org
>> >>>>> > > > > >>>For additional commands, e-mail:
>> >>>>> > velocity-user-help@jakarta.apache.org
>> >>>>> > > > > >>>
>> >>>>> > > > > >>>
>> >>>>> > > > > >>
>> >>>>> > > > >
>> >>>>> >
>> >>>>>
>> >>>>---------------------------------------------------------------------
>> >>>>> > > > > >>To unsubscribe, e-mail:
>> >>>>> > velocity-user-unsubscribe@jakarta.apache.org
>> >>>>> > > > > >>For additional commands, e-mail:
>> >>>>> > velocity-user-help@jakarta.apache.org
>> >>>>> > > > > >
>> >>>>> > > > > >
>> >>>>> > > > >
>> >>>>> >
>> >>>---------------------------------------------------------------------
>> >>>>> > > > > >To unsubscribe, e-mail:
>> >>>>> > velocity-user-unsubscribe@jakarta.apache.org
>> >>>>> > > > > >For additional commands, e-mail:
>> >>>>> > velocity-user-help@jakarta.apache.org
>> >>>>> > > > > >
>> >>>>> > > > > >.
>> >>>>> > > > > >
>> >>>>> > > > >
>> >>>>> > > > >
>> >>>>> > > > >
>> >>>>> >
>> >>---------------------------------------------------------------------
>> >>>>> > > > > To unsubscribe, e-mail:
>> >>>>> > velocity-user-unsubscribe@jakarta.apache.org
>> >>>>> > > > > For additional commands, e-mail:
>> >>>>> > velocity-user-help@jakarta.apache.org
>> >>>>> > > >
>> >>>>> > > >
>> >>>>> >
>> >>---------------------------------------------------------------------
>> >>>>> > > > To unsubscribe, e-mail:
>> >>>>> velocity-user-unsubscribe@jakarta.apache.org
>> >>>>> > > > For additional commands, e-mail:
>> >>>>> > velocity-user-help@jakarta.apache.org
>> >>>>> > > >
>> >>>>> > > >
>> >>>>> > >
>> >>>>> > >
>> >>>>> ---------------------------------------------------------------------
>> >>>>> > > To unsubscribe, e-mail:
>> >>>>> > > velocity-user-unsubscribe@jakarta.apache.org
>> >>>>> > > For additional commands, e-mail:
>> >>>>> velocity-user-help@jakarta.apache.org
>> >>>>> >
>> >>>>> >
>> >>---------------------------------------------------------------------
>> >>>>> > To unsubscribe, e-mail:
>> >>>>> > velocity-user-unsubscribe@jakarta.apache.org
>> >>>>> > For additional commands, e-mail:
>> >>>>> > velocity-user-help@jakarta.apache.org
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>---------------------------------------------------------------------
>> >>>>> > To unsubscribe, e-mail:
>> >>>>> > velocity-user-unsubscribe@jakarta.apache.org
>> >>>>> > For additional commands, e-mail:
>> >>>>> > velocity-user-help@jakarta.apache.org
>> >>>>>
>> >>>>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail:
>> >>>>> velocity-user-unsubscribe@jakarta.apache.org
>> >>>>> For additional commands, e-mail:
>> >>>>> velocity-user-help@jakarta.apache.org
>> >>>>>
>> >>>>>
>> >>>>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail:
>> >>>>> velocity-user-unsubscribe@jakarta.apache.org
>> >>>>> For additional commands, e-mail:
>> >>>>> velocity-user-help@jakarta.apache.org
>> >>>>
>> >>>>---------------------------------------------------------------------
>> >>>>To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
>> >>>>For additional commands, e-mail:
>> >>>>velocity-user-help@jakarta.apache.org
>> >>>
>> >>>---------------------------------------------------------------------
>> >>>To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
>> >>>For additional commands, e-mail: velocity-user-help@jakarta.apache.org
>> >>>
>> >>
>> >>
>> >>---------------------------------------------------------------------
>> >>To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
>> >>For additional commands, e-mail: velocity-user-help@jakarta.apache.org
>> >
>> >---------------------------------------------------------------------
>> >To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
>> >For additional commands, e-mail: velocity-user-help@jakarta.apache.org
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: velocity-user-help@jakarta.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: velocity-user-help@jakarta.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: velocity-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: velocity-user-help@jakarta.apache.org


Mime
View raw message