synapse-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ruwan Linton <ruwan.lin...@gmail.com>
Subject Re: VFS - Synapse Memory Leak
Date Sun, 08 Mar 2009 02:47:52 GMT
Andreas,

Can we have the caching at the file system as a property to support the
multiple layers touching the full message and is it possible make it to
specify a threshold for streaming? For example if the message is touched
several time we might still need streaming but not for the 100KB or lesser
files.

Thanks,
Ruwan

On Sun, Mar 8, 2009 at 1:12 AM, Andreas Veithen
<andreas.veithen@gmail.com>wrote:

> I've done an initial implementation of this feature. It is available
> in trunk and should be included in the next nightly build. In order to
> enable this in your configuration, you need to add the following
> property to the proxy:
>
> <parameter name="transport.vfs.Streaming">true</parameter>
>
> You also need to add the following mediators just before the <send>
> mediator:
>
> <property action="remove" name="transportNonBlocking" scope="axis2"/>
> <property action="set" name="OUT_ONLY" value="true"/>
>
> With this configuration Synapse will stream the data directly from the
> incoming to the outgoing transport without storing it in memory or in
> a temporary file. Note that this has two other side effects:
> * The incoming file (or connection in case of a remote file) will only
> be opened on demand. In this case this happens during execution of the
> <send> mediator.
> * If during the mediation the content of the file is needed several
> time (which is not the case in your example), it will be read several
> times. The reason is of course that the content is not cached.
>
> I tested the solution with a 2GB file and it worked fine. The
> performance of the implementation is not yet optimal, but at least the
> memory consumption is constant.
>
> Some additional comments:
> * The transport.vfs.Streaming property has no impact on XML and SOAP
> processing: this type of content is processed exactly as before.
> * With the changes described here, we have now two different policies
> for plain text and binary content processing: in-memory caching + no
> streaming (transport.vfs.Streaming=false) and no caching + deferred
> connection + streaming (transport.vfs.Streaming=true). Probably we
> should define a wider range of policies in the future, including file
> system caching + streaming.
> * It is necessary to remove the transportNonBlocking property
> (MessageContext.TRANSPORT_NON_BLOCKING) to prevent the <send> mediator
> (more precisely the OperationClient) from executing the outgoing
> transport in a separate thread. This property is set by the incoming
> transport. I think this is a bug since I don't see any valid reason
> why the transport that handles the incoming request should determine
> the threading behavior of the transport that sends the outgoing
> request to the target service. Maybe Asankha can comment on this?
>
> Andreas
>
> On Thu, Mar 5, 2009 at 07:21, kimhorn <kim.horn@icsglobal.net> wrote:
> >
> > Thats good; as this stops us using Synapse.
> >
> >
> >
> > Asankha C. Perera wrote:
> >>
> >>
> >>> Exception in thread "vfs-Worker-4" java.lang.OutOfMemoryError: Java
> heap
> >>> space
> >>>         at
> >>>
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:99)
> >>>         at
> >>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:518)
> >>>         at java.lang.StringBuffer.append(StringBuffer.java:307)
> >>>         at java.io.StringWriter.write(StringWriter.java:72)
> >>>         at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1129)
> >>>         at org.apache.commons.io.IOUtils.copy(IOUtils.java:1104)
> >>>         at org.apache.commons.io.IOUtils.copy(IOUtils.java:1078)
> >>>         at org.apache.commons.io.IOUtils.toString(IOUtils.java:382)
> >>>         at
> >>>
> org.apache.synapse.format.PlainTextBuilder.processDocument(PlainTextBuilder.java:68)
> >>>
> >> Since the content type is text, the plain text formatter is trying to
> >> use a String to parse as I see.. which is a problem for large content..
> >>
> >> A definite bug we need to fix ..
> >>
> >> cheers
> >> asankha
> >>
> >> --
> >> Asankha C. Perera
> >> AdroitLogic, http://adroitlogic.org
> >>
> >> http://esbmagic.blogspot.com
> >>
> >>
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> >> For additional commands, e-mail: dev-help@synapse.apache.org
> >>
> >>
> >>
> >
> > --
> > View this message in context:
> http://www.nabble.com/VFS---Synapse-Memory-Leak-tp22344176p22345904.html
> > Sent from the Synapse - Dev mailing list archive at Nabble.com.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> > For additional commands, e-mail: dev-help@synapse.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> For additional commands, e-mail: dev-help@synapse.apache.org
>
>


-- 
Ruwan Linton
http://wso2.org - "Oxygenating the Web Services Platform"
http://ruwansblog.blogspot.com/

Mime
View raw message