synapse-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Veithen <andreas.veit...@gmail.com>
Subject Re: VFS - Synapse Memory Leak
Date Sat, 07 Mar 2009 19:42:40 GMT
I've done an initial implementation of this feature. It is available
in trunk and should be included in the next nightly build. In order to
enable this in your configuration, you need to add the following
property to the proxy:

<parameter name="transport.vfs.Streaming">true</parameter>

You also need to add the following mediators just before the <send> mediator:

<property action="remove" name="transportNonBlocking" scope="axis2"/>
<property action="set" name="OUT_ONLY" value="true"/>

With this configuration Synapse will stream the data directly from the
incoming to the outgoing transport without storing it in memory or in
a temporary file. Note that this has two other side effects:
* The incoming file (or connection in case of a remote file) will only
be opened on demand. In this case this happens during execution of the
<send> mediator.
* If during the mediation the content of the file is needed several
time (which is not the case in your example), it will be read several
times. The reason is of course that the content is not cached.

I tested the solution with a 2GB file and it worked fine. The
performance of the implementation is not yet optimal, but at least the
memory consumption is constant.

Some additional comments:
* The transport.vfs.Streaming property has no impact on XML and SOAP
processing: this type of content is processed exactly as before.
* With the changes described here, we have now two different policies
for plain text and binary content processing: in-memory caching + no
streaming (transport.vfs.Streaming=false) and no caching + deferred
connection + streaming (transport.vfs.Streaming=true). Probably we
should define a wider range of policies in the future, including file
system caching + streaming.
* It is necessary to remove the transportNonBlocking property
(MessageContext.TRANSPORT_NON_BLOCKING) to prevent the <send> mediator
(more precisely the OperationClient) from executing the outgoing
transport in a separate thread. This property is set by the incoming
transport. I think this is a bug since I don't see any valid reason
why the transport that handles the incoming request should determine
the threading behavior of the transport that sends the outgoing
request to the target service. Maybe Asankha can comment on this?

Andreas

On Thu, Mar 5, 2009 at 07:21, kimhorn <kim.horn@icsglobal.net> wrote:
>
> Thats good; as this stops us using Synapse.
>
>
>
> Asankha C. Perera wrote:
>>
>>
>>> Exception in thread "vfs-Worker-4" java.lang.OutOfMemoryError: Java heap
>>> space
>>>         at
>>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:99)
>>>         at
>>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:518)
>>>         at java.lang.StringBuffer.append(StringBuffer.java:307)
>>>         at java.io.StringWriter.write(StringWriter.java:72)
>>>         at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1129)
>>>         at org.apache.commons.io.IOUtils.copy(IOUtils.java:1104)
>>>         at org.apache.commons.io.IOUtils.copy(IOUtils.java:1078)
>>>         at org.apache.commons.io.IOUtils.toString(IOUtils.java:382)
>>>         at
>>> org.apache.synapse.format.PlainTextBuilder.processDocument(PlainTextBuilder.java:68)
>>>
>> Since the content type is text, the plain text formatter is trying to
>> use a String to parse as I see.. which is a problem for large content..
>>
>> A definite bug we need to fix ..
>>
>> cheers
>> asankha
>>
>> --
>> Asankha C. Perera
>> AdroitLogic, http://adroitlogic.org
>>
>> http://esbmagic.blogspot.com
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
>> For additional commands, e-mail: dev-help@synapse.apache.org
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/VFS---Synapse-Memory-Leak-tp22344176p22345904.html
> Sent from the Synapse - Dev mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> For additional commands, e-mail: dev-help@synapse.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org


Mime
View raw message