axis-c-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Senaka Fernando" <sen...@wso2.com>
Subject Re: Caching support for large attachments
Date Sat, 15 Mar 2008 07:50:50 GMT
>>  BTW, this whole discussion is about in path, that is reading an
>>  incomming message. How about the out path? We have the same problems
>>  when sending attachments. Right now, we read the whole file into memory
>>  and then only we send over the wire.
> hmm... Why not write it in chunks.. Read a chunk from the file, then
> write it to the outstream.. Use size of the file for content-type
> calculation in case of non-chunking.. But mostly people will use
> chunking when using MTOM..

No, chunking is not required. You also don't need to write the entire data
to be sent, to the stream at once. Because any HTTP Receiver will pull
from the stream until it sees a valid ending character sequence.

I believe that you should be able to write part by part to the stream, and
send it, then reuse the buffer and write part 2, and send and so on. This
argument can be justified, because on the receiving end, we must read the
multi-part data until we encounter the mime boundary, unlike an ordinary
payload where it can be terminated by a valid terminating character
sequence . We'll only have issues if we are to write large soap payloads
which of course can be dealt with once we've implemented Session in
Axis2/C.

Regards,
Senaka

>
> thanks,
> Thilina
>
>
>>
>>  Samisa...
>>
>>
>>
>>  > Regards,
>>  > Senaka
>>  >
>>  >
>>  >> Hi,
>>  >>
>>  >>>  > In Axis2/Java case we do write the attachment content directly
>> from
>>  >>>  > the InputStream to the File when the attachment size is larger
>> than
>>  >>>  > the threshold.  This avoids loading the whole attachment to the
>>  >>> memory
>>  >>>  > at all.
>>  >>>
>>  >>>  In this case to find out the attachment size don't you need to do
>> any
>>  >>>  mime parsing? How do you find the attachment size with out
>> searching
>>  >>> for
>>  >>>  the mime boundaries ?
>>  >>>
>>  >> Yes.. MIME is a boundary based packaging mechanism and you does not
>>  >> need to specify the length for each of the parts...Even the HTTP
>>  >> content length is not there if the message is chunked.
>>  >>
>>  >> What we did in Axis2/Java to overcome this is to read the data to a
>>  >> byte[] buffer of up to a certain size (the size threshold). If there
>>  >> are more data available in the mime part (if we have not encountered
>>  >> the boundary yet) then we know this attachment is bigger than the
>>  >> threshold. So we create the temp file, pump the content in the
>> buffer
>>  >> to the file, then pump the rest of the stream to the file.. In this
>>  >> way we do not need to know the size of the attachment upfront.. BTW
>> we
>>  >> do all of the above while we are parsing the MIME message at the
>> MIME
>>  >> parser level..
>>  >>
>>  >>
>>  >>>  > This has the plus point that the attachment size will be
>>  >>>  > limited only by the available free space in the Temp Directory..
>>  >>>  > Will that be possible in Axis2/C.. Or is that wat you have in
>> mind
>>  >>> :)..
>>  >>>
>>  >>>  Yes this is possible.
>>  >>>
>>  >> But in Axis2/JAVA we will get a OutOfMemory if we parse a large MIME
>>  >> part upfront, since it reads the attachment to memory. May be you
>> can
>>  >> have a larger limit with C than in Java, but ultimately you'll come
>> to
>>  >> a situation where you will not have enough memory to store that MIME
>>  >> part in memory in the parsing time, unless you write in to a File
>>  >> while parsing,..
>>  >>
>>  >> thanks,
>>  >> Thilina
>>  >>
>>  >>
>>  >>>
>>  >>>  >
>>  >>>  > thanks,
>>  >>>  > Thilina
>>  >>>  >
>>  >>>  >  >and keeping the file name inside
>>  >>>  > >  data_handler instead of the whole buffer. So the service
or
>> the
>>  >>> client
>>  >>>  > >  will get the file name instead of the buffered stream,
when
>> it
>>  >>> receives
>>  >>>  > >  an attachment. This will not prevent buffering the attachment
>> at
>>  >>> the
>>  >>>  > >  transport but will prevent keeping it inside the om_tree
till
>> it
>>  >>> reaches
>>  >>>  > >  the receiver.
>>  >>>  > >
>>  >>>  > >  Before implementing this I would like to know your
>> suggestions
>>  >>> regarding
>>  >>>  > >  this.
>>  >>>  > >
>>  >>>  > >  [1] https://issues.apache.org/jira/browse/AXIS2C-672
>>  >>>  > >
>>  >>>  > >  Thanks,
>>  >>>  > >  -Manjula
>>  >>>  > >
>>  >>>  > >  --
>>  >>>  > >  Manjula Peiris: http://manjula-peiris.blogspot.com/
>>  >>>  > >
>>  >>>  > >
>>  >>>  > >  ---------------------------------------------------------------------
>>  >>>  > >  To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>  >>>  > >  For additional commands, e-mail:
>> axis-c-dev-help@ws.apache.org
>>  >>>  > >
>>  >>>  > >
>>  >>>  >
>>  >>>  >
>>  >>>  >
>>  >>>
>>  >>>
>>  >>>  ---------------------------------------------------------------------
>>  >>>  To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>  >>>  For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>  >>>
>>  >>>
>>  >>>
>>  >>
>>  >> --
>>  >> Thilina Gunarathne - http://thilinag.blogspot.com
>>  >>
>>  >> ---------------------------------------------------------------------
>>  >> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>  >> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>  >>
>>  >>
>>  >>
>>  >
>>  >
>>  > ---------------------------------------------------------------------
>>  > To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>  > For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>  >
>>  >
>>  >
>>  >
>>
>>
>>  --
>>  Samisa Abeysinghe
>>  Software Architect; WSO2 Inc.
>>
>>  http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>
>>
>>
>>
>>  ---------------------------------------------------------------------
>>  To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>  For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>
>
>
> --
> Thilina Gunarathne - http://thilinag.blogspot.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org


Mime
View raw message