uima-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thilo Goetz <twgo...@gmx.de>
Subject Re: CAS serialization performance: XMI vs. Java serialization
Date Wed, 15 Aug 2012 09:00:31 GMT
On 15/08/12 10:09, Richard Eckart de Castilho wrote:
> Hi,
> 
> I am looking for a way to improve loading times in an application, so I did a little
experiment with binary CAS serialization to see if it was superior to XMI serialization. For
serialization I used the CASCompleteSerializer to serialize the type-system and heaps into
the same file using Java object serialization - at least that is what I understood it should
do. To read in these files, I would deserialize the CASCompleteSerializer and initialize a
CAS from it using CASImpl.reinit().
> 
> 96.400 files
> 
> plain text (uncompressed)      :                 581.865.593 Byte
> binary (serialized java, gzip) : 0:47:02.835   3.555.449.597 Byte 
> xmi (gzip)                     : 1:20:31.535   4.712.633.769 Byte
> 
> So binary takes about 60% of the time xmi serialization would need and uses about 75%
of the space.
> I didn't do reading experiment yet, but I suppose the improvement should be on a similar
level, if not better.
> 
> I am also not sure yet about the draw-backs of binary serialization and in which scenarios
they apply. The draw-backs I saw so far are:
> 
> - Type-system is stored redudantly in every output file.
> - The type system configured with CASImpl.reinit() may be different from the one which
was used to initialize the pipeline, CAS-based annotators relying on typeSystemInit() may
not be configured with the correct types - this is a hypothesis I didn't test.
> - Serialized Java objects may become due to refactoring within the UIMA framework. However,
there is yet another binary CAS serialization in UIMA which uses the DataOutputStream and
may be more stable.
> 
> Did anybody ever use any form of binary CAS serialization outside Vinci/UIMA-AS?

Not sure this will help, but we originally implemented the binary
serialization to pass the CAS between Java and C++ (in-process).  Used
that way it's blindingly fast because Java and C++ use the same heap
layout for the CAS data.  We have also used it for communication via
SOAP, but I'm not sure I would do that today.  I might prefer a format
that's at least somewhat human readable.  I do not recall ever using the
binary format for serialization to disk, just because I never had that
use case.

The thing with the binary serialization is that type information is
encoded in a binary format as well.  So you need to be sure that when
you read it back in, every type and feature gets assigned the same code,
otherwise the heap is garbage.  That's why you need to be sure to use
the correct, encoded type system as well.

However, as I recall, there was a way you could serialize the CAS
without the type system if you were sure you didn't need it.  Isn't that
the difference between the CasCompleteSerializer and the
NotSoCompleteSerializer (making that up here)?  On the way back, you can
deserialize into an existing CAS that has the right type system.

Your times above, do they include time needed to do the compression?
I'm surprised binary serialization is not even twice as fast.  Or is
this gated by the disk I/O?

--Thilo

> 
> Cheers,
> 
> -- Richard
> 


Mime
View raw message