ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: Ignite as distributed file storage
Date Sat, 04 Aug 2018 04:49:58 GMT
Dmitriy,

I would suggest us not limiting blobs use case to a dedicated cache. If to
look at other databases they usually have BLOB/LONGBLOB/etc. as a type
meaning that users mix simple and BLOB types in the same tables.

Should we start with Ignite SQL adding blobs through its APIs?

--
Denis

On Fri, Aug 3, 2018 at 1:52 PM Dmitriy Setrakyan <dsetrakyan@apache.org>
wrote:

> Pavel,
>
> Not everything that gets put in Ignite has a class, and not everything can
> be annotated. For example, what about byte[] or a binary object?
>
> I would prefer a separate cache with specific purpose of storing blobs. It
> will be easier for users to understand and configure.  It can also have a
> custom logic of splitting a blob into multiple batches to be able to
> transfer it over the network faster.
>
> D.
>
>
> On Fri, Aug 3, 2018 at 3:21 AM, Pavel Kovalenko <jokserfn@gmail.com>
> wrote:
>
> > Dmitriy,
> >
> > I think we don't need a separate implementation of cache like BLOB cache.
> > Instead of it, a user can mark value class or value class field with the
> > special annotation "@BLOB".
> > During cache put, marshaller will place a special placeholder on such
> > fields, write byte[] array payload of a field to special internal blob
> > storage and place the only reference to actual DataEntry in the page
> > memory.
> > During cache get, marshaller will place a special proxy instead of an
> > actual class that can be downloaded and unmarshalled by demand from
> > internal storage on the user side.
> > Using such approach we will also solve eager/lazy problem, this will also
> > give user possibility to adjust his own marshallers (like Jackson, Jaxb,
> > etc.) to marshal/unmarshal his blob classes from/to byte[] arrays.
> > No major changes in public API are required, it can be pluggable
> component.
> >
> >
> > 2018-08-03 0:25 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org>:
> >
> > > On Thu, Aug 2, 2018 at 1:08 AM, Pavel Kovalenko <jokserfn@gmail.com>
> > > wrote:
> > >
> > > > Dmitriy,
> > > >
> > > > I still don't understand why do you think that it will be file
> system?
> > > > In all my previous messages I emphasized that this storage shouldn't
> be
> > > > considered as a file system. It's just a large data storage, whose
> > > entities
> > > > can be easily accessed using key/link (internally, or externally
> using
> > > > web/binary protocol interfaces).
> > > >
> > > > > Instead, if we must focus on large blobs, I would solve the problem
> > of
> > > > supporting large blobs in regular Ignite caches, as I suggested
> before.
> > > >
> > > > This is impossible. Our page memory can't handle efficiently it by
> > > design.
> > > >
> > >
> > > But our API does. What is stopping us from creating a cache as a BLOB
> > cache
> > > and using whatever storage we need?
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message