calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enrico Olivelli <eolive...@gmail.com>
Subject Re: Adding UnsafeRow to Calcite
Date Thu, 23 Nov 2017 21:04:53 GMT
Il mer 22 nov 2017, 22:28 IƱigo Mediavilla <imediava@gmail.com> ha scritto:

> Hello,
>
> I'm using Calcite to provide an SQL interface (Read-Only) for a Java
> Service that contains its data in memory. However most of the fields are
> primitive types and exposing them in a ProjectableFilterableTable forces
> boxing of the fields values since the scan method returns
> Enumerable<Object[]>.
>

I see another issue, maybe from a different perspective.
Having to deal with Object[] means that you always have to unpack your
record to have this particular representation. For instance in my system I
have a page of data loaded in memory and it contains a bunch of
records/tuples.
If I want to feed Calcite I always have to unpack each record to Object[],
and as there is a projection to apply very often a new Object[] is to be
created. This is not efficient and not GC friendly.
A great enhancement maybe it would be to have some interface instead of the
bare array.
This will enable the underlying implementation not to deserialize cells
Object[]

As far as my limited experience of Calcite suggests me this change will be
very difficult to introduce as it will break binary compatibility, as most
of the Enumerable stuff is done with reflection.

Just my 2 cents
Enrico


>
>
>
>
> When I look at how ProjectableFilterableTable is used, it seems that for
> every Object array calcite is generating a Row object that relies on an
> array of Object internally.
>
> I was wondering if an UnsafeRow similar to what Spark has implemented could
> be considered for Calcite given the possible savings that it could bring in
> terms of memory and how it could in some cases like mine help avoiding
> unnecessary boxing / unboxing.
>
>
> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java
>
> Kind regards,
>
> Inigo Mediavilla
>
-- 


-- Enrico Olivelli

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message