calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Beikov <christian.bei...@gmail.com>
Subject Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table
Date Thu, 28 Jun 2018 23:30:46 GMT
Is there an API to discover indexes? If there is, I'd suggest we allow a 
config option that to make the adapter discover the possible indexes. 
We'd still have to adapt the code a bit, but internally, the schema 
could just keep a cache of type name to index name map and be able to 
support both scenarios.


Mit freundlichen Grüßen,
------------------------------------------------------------------------
*Christian Beikov*
Am 29.06.2018 um 00:12 schrieb Andrei Sereda:
>> 1) What's the time horizon for the current adapter no longer working with these
> changes to ES ?
> Current adapter will be working for a while with existing setup. The
> problem is nomenclature and ease of use.
>
> Their new SQL concepts mapping
> <https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html>
> drops
> the notion of ES type (which before was equivalent of RDBMS table) and uses
> ES index as new table equivalent (before ES index was equal to database).
> Most users use elastic this way (one type , one index) index == table.
>
> Currently calcite requires schema per index. In RDBMS parlance database per
> table (I'd like to change that).
>
>> 2) Any guess how complicated it would be to maintain code paths for both
>> behaviours? I know this is probably really challenging to estimate, but I
>> really have no idea of the scope of these changes. Would it mean two
>> different ES adapters?
> One can have just a separate calcite schema implementations (same adapter /
> module) :
> 1)  LegacySchema (old). Schema can have only one index (but multiple
> types). Type == table in this case.
> 2)  NewSchema (new). Single schema can have multiple indexes (type is
> dropped). Index == table in this case
>
>> 3) Do we really need compatibility with the current version of the
> adapter?
>> IMO this depends on what versions of ES we would lose support for and how
>> complex it would be for users of the current ES adapter to make updates
> for
>> any Calcite API changes.
> The issue is not in adapter but how calcite schema exposes tables.  Should
> it expose index as individual table (new), or ES type (old) ?
>
> Andrei.
>
> On Thu, Jun 28, 2018 at 5:23 PM Michael Mior <mmior@apache.org> wrote:
>
>> Unfortunately I know very little about ES so I'm not in a great position to
>> asses the impact of these changes. I will say that that legacy
>> compatibility is great, but maintaining two sets of logic is always a
>> challenge. A few follow up questions:
>>
>> 1) What's the time horizon for the current adapter no longer working with
>> these changes to ES?
>>
>> 2) Any guess how complicated it would be to maintain code paths for both
>> behaviours? I know this is probably really challenging to estimate, but I
>> really have no idea of the scope of these changes. Would it mean two
>> different ES adapters?
>>
>> 3) Do we really need compatibility with the current version of the adapter?
>> IMO this depends on what versions of ES we would lose support for and how
>> complex it would be for users of the current ES adapter to make updates for
>> any Calcite API changes.
>>
>> Thanks for your continued work on the ES adapter Andrei!
>>
>> --
>> Michael Mior
>> mmior@apache.org
>>
>>
>>
>> Le jeu. 28 juin 2018 à 12:57, Andrei Sereda <andrei@sereda.cc> a écrit :
>>
>>> Hello,
>>>
>>> Elastic announced
>>> <
>>>
>> https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html
>>> that they will be deprecating mapping types in ES6 and indexes will be
>>> single-typed only.
>>>
>>> Historical analogy <https://www.elastic.co/blog/index-vs-type> between
>>> RDBMS and elastic was that index is equivalent to a database and type
>>> corresponds to table in that database. In a couple of releases (ES6-8)
>> this
>>> shall not longer be true.
>>>
>>> Recent SQL addition
>>> <https://www.elastic.co/blog/elasticsearch-6-3-0-released> to elastic
>>> confirms
>>> this trend
>>> <
>>>
>> https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html
>>>> .
>>> Index is equivalent to a table and there are no more ES types.
>>>
>>> I would like to propose to include this logic in Calcite ES adapter. IE,
>>> expose each ES single-typed index as a separate table inside calcite
>>> schema. This is in contrast to  current integration where schema can only
>>> have a single index. Current approach forces you to create multiple
>> schemas
>>> to query single-typed indexes (on the same ES cluster).
>>>
>>> Legacy compatibility can always be controlled with configuration
>>> parameters.
>>>
>>> Do you agree with such changes ? If yes, would you consider a PR ?
>>>
>>> Regards,
>>> Andrei.
>>>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message