calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Beikov <christian.bei...@gmail.com>
Subject Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table
Date Fri, 29 Jun 2018 08:19:07 GMT
IMO the best solution would be to make it configurable by introducing a 
"table_mapping" config with values

  * type - every type in the known indices is mapped as table
  * index - every known index is mapped as table

We'd probably also need a "type_field" configuration for defining which 
field to use for the type determination as one of the possible future 
ways to do things is to introduce a custom field: 
https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html#_custom_type_field_2

We already detect the ES version, so we can set a smart default for this 
setting. Let's make the index config param optional.

  * When no index is given, we discover indexes, the default for
    "table_mapping" then is "index"
  * When index is given, the we only discover types according to the
    "type_field" configuration and the default for "table_mapping" is "type"

This would also allow to discover indexes but still use "type" as 
"table_mapping".

What do you think?

Mit freundlichen Grüßen,
------------------------------------------------------------------------
*Christian Beikov*
Am 29.06.2018 um 02:41 schrieb Andrei Sereda:
> Yes. There is an API to list all indexes / types in elastic. They can be
> automatically imported into a schema.
>
> What needs to be agreed upon is how to expose those elements in calcite
> schema (naming / behaviour).
>
> 1) Many (most?) of setups are single type per index. Natural way to name
> would be  "elastic.$index" (elastic being schema name). Multiple indexes
> would be under same schema "elastic.index1" "elastic.index2" etc.
>
> 2) What if index has several types should they exported as calcite tables:
> "elastic.$index_type1" "elastic.$index_type2" ?  Or (current behaviour) as
> "elastic.type1" and "elastic.type2". Or as subschema
> "elastic.$index.type1" ?
>
> Now what if one has combination of (1) and (2) ?
> Setup (2) is already deprecated (and will be unsupported in next version)
>
>
> On Thu, Jun 28, 2018 at 7:31 PM Christian Beikov <christian.beikov@gmail.com>
> wrote:
>
>> Is there an API to discover indexes? If there is, I'd suggest we allow a
>> config option that to make the adapter discover the possible indexes.
>> We'd still have to adapt the code a bit, but internally, the schema
>> could just keep a cache of type name to index name map and be able to
>> support both scenarios.
>>
>>
>> Mit freundlichen Grüßen,
>> ------------------------------------------------------------------------
>> *Christian Beikov*
>> Am 29.06.2018 um 00:12 schrieb Andrei Sereda:
>>>> 1) What's the time horizon for the current adapter no longer working
>> with these
>>> changes to ES ?
>>> Current adapter will be working for a while with existing setup. The
>>> problem is nomenclature and ease of use.
>>>
>>> Their new SQL concepts mapping
>>> <
>> https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html
>>> drops
>>> the notion of ES type (which before was equivalent of RDBMS table) and
>> uses
>>> ES index as new table equivalent (before ES index was equal to database).
>>> Most users use elastic this way (one type , one index) index == table.
>>>
>>> Currently calcite requires schema per index. In RDBMS parlance database
>> per
>>> table (I'd like to change that).
>>>
>>>> 2) Any guess how complicated it would be to maintain code paths for both
>>>> behaviours? I know this is probably really challenging to estimate, but
>> I
>>>> really have no idea of the scope of these changes. Would it mean two
>>>> different ES adapters?
>>> One can have just a separate calcite schema implementations (same
>> adapter /
>>> module) :
>>> 1)  LegacySchema (old). Schema can have only one index (but multiple
>>> types). Type == table in this case.
>>> 2)  NewSchema (new). Single schema can have multiple indexes (type is
>>> dropped). Index == table in this case
>>>
>>>> 3) Do we really need compatibility with the current version of the
>>> adapter?
>>>> IMO this depends on what versions of ES we would lose support for and
>> how
>>>> complex it would be for users of the current ES adapter to make updates
>>> for
>>>> any Calcite API changes.
>>> The issue is not in adapter but how calcite schema exposes tables.
>> Should
>>> it expose index as individual table (new), or ES type (old) ?
>>>
>>> Andrei.
>>>
>>> On Thu, Jun 28, 2018 at 5:23 PM Michael Mior <mmior@apache.org> wrote:
>>>
>>>> Unfortunately I know very little about ES so I'm not in a great
>> position to
>>>> asses the impact of these changes. I will say that that legacy
>>>> compatibility is great, but maintaining two sets of logic is always a
>>>> challenge. A few follow up questions:
>>>>
>>>> 1) What's the time horizon for the current adapter no longer working
>> with
>>>> these changes to ES?
>>>>
>>>> 2) Any guess how complicated it would be to maintain code paths for both
>>>> behaviours? I know this is probably really challenging to estimate, but
>> I
>>>> really have no idea of the scope of these changes. Would it mean two
>>>> different ES adapters?
>>>>
>>>> 3) Do we really need compatibility with the current version of the
>> adapter?
>>>> IMO this depends on what versions of ES we would lose support for and
>> how
>>>> complex it would be for users of the current ES adapter to make updates
>> for
>>>> any Calcite API changes.
>>>>
>>>> Thanks for your continued work on the ES adapter Andrei!
>>>>
>>>> --
>>>> Michael Mior
>>>> mmior@apache.org
>>>>
>>>>
>>>>
>>>> Le jeu. 28 juin 2018 à 12:57, Andrei Sereda <andrei@sereda.cc> a écrit
>> :
>>>>> Hello,
>>>>>
>>>>> Elastic announced
>>>>> <
>>>>>
>> https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html
>>>>> that they will be deprecating mapping types in ES6 and indexes will be
>>>>> single-typed only.
>>>>>
>>>>> Historical analogy <https://www.elastic.co/blog/index-vs-type>
between
>>>>> RDBMS and elastic was that index is equivalent to a database and type
>>>>> corresponds to table in that database. In a couple of releases (ES6-8)
>>>> this
>>>>> shall not longer be true.
>>>>>
>>>>> Recent SQL addition
>>>>> <https://www.elastic.co/blog/elasticsearch-6-3-0-released> to elastic
>>>>> confirms
>>>>> this trend
>>>>> <
>>>>>
>> https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html
>>>>>> .
>>>>> Index is equivalent to a table and there are no more ES types.
>>>>>
>>>>> I would like to propose to include this logic in Calcite ES adapter.
>> IE,
>>>>> expose each ES single-typed index as a separate table inside calcite
>>>>> schema. This is in contrast to  current integration where schema can
>> only
>>>>> have a single index. Current approach forces you to create multiple
>>>> schemas
>>>>> to query single-typed indexes (on the same ES cluster).
>>>>>
>>>>> Legacy compatibility can always be controlled with configuration
>>>>> parameters.
>>>>>
>>>>> Do you agree with such changes ? If yes, would you consider a PR ?
>>>>>
>>>>> Regards,
>>>>> Andrei.
>>>>>
>>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message