ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Needham <needb...@gmail.com>
Subject Re: Access a cache loaded by DataStreamer with SQL
Date Fri, 15 Mar 2019 12:44:25 GMT
OK, that appears to be part of the way, but they caches are not queryable
from any SQL tools like Tableau or DBeaver.  What am i missing?

On Fri, Mar 15, 2019 at 5:29 AM Ilya Kasnacheev <ilya.kasnacheev@gmail.com>
wrote:

> Hello!
>
> Yes, I understand your confusuion here.
>
> Take a look at following elaborate snippet:
>
> package org.apache.ignite.examples;
> import java.util.Collections;
> import java.util.LinkedHashMap;
> import org.apache.ignite.Ignite;
> import org.apache.ignite.IgniteCache;
> import org.apache.ignite.IgniteDataStreamer;
> import org.apache.ignite.Ignition;
> import org.apache.ignite.cache.QueryEntity;
> import org.apache.ignite.cache.query.SqlFieldsQuery;
> import org.apache.ignite.configuration.CacheConfiguration;
>
> public class LoadTableWithDataStreamer {
>     public static void main(String[] args) {
>         try (Ignite ignite = Ignition.start()) {
>             IgniteCache<Object, Object> personCache = ignite.getOrCreateCache(new
CacheConfiguration<>("person")
>                 .setQueryEntities(Collections.singleton(
>                     new QueryEntity(Integer.class, String.class).setTableName("person_table"))));
>
>             IgniteCache<Object, Object> placeCache = ignite.getOrCreateCache(new
CacheConfiguration<>("place")
>                 .setQueryEntities(Collections.singleton(
>                     new QueryEntity(Integer.class, String.class).setTableName("place_table")
>                         // more decoration
>                         .setKeyFieldName("id").setValueFieldName("name")
>                 .setFields(new LinkedHashMap<String, String>() {{
>                     // Note that extending LinkedHashMap isn't production-ready
>                     put("id", Integer.class.getCanonicalName());
>                     put("name", String.class.getCanonicalName());
>                 }}))));
>
>             try (IgniteDataStreamer<Object, Object> ds = ignite.dataStreamer("person"))
{
>                 ds.addData(1, "John");
>             }
>
>             try (IgniteDataStreamer<Object, Object> ds = ignite.dataStreamer("place"))
{
>                 ds.addData(1, "Siberia");
>             }
>
>             System.err.println("Query result");
>             personCache.query(new SqlFieldsQuery("select _key, _val from person_table")).getAll().forEach(System.err::println);
>
>             // refer to different cache's table
>             personCache.query(new SqlFieldsQuery("select id, name from \"place\".place_table")).getAll().forEach(System.err::println);
>         }
>     }
> }
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 14 мар. 2019 г. в 22:59, Mike Needham <needbrew@gmail.com>:
>
>> Hi,
>>
>> Here is the code I am using
>>
>> package org.apache.ignite.examples;
>> import java.util.Collections;
>> import org.apache.ignite.Ignite;
>> import org.apache.ignite.IgniteCache;
>> import org.apache.ignite.IgniteDataStreamer;
>> import org.apache.ignite.Ignition;
>> import org.apache.ignite.cache.QueryEntity;
>> import org.apache.ignite.cache.query.SqlFieldsQuery;
>> import org.apache.ignite.configuration.CacheConfiguration;
>> import org.apache.ignite.configuration.IgniteConfiguration;
>>
>> public class LoadTableWithDataStreamer {
>>     public static void main(String[] args) {
>>         try (Ignite ignite =
>> Ignition.start("E:\\ignite\\apache-ignite-2.7.0-src\\examples\\config\\example-ignite.xml"))
>> {
>>             IgniteCache<Object, Object> personCache =
>> ignite.getOrCreateCache(new CacheConfiguration<>("PUBLIC")
>>                 .setQueryEntities(Collections.singleton(
>>                     new QueryEntity(Integer.class,
>> String.class).setTableName("person_table"))));
>>
>>             IgniteCache<Object, Object> placeCache =
>> ignite.getOrCreateCache(new CacheConfiguration<>("PUBLIC")
>>                     .setQueryEntities(Collections.singleton(
>>                         new QueryEntity(Integer.class,
>> String.class).setTableName("place_table"))));
>>             try (IgniteDataStreamer<Object, Object> ds =
>> ignite.dataStreamer("person")) {
>>                 ds.addData(1, "John");
>>             }
>>
>>             System.err.println("Query result");
>>             personCache.query(new SqlFieldsQuery("select * from
>> person_table")).getAll().forEach(System.err::println);
>>         }
>>     }
>> }
>>
>> I want to create to queryable tables that I can load using the data
>> streamer.  The config is the default config in the java examples
>>
>>
>> On Mon, Mar 11, 2019 at 3:06 AM Ilya Kasnacheev <
>> ilya.kasnacheev@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> Can you perhaps post your config and code? Much easier than writing my
>>> own boilerplate.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пт, 8 мар. 2019 г. в 00:36, Mike Needham <needbrew@gmail.com>:
>>>
>>>> Would it be possible for someone to provide a sample that uses the
>>>> DataStreamer for multiple large caches and the resulting caches are
>>>> queryable from ODBC tools like Tableau and DBeaver?  I can get one to work,
>>>> but cannot get the cache naming to work for a second one.
>>>>
>>>> On Thu, Mar 7, 2019 at 7:53 AM Ilya Kasnacheev <
>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> JDBC with SET STREAMING ON should work reasonably well. You could also
>>>>> use CacheStore.
>>>>>
>>>>> With .Net I guess you will have to stick to DataStreamer & learn
how
>>>>> to integrate it with Indexing properly. Or use JDBC from .Net.
>>>>>
>>>>> Regards,
>>>>> --
>>>>> Ilya Kasnacheev
>>>>>
>>>>>
>>>>> чт, 7 мар. 2019 г. в 16:31, Mike Needham <needbrew@gmail.com>:
>>>>>
>>>>>> Is that the most efficient way to load millions of rows from a
>>>>>> database table into a cache? I guess I am not seeing how this would
work
>>>>>> for millions of rows in the source database.  That also appears to
be a
>>>>>> JAVA only solution and it would be preferable to have it be usable
for .NET
>>>>>> as well as java and hopefully python.
>>>>>>
>>>>>> On Thu, Mar 7, 2019 at 6:55 AM Ilya Kasnacheev <
>>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>>
>>>>>>> Hello!
>>>>>>>
>>>>>>> What prevents you from reading rows from DB and feeding them
to JDBC
>>>>>>> prepared statement?
>>>>>>>
>>>>>>> Regards,
>>>>>>> --
>>>>>>> Ilya Kasnacheev
>>>>>>>
>>>>>>>
>>>>>>> чт, 7 мар. 2019 г. в 15:51, Mike Needham <needbrew@gmail.com>:
>>>>>>>
>>>>>>>> And what if the data is already in a database table?  I do
not want
>>>>>>>> to read from the table to write to a file to load a cache.
>>>>>>>>
>>>>>>>> On Tue, Mar 5, 2019 at 4:06 AM Ilya Kasnacheev <
>>>>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hello!
>>>>>>>>>
>>>>>>>>> The preferred approach is to use Thin JDBC client:
>>>>>>>>> https://apacheignite-sql.readme.io/docs/jdbc-driver
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> --
>>>>>>>>> Ilya Kasnacheev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> пн, 4 мар. 2019 г. в 19:39, Mike Needham <needbrew@gmail.com>:
>>>>>>>>>
>>>>>>>>>> Thanks for the links,  If SET STREAMING ON is the
preferred
>>>>>>>>>> method, how would you do this in code rather than
from a file with all the
>>>>>>>>>> insert statements.
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 4, 2019 at 1:44 AM Ilya Kasnacheev <
>>>>>>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello!
>>>>>>>>>>>
>>>>>>>>>>> You can see at this page:
>>>>>>>>>>> https://apacheignite-sql.readme.io/docs/sql-and-key-value-usage
>>>>>>>>>>>
>>>>>>>>>>> And then substitite cache.put() with addData()
and cache name
>>>>>>>>>>> will be SQL_PUBLIC_{table name in caps}
>>>>>>>>>>>
>>>>>>>>>>> There are not many examples since this is being
discouraged in
>>>>>>>>>>> favor of SET STREAMING ON, actually:
>>>>>>>>>>> https://apacheignite-sql.readme.io/docs/set
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> --
>>>>>>>>>>> Ilya Kasnacheev
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> пт, 1 мар. 2019 г. в 22:48, Mike Needham
<needbrew@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>> I have looked at the documentation and the
code samples and
>>>>>>>>>>>> nothing is doing what I am trying to do.
 I want to be able to use the
>>>>>>>>>>>> datastreamer to load 3 or 4 TABLES in a cache
for an application that we
>>>>>>>>>>>> use.  If I create the tables using a create
table syntax how do attach a
>>>>>>>>>>>> datastreamer to the different caches if the
cache name is PUBLIC for all of
>>>>>>>>>>>> them?
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Feb 28, 2019 at 8:13 AM Ilya Kasnacheev
<
>>>>>>>>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello!
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have linked the documentation page,
there are also some code
>>>>>>>>>>>>> examples in distribution.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Ilya Kasnacheev
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> чт, 28 февр. 2019 г. в 17:10,
Mike Needham <needbrew@gmail.com
>>>>>>>>>>>>> >:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is there any examples that show the
steps to do this
>>>>>>>>>>>>>> correctly?  I stumbled upon this
but have no idea if it is the best way to
>>>>>>>>>>>>>> do this
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Feb 28, 2019 at 6:27 AM Ilya
Kasnacheev <
>>>>>>>>>>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There's no restriction on cache
name but setting it up for
>>>>>>>>>>>>>>> the first time may be tricky
indeed.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Ilya Kasnacheev
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ср, 27 февр. 2019 г. в
19:48, needbrew99 <needbrew@gmail.com
>>>>>>>>>>>>>>> >:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> OK, was able to get it working.
 Apparently the cache name
>>>>>>>>>>>>>>>> has to be PUBLIC
>>>>>>>>>>>>>>>> and it will create a table
based on the object definition
>>>>>>>>>>>>>>>> that I have.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> *Some days it just not worth chewing
through the restraints*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> *Some days it just not worth chewing through
the restraints*
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> *Some days it just not worth chewing through the
restraints*
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Some days it just not worth chewing through the restraints*
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Some days it just not worth chewing through the restraints*
>>>>>>
>>>>>
>>>>
>>>> --
>>>> *Some days it just not worth chewing through the restraints*
>>>>
>>>
>>
>> --
>> *Some days it just not worth chewing through the restraints*
>>
>

-- 
*Some days it just not worth chewing through the restraints*

Mime
View raw message