ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ilya Kasnacheev <ilya.kasnach...@gmail.com>
Subject Re: Access a cache loaded by DataStreamer with SQL
Date Fri, 15 Mar 2019 10:28:50 GMT
Hello!

Yes, I understand your confusuion here.

Take a look at following elaborate snippet:

package org.apache.ignite.examples;
import java.util.Collections;
import java.util.LinkedHashMap;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.IgniteDataStreamer;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.QueryEntity;
import org.apache.ignite.cache.query.SqlFieldsQuery;
import org.apache.ignite.configuration.CacheConfiguration;

public class LoadTableWithDataStreamer {
    public static void main(String[] args) {
        try (Ignite ignite = Ignition.start()) {
            IgniteCache<Object, Object> personCache =
ignite.getOrCreateCache(new CacheConfiguration<>("person")
                .setQueryEntities(Collections.singleton(
                    new QueryEntity(Integer.class,
String.class).setTableName("person_table"))));

            IgniteCache<Object, Object> placeCache =
ignite.getOrCreateCache(new CacheConfiguration<>("place")
                .setQueryEntities(Collections.singleton(
                    new QueryEntity(Integer.class,
String.class).setTableName("place_table")
                        // more decoration
                        .setKeyFieldName("id").setValueFieldName("name")
                .setFields(new LinkedHashMap<String, String>() {{
                    // Note that extending LinkedHashMap isn't production-ready
                    put("id", Integer.class.getCanonicalName());
                    put("name", String.class.getCanonicalName());
                }}))));

            try (IgniteDataStreamer<Object, Object> ds =
ignite.dataStreamer("person")) {
                ds.addData(1, "John");
            }

            try (IgniteDataStreamer<Object, Object> ds =
ignite.dataStreamer("place")) {
                ds.addData(1, "Siberia");
            }

            System.err.println("Query result");
            personCache.query(new SqlFieldsQuery("select _key, _val
from person_table")).getAll().forEach(System.err::println);

            // refer to different cache's table
            personCache.query(new SqlFieldsQuery("select id, name from
\"place\".place_table")).getAll().forEach(System.err::println);
        }
    }
}

Regards,
-- 
Ilya Kasnacheev


чт, 14 мар. 2019 г. в 22:59, Mike Needham <needbrew@gmail.com>:

> Hi,
>
> Here is the code I am using
>
> package org.apache.ignite.examples;
> import java.util.Collections;
> import org.apache.ignite.Ignite;
> import org.apache.ignite.IgniteCache;
> import org.apache.ignite.IgniteDataStreamer;
> import org.apache.ignite.Ignition;
> import org.apache.ignite.cache.QueryEntity;
> import org.apache.ignite.cache.query.SqlFieldsQuery;
> import org.apache.ignite.configuration.CacheConfiguration;
> import org.apache.ignite.configuration.IgniteConfiguration;
>
> public class LoadTableWithDataStreamer {
>     public static void main(String[] args) {
>         try (Ignite ignite =
> Ignition.start("E:\\ignite\\apache-ignite-2.7.0-src\\examples\\config\\example-ignite.xml"))
> {
>             IgniteCache<Object, Object> personCache =
> ignite.getOrCreateCache(new CacheConfiguration<>("PUBLIC")
>                 .setQueryEntities(Collections.singleton(
>                     new QueryEntity(Integer.class,
> String.class).setTableName("person_table"))));
>
>             IgniteCache<Object, Object> placeCache =
> ignite.getOrCreateCache(new CacheConfiguration<>("PUBLIC")
>                     .setQueryEntities(Collections.singleton(
>                         new QueryEntity(Integer.class,
> String.class).setTableName("place_table"))));
>             try (IgniteDataStreamer<Object, Object> ds =
> ignite.dataStreamer("person")) {
>                 ds.addData(1, "John");
>             }
>
>             System.err.println("Query result");
>             personCache.query(new SqlFieldsQuery("select * from
> person_table")).getAll().forEach(System.err::println);
>         }
>     }
> }
>
> I want to create to queryable tables that I can load using the data
> streamer.  The config is the default config in the java examples
>
>
> On Mon, Mar 11, 2019 at 3:06 AM Ilya Kasnacheev <ilya.kasnacheev@gmail.com>
> wrote:
>
>> Hello!
>>
>> Can you perhaps post your config and code? Much easier than writing my
>> own boilerplate.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пт, 8 мар. 2019 г. в 00:36, Mike Needham <needbrew@gmail.com>:
>>
>>> Would it be possible for someone to provide a sample that uses the
>>> DataStreamer for multiple large caches and the resulting caches are
>>> queryable from ODBC tools like Tableau and DBeaver?  I can get one to work,
>>> but cannot get the cache naming to work for a second one.
>>>
>>> On Thu, Mar 7, 2019 at 7:53 AM Ilya Kasnacheev <
>>> ilya.kasnacheev@gmail.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> JDBC with SET STREAMING ON should work reasonably well. You could also
>>>> use CacheStore.
>>>>
>>>> With .Net I guess you will have to stick to DataStreamer & learn how
to
>>>> integrate it with Indexing properly. Or use JDBC from .Net.
>>>>
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>>
>>>>
>>>> чт, 7 мар. 2019 г. в 16:31, Mike Needham <needbrew@gmail.com>:
>>>>
>>>>> Is that the most efficient way to load millions of rows from a
>>>>> database table into a cache? I guess I am not seeing how this would work
>>>>> for millions of rows in the source database.  That also appears to be
a
>>>>> JAVA only solution and it would be preferable to have it be usable for
.NET
>>>>> as well as java and hopefully python.
>>>>>
>>>>> On Thu, Mar 7, 2019 at 6:55 AM Ilya Kasnacheev <
>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>
>>>>>> Hello!
>>>>>>
>>>>>> What prevents you from reading rows from DB and feeding them to JDBC
>>>>>> prepared statement?
>>>>>>
>>>>>> Regards,
>>>>>> --
>>>>>> Ilya Kasnacheev
>>>>>>
>>>>>>
>>>>>> чт, 7 мар. 2019 г. в 15:51, Mike Needham <needbrew@gmail.com>:
>>>>>>
>>>>>>> And what if the data is already in a database table?  I do not
want
>>>>>>> to read from the table to write to a file to load a cache.
>>>>>>>
>>>>>>> On Tue, Mar 5, 2019 at 4:06 AM Ilya Kasnacheev <
>>>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hello!
>>>>>>>>
>>>>>>>> The preferred approach is to use Thin JDBC client:
>>>>>>>> https://apacheignite-sql.readme.io/docs/jdbc-driver
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> --
>>>>>>>> Ilya Kasnacheev
>>>>>>>>
>>>>>>>>
>>>>>>>> пн, 4 мар. 2019 г. в 19:39, Mike Needham <needbrew@gmail.com>:
>>>>>>>>
>>>>>>>>> Thanks for the links,  If SET STREAMING ON is the preferred
>>>>>>>>> method, how would you do this in code rather than from
a file with all the
>>>>>>>>> insert statements.
>>>>>>>>>
>>>>>>>>> On Mon, Mar 4, 2019 at 1:44 AM Ilya Kasnacheev <
>>>>>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello!
>>>>>>>>>>
>>>>>>>>>> You can see at this page:
>>>>>>>>>> https://apacheignite-sql.readme.io/docs/sql-and-key-value-usage
>>>>>>>>>>
>>>>>>>>>> And then substitite cache.put() with addData() and
cache name
>>>>>>>>>> will be SQL_PUBLIC_{table name in caps}
>>>>>>>>>>
>>>>>>>>>> There are not many examples since this is being discouraged
in
>>>>>>>>>> favor of SET STREAMING ON, actually:
>>>>>>>>>> https://apacheignite-sql.readme.io/docs/set
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> --
>>>>>>>>>> Ilya Kasnacheev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> пт, 1 мар. 2019 г. в 22:48, Mike Needham <needbrew@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> I have looked at the documentation and the code
samples and
>>>>>>>>>>> nothing is doing what I am trying to do.  I want
to be able to use the
>>>>>>>>>>> datastreamer to load 3 or 4 TABLES in a cache
for an application that we
>>>>>>>>>>> use.  If I create the tables using a create table
syntax how do attach a
>>>>>>>>>>> datastreamer to the different caches if the cache
name is PUBLIC for all of
>>>>>>>>>>> them?
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Feb 28, 2019 at 8:13 AM Ilya Kasnacheev
<
>>>>>>>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello!
>>>>>>>>>>>>
>>>>>>>>>>>> I have linked the documentation page, there
are also some code
>>>>>>>>>>>> examples in distribution.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> --
>>>>>>>>>>>> Ilya Kasnacheev
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> чт, 28 февр. 2019 г. в 17:10, Mike
Needham <needbrew@gmail.com
>>>>>>>>>>>> >:
>>>>>>>>>>>>
>>>>>>>>>>>>> Is there any examples that show the steps
to do this
>>>>>>>>>>>>> correctly?  I stumbled upon this but
have no idea if it is the best way to
>>>>>>>>>>>>> do this
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Feb 28, 2019 at 6:27 AM Ilya
Kasnacheev <
>>>>>>>>>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There's no restriction on cache name
but setting it up for
>>>>>>>>>>>>>> the first time may be tricky indeed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Ilya Kasnacheev
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ср, 27 февр. 2019 г. в 19:48,
needbrew99 <needbrew@gmail.com
>>>>>>>>>>>>>> >:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> OK, was able to get it working.
 Apparently the cache name
>>>>>>>>>>>>>>> has to be PUBLIC
>>>>>>>>>>>>>>> and it will create a table based
on the object definition
>>>>>>>>>>>>>>> that I have.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> *Some days it just not worth chewing
through the restraints*
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *Some days it just not worth chewing through
the restraints*
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Some days it just not worth chewing through the restraints*
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Some days it just not worth chewing through the restraints*
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> *Some days it just not worth chewing through the restraints*
>>>>>
>>>>
>>>
>>> --
>>> *Some days it just not worth chewing through the restraints*
>>>
>>
>
> --
> *Some days it just not worth chewing through the restraints*
>

Mime
View raw message