storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dima Dragan <dima.dra...@belleron.net>
Subject Re: write data to file then bulk copy to database
Date Sat, 06 Dec 2014 08:37:35 GMT
Sure it depends from database, but you can try using statement in batch
mode.

Please look at:
https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#addBatch(java.lang.String)

Best regards,
Dmytro Dragan
On Dec 6, 2014 12:52 AM, "Sa Li" <sa.in.vanc@gmail.com> wrote:

> Hi, Dmytro
>
> Thanks for the reply, I am using trident topology, here is my static
> class, but I think this is incorrect since trident process data in batch,
> but following code process single row.
>
> public static class WriteDB extends BaseFunction {
>       private Connection conn = null ;
>       PreparedStatement ps = null;
>
>      @Override
>      public final void execute(final TridentTuple tuple, final
> TridentCollector collector) {
>                       int user = tuple.getInteger(0);
>                       String value = tuple.getString(1);
>                       final StringBuilder queryBuilder = new
> StringBuilder()
>                                .append("INSERT INTO test.state(userid,
> event) VALUES(")
>                                .append(user)
>                                .append(", '")
>                                .append(value)
>                                .append("')");
>                      System.out.println(queryBuilder.toString());
>                      try {
>                               ps =
> conn.prepareStatement(queryBuilder.toString()) ;
>                               ps.execute();
>                               collector.emit(new
> Values(tuple.getStringByField("event")));
>                      }
>                      catch (SQLException ex) {
>                               System.err.println("Caught IOException: " +
> ex.getMessage());
>                      }      finally {
>                                             if (ps != null) {
>                                                              try {
>
> ps.close();
>                                                              } catch
> (SQLException ex) {
>                                                              }
>                                             }
>                      }
>         }
>  }
>
>
> Any idea?
>
> thanks
>
> On Fri, Dec 5, 2014 at 2:35 PM, Dima Dragan <dima.dragan@belleron.net>
> wrote:
>
>> Hi,
>>
>> I think better approach is creating static class with thread-safe
>> singleton initialization of connection pool (for example, I use HikariCP
>> for SQL db and JedisPool for Redis) and provide public access for getting
>> connection from it to bolt. Init pool in prepare method, take
>> connection,execute smth, give it back in execute method.
>>
>> So every worker will get it's own pool which is shared between bolts.
>>
>> Best regards,
>> Dmytro Dragan
>> On Dec 5, 2014 11:57 PM, "Sa Li" <sa.in.vanc@gmail.com> wrote:
>>
>>> Hi, all
>>>
>>> Right now I am able to write tuples into a file, and use copy command to
>>> load into database, but this is obviously not a perfect solution, increase
>>> complexity and overheads. Any idea to make it simpler?
>>>
>>>
>>> thanks
>>>
>>>
>>> Alec
>>>
>>
>

Mime
View raw message