storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sa Li <sa.in.v...@gmail.com>
Subject Re: write data to file then bulk copy to database
Date Fri, 05 Dec 2014 22:51:45 GMT
Hi, Dmytro

Thanks for the reply, I am using trident topology, here is my static class,
but I think this is incorrect since trident process data in batch, but
following code process single row.

public static class WriteDB extends BaseFunction {
      private Connection conn = null ;
      PreparedStatement ps = null;

     @Override
     public final void execute(final TridentTuple tuple, final
TridentCollector collector) {
                      int user = tuple.getInteger(0);
                      String value = tuple.getString(1);
                      final StringBuilder queryBuilder = new StringBuilder()
                               .append("INSERT INTO test.state(userid,
event) VALUES(")
                               .append(user)
                               .append(", '")
                               .append(value)
                               .append("')");
                     System.out.println(queryBuilder.toString());
                     try {
                              ps =
conn.prepareStatement(queryBuilder.toString()) ;
                              ps.execute();
                              collector.emit(new
Values(tuple.getStringByField("event")));
                     }
                     catch (SQLException ex) {
                              System.err.println("Caught IOException: " +
ex.getMessage());
                     }      finally {
                                            if (ps != null) {
                                                             try {

ps.close();
                                                             } catch
(SQLException ex) {
                                                             }
                                            }
                     }
        }
 }


Any idea?

thanks

On Fri, Dec 5, 2014 at 2:35 PM, Dima Dragan <dima.dragan@belleron.net>
wrote:

> Hi,
>
> I think better approach is creating static class with thread-safe
> singleton initialization of connection pool (for example, I use HikariCP
> for SQL db and JedisPool for Redis) and provide public access for getting
> connection from it to bolt. Init pool in prepare method, take
> connection,execute smth, give it back in execute method.
>
> So every worker will get it's own pool which is shared between bolts.
>
> Best regards,
> Dmytro Dragan
> On Dec 5, 2014 11:57 PM, "Sa Li" <sa.in.vanc@gmail.com> wrote:
>
>> Hi, all
>>
>> Right now I am able to write tuples into a file, and use copy command to
>> load into database, but this is obviously not a perfect solution, increase
>> complexity and overheads. Any idea to make it simpler?
>>
>>
>> thanks
>>
>>
>> Alec
>>
>

Mime
View raw message