Hi,

I'm closing up and opening the connections after execution of insert query by my bolt.Do I need to increase the max connections db to a level of 500 to 1000? Here in log file of supervisor it tells  me the connection made by bolt  is closed and again taking new connection.I think after closing it is going into ideal and not used for new connections.

2016-05-05T10:44:37.761+0530 STDIO [INFO] COMMAND TO CLOSE CONNECTION IS DONE
2016-05-05T10:44:37.761+0530 STDIO [INFO] DB connected....
 
if I'm right what is the solution for those ideal connections?

 

Best regards,
K.Sai Dilip Reddy.


On Thu, May 5, 2016 at 1:01 AM, Spico Florin <spicoflorin@gmail.com> wrote:
hi!
 you have 9 bolts with 50 max db connections. so for each bolt you get a conection pool. try to decresae this number for example to 5 and check if your performance if fine with your db
regards,
 florin

On Wednesday, May 4, 2016, Sai Dilip Reddy Kiralam <dkiralam@aadhya-analytics.com> wrote:
sorry ! my db is not yet started, so it given me the error ! but when I gave that statement and run the topology then it is using more connections than the specified number.

 

Best regards,
K.Sai Dilip Reddy.


On Wed, May 4, 2016 at 11:22 AM, Sai Dilip Reddy Kiralam <dkiralam@aadhya-analytics.com> wrote:

Hi,

I just added

a statement  // config.setMaximumPoolSize(50);

    public synchronized void prepare() {
        if(dataSource == null) {
            Properties properties = new Properties();
            properties.putAll(configMap);
            HikariConfig config = new HikariConfig(properties);
            config.setMaximumPoolSize(50);
            this.dataSource = new HikariDataSource(config);
            this.dataSource.setAutoCommit(false);
        }
    }


and I'm getting the following error

java.lang.RuntimeException: Fail-fast during pool initialization at com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:475) at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:159) at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112) at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:78) at com.aail.config.MConnectionProvider.prepare(MConnectionProvider.java:27) at com.aail.storm.bolts.AbstractJdbcBolt.prepare(AbstractJdbcBolt.java:34) at com.aail.storm.bolts.Inserts.prepare(Inserts.java:290) at backtype.storm.daemon.executor$fn__6647$fn__6659.invoke(executor.clj:692) at backtype.storm.util$async_loop$fn__459.invoke(util.clj:461) at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745) Caused by: org.postgresql.util.PSQLException: Connection attempt timed out. at org.postgresql.Driver$ConnectThread.getResult(Driver.java:372) at org.postgresql.Driver.connect(Driver.java:284) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:247) at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:99) at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:82) at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:398) at com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:474) ... 10 more


 

Best regards,
K.Sai Dilip Reddy.


On Wed, May 4, 2016 at 9:53 AM, Sai Dilip Reddy Kiralam <dkiralam@aadhya-analytics.com> wrote:
Hi,

Thank you Spico Florin.

 

Best regards,
K.Sai Dilip Reddy.


On Wed, May 4, 2016 at 1:18 AM, Spico Florin <spicoflorin@gmail.com> wrote:
hi!
  please have a look https://github.com/apache/storm/tree/master/external/storm-jdbc where you hve to implemnt thw interface connectionprovider interface. you have to dins a third paty librry that provides conection pooloing for postgresql and use that library to implement the interface connection provider. i hope that other community member will give you some other ideas.
 i hope that these help.
regards,
 florin

On Tuesday, May 3, 2016, Sai Dilip Reddy Kiralam <dkiralam@aadhya-analytics.com> wrote:
to implemI'm using storm jdbc for connecting to db.how can I use pooling on codes on topology.please share information on connection pooling used on storm topologies.

 

Best regards,
K.Sai Dilip Reddy.


On Thu, Apr 28, 2016 at 12:28 PM, Sai Dilip Reddy Kiralam <dkiralam@aadhya-analytics.com> wrote:
Hi Spico,

I use 9 bolts with parallelism of 1 and with 1 task for each bolt(default) and I'm not using any ConnectionPool for connecting to postgres.Just using jdbc classe examples here is source example http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-jdbc.html.
I will check by using the ConnectionPool.

Thank you.
 

 

Best regards,
K.Sai Dilip Reddy.


On Thu, Apr 28, 2016 at 12:19 PM, Spico Florin <spicoflorin@gmail.com> wrote:
Hello!
   How many tasks do you have for inserting the data to your database? Are you using ConnectionPool for connecting to Postgres? If your number of task superseeds the number of max connections provided in connection pool then your have a problem.
Please also check the number of max connections that your db accepts.
I hope that these help.
Regards,
  Florin

On Thu, Apr 28, 2016 at 7:09 AM, Sai Dilip Reddy Kiralam <dkiralam@aadhya-analytics.com> wrote:
Hi,

I have topology that make connection with postgresdb and insert the fields into tables of a test db.my topology is working fine but when I submit the topology it is establishing all the connections of db.but I don’t know why it is taking all the max connections.
 

below attached the pics of pg_stat_activity.
 

Best regards,
K.Sai Dilip Reddy.