beehive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chad Schoettger" <chad.schoett...@gmail.com>
Subject Re: Idea for JDBC system control
Date Mon, 18 Dec 2006 21:48:09 GMT
Hi Drew,

I may not have much time to work on this for the rest of the week
(Holidays rapidly approaching) so I'm going to go ahead and commit
your patch to add the setConnection() API to the JDBC control (minus
the lazy connection issue).

I think we are both in agreement that it would be very useful to be
able to configure a control's resource management.  I am not convinced
that only changing how the JDBC control manages its resources is the
best long term approach.

I would prefer that we try to come up with an enhancement to the
controls framework so that any control could have its resource
management tuned by the user.  I'll create a JIRA enhancement request
and we can start a new thread on the dev list about the best possible
solution for this issue.

  - Chad






On 12/15/06, Drew Varner <drew.varner@redops.org> wrote:
> Hey Chad,
>
> I am trying to put my arms around what implementations we could break
> by moving to lazy connections and closing them when possible. There
> are four that come to mind. The first is a case where someone set's a
> transaction on a JDBC control via setAutoCommit(false).
>
> For example:
>
> myDatabaseControl.getConnection().setAutoCommit(false);
> myDatabaseControl.someTransactionalMethod();
> myDatabaseControl.getConnection().commit();
> myDatabaseControl.getConnection().setAutoCommit(true);
>
> This case is covered in unit tests
> (org.apache.beehive.controls.system.jdbc.units.results.TxResultsTest). I
> t's easy to preserve this case and my yet-to-be-submitted patch
> passes this test. We can't close out a connection after invoke() if
> the Connection is not in auto-commit mode. I provide a private flag
> inside the Control (boolean _connectionCloseable, defaults to true)
> to prevent this from happening. We still close out a connection if at
> release if it wasn't passed in via setConnection().
>
> The second case is where a user pulls a connection out of the Control
> to use elsewhere. For example:
>
> Connection conn = myDatabaseControl.getConnection();
> myDatabaseControl.someMethod(9);
> myDAO.deleteById(9,conn);
> conn.close();
>
> This case is not present in the unit tests. However, it is easy to
> solve this case. Once a user pulls the Connection out, we set
> _connectionCloseable to false. As the Javadoc for it suggests, this
> method is rarely called.
>
> The third case is where we return ResultSets, Iterators or RowSets.
> These are covered in the unit tests and my not-yet-submitted patch
> passes them. Flagging the connection as unsafe to close is easy.
> Calling ResultSetMapper's canCloseResultSet() method seems to be enough.
>
> The fourth case is not present in Beehive. It involves transaction
> annotations like the ones in WebLogic now. I have no way to test
> against these or know how to make this condition safe. Perhaps as
> Beehive uses Geronimo for tests we could evaluate this. Would a test
> for see if the connection is an XA connection be enough?
>
> I am going to setup a test case to measure the difference between the
> two implementations including:
>
> 1) Scalability with a small, fixed pool
> 2) Scalability with a small pool that can grow large
> 3) The number of file descriptors that are chewed up by each
> implementation
> 4) The behavior of the DataSource's connection pool
>
> If the difference is significant, perhaps behavior change could be
> could be considered for Beehive 2.x. If the difference is dramatic,
> perhaps we could evaluate a safe way to incorporate the behavior.
> It'd be nice to have an example of the cases my patch would break in
> unit tests.
>
> - Drew
>
>
> On Dec 15, 2006, at 6:21 PM, Chad Schoettger wrote:
>
> > Hi Drew,
> >
> > I'm in the process of applying your patch -- but there are a few
> > changes I'll need to make.  Due to backwards compatibility issues I
> > don't think we can  just switch the JDBC control to acquire a
> > connection in its invoke() and release it at the end of the invoke().
> > Existing applications may depend on the current behavior.
> >
> > Here are some options:
> >
> > 1) I've tweaked your patch so that it is now valid to have a JDBC
> > control without either a ConnectionDataSource or a ConnectionDriver
> > annotation.
> >
> > A JDBC control which does not include either of these annotations will
> > NOT try to acquire a connection when it receives an onAcquire event.
> > The connection must be set with the new 'setConnection()' api.  This
> > should allow developers to have full control over connections being
> > used by the control.
> >
> > If the annotations are present the setConnection() api can still be
> > used to replace a connection the jdbc control has acquired.
> >
> >
> >
> > 2) It would still useful to introduce the concept of 'short-lived'
> > resources into the controls framework.  I can imagine the scope being
> > that of a method call to the control (each method call causes an
> > onAcquire/onRelease event to be fired).   I haven't given it a lot of
> > thought but it seems like we could add a property to the
> > @BaseProperites annotation for controls something like
> > 'methodResourceScope' which would trigger this behavior.
> >
> > @ControlInterface
> > @BaseProperites(methodResourceScope=true)
> > public interface MyCtrl {
> > .
> > .
> > .
> > }
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On 12/12/06, Drew Varner <drew.varner@redops.org> wrote:
> >> Ignore the "must setup JNDI if you don't need it" at the end, you
> >> must setup JNDI or explicitly null it out via a bean method.
> >>
> >> - Drew
> >>
> >> On Dec 13, 2006, at 1:28 AM, Drew Varner wrote:
> >>
> >> > Hey Chad,
> >> >
> >> > I was not sure where those events fired.
> >> >
> >> > My concern is picking up multiple connections where one is needed.
> >> > In an example where one database control is used the results are no
> >> > different than using any other persistence mechanism. However, if
> >> > we have many controls, it becomes problematic. If I use controls as
> >> > my basic building block for data abstraction, I could have one per
> >> > domain object. It's not beyond reason to expect 10 connections to
> >> > be used in a method. This is definitely the case where we wrap
> >> > multiple JDBC Controls in a custom control. Asking the architect to
> >> > accept a database connection pool five times or more than that of
> >> > another persistence solution seems steep. When load surges, you may
> >> > need one file descriptor per request to service the HTTP request
> >> > but ten to connect to the database. Manipulating 10 domain objects
> >> > in a call may seem extreme,
> >> >
> >> > We could work around this problem by creating one control per
> >> > database. This control would quickly become a maintenance monster.
> >> >
> >> > There are other applications for controls outside of short-lived
> >> > HTTP requests. BEA's WebLogic Integration product uses Controls
> >> > extensively in JPDs. I can't speak to the details of a JpdConext
> >> > but they don't sound like short lifespans. Assuming a rich data
> >> > model, the impact of a control keeping a tight grip on a Connection
> >> > could be high.
> >> >
> >> > I think we know whether or not it is safe to close the connection
> >> > after use. The ResultSetMapper's canCloseResultSet() should be
> >> > enough information. If you can close the ResultSet and the
> >> > Connection is in auto-commit mode, you should be able to close it
> >> > safely (according to the unit tests).
> >> >
> >> > While I appreciate that part of the controls mantra is
> >> > simplification when it comes to managing resources, DataSources
> >> > probably do a better job than a control of managing Connections
> >> > (running queries to identify stale connections, etc.). But, if we
> >> > limit controls to Pageflows or other web applications, these
> >> > features aren't a big deal.
> >> >
> >> > Having the database connection instantiated lazily makes testing
> >> > easier. Look at Chris' example of testing a Control with a JNDI
> >> > DataSource:
> >> >
> >> > http://dev2dev.bea.com/blog/hogue/archive/2006/07/
> >> > outofcontainer_1.html
> >> >
> >> > You have to reference the bean, not the interface. You have to
> >> > setup a JNDI Context if you use it in testing or not. If you lazily
> >> > instantiate, these hassles go away.
> >> >
> >> > - Drew
> >> >
> >> > On Dec 12, 2006, at 11:13 PM, Chad Schoettger wrote:
> >> >
> >> >> Hi Drew,
> >> >>
> >> >> I haven't had a chance to look at your patch yet (hopefully
> >> I'll have
> >> >> some time tomorrow) but I think the best solution may be to
> >> leave the
> >> >> aquisition/release of the connection in the onAcquire/onRelease
> >> >> methods.
> >> >>
> >> >> When using a JDBC control from a page flow these events provide a
> >> >> reasonably nice way of managing the connections. The connection is
> >> >> basically scoped for the page flow method the control is being
> >> >> invoked
> >> >> from:
> >> >>
> >> >> MyPageFlow.java
> >> >>
> >> >>   @Control private JDBCControl _myJdbcControl;
> >> >>
> >> >>   myAction() {
> >> >>       _myJdbcControl.getRow()   /* causes onAcquire() to be
> >> >> invoked */
> >> >>       _myJdbcControl.getAnotherRow() /* does NOT cause onAquire
> >> () to
> >> >> be invoked */
> >> >>       ......
> >> >>       ......
> >> >>       return ....   /* causes onRelease() to be invoked() */
> >> >>   }
> >> >>
> >> >> I really like the idea Eddie mentioned about having a set
> >> method for
> >> >> the connection -- if the user sets the connection, the jdbc
> >> control
> >> >> closes an open connection if it has one and uses the one
> >> provided by
> >> >> the user and basically noops in the onAcquire/onRelease
> >> methods.  If
> >> >> the user sets the connection to null, the onAcquire/onRelease
> >> default
> >> >> behavior resumes.
> >> >>
> >> >> I do apologize if you were already aware of when onAcquire/
> >> onRelease
> >> >> were invoked, it seems to be an area which causes a fair amount of
> >> >> confusion.
> >> >>
> >> >>   - Chad
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On 12/11/06, Drew Varner <drew.varner@redops.org> wrote:
> >> >>> Ok, I now see why the connection cannot be closed immediately
> >> after
> >> >>> invoke: ResultSet results and possibly connected RowSets.
> >> >>>
> >> >>> Would it be possible to flag SQL methods to indicate that they
> >> don't
> >> >>> depend on an open connection? This option would not break
> >> existing
> >> >>> implementations.
> >> >>>
> >> >>> Another option would be to check the return type of the
> >> method. If
> >> >>> the return type is ResultSet or RowSet, we don't close it. I
> >> don't
> >> >>> like this solution as much. If a custom mapper does something
> >> weird
> >> >>> like wrapping a RowSet, we'll never know. This solution seems
> >> like
> >> >>> more of a hack.
> >> >>>
> >> >>> - Drew
> >> >>>
> >> >>>
> >> >>> On Dec 11, 2006, at 10:57 PM, Drew Varner wrote:
> >> >>>
> >> >>> > I attached the patch to BEEHIVE-1019 (http://issues.apache.org/
> >> >>> jira/
> >> >>> > browse/BEEHIVE-1019).
> >> >>> >
> >> >>> > The patch I attached moves getConnection() from onAcquire()
to
> >> >>> > invoke().
> >> >>> >
> >> >>> > In the current version (grab connection when control is
> >> acquired)
> >> >>> > or the new version (if my patch is accepted) of the JDBC
> >> Control,
> >> >>> > the connection is pulled and held for quite a while. Imagine
an
> >> >>> app
> >> >>> > has 10 database controls that point to the same database.
If
> >> they
> >> >>> > are used in the same Pageflow, it eats up 10 connections to
the
> >> >>> > database. That's a lot of overhead for one Pageflow to inflict
> >> >>> on a
> >> >>> > DataSource.
> >> >>> >
> >> >>> > Is there a reason not to move getConnection() at the
> >> beginning of
> >> >>> > invoke() and then close it out at the end of invoke()? This
> >> would
> >> >>> > be nice for JDBC controls with ConnectionDataSource
> >> >>> annotations. If
> >> >>> > there is some reason to hold on the connection like the current
> >> >>> > implementation, could we set up an annotation to allow this
> >> >>> > connection-conserving behavior?
> >> >>> >
> >> >>> > - Drew
> >> >>> >
> >> >>> > On Dec 1, 2006, at 10:07 AM, Eddie O'Neil wrote:
> >> >>> >
> >> >>> >> Drew--
> >> >>> >>
> >> >>> >>  Definitely -- this would be a great feature that would
help
> >> >>> with the
> >> >>> >> situations you described.   Today, it's not possible to
manage
> >> >>> the
> >> >>> >> Connection directly which makes it impossible to share
it
> >> >>> between two
> >> >>> >> JdbcControl instances.  It might be possible to just add
a
> >> method
> >> >>> >> like:
> >> >>> >>
> >> >>> >>  public void setConnection(Connection conn)
> >> >>> >>
> >> >>> >> that if called will short circuit the Connection creation
code
> >> >>> in the
> >> >>> >> JdbcControl.
> >> >>> >>
> >> >>> >>  This might be easy to add to the Control -- feel free
to
> >> give it
> >> >>> >> a shot.  :)
> >> >>> >>
> >> >>> >> Eddie
> >> >>> >>
> >> >>> >>
> >> >>> >> On 11/30/06, Drew Varner <drew.varner@redops.org>
wrote:
> >> >>> >>> I was playing around with an idea for the JDBC system
> >> >>> control. The
> >> >>> >>> ability to pass it a Connection object other than
one
> >> >>> declared in
> >> >>> >>> it's annotations.
> >> >>> >>>
> >> >>> >>> Here are a couple of use cases:
> >> >>> >>>
> >> >>> >>> 1) Integration with other persistence mechanisms.
Let's say I
> >> >>> have a
> >> >>> >>> DAO that I can pass a Connection to. I can place the
DAO
> >> and and
> >> >>> >>> JDBC
> >> >>> >>> Control operations in the same transaction.
> >> >>> >>>
> >> >>> >>> Connection conn = JDBCUtils.getConnection();
> >> >>> >>> conn.setAutoCommit(false);
> >> >>> >>> myJdbcControl.setExternalConnection(conn);
> >> >>> >>> myJdbcControl.depositMoney(1000)
> >> >>> >>> AccountAuditDAO.logDepositTransaction(1000,conn);
> >> >>> >>> conn.commit();
> >> >>> >>> conn.close();
> >> >>> >>>
> >> >>> >>> 2) Use it to put multiple JDBC Controls in the same
db
> >> >>> transaction
> >> >>> >>> (same as above, replace the DAO with another JDBC
control)
> >> >>> >>>
> >> >>> >>> 3) Make unit testing easier. You can pass a connection
to
> >> a test
> >> >>> >>> database into the control during testing and don't
have to
> >> >>> mess with
> >> >>> >>> the JNDI setup for out-of-container testing.
> >> >>> >>>
> >> >>> >>> 4) Support for non-standard Connection sources. If
you are
> >> >>> using a
> >> >>> >>> JDBC Control in a Struts app, you can grab the DataSource
via
> >> >>> the
> >> >>> >>> Struts DataSource mechanism.
> >> >>> >>>
> >> >>> >>> Setting an external connection would be manipulated
by
> >> >>> methods, not
> >> >>> >>> annotations.
> >> >>> >>>
> >> >>> >>> public void setExternalConnection(Connection conn);
> >> >>> >>> public Connection getExternalConnection();
> >> >>> >>>
> >> >>> >>> The control implementation would never close the external
> >> >>> >>> connection.
> >> >>> >>>
> >> >>> >>> Would it be worth adding?
> >> >>> >>>
> >> >>> >>> - Drew
> >> >>> >>>
> >> >>> >>>
> >> >>> >>>
> >> >>> >
> >> >>>
> >> >>>
> >> >
> >>
> >>
>
>

Mime
View raw message