manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Derby DB up-to-date?
Date Tue, 05 Jun 2012 10:00:51 GMT
The service interruption warning is just acknowledgement that you
paused or aborted the job.  It means nothing.

The warning about long-running queries means that some queries were
probably blocked for a fairly long time.  This can happen because
ManifoldCF periodically analyzes its tables and/or reindexes them.
It's usually not a problem except if one particular query appears
again and again and seems to have a bad plan, in which case we'd want
to know about it so we can fix it.

Karl

On Tue, Jun 5, 2012 at 5:04 AM, Marcin Goss <marcin.goss@findwise.com> wrote:
> Ok, after applying your patch, it seems that the problem with duplicate values is solved.
Nice!
> However, there are two new warnings that I observed in log:
>
> 1. WARN 2012-06-05 09:58:03,822 (Worker thread '33') - Pre-ingest service interruption
reported for job 1338817906566 connection 'Wiki': Job no longer active
>
> This one is always in series with different numbers of threads. It seems that I'm getting
it after pausing a job. Restarting the job does not help. The only way to get rid of those
warnings is to abort a job and then start it again.
>
> 2. WARN 2012-06-05 10:16:52,967 (Worker thread '5') - Found a long-running query (62959
ms): [UPDATE hopcount SET deathmark=?,distance=? WHERE id IN(SELECT ownerid FROM hopdeletedeps
t0 WHERE t0.jobid=? AND t0.childidhash=? AND EXISTS(SELECT 'x' FROM intrinsiclink t1 WHERE
t1.jobid=t0.jobid AND t1.linktype=t0.linktype AND t1.parentidhash=t0.parentidhash AND t1.childidhash=t0.childidhash
AND t1.isnew=?))]
>  WARN 2012-06-05 10:16:52,968 (Worker thread '5') -   Parameter 0: 'D'
>  WARN 2012-06-05 10:16:52,968 (Worker thread '5') -   Parameter 1: '-1'
>  WARN 2012-06-05 10:16:52,968 (Worker thread '5') -   Parameter 2: '1338817906566'
>  WARN 2012-06-05 10:16:52,968 (Worker thread '5') -   Parameter 3: '9C9DE99F3162AC861D4B92764B9F3CC47F120704'
>  WARN 2012-06-05 10:16:52,968 (Worker thread '5') -   Parameter 4: 'B'
>
> This one also goes in series. Usually > 15 threads reporting the same warning one
after another. I'm not really sure when it happens though...
>
> Marcin
>
> -----Original Message-----
> From: Karl Wright [mailto:daddywri@gmail.com]
> Sent: Monday, June 04, 2012 3:03 PM
> To: user@manifoldcf.apache.org
> Subject: Re: Derby DB up-to-date?
>
> Also, for what it is worth, we're planning to release 0.6 sometime around June 30, provided
all the various tickets are completed as expected.
> Karl
>
> On Mon, Jun 4, 2012 at 8:17 AM, Karl Wright <daddywri@gmail.com> wrote:
>> Generally we don't release binaries except as a point release.  That
>> takes at least a four-day process.  I can, of course, build the
>> binaries here for you but then I'd have no means of transfering them
>> to you.
>>
>> Can you describe the problems you are having building?  I suspect you
>> need to download the -lib package and put that in place first.  The
>> steps are:
>>
>> (1) download the -src package
>> (2) download the -lib package
>> (3) unpack them both and place the contents of the -lib package into
>> the "lib" directory under the root directory of the -src package
>> (where LICENSE.txt is found)
>> (4) type "ant build".
>>
>> There are clear directions in the README.txt file for each of the two
>> packages, if you need further help.
>>
>> Thanks,
>> Karl
>>
>>
>> On Mon, Jun 4, 2012 at 8:10 AM, Marcin Goss <marcin.goss@findwise.com> wrote:
>>> Thanks. Any chance of getting binaries with this fix? I have some problems building
everything with the patch applied.
>>>
>>> -----Original Message-----
>>> From: Karl Wright [mailto:daddywri@gmail.com]
>>> Sent: Monday, June 04, 2012 2:05 PM
>>> To: user@manifoldcf.apache.org
>>> Subject: Re: Derby DB up-to-date?
>>>
>>> I committed the pertinent change (without debugging output) to trunk as well.
 So if you want to upgrade to 0.6-dev, you will get the same fix.
>>>
>>> Karl
>>>
>>>
>>> On Mon, Jun 4, 2012 at 6:48 AM, Karl Wright <daddywri@gmail.com> wrote:
>>>> I've attached a test patch to the CONNECTORS-478 ticket that you can
>>>> download and use to patch 0.5.1 sources.  The test patch *should*
>>>> fix the problem, but if it doesn't, it will certainly log more
>>>> diagnostics, enough so I can see what the change in behavior has
>>>> been for Derby in this release.
>>>>
>>>> FWIW, it looks like Derby is throwing a slightly different kind of
>>>> exception for this case than it did before.
>>>>
>>>> Can you please try it out and let me know what happens?
>>>>
>>>> Thanks!
>>>> Karl
>>>>
>>>>
>>>> On Mon, Jun 4, 2012 at 6:29 AM, Karl Wright <daddywri@gmail.com> wrote:
>>>>> Thanks, this is exactly what I was looking for.
>>>>>
>>>>> I'll put it in the ticket and have a look at this later today.
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>> On Mon, Jun 4, 2012 at 6:20 AM, Marcin Goss <marcin.goss@findwise.com>
wrote:
>>>>>> This is all I've got. I am sorry for the polish fragments, I can't
seem to change it anywhere. Anyway, hopefully you should be able to get all the information
you need.
>>>>>>
>>>>>> ERROR 2012-06-04 11:22:25,449 (Worker thread '42') - Worker thread
aborting and restarting due to database connection reset: Database exception: Exception doing
query: Instrukcja została przerwana, ponieważ mogła spowodować powstanie duplikatu wartości
klucza w regule ograniczającej klucz unikalny lub podstawowy albo w unikalnym indeksie identyfikowanym
przez obiekt 'C1338469165883' zdefiniowany dla 'EVENTS'.
>>>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
exception: Exception doing query: Instrukcja została przerwana, ponieważ mogła spowodować
powstanie duplikatu wartości klucza w regule ograniczającej klucz unikalny lub podstawowy
albo w unikalnym indeksie identyfikowanym przez obiekt 'C1338469165883' zdefiniowany dla 'EVENTS'.
>>>>>>        at
>>>>>> org.apache.manifoldcf.core.database.Database.executeViaThread(Data
>>>>>> ba
>>>>>> se.java:680)
>>>>>>        at
>>>>>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(
>>>>>> Da
>>>>>> tabase.java:726)
>>>>>>        at
>>>>>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.cr
>>>>>> ea
>>>>>> te(Database.java:1393)
>>>>>>        at
>>>>>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAn
>>>>>> dE
>>>>>> xecute(CacheManager.java:144)
>>>>>>        at
>>>>>> org.apache.manifoldcf.core.database.Database.executeQuery(Database
>>>>>> .j
>>>>>> ava:185)
>>>>>>        at
>>>>>> org.apache.manifoldcf.core.database.DBInterfaceDerby.performModifi
>>>>>> ca
>>>>>> tion(DBInterfaceDerby.java:831)
>>>>>>        at
>>>>>> org.apache.manifoldcf.core.database.DBInterfaceDerby.performInsert
>>>>>> (D
>>>>>> BInterfaceDerby.java:222)
>>>>>>        at
>>>>>> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTa
>>>>>> bl
>>>>>> e.java:68)
>>>>>>        at
>>>>>> org.apache.manifoldcf.crawler.jobs.EventManager.createEvent(EventM
>>>>>> an
>>>>>> ager.java:123)
>>>>>>        at
>>>>>> org.apache.manifoldcf.crawler.jobs.JobManager.beginEventSequence(J
>>>>>> ob
>>>>>> Manager.java:4217)
>>>>>>        at
>>>>>> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.
>>>>>> be
>>>>>> ginEventSequence(WorkerThread.java:1236)
>>>>>>        at
>>>>>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConn
>>>>>> ec
>>>>>> tor.lookupIPAddress(WebcrawlerConnector.java:4737)
>>>>>>        at
>>>>>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConn
>>>>>> ec
>>>>>> tor.getDocumentVersions(WebcrawlerConnector.java:634)
>>>>>>        at
>>>>>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread
>>>>>> .j
>>>>>> ava:318) Caused by:
>>>>>> java.sql.SQLIntegrityConstraintViolationException: Instrukcja została
przerwana, ponieważ mogła spowodować powstanie duplikatu wartości klucza w regule ograniczającej
klucz unikalny lub podstawowy albo w unikalnym indeksie identyfikowanym przez obiekt 'C1338469165883'
zdefiniowany dla 'EVENTS'.
>>>>>>        at
>>>>>> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(U
>>>>>> nk
>>>>>> nown Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
>>>>>> Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLExcept
>>>>>> io
>>>>>> n(Unknown Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException
>>>>>> (U
>>>>>> nknown Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
>>>>>> Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
>>>>>> Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
>>>>>> Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement
>>>>>> (U
>>>>>> nknown Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Un
>>>>>> kn
>>>>>> own Source)
>>>>>>        at
>>>>>> org.apache.manifoldcf.core.database.Database.execute(Database.java
>>>>>> :8
>>>>>> 39)
>>>>>>        at
>>>>>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.ru
>>>>>> n(
>>>>>> Database.java:640) Caused by: java.sql.SQLException: Instrukcja
>>>>>> została przerwana, ponieważ mogła spowodować powstanie duplikatu
wartości klucza w regule ograniczającej klucz unikalny lub podstawowy albo w unikalnym indeksie
identyfikowanym przez obiekt 'C1338469165883' zdefiniowany dla 'EVENTS'.
>>>>>>        at
>>>>>> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unk
>>>>>> no
>>>>>> wn Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransp
>>>>>> or
>>>>>> tAcrossDRDA(Unknown Source)
>>>>>>        ... 11 more
>>>>>> Caused by: ERROR 23505: Instrukcja została przerwana, ponieważ
mogła spowodować powstanie duplikatu wartości klucza w regule ograniczającej klucz unikalny
lub podstawowy albo w unikalnym indeksie identyfikowanym przez obiekt 'C1338469165883' zdefiniowany
dla 'EVENTS'.
>>>>>>        at
>>>>>> org.apache.derby.iapi.error.StandardException.newException(Unknown
>>>>>> Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.sql.execute.IndexChanger.insertAndCheckDups(
>>>>>> Un
>>>>>> known Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.sql.execute.IndexChanger.doInsert(Unknown
>>>>>> Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.sql.execute.IndexChanger.insert(Unknown
>>>>>> Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.sql.execute.IndexSetChanger.insert(Unknown
>>>>>> Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(Unknown
>>>>>> Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore
>>>>>> (U
>>>>>> nknown Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown
>>>>>> Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unk
>>>>>> no
>>>>>> wn Source)
>>>>>>        at
>>>>>> org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown
>>>>>> Source)
>>>>>>        ... 5 more
>>>>>>
>>>>>>
>>>>>> /Marcin
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Karl Wright [mailto:daddywri@gmail.com]
>>>>>> Sent: Monday, June 04, 2012 12:17 PM
>>>>>> To: user@manifoldcf.apache.org
>>>>>> Subject: Re: Derby DB up-to-date?
>>>>>>
>>>>>> I've created CONNECTORS-478 to track this issue.  But please do
send along the rest of the stack trace context because without that I cannot know where, exactly,
the constraint violation is not getting caught.
>>>>>>
>>>>>> Thanks,
>>>>>> Karl
>>>>>>
>>>>>> On Mon, Jun 4, 2012 at 6:08 AM, Karl Wright <daddywri@gmail.com>
wrote:
>>>>>>> Can you provide the rest of the exception?  (The "caused by"
>>>>>>> clauses
>>>>>>> etc.?)
>>>>>>>
>>>>>>> The fact is that we discovered that 10.8.2.2 throws deadlock
and
>>>>>>> constraint violation exceptions in different places (as compared
>>>>>>> to
>>>>>>> 10.7.1.1) under multi-threaded conditions.  That was part of
the
>>>>>>> reason why the 0.5.1 release was necessary.  It's of course
>>>>>>> possible that we missed some places that (for instance) the wiki
>>>>>>> connector especially tweaks.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jun 4, 2012 at 5:29 AM, Marcin Goss <marcin.goss@findwise.com>
wrote:
>>>>>>>> Hello Karl,
>>>>>>>>
>>>>>>>> Thank you for answering.
>>>>>>>>  I am using Manifold 0.5.1 and what I am doing is a simple
indexing of a wiki page. The error that I am getting looks like this:
>>>>>>>>
>>>>>>>> Caused by: java.sql.SQLIntegrityConstraintViolationException:
The statement was aborted because it would have caused a duplicate key value in a unique or
primary key constraint or unique index identified by 'XXXXXXXX' defined on 'EVENTS'.
>>>>>>>>        at
>>>>>>>> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException
>>>>>>>> (U
>>>>>>>> nkn
>>>>>>>> own Source)
>>>>>>>>        at
>>>>>>>> org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
>>>>>>>> Source)
>>>>>>>>        at
>>>>>>>> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLExce
>>>>>>>> pt
>>>>>>>> ion
>>>>>>>> (Unknown Source)
>>>>>>>>        at
>>>>>>>> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleExcepti
>>>>>>>> on
>>>>>>>> (Un
>>>>>>>> known Source)
>>>>>>>>        at
>>>>>>>> org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unkno
>>>>>>>> wn
>>>>>>>> Source)
>>>>>>>>        at
>>>>>>>> org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unkno
>>>>>>>> wn
>>>>>>>> Source)
>>>>>>>>        at
>>>>>>>> org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unkno
>>>>>>>> wn
>>>>>>>> Source)
>>>>>>>>
>>>>>>>> /Marcin
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Karl Wright [mailto:daddywri@gmail.com]
>>>>>>>> Sent: Wednesday, May 30, 2012 12:42 PM
>>>>>>>> To: user@manifoldcf.apache.org
>>>>>>>> Subject: Re: Derby DB up-to-date?
>>>>>>>>
>>>>>>>> Also, the ManifoldCF tests passed for these releases, so
I'm wondering exactly what you are seeing.  Can you provide more details, including exceptions?
>>>>>>>>
>>>>>>>> Karl
>>>>>>>>
>>>>>>>> On Wed, May 30, 2012 at 6:38 AM, Karl Wright <daddywri@gmail.com>
wrote:
>>>>>>>>> Which version of ManifoldCF are you using?  The 0.5
release uses 10.8.2.2.
>>>>>>>>>
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>> On Wed, May 30, 2012 at 5:21 AM, Marcin Goss <marcin.goss@findwise.com>
wrote:
>>>>>>>>>> I'm getting an error in a default manifoldCF setup
(jetty
>>>>>>>>>> server and derby
>>>>>>>>>> db) that is probably caused (my supposition) by this
bug in Derby:
>>>>>>>>>> https://issues.apache.org/jira/browse/DERBY-3343.
The bug was
>>>>>>>>>> fixed in January 2011. Which version of Derby db
is coming with ManifoldCF?
>>>>>>>>>> Is it the one with this bug fixed?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>
>>>
>
>

Mime
View raw message