manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Schuch <markus_sch...@web.de>
Subject Re: Manifoldcf server Error
Date Fri, 20 Dec 2019 11:09:28 GMT
Hi Priya,

the container you trying to interactivily executing a command with is no
longer running. It is not possible to execute command with stopped
containers.

The logger issues might be related to missing file system permissions.
But thats a wild guess. Is there a "Caused by" part in the stacktrace of
the AppenderLoggingException?

Cheers,
Markus

Am 20.12.2019 um 11:48 schrieb Priya Arora:
> Hi All,
>
> When i am trying to execute bash command inside manifoldcf container
> getting error. 
> image.png
> And when checking logs Sudo docker logs <CID>
> 2019-12-19 18:09:05,848 Job start thread ERROR Unable to write to stream
> logs/manifoldcf.log for appender MyFile
> 2019-12-19 18:09:05,848 Seeding thread ERROR Unable to write to stream
> logs/manifoldcf.log for appender MyFile
> 2019-12-19 18:09:05,848 Job reset thread ERROR Unable to write to stream
> logs/manifoldcf.log for appender MyFile
> 2019-12-19 18:09:05,848 Job notification thread ERROR Unable to write to
> stream logs/manifoldcf.log for appender MyFile
> 2019-12-19 18:09:05,849 Seeding thread ERROR An exception occurred
> processing Appender MyFile org             
>  .apache.logging.log4j.core.appender.AppenderLoggingException: Error
> flushing stream logs/manifoldcf.log
>         at
> org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:159)
>
> Can any body suggest reason behind this error?
>
> Thanks
> Priya
>
> On Fri, Dec 20, 2019 at 3:37 PM Priya Arora <priya@smartshore.nl
> <mailto:priya@smartshore.nl>> wrote:
>
>     Hi Markus,
>
>     Many thanks for your reply!!.
>
>     I tried this approach to reproduce the scenario in a different
>     environment, but the case  where I listed the error above is when I
>     am crawling INTRANET sites which can be accessible over a remote
>     server. Also I have used Transformation connectors:-Allow Documents,
>     Tika Parser, Content Limiter( 10000000), Metadata Adjuster.
>
>     When tried reproducing the error with Public sites of the same
>     domain and on a different server(DEV), it was successful, with no
>     error.Also there was no any postgres related error.
>
>     Can it depends observer related configurations like Firewall etc, as
>     this case include some firewall,security related configurations.
>
>     Thanks 
>     Priya
>
>
>
>
>     On Fri, Dec 20, 2019 at 3:23 PM Markus Schuch <markus_schuch@web.de
>     <mailto:markus_schuch@web.de>> wrote:
>
>         Hi Priya,
>
>         in my experience, i would focus on the OutOfMemoryError (OOME).
>         8 Gigs can be enough, but they don't have to.
>
>         At first i would check if the jvm is really getting the desired heap
>         size. The dockered environment make that a little harder find
>         find out,
>         since you need to get access to the jvm metrics, e.g. via jmxremote.
>         Beeing able to monitor the jvm metrics helps you with
>         correlating the
>         errors with the heap and garbage collection activity.
>
>         The errors you see on postgresql jdbc driver might be very
>         related to
>         the OOME.
>
>         Some question i would ask myself:
>
>         Do the problems repeatingly occur only when crawling this specific
>         content source or only with this specific output connection? Can you
>         reproduce it outside of docker in a controlled dev environment?
>         Or is it
>         a more general problem with your manifoldcf instance?
>
>         May be there are some huge files beeing crawled in your content
>         source?
>         To you have any kind of transformations configured? (e.g.
>         content size
>         limit?) You should try to see in the job's history if there are any
>         patterns, like the error rises always after encountering the same
>         document xy.
>
>         Cheers
>         Markus
>
>
>
>         Am 20.12.2019 um 09:59 schrieb Priya Arora:
>         > Hi  Markus ,
>         >
>         > Heap size defined is 8GB. Manifoldcf start-options-unix file 
>         Xmx etc
>         > parameters is defined to have memory 8192mb.
>         >
>         > It seems to be an issue with memory also, and also when
>         manifoldcf tries
>         > to communicate to Database. Do you explicitly define somewhere
>         > connection timer when to communicate to postgres.
>         > Postgres is installed as a part of docker image pull and then some
>         > changes in properties.xml(of manifoldcf) to connect to database.
>         > On the other hand Elastic search is also holding sufficient
>         memory and
>         > Manifoldcf is also provided with 8 cores CPU.
>         >
>         > Can you suggest some solution.
>         >
>         > Thanks
>         > Priya
>         >
>         > On Fri, Dec 20, 2019 at 2:23 PM Markus Schuch
>         <markus_schuch@web.de <mailto:markus_schuch@web.de>
>         > <mailto:markus_schuch@web.de <mailto:markus_schuch@web.de>>>
>         wrote:
>         >
>         >     Hi Priya,
>         >
>         >     your manifoldcf JVM suffers from high garbage collection
>         pressure:
>         >
>         >         java.lang.OutOfMemoryError: GC overhead limit exceeded
>         >
>         >     What is your current heap size?
>         >     Without knowing that, i suggest to increase the heap size.
>         (java
>         >     -Xmx...)
>         >
>         >     Cheers,
>         >     Markus
>         >
>         >     Am 20.12.2019 um 09:02 schrieb Priya Arora:
>         >     > Hi All,
>         >     >
>         >     > I am facing below error while accessing Manifoldcf.
>         Requirement is to
>         >     > crawl data from a website using Repository as "Web" and
>         Output
>         >     connector
>         >     > as "Elastic Search"
>         >     > Manifoldcf is configured inside a docker container and also
>         >     postgres is
>         >     > used a docker container.
>         >     > When launching manifold getting below error
>         >     > image.png
>         >     >
>         >     > When checked logs:-
>         >     > *1)sudo docker exec -it 0b872dfafc5c tail -1000
>         >     > /usr/share/manifoldcf/example/logs/manifoldcf.log*
>         >     > FATAL 2019-12-20T06:06:13,176 (Stuffer thread) - Error
>         tossed: Timer
>         >     > already cancelled.
>         >     > java.lang.IllegalStateException: Timer already cancelled.
>         >     >         at java.util.Timer.sched(Timer.java:397)
>         ~[?:1.8.0_232]
>         >     >         at java.util.Timer.schedule(Timer.java:193)
>         ~[?:1.8.0_232]
>         >     >         at
>         >     >
>         org.postgresql.jdbc.PgConnection.addTimerTask(PgConnection.java:1113)
>         >     > ~[postgresql-42.1.3.jar:42.1.3]
>         >     >         at
>         >     >
>         org.postgresql.jdbc.PgStatement.startTimer(PgStatement.java:887)
>         >     > ~[postgresql-42.1.3.jar:42.1.3]
>         >     >         at
>         >     >
>         org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:427)
>         >     > ~[postgresql-42.1.3.jar:42.1.3]
>         >     >         at
>         >     org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354)
>         >     > ~[postgresql-42.1.3.jar:42.1.3]
>         >     >         at
>         >     >
>         >   
>          org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:169)
>         >     > ~[postgresql-42.1.3.jar:42.1.3]
>         >     >         at
>         >     >
>         >   
>          org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:136)
>         >     > ~[postgresql-42.1.3.jar:42.1.3]
>         >     >         at
>         >     >
>         org.postgresql.jdbc.PgConnection.isValid(PgConnection.java:1311)
>         >     > ~[postgresql-42.1.3.jar:42.1.3]
>         >     >         at
>         >     >
>         >   
>          org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:92)
>         >     > ~[mcf-core.jar:?]
>         >     >         at
>         >     >
>         >   
>          org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:126)
>         >     > ~[mcf-core.jar:?]
>         >     >         at
>         >     >
>         >   
>          org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:75)
>         >     > ~[mcf-core.jar:?]
>         >     >         at
>         >     >
>         >   
>          org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:797)
>         >     > ~[mcf-core.jar:?]
>         >     >         at
>         >     >
>         >   
>          org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1457)
>         >     > ~[mcf-core.jar:?]
>         >     >         at
>         >     >
>         >   
>          org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146)
>         >     > ~[mcf-core.jar:?]
>         >     >         at
>         >     >
>         >   
>          org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
>         >     > ~[mcf-core.jar:?]
>         >     >         at
>         >     >
>         >   
>          org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:837)
>         >     > ~[mcf-core.jar:?]
>         >     >         at
>         >     >
>         >   
>          org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
>         >     > ~[mcf-core.jar:?]
>         >     >         at
>         >     >
>         >   
>          org.apache.manifoldcf.crawler.jobs.Jobs.getActiveJobConnections(Jobs.java:736)
>         >     > ~[mcf-pull-agent.jar:?]
>         >     >         at
>         >     >
>         >   
>          org.apache.manifoldcf.crawler.jobs.JobManager.getNextDocuments(JobManager.java:2869)
>         >     > ~[mcf-pull-agent.jar:?]
>         >     >         at
>         >     >
>         >   
>          org.apache.manifoldcf.crawler.system.StufferThread.run(StufferThread.java:186)
>         >     > [mcf-pull-agent.jar:?]
>         >     > *2)sudo docker logs <CID> --tail 1000*
>         >     > Exception in thread "PostgreSQL-JDBC-SharedTimer-1"
>         >     > java.lang.OutOfMemoryError: GC overhead limit exceeded
>         >     >         at java.util.ArrayList.iterator(ArrayList.java:840)
>         >     >         at
>         >     >
>         >   
>          java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1316)
>         >     >         at
>         java.net.InetAddress.getAllByName0(InetAddress.java:1277)
>         >     >         at
>         java.net.InetAddress.getAllByName(InetAddress.java:1193)
>         >     >         at
>         java.net.InetAddress.getAllByName(InetAddress.java:1127)
>         >     >         at
>         java.net.InetAddress.getByName(InetAddress.java:1077)
>         >     >         at
>         >     java.net.InetSocketAddress.<init>(InetSocketAddress.java:220)
>         >     >         at
>         org.postgresql.core.PGStream.<init>(PGStream.java:66)
>         >     >         at
>         >     >
>         >   
>          org.postgresql.core.QueryExecutorBase.sendQueryCancel(QueryExecutorBase.java:155)
>         >     >         at
>         >     >
>         org.postgresql.jdbc.PgConnection.cancelQuery(PgConnection.java:971)
>         >     >         at
>         >     org.postgresql.jdbc.PgStatement.cancel(PgStatement.java:812)
>         >     >         at
>         org.postgresql.jdbc.PgStatement$1.run(PgStatement.java:880)
>         >     >         at java.util.TimerThread.mainLoop(Timer.java:555)
>         >     >         at java.util.TimerThread.run(Timer.java:505)
>         >     > 2019-12-19 18:09:05,848 Job start thread ERROR Unable to
>         write to
>         >     stream
>         >     > logs/manifoldcf.log for appender MyFile
>         >     > 2019-12-19 18:09:05,848 Seeding thread ERROR Unable to
>         write to stream
>         >     > logs/manifoldcf.log for appender MyFile
>         >     > 2019-12-19 18:09:05,848 Job reset thread ERROR Unable to
>         write to
>         >     stream
>         >     > logs/manifoldcf.log for appender MyFile
>         >     > 2019-12-19 18:09:05,848 Job notification thread ERROR
>         Unable to
>         >     write to
>         >     > stream logs/manifoldcf.log for appender MyFile
>         >     > 2019-12-19 18:09:05,849 Seeding thread ERROR An
>         exception occurred
>         >     > processing Appender MyFile
>         >     >
>         org.apache.logging.log4j.core.appender.AppenderLoggingException:
>         Error
>         >     > flushing stream logs/manifoldcf.log
>         >     >         at
>         >     >
>         >   
>          org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:159).
>         >     >
>         >     > _Also tried the approach to clean up Database by
>         truncating all
>         >     > manifoldcf related tables, but still getting this error._
>         >     >
>         >     > Parameters defined in *postgresql conf *file is as
>         suggested :- and
>         >     > "max_pred_per_locks_transctions" is set to value "256".
>         >     > image.png
>         >
>

Mime
View raw message