manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: "Connection already in use" when crawling Windows Share repositories
Date Tue, 22 Feb 2011 11:09:01 GMT
I've seen this before, when I was using HTTP and setting a connection
up and tearing it down for each communication.  This is why HTTP
keep-alive was invented in fact.

TIME_WAIT is the result of an oddity in the way sockets are managed.
A socket that has been closed but has not received an ACK from the
other end is in this state.  Read here:
http://stackoverflow.com/questions/813790/too-many-time-wait-connections

If you can figure out what socket is being left dangling, then the
right thing to do is create the socket using NO_WAIT, provided you
have access to that code.  It's one of the socket parameters that you
can use only at socket create time.  It tells the socket to not stick
around after close.

If this is in the ManifoldCF code, or in JCifs, I'd love to know about it! ;-)

Thanks,
Karl


On Tue, Feb 22, 2011 at 4:40 AM,  <c.a.r.e@gmx.de> wrote:
> Hi Karl,
>
> your guess was right. There has been a problem with too many connections in status "TIME_WAIT".
> Even if I set max. connections per JVM to 1 for the Repository and Authority Connection
I hat up to 3500 connections waiting.
> When investingating this further I found out that an unbind response from the LDAP server
was missing.
>
> Sometimes this error occurs and sometimes it works fine. It's a bit weird, I wasn't able
to figure out where it came from.
>
> Thanks for your hint.
>
> Carina
>
>
> -------- Original-Nachricht --------
>> Datum: Fri, 18 Feb 2011 06:50:36 -0500
>> Von: Karl Wright <daddywri@gmail.com>
>> An: connectors-user@incubator.apache.org
>> Betreff: Re: "Connection already in use" when crawling Windows Share repositories
>
>> I don't think this is directly a ManifoldCF issue.  My guess is that
>> you are running out of file handles.  If you are on Windows, you might
>> want to track the number of handles you are using via netstat.  If on
>> linux, then lsof is the right tool.
>>
>> If you decide that this is indeed what the issue is, then on linux you
>> need to either rebuild the kernel, or do stuff with ulimit to set the
>> number of handles.  On Windows, I don't know how to do it.  Limiting
>> the number of handles used by ManifoldCF is done by setting the number
>> of connections under the "Throttling" tab of the various connection
>> definitions you create, but that's not going to help with Tomcat.
>>
>> It may be worth checking if you aren't closing something.  That could
>> easily cause this problem, eventually.  Since it is happening at
>> crawling time, the Active Directory Authority is not actually
>> communicating with the LDAP server yet.  JCIFS probably is, though.
>>
>> Let us know what you find.
>>
>> Thanks,
>> Karl
>>
>>
>> On Fri, Feb 18, 2011 at 6:31 AM,  <c.a.r.e@gmx.de> wrote:
>> > I'm running MCF rev. 1071535 with Apache Tomcat 7. Solr and MCF are
>> using HTTPS. When crawling data from a Windows Share Connection with Active
>> Directory Authority after some time of crawling my job aborts with the status
>> "Error: Couldn't set up SSL connection to ingestion API: Address already in
>> use: connect".
>> >
>> > In the mcf logfile I get:
>> >
>> > [2011-02-18 11:24:33,695]ERROR Exception tossed: Couldn't set up SSL
>> connection to ingestion API: Address already in use: connect
>> > org.apache.manifoldcf.core.interfaces.ManifoldCFException: Couldn't set
>> up SSL connection to ingestion API: Address already in use: connect
>> >        at
>> org.apache.manifoldcf.agents.output.solr.HttpPoster.createSocket(HttpPoster.java:736)
>> >        at
>> org.apache.manifoldcf.agents.output.solr.HttpPoster$IngestThread.run(HttpPoster.java:1006)
>> > Caused by: java.net.BindException: Address already in use: connect
>> >        at java.net.PlainSocketImpl.socketConnect(Native Method)
>> >        at
>> java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>> >        at
>> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
>> >        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>> >        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>> >        at java.net.Socket.connect(Socket.java:529)
>> >        at
>> com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:550)
>> >        at
>> com.sun.net.ssl.internal.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:353)
>> >        at
>> com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:71)
>> >        at
>> org.apache.manifoldcf.agents.output.solr.HttpPoster.createSocket(HttpPoster.java:728)
>> >        ... 1 more
>> >
>> > I use the same LDAP user
>> > - to authenticate at MCF
>> > - login credentials for the Solr Connection
>> > - credentials for authority connection
>> > - and for connecting to our LDAP server itself (in tomcat JNDI Realm
>> config).
>> >
>> > It seems to me that the connection to LDAP might be a problem. I tried
>> to configure connection pooling for tomcat. (I'm afraid I'm not so familiar
>> with that.)
>> > "-Dcom.sun.jndi.ldap.connect.pool=true
>> -Dcom.jndi.ldap.connect.pool.protocol=ssl"
>> >
>> > Did I miss sth in the config? Or how do I have to configure MCF?
>> >
>> > Thanks in advance.
>> >
>> > Carina
>> > --
>> > GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit
>> > gratis Handy-Flat! http://portal.gmx.net/de/go/dsl
>> >
>
> --
> GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit
> gratis Handy-Flat! http://portal.gmx.net/de/go/dsl
>

Mime
View raw message