manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Problem crawling windows share
Date Fri, 09 Dec 2011 18:16:53 GMT
Michael Allen's response to your capture:

"It looks like the non-JCIFS communication is successful because the
anonymous credential is used.

Note that there is still no sign of DFS going on anywhere. Actually
JCIFS tries to do a referral for the DFS root and gets an error of
STATUS_FS_DRIVER_REQUIRED so it looks like DFS may not even be setup
at all in the target environment. At least not domain based DFS.

So if you want to emulate Windows behavior, I guess you need to retry
the operation with credential NtlmPasswordAuthentication.ANONYMOUS (or
all empty strings should work too I think like ";:")."

In other words, the issue is that the server in question does not
actually recognize the credentials given at all, and gives an error.
But while Windows then retries with no credentials whatsoever, nothing
in JCIFS does that retry.

You can confirm that this is what is happening by using an empty user
name, password, and domain in the JCIFS test code you wrote to obtain
the captures.  If, then, the test code works against that server, I'll
create a ticket for downgrading to ANONYMOUS whenever there is an
authentication error in the JCIFS connector.

Thanks,
Karl

On Wed, Dec 7, 2011 at 6:41 AM, Karl Wright <daddywri@gmail.com> wrote:
> The next step is to obtain two packet captures using WireShark.  These
> should both be done from the machine that is not joined to the domain.
>  The first capture should be the interaction that takes place when you
> try to go to the foreign directory, and you supply the credentials,
> and then you list the folders.  The second capture will require you to
> do the same using JCIFS.  For that capture, you will need jcifs.jar
> and a simple example class that provides jcifs with credentials and
> tries to list the directory.  A simple class should be enough to do
> this, including code such as:
>
> import jcifs.smb.*;
>
> NtlmPasswordAuthentication pa = new
> NtlmPasswordAuthentication(domain,username,password);
> SmbFile smbconnection = new SmbFile("smb://" + server + "/" + share,pa);
> SmbFile[] files = smbconnection.listFiles();
>
> for (int i = 0 ; i < files.length ; i++)
> {
>        System.out.println(file.toString());
> }
>
> I suggest you put together your test class first and make sure it
> works, and then go after the captures.
>
> Let me know how it goes.
>
> Karl
>
> On Wed, Dec 7, 2011 at 6:10 AM, Swapna Vuppala
> <swapna.kollipara@gmail.com> wrote:
>> Hi Karl,
>>
>> Thanks for the detailed reply.
>>
>> As per your suggestion, I tried connecting to the problematic server via
>> windows using a machine that is not in the domain. On supplying the
>> credentials via windows, I could connect to the server, although it did not
>> list any folders. But when I tried accessing the folder of my interest via
>> \\server\folder$, I could see all the sub folders. All this is when I
>> connected via windows from a machine that's not joined to the domain.
>>
>> The behavior is same when I connect to the server via windows from a machine
>> that's joined to the domain.
>>
>> So I assume this means that through both Kerberos and NTLM, I see the same
>> behavior. Whereas ManifoldCF throws an exception when trying to define the
>> repository connection.
>>
>> Please advise me on what I should be doing next to resolve this.
>>
>> Thanks and Regards,
>> Swapna.
>>
>>
>>
>>
>> On Tue, Dec 6, 2011 at 12:55 PM, Karl Wright <daddywri@gmail.com> wrote:
>>>
>>> About your capture - Michael Allen says the following:
>>>
>>> "Actually this has nothing to do with DFS. JCIFS does not get to the
>>> point where it does DFS anything. The capture shows a vanilla
>>> STATUS_LOGON_FAILURE when GLOBAL\swapna.vuppala tries to auth with
>>> l-carx01.global.arup.com. So the possible causes for this are 1) the
>>> account name is not valid 2) the supplied password is incorrect 3)
>>> some security policy is deliberately blocking that user or particular
>>> type of auth or 4) some server configuration is incompatible with
>>> JCIFS. I only mention this last option because I noticed the target
>>> server has security signatures disabled. That's strange. If they're
>>> messing around with things like that, who knows what their clients are
>>> expected to do.
>>>
>>> Try a Windows client that uses NTLM instead of Kerberos. Meaning try a
>>> machine that is not joined to the domain so that when you try to
>>> access the target it asks you for credentials at which point you can
>>> test with GLOBAL\swapna.vuppala. Then it will use NTLM and you can
>>> actually compare captures. If the operator doesn't have a laptop or
>>> something not joined to the domain, it might be sufficient to log into
>>> a workstation using machine credentials and not domain credentials.
>>>
>>> Also when testing JCIFS you should use a simple stand-alone program
>>> like examples/ListFiles.java."
>>>
>>> In other words:
>>> (a) Since JCIFS does not use Kerberos for authentication, you need to
>>> try to log into the recalcitrant server via Windows without using
>>> Kerberos to be able to do a side-by-side comparison.  Michael has some
>>> ways of doing that, above.
>>> (b) You may find that it doesn't work, in which case JCIFS is not
>>> going to work either.
>>> (c) If it *does* work, then try to generate your side-by-side
>>> comparisons using a simpler example rather than ManifoldCF en toto;
>>> you can see how at jcifs.samba.org, or I can help you further.
>>>
>>> He also mentions that there is some bizarreness on the response that
>>> indicates that the server is configured in a way that he's never seen
>>> before.  And believe me, Michael has seen a *lot* of strange
>>> configurations...
>>>
>>> Hope this helps.
>>> Karl
>>>
>>> On Mon, Nov 28, 2011 at 4:12 AM, Karl Wright <daddywri@gmail.com> wrote:
>>> > That should read "properties.xml", not "properties.ini".  It looks
>>> > like this page needs updating.
>>> >
>>> > The debug property in the XML form is:
>>> >
>>> > <property name="org.apache.manifoldcf.connectors" value="DEBUG"/>
>>> >
>>> > I don't think it will provide you with any additional information that
>>> > is useful for debugging your authentication issue, however, if that is
>>> > why you are looking at it.  There may be some jcifs.jar debugging
>>> > switches that might be of more help, but in the end I suspect you will
>>> > need a packet capture of both a successful connection (via Windows)
>>> > and an unsuccessful one (via MCF).  The guy you will need to talk with
>>> > after that is the jcifs author Michael Allen; I can give you his email
>>> > address if you get that far.
>>> >
>>> > Karl
>>> >
>>> >
>>> > On Mon, Nov 28, 2011 at 1:30 AM, Swapna Vuppala
>>> > <swapna.kollipara@gmail.com> wrote:
>>> >> Hi Karl,
>>> >>
>>> >> I was planning to debug jCIFS repository connection using WireShark
and
>>> >> I
>>> >> came across this
>>> >> https://cwiki.apache.org/CONNECTORS/debugging-connections.html
>>> >> Here, I see something as add "org.apache.manifoldcf.connectors=DEBUG"
>>> >> to the
>>> >> properties.ini file. Is it the properties.xml file that is being
>>> >> referred
>>> >> here ? If not, where do I find properties.ini file ?
>>> >>
>>> >> Thanks and Regards,
>>> >> Swapna.
>>> >>
>>> >> On Thu, Nov 17, 2011 at 1:31 PM, Karl Wright <daddywri@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> See http://jcifs.samba.org/src/docs/api/overview-summary.html#scp.
>>> >>> The properties jcifs.smb.lmCompatibility and
>>> >>> jcifs.smb.client.useExtendedSecurity are the ones you may want to
>>> >>> change.  These two properties go together so certain combinations
make
>>> >>> sense and others don't, so there's really only combinations you
need
>>> >>> but I'll need to look at what they are and get back to you later
>>> >>> today.
>>> >>>
>>> >>> As far as setting the switches are concerned, if you are using the
>>> >>> Quick Start you do this trivially by:
>>> >>>
>>> >>> <java> -Dxxx -Dyyy -jar start.jar
>>> >>>
>>> >>> If you are using the multi-process configuration, that is what the
>>> >>> "defines" directory is for; you only need to create files in that
>>> >>> directory with the names "jcifs.smb.lmCompatibility" and
>>> >>> "jcifs.smb.client.useExtendedSecurity" containing the values you
want
>>> >>> to set.
>>> >>>
>>> >>> Karl
>>> >>>
>>> >>>
>>> >>> On Thu, Nov 17, 2011 at 1:11 AM, Swapna Vuppala
>>> >>> <swapna.kollipara@gmail.com> wrote:
>>> >>> > Hi Karl,
>>> >>> >
>>> >>> > Am able to access the folders on the problem server through
windows
>>> >>> > explorer, (\\server3\Folder1). I tried couple of things with
the
>>> >>> > credentials
>>> >>> > form, changing username, domain etc.. but I keep getting the
same
>>> >>> > error
>>> >>> > "Couldn't connect to server: Logon failure: unknown user name
or bad
>>> >>> > password"
>>> >>> >
>>> >>> > Can you tell me more about the -D switch you were talking of
?
>>> >>> >
>>> >>> > Thanks and Regards,
>>> >>> > Swapna.
>>> >>> >
>>> >>> > On Tue, Nov 15, 2011 at 12:40 PM, Karl Wright <daddywri@gmail.com>
>>> >>> > wrote:
>>> >>> >>
>>> >>> >> Glad you chased it down this far.
>>> >>> >>
>>> >>> >> First thing to try is whether you can get into the problem
server
>>> >>> >> using Windows Explorer.  Obviously ManifoldCF is not going
to be
>>> >>> >> able
>>> >>> >> to do it if Windows can't.  If you *can* get in, then
just playing
>>> >>> >> with the form of the credentials in the MCF connection
might do the
>>> >>> >> trick.  Some Windows or net appliance servers are picky
about this.
>>> >>> >> Try various things like leaving the domain blank and specifying
the
>>> >>> >> user as "abc@domain.com", for instance. There's also a
different
>>> >>> >> NTLM
>>> >>> >> mode you can operation jcifs in that some servers may be
configured
>>> >>> >> to
>>> >>> >> require; this would need you to set a -D switch on the
command line
>>> >>> >> to
>>> >>> >> enable.
>>> >>> >>
>>> >>> >> Karl
>>> >>> >>
>>> >>> >> On Tue, Nov 15, 2011 at 12:10 AM, Swapna Vuppala
>>> >>> >> <swapna.kollipara@gmail.com> wrote:
>>> >>> >> > Hi Karl,
>>> >>> >> >
>>> >>> >> > Thanks for the input. It looks like my problem is
related to the
>>> >>> >> > second
>>> >>> >> > one
>>> >>> >> > that you specified. One of the directories in the
path am trying
>>> >>> >> > to
>>> >>> >> > index is
>>> >>> >> > actually redirecting to a different server. And when
I specify
>>> >>> >> > this
>>> >>> >> > new
>>> >>> >> > server in defining the repository connection, with
my
>>> >>> >> > credentials,
>>> >>> >> > the
>>> >>> >> > connection fails with the message:  "Couldn't connect
to server:
>>> >>> >> > Logon
>>> >>> >> > failure: unknown user name or bad password"
>>> >>> >> >
>>> >>> >> > I'll look into why am not able to connect to this
server.
>>> >>> >> >
>>> >>> >> > Thanks and Regards,
>>> >>> >> > Swapna.
>>> >>> >> >
>>> >>> >> > On Mon, Nov 14, 2011 at 4:56 PM, Karl Wright <daddywri@gmail.com>
>>> >>> >> > wrote:
>>> >>> >> >>
>>> >>> >> >> There's two kinds of problem you might be having.
 The first is
>>> >>> >> >> intermittent, and the second is not intermittent
but would have
>>> >>> >> >> something to do with specific directories.
>>> >>> >> >>
>>> >>> >> >> Intermittent problems might include a domain controller
that is
>>> >>> >> >> not
>>> >>> >> >> always accessible.  In such cases, the crawl
will proceed but
>>> >>> >> >> will
>>> >>> >> >> tend to fail unpredictably.  On the other hand,
if you have a
>>> >>> >> >> directory that is handled by a DFS redirection,
it is possible
>>> >>> >> >> that
>>> >>> >> >> the redirection is indicating a new server (lets
call it
>>> >>> >> >> server3)
>>> >>> >> >> which may not like the precise form of your login
credentials.
>>> >>> >> >>  Can
>>> >>> >> >> you determine which scenario you are seeing?
>>> >>> >> >>
>>> >>> >> >> Karl
>>> >>> >> >>
>>> >>> >> >> On Mon, Nov 14, 2011 at 3:11 AM, Swapna Vuppala
>>> >>> >> >> <swapna.kollipara@gmail.com> wrote:
>>> >>> >> >> > Hi,
>>> >>> >> >> >
>>> >>> >> >> > I have been using windows share repository
connection to crawl
>>> >>> >> >> > and
>>> >>> >> >> > get
>>> >>> >> >> > data
>>> >>> >> >> > from a particular server (server 1). Its
working perfectly
>>> >>> >> >> > fine.
>>> >>> >> >> > However, am
>>> >>> >> >> > having trouble when I try with data from
another server
>>> >>> >> >> > (server
>>> >>> >> >> > 2).
>>> >>> >> >> >
>>> >>> >> >> > When I define a repository connection of
type windows share
>>> >>> >> >> > and
>>> >>> >> >> > specify
>>> >>> >> >> > the
>>> >>> >> >> > server name (server 2) with my credentials,
the connection
>>> >>> >> >> > status
>>> >>> >> >> > shows
>>> >>> >> >> > "Connection working". But when I run a job
to use this
>>> >>> >> >> > repository
>>> >>> >> >> > connection
>>> >>> >> >> > and index data from a location on this server
2, I keep
>>> >>> >> >> > getting
>>> >>> >> >> > the
>>> >>> >> >> > exception below:
>>> >>> >> >> >
>>> >>> >> >> > JCIFS: Possibly transient exception detected
on attempt 3
>>> >>> >> >> > while
>>> >>> >> >> > checking
>>> >>> >> >> > if
>>> >>> >> >> > file exists: Logon failure: unknown user
name or bad password.
>>> >>> >> >> > jcifs.smb.SmbAuthException: Logon failure:
unknown user name
>>> >>> >> >> > or
>>> >>> >> >> > bad
>>> >>> >> >> > password.
>>> >>> >> >> >     at
>>> >>> >> >> > jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:544)
>>> >>> >> >> >     at jcifs.smb.SmbTransport.send(SmbTransport.java:661)
>>> >>> >> >> >     at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
>>> >>> >> >> >     at jcifs.smb.SmbSession.send(SmbSession.java:218)
>>> >>> >> >> >     at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>>> >>> >> >> >     at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>>> >>> >> >> >     at jcifs.smb.SmbFile.connect(SmbFile.java:954)
>>> >>> >> >> >     at jcifs.smb.SmbFile.connect0(SmbFile.java:880)
>>> >>> >> >> >     at jcifs.smb.SmbFile.queryPath(SmbFile.java:1335)
>>> >>> >> >> >     at jcifs.smb.SmbFile.exists(SmbFile.java:1417)
>>> >>> >> >> >     at
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2064)
>>> >>> >> >> >     at
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getDocumentVersions(SharedDriveConnector.java:521)
>>> >>> >> >> >     at
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> >
>>> >>> >> >> > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:318)
>>> >>> >> >> >
>>> >>> >> >> > Am able to access this location from windows
explorer. What
>>> >>> >> >> > else
>>> >>> >> >> > should
>>> >>> >> >> > I be
>>> >>> >> >> > checking or what could be the reasons/factors
causing this to
>>> >>> >> >> > fail
>>> >>> >> >> > ?
>>> >>> >> >> >
>>> >>> >> >> > Thanks and Regards,
>>> >>> >> >> > Swapna.
>>> >>> >> >> >
>>> >>> >> >
>>> >>> >> >
>>> >>> >
>>> >>> >
>>> >>
>>> >>
>>
>>

Mime
View raw message