Hi Ameya,

Have a look at this one line:

INFO  - 2014-07-09 15:30:56.243; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/solr path=/update/extract params={literal.deny_token_document=DEAD_AUTHORITY&literal.DocIcon=docx&resource.name=Anarchism-201310091123505625.docx&literal.allow_token_document=S-1-5-21-1482846375-2276666860-3536682573-500&literal.allow_token_document=S-1-5-21-1482846375-2276666860-3536682573-68651&literal.FolderChildCount=0&version=2.2&literal.ItemChildCount=0&literal.GUID=Ameya&literal.ParentVersionString=&literal._CopySource=&literal.cat=&literal.FileSizeDisplay=1264155&literal._CheckinComment=&literal.Edit=0&literal.id=http://sharepointten:10800/sites/siteecho/Shared%2520Documents/Anarchism-201310091123505625.docx&literal.LinkFilenameNoMenu=Anarchism-201310091123505625.docx&literal.Created=2014-06-03+11:21:53&literal._UIVersionString=1.0&wt=xml&literal.Title=Anarchism&literal.Modified=2014-06-03+11:21:53&literal.Author=Sharepoint+Backup&literal.FileLeafRef=Anarchism-201310091123505625.docx&literal.LinkFilename=Anarchism-201310091123505625.docx&literal.lcf_metadata_id=81&literal.Editor=Administrator&literal.ParentLeafName=&literal.CheckoutUser=&literal.ContentType=Document} {add=[http://sharepointten:10800/sites/siteecho/Shared%20Documents/Anarchism-201310091123505625.docx (1473180342671114240)]} 0 66


Note that literal.Created and literal.Modified are both reasonable dates.

So MCF is not supplying the last_modified field; probably that is being extracted by Solr Cell.

Karl



On Wed, Jul 9, 2014 at 3:33 PM, Ameya Aware <ameya.aware@gmail.com> wrote:
Hi Karl,

Please find attached solr log file.

Please let me know if you need anything else.

Thanks,
Ameya




On Wed, Jul 9, 2014 at 3:07 PM, Karl Wright <daddywri@gmail.com> wrote:
Hi Ameya,

I need to see the solr log output.

Thanks,
Karl



On Wed, Jul 9, 2014 at 3:04 PM, Ameya Aware <ameya.aware@gmail.com> wrote:
Please see if you needed below information.

Inline image 1

Or please let me know where would i get information from.

Thanks,
Ameya


On Wed, Jul 9, 2014 at 2:51 PM, Karl Wright <daddywri@gmail.com> wrote:
Hi Ameya,

Can you include the Solr [INFO] log entry for one of these indexing actions?  I want to see if last_modified is set incorrectly there.  If you are running these documents through Solr Cell, it may well be Tika that is providing the last_modified date, not ManifoldCF.

Karl



On Wed, Jul 9, 2014 at 2:37 PM, Ameya Aware <ameya.aware@gmail.com> wrote:
Hi Karl,

Please find below screenshot.

Inline image 1

looks like indexing date comes out to be good.

Also, just for your reference pasting my Solr screenshot as well.

Inline image 2


Thanks,
Ameya



On Wed, Jul 9, 2014 at 2:20 PM, Karl Wright <daddywri@gmail.com> wrote:
On second look, it all looks fine.

The last thing to check is to look at what is getting set as data.  Around line 1996 in SharePointRepository, there is this code:


>>>>>>
                if (modifiedDate != null)
                  data.setModifiedDate(modifiedDate);
                if (createdDate != null)
                  data.setCreatedDate(createdDate);
<<<<<<

Can you add this line:

>>>>>>
               System.out.println("Indexing modified date: "+modifiedDate);
<<<<<<

... and recrawl?

If that works, we'll have to start looking at Solr.

Thanks,
Karl



On Wed, Jul 9, 2014 at 2:13 PM, Karl Wright <daddywri@gmail.com> wrote:
Actually, looking at your screen shot, it is harder to see for sure since there are multiple threads active.  So it may well be that there is no issue with the parsing.  Let me see if I can confirm that.

Karl



On Wed, Jul 9, 2014 at 2:05 PM, Karl Wright <daddywri@gmail.com> wrote:


On Wed, Jul 9, 2014 at 2:04 PM, Ameya Aware <ameya.aware@gmail.com> wrote:
how do i open the ticket?


Thanks,
Ameya


On Wed, Jul 9, 2014 at 2:03 PM, Karl Wright <daddywri@gmail.com> wrote:
bq. How is that coming good?

No idea.  It may be a bug in the SimpleDateFormat class pertaining to only specific dates.

Karl



On Wed, Jul 9, 2014 at 2:01 PM, Ameya Aware <ameya.aware@gmail.com> wrote:
Ok.

But then same thing should happen in created date also , isnt it?

How is that coming good?

Thanks,
Ameya


On Wed, Jul 9, 2014 at 1:56 PM, Karl Wright <daddywri@gmail.com> wrote:
This shows clearly that the parsing is not doing the expected thing.  It's not clear why, since it's a pretty straight usage of SimpleDateFormat, but that is what is going wrong.

Please open a ticket for us to look at this.

Thanks,
Karl



On Wed, Jul 9, 2014 at 1:53 PM, Ameya Aware <ameya.aware@gmail.com> wrote:
Hi Karl,

Please find screenshot below to show modified date values as date object and string as well.

Inline image 1

Thanks,
Ameya


On Wed, Jul 9, 2014 at 12:32 PM, Karl Wright <daddywri@gmail.com> wrote:
Hi Ameya,

The ability to parse Microsoft's special 8601 dates is in fact already there.  So what might be happening is a timezone issue, since the timezone is not being explicitly set during parsing.  Printing the value of modifiedDateValue will show us if that is indeed the problem.

Karl



On Wed, Jul 9, 2014 at 12:19 PM, Karl Wright <daddywri@gmail.com> wrote:
Hi Ameya,

Try printing "modifiedDateValue", in addition to printing "modifiedDate".  The parsed form is a date object, not a string.

Karl



On Wed, Jul 9, 2014 at 11:20 AM, Ameya Aware <ameya.aware@gmail.com> wrote:
Ok.

But created date for all files is coming good.
Also,


>>>>>>>>
if (modifyDate != null)
                {
                  // Item has a modified date, so we presume it exists
                  Date modifiedDateValue = DateParser.parseISO8601Date(modifiedDate);
                  Date createdDateValue = DateParser.parseISO8601Date(createdDate);
                  
                  System.out.println("Modified date string is: '"+modifiedDate+"'");
                  System.out.println("Modify Date:" + modifyDate);


<<<<<<<<

Above code prints out correct date even after parsing.
So is the issue coming after this step??

I am using Sharepoint 2010.


Thanks,
Ameya


On Wed, Jul 9, 2014 at 11:10 AM, Karl Wright <daddywri@gmail.com> wrote:
Ok, well SharePoint claims these dates are ISO8601 dates, but they are clearly not in this case.  Here are the tests for 8601 dates in the MCF core code:

>>>>>>
    Date d = DateParser.parseISO8601Date("96-11-15T01:32:33.344GMT");
    assertNotNull(d);
    d = DateParser.parseISO8601Date("2012-11-15T01:32:33.344Z");
    assertNotNull(d);
    d = DateParser.parseISO8601Date("2012-11-15T01:32:33Z");
    assertNotNull(d);
    d = DateParser.parseISO8601Date("2012-11-15T01:32:33+0100");
    assertNotNull(d);
    d = DateParser.parseISO8601Date("2012-11-15T01:32:33-03:00");
    assertNotNull(d);
    d = DateParser.parseISO8601Date("2012-11-15T01:32:33GMT-03:00");
    assertNotNull(d);
    d = DateParser.parseISO8601Date("2012-11-15T01:32:33.001-04:00");
    assertNotNull(d);
<<<<<<

You will note that there is supposed to be a "T" and a timezone in an ISO-8601 date.

What version of SharePoint are you using, and what is the locale settings for the server that your SharePoint is running on?

Karl



On Wed, Jul 9, 2014 at 11:06 AM, Ameya Aware <ameya.aware@gmail.com> wrote:
Hi

Please find below screenshot for dates.

Modify Date i added on my own.

Inline image 1


Thanks,
Ameya


On Wed, Jul 9, 2014 at 11:03 AM, Karl Wright <daddywri@gmail.com> wrote:
Hi Ameya,

The SharePoint connector parses the date.  Can you send me some EXAMPLES of the dates coming back so that I can be sure they will parse correctly?

Thanks,
Karl



On Wed, Jul 9, 2014 at 10:59 AM, Ameya Aware <ameya.aware@gmail.com> wrote:
Hi Karl,

I ran the job and at this point values coming for modified date are correct.


Ameya


On Wed, Jul 9, 2014 at 10:25 AM, Karl Wright <daddywri@gmail.com> wrote:
Hi Ameya,

I will provide instructions for how I'd like you to research this.  I don't suggest running under eclipse for this research.

Around line 1047 in SharePointRepository.java, there is this code:

>>>>>
                String modifiedDate = values.get("Modified");
                String createdDate = values.get("Created");
                String guid = values.get("GUID");
                String modifyDate = values.get("Last_x0020_Modified");

<<<<<

Please add this line:

>>>>>
               System.out.println("Modified date string is: '"+modifiedDate+"'");
<<<<<

Please run the job and send me some examples of the modified date string.

Thanks,
Karl





On Wed, Jul 9, 2014 at 10:13 AM, Ameya Aware <ameya.aware@gmail.com> wrote:
i am finding it difficult debug the code.. Whatever changes i do,i just build from scratch and check if changes are done or not.

Can you help me with how can i debug this code? (i am using eclipse IDE)


Thanks,
Ameya


On Wed, Jul 9, 2014 at 10:08 AM, Karl Wright <daddywri@gmail.com> wrote:
Hi Ameya,

Other users have had no trouble with this attribute in the past.  SharePoint, though, has been known to use non-ISO-8601-format dates in some cases.  I wonder if this is one of those cases?

In order to determine this, you will need to edit the code for the SharePoint connector and add debugging output.  Are you in a position to do that?

Karl



On Wed, Jul 9, 2014 at 9:51 AM, Ameya Aware <ameya.aware@gmail.com> wrote:
Hi Karl,

I am seeing Shared documents from Sharepoint.

Thanks,
Ameya


On Tue, Jul 8, 2014 at 5:52 PM, Karl Wright <daddywri@gmail.com> wrote:
Hi Ameya,

There is no difference in treatment between created date and modified date that I can find in the connector.  Can you tell me what kind of SharePoint entity you are seeing this on?  Eg documents, list items, attachments?


Karl

Sent from my Windows Phone

From: Ameya Aware
Sent: 7/8/2014 3:41 PM
To: Karl Wright
Subject: Re: Query in Sharepoint connector

I did not get you properly. Please see below if it satisfies your query.

Last modified is date field in Sharepoint. When i run job and send metadata to Solr, the date which is being sent to Solr is far different than that of it is in Sharepoint.

Please let me know if you need any more details.

Thanks,
Ameya



On Tue, Jul 8, 2014 at 3:35 PM, Karl Wright <daddywri@gmail.com> wrote:
What does this field look like in SharePoint?

Karl

Sent from my Windows Phone
From: Ameya Aware
Sent: 7/8/2014 1:50 PM
To: dev@manifoldcf.apache.org
Subject: Query in Sharepoint connector
Hi

Last_modified metadata sent from Sharepoint to Solr is not giving correct
values.

Even last_modified showing lesser value than created_by date. (Created_by
date is coming good).

Is this bug?

Thanks,
Ameya