manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Has anyone successfully populated Elastic Search properties from custom SharePoint Meta data fields?
Date Thu, 01 Sep 2016 07:00:47 GMT
Ok, the packaging should now be updated as well.

Karl


On Thu, Sep 1, 2016 at 2:51 AM, Karl Wright <daddywri@gmail.com> wrote:

> Hi Mike,
>
> The code all now builds, and the unit tests pass.  I don't know if it
> works as a plugin, however.  And I haven't changed the jar packaging yet to
> include the properties file.  I will try to look at that today at some
> point.
>
> Note well: I am building this new plugin against ES 2.0.2.  The reason I
> chose this version is because I'm hoping to limit the number of plugin
> versions we need to release, and this might be compatible with all of 2.x,
> with luck.  If you can tell me which version of ES you want to use, I can
> try building against that as a trial when the packaging work is done.
>
> Note also that I've needed to make *no* functional changes to the
> authorization code -- all the work has been in making the code function as
> a proper plugin under 2.0.2.  If the plugin API changes between each
> version of ES 2.x are as significant as the changes between 1.7 and 2.0,
> and the release schedule of ES continues to be essentially one release
> every couple of months, there is no way that ManifoldCF can possibly keep
> up.  I think if that turns out to be the reality I'd recommend that we just
> go back to the original toolkit and have people do their own plugin
> wrappers in the future.
>
> Thanks,
> Karl
>
>
> On Wed, Aug 31, 2016 at 5:09 AM, Karl Wright <daddywri@gmail.com> wrote:
>
>> Ok, I made some progress disentangling what we truly need to get this
>> plugin to build and what's just noise.
>>
>> There are two main problems.
>>
>> The first main problem is that I can't see how you're supposed to
>> register a REST action now.  See MCFAuthorizerPlugin.java, and note the
>> fact that onModule() for RestModule objects doesn't exist, and yet that's
>> what the few examples I've found say you're supposed to do.
>>
>> The second problem is that the way the plugin overrides RestSearchAction
>> is no longer viable; the way requests are processed in the base class is in
>> place into member variables and not into a return variable.  That's going
>> to require some structural changes to address.  It also looks to me like
>> the code that was confusing me before was just modified code from the
>> original RestSearchAction.parse() method, so I can just lift the 2.0.2 code
>> and modify it in a similar manner.  Anyway, I'll push at that one later
>> today.
>>
>> Karl
>>
>>
>> On Tue, Aug 30, 2016 at 7:43 PM, Karl Wright <daddywri@gmail.com> wrote:
>>
>>> I tinkered around with this further a bit but the API indeed changed
>>> dramatically from 1.x to 2.x.  The MCFAuthorizerRestSearchAction class is
>>> the one that needs serious help -- it looks like a lot of specialty code
>>> was needed in 1.x to build a RestSearchAction class that is quite different
>>> in 2.x.  If you can find an example of a RestSearchAction class written
>>> against the 2.0.x API then I might be able to use that as a basis for
>>> completing the plugin; otherwise I don't know where to begin.
>>>
>>> Karl
>>>
>>>
>>> On Tue, Aug 30, 2016 at 5:31 PM, Karl Wright <daddywri@gmail.com> wrote:
>>>
>>>> I set up an ES-2.0 plugin and pointed it at ES 2.0.2.  Build fails
>>>> pretty dramatically.  I'd say that pretty much all the API's changed.  I
>>>> committed the code if you want to try for yourself.
>>>>
>>>> You can find the plugin at:
>>>>
>>>> https://svn.apache.org/repos/asf/manifoldcf/integration/elas
>>>> ticsearch-2.0/trunk
>>>>
>>>> Karl
>>>>
>>>>
>>>> On Tue, Aug 30, 2016 at 3:27 PM, Holtz, Michael (IT Consultant) <
>>>> MHoltz@consultantemail.com> wrote:
>>>>
>>>>> Ah got it. I was wondering if it was something like that.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Tuesday, August 30, 2016 2:55 PM
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Hi Mike,
>>>>>
>>>>>
>>>>>
>>>>> There are TWO versions for the plugin: (1) the version of ES it was
>>>>> written for (1.5), and the functional compatibility with MCF itself (that's
>>>>> where the 2.1 comes from).
>>>>>
>>>>>
>>>>>
>>>>> There is no version of the ES plugin that we built for ES 2.x.  That's
>>>>> what we're trying to create right now.
>>>>>
>>>>>
>>>>>
>>>>> So when you see:
>>>>>
>>>>>
>>>>>
>>>>> apache-manifoldcf-elasticsearch-1.5-plugin-2.2
>>>>>
>>>>>
>>>>>
>>>>> ... the 1.5 means the ES version and the 2.2 is the actual version of
>>>>> the ES-1.5 plugin.
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 1:25 PM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> So the plugin install fails in Elastic Search 2.1.0 with the same
>>>>> error:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> C:\APPS\elasticsearch-2.1.0>bin\plugin install
>>>>> file:///C:/Users/holtzm/Downloads/apache-manifoldcf-elastics
>>>>> earch-plugin-2.1-bin.zip
>>>>>
>>>>> -> Installing from file:/C:/Users/holtzm/Download
>>>>> s/apache-manifoldcf-elasticsearch-plugin-2.1-bin.zip...
>>>>>
>>>>> Trying file:/C:/Users/holtzm/Downloads/apache-manifoldcf-elasticsearch-plugin-2.1-bin.zip
>>>>> ...
>>>>>
>>>>> Downloading .DONE
>>>>>
>>>>> Verifying file:/C:/Users/holtzm/Downloads/apache-manifoldcf-elasticsearch-plugin-2.1-bin.zip
>>>>> checksums if available ...
>>>>>
>>>>> NOTE: Unable to verify checksum for downloaded plugin (unable to find
>>>>> .sha1 or .md5 file to verify)
>>>>>
>>>>> ERROR: Could not find plugin descriptor 'plugin-descriptor.properties'
>>>>> in plugin zip
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The documentation says for 2.1 though so I’m confused at this point.
>>>>>
>>>>>
>>>>>
>>>>> I will try adding the ‘plugin-descriptor.properties ‘ file
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From:* Holtz, Michael (IT Consultant)
>>>>> *Sent:* Tuesday, August 30, 2016 1:03 PM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* RE: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> I think I’ll switch to Elastic Search version 2.1, just to see it work
>>>>> first:
>>>>>
>>>>>
>>>>>
>>>>> *Plugin for Elastic Search (toolkit only)*
>>>>>
>>>>> Latest release (Apache ManifoldCF Plugin for Elastic Search, version
>>>>> 2.1, 2015 Jul 25)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> BTW I was able to see my first use case work. It turns out that I
>>>>> needed to explicitly include the custom meta data fields, “include
all
>>>>> metadata” didn’t work.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> That leaves me with #4
>>>>>
>>>>>
>>>>>
>>>>> 1.       Index custom meta data fields from SharePoint into the
>>>>> ElasticSearch index *done*
>>>>>
>>>>> 2.       Full text indexing of SharePoint content in ElasticSearch
>>>>> *done*
>>>>>
>>>>> 3.       SharePoint security information indexed in ElasticSearch*
>>>>> done*
>>>>>
>>>>> 4.       ElasticSearch Plugin to enforce SharePoint security settings
>>>>> for search *todo*
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com <daddywri@gmail.com>]
>>>>> *Sent:* Tuesday, August 30, 2016 12:34 PM
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> We'll probably need to build against ES 2.x, and will need a whole new
>>>>> plugin.
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 12:29 PM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> Actually I see a classpath exception when starting elastic search so
>>>>> it doesn’t appear to be that simple.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Tuesday, August 30, 2016 11:55 AM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Ok, you just included this in the jar and all was OK?  It should be
>>>>> straightforward to set that up if so...
>>>>>
>>>>>
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 11:27 AM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> Karl,
>>>>>
>>>>>
>>>>>
>>>>> Ok, I will see what I can do.
>>>>>
>>>>>
>>>>>
>>>>> I just created the attached file for the 1.5 version of the plugin and
>>>>> was able to install it. Let me know if you see any issues with it.
>>>>>
>>>>>
>>>>>
>>>>> I’m going to start with a clean version of the new 2.5 release for
my
>>>>> use cases just using the example setup, let me know if you don’t think
that
>>>>> is a good idea.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Tuesday, August 30, 2016 11:17 AM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> The 2.0 spec has clearly changed from that needed for the 1.5-level
>>>>> plugins.
>>>>>
>>>>>
>>>>>
>>>>> The plugin code is likely to be pretty much unchanged but the
>>>>> packaging will need to change.  Now, I did not develop the 1.5-level
>>>>> plugin, and don't know that much about ES in the first place, so if you
>>>>> want to undertake the plugin transformation and contribute it, we'd be
very
>>>>> grateful.
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 11:09 AM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> It looks like a ‘plugin-descriptor.properties’ file needs to be added
>>>>> to the jar:
>>>>>
>>>>>
>>>>>
>>>>> https://www.elastic.co/guide/en/elasticsearch/plugins/2.0/pl
>>>>> ugin-authors.html
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From:* Holtz, Michael (IT Consultant)
>>>>> *Sent:* Tuesday, August 30, 2016 10:53 AM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* RE: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Karl,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> FYI, Now I’m seeing this:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> C:\APPS\elasticsearch-2.3.3>bin\plugin install
>>>>> file:///C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist
>>>>> /plugins/elasticsearch/elasticsearch-1.5/target/elasticsearc
>>>>> h-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar
>>>>>
>>>>> -> Installing from file:/C:/Users/holtzm/workspac
>>>>> e/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elasticse
>>>>> arch-1.5/target/elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-
>>>>> dependencies.jar...
>>>>>
>>>>> Trying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/p
>>>>> lugins/elasticsearch/elasticsearch-1.5/target/elasticsearch-
>>>>> 1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar ...
>>>>>
>>>>> Downloading ...........................DONE
>>>>>
>>>>> Verifying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/p
>>>>> lugins/elasticsearch/elasticsearch-1.5/target/elasticsearch-
>>>>> 1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar checksums if available
>>>>> ...
>>>>>
>>>>> NOTE: Unable to verify checksum for downloaded plugin (unable to find
>>>>> .sha1 or .md5 file to verify)
>>>>>
>>>>> ERROR: Could not find plugin descriptor 'plugin-descriptor.properties'
>>>>> in plugin zip
>>>>>
>>>>>
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From:* Holtz, Michael (IT Consultant)
>>>>> *Sent:* Tuesday, August 30, 2016 10:48 AM
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* RE: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Karl,
>>>>>
>>>>>
>>>>>
>>>>> I got what your’re saying, I should use the 1.5 plugin. Got it.
>>>>>
>>>>>
>>>>>
>>>>> I will try that
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Holtz, Michael (IT Consultant)
>>>>> *Sent:* Tuesday, August 30, 2016 10:47 AM
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* RE: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Hmm that’s confusing, here is the documentation:
>>>>>
>>>>>
>>>>>
>>>>> *Apache ManifoldCF ElasticSearch plugin compatibility*
>>>>>
>>>>> *ManifoldCF versions*
>>>>>
>>>>> *Plugin version*
>>>>>
>>>>> 0.1.x-1.4.x
>>>>>
>>>>> 0.x
>>>>>
>>>>> 1.5.x
>>>>>
>>>>> 1.x
>>>>>
>>>>> >=1.6.x
>>>>>
>>>>> 2.x
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I’m running version 2.4 of manifold so based on this I went with the
>>>>> 2.1 plugin.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Where can I get the 1.0 plugin? I don’t see it on the site or in the
>>>>> 2.4 build.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com <daddywri@gmail.com>]
>>>>> *Sent:* Tuesday, August 30, 2016 10:38 AM
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> You need the ES-1.5 plugin.  That's the older (deprecated) 1.0 plugin.
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 10:36 AM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> Elasticsearch 2.3.3
>>>>>
>>>>> Plugin 2.1
>>>>>
>>>>>
>>>>>
>>>>> Here is the current error
>>>>>
>>>>>
>>>>>
>>>>> C:\APPS\elasticsearch-2.3.3>bin\plugin install
>>>>> file:///C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist
>>>>> /plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar
>>>>>
>>>>> -> Installing from file:/C:/Users/holtzm/workspac
>>>>> e/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elasticse
>>>>> arch/elasticsearch-plugin-mcf-2.1.jar...
>>>>>
>>>>> Trying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/p
>>>>> lugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar
>>>>> ...
>>>>>
>>>>> Downloading .DONE
>>>>>
>>>>> Verifying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/p
>>>>> lugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar
>>>>> checksums if available ...
>>>>>
>>>>> NOTE: Unable to verify checksum for downloaded plugin (unable to find
>>>>> .sha1 or .md5 file to verify)
>>>>>
>>>>> ERROR: Could not find plugin descriptor 'plugin-descriptor.properties'
>>>>> in plugin zip
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Tuesday, August 30, 2016 10:22 AM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Hi Mike,
>>>>>
>>>>>
>>>>>
>>>>> If you are just trying things out you can use the single-process
>>>>> example and it will make your life easier for experimentation.
>>>>>
>>>>>
>>>>>
>>>>> As far as the ES plugin, there are two: one for ES up to 1.5, and one
>>>>> for 1.5 onward.  Which did you try to install?
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 9:56 AM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> Karl,
>>>>>
>>>>>
>>>>>
>>>>> You are correct.
>>>>>
>>>>>
>>>>>
>>>>> Before I setup zookeeper I would like to prove out a few use cases.
>>>>>
>>>>>
>>>>>
>>>>> 1.       Index custom meta data fields from SharePoint into the
>>>>> ElasticSearch index
>>>>>
>>>>> 2.       Full text indexing of SharePoint content in ElasticSearch
>>>>>
>>>>> 3.       SharePoint security information indexed in ElasticSearch
>>>>>
>>>>> 4.       ElasticSearch Plugin to enforce SharePoint security settings
>>>>> for search
>>>>>
>>>>>
>>>>>
>>>>> At this point I’ve only managed to see #2 and #3 work
>>>>>
>>>>>
>>>>>
>>>>> For #4 I was getting an error when installing the ElasticSearch plugin.
>>>>>
>>>>> I would like to see #1 work before I do anything else. Are there any
>>>>> example configurations that show how to do this?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Tuesday, August 30, 2016 9:49 AM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Hi Mike,
>>>>>
>>>>>
>>>>>
>>>>> This sounds like you're having a lock problem.
>>>>>
>>>>>
>>>>>
>>>>> Let me guess: you are using the multiprocess example with file-based
>>>>> locking?  If so, you need to stop everything and do the "lock clean"
>>>>> procedure.  I would also strongly recommend that you switch to the
>>>>> zookeeper multiprocess example.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 9:44 AM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> I currently cannot view the job. The UI is hanging when I click the
>>>>> “view” link for the job. I’m wondering if that has anything to
do with the
>>>>> “Metadata Adjuster” I added?
>>>>>
>>>>>
>>>>>
>>>>> I did not schedule the job.
>>>>>
>>>>>
>>>>>
>>>>> At this point the job has finished running and the Document Status and
>>>>> Queue status reports. I do not see any errors reported there.
>>>>>
>>>>>
>>>>>
>>>>> Also I still do not see my custom meta data fields from SharePoint in
>>>>> the ElasticSearch index.
>>>>>
>>>>>
>>>>>
>>>>> We are hoping to use this in production, but at this point it is not
>>>>> looking too good.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Tuesday, August 30, 2016 9:09 AM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> It is also important to know whether you have your job set up as
>>>>> "continuous".  If so, it will run forever.
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 8:48 AM, Furkan KAMACI <furkankamaci@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Mike,
>>>>>
>>>>>
>>>>>
>>>>> Could you tell us what "Document Status" and "Queue Status" says as
>>>>> Karl mentioned?
>>>>>
>>>>>
>>>>>
>>>>> Kind Regards,
>>>>>
>>>>> Furkan KAMACI
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 3:39 PM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> I’m looking at manifoldcf.log and I see no errors.
>>>>>
>>>>>
>>>>>
>>>>> I have only 13 files and they are all very tiny < 15kb .docx files.
>>>>>
>>>>>
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Monday, August 29, 2016 5:26 PM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> You should be looking at the ManifoldCF.log file, and the examples
>>>>> have logging for errors at least as being the default.
>>>>>
>>>>>
>>>>>
>>>>> The only other possibility is that you have a very large document that
>>>>> is taking a long time to transfer/index.
>>>>>
>>>>>
>>>>>
>>>>> Also, you can generally see what MCF is up to using the "document
>>>>> status" and "queue status" reports.
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 29, 2016 at 5:22 PM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> Karl,
>>>>>
>>>>>
>>>>>
>>>>> I see no errors in the log.
>>>>>
>>>>>
>>>>>
>>>>> I probably need to configure the logging I think.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Monday, August 29, 2016 5:20 PM
>>>>>
>>>>>
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Hi Mike,
>>>>>
>>>>>
>>>>>
>>>>> Have a look in the log.  If there are errors there please send them to
>>>>> us.  Generally, "hangs" are due to MCF retrying documents that have errors.
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 29, 2016 at 5:18 PM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> Karl,
>>>>>
>>>>>
>>>>>
>>>>> Yes, it shows up. I include all meta-data.
>>>>>
>>>>>
>>>>>
>>>>> I’m no trying to use a MetaData adjuster like this
>>>>>
>>>>>
>>>>>
>>>>> Parameter
>>>>>
>>>>> field1                     “${Field 1}”
>>>>>
>>>>>
>>>>>
>>>>> And now I see “field1”=”” in Elastic Search.
>>>>>
>>>>>
>>>>>
>>>>> Also it seems each time I make a change the job will hang and I need
>>>>> to restart everything including ElasticSearch to get it to complete.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike H.
>>>>>
>>>>>
>>>>>
>>>>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>>>>> *Sent:* Monday, August 29, 2016 5:13 PM
>>>>> *To:* user@manifoldcf.apache.org
>>>>> *Subject:* Re: Has anyone successfully populated Elastic Search
>>>>> properties from custom SharePoint Meta data fields?
>>>>>
>>>>>
>>>>>
>>>>> Hi Mike,
>>>>>
>>>>>
>>>>>
>>>>> If I were you I'd try to independently verify that the custom field
>>>>> shows up.  Do you see it appear in the pulldown for selecting metadata?
 If
>>>>> not, it probably is not accessible through the inspection methods that
give
>>>>> us access into SharePoint fields.
>>>>>
>>>>>
>>>>>
>>>>> If it appears there then we should look at what ES gets sent.  It's
>>>>> possible it appears but under a different field name.
>>>>>
>>>>>
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 29, 2016 at 5:01 PM, Holtz, Michael (IT Consultant) <
>>>>> MHoltz@consultantemail.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> I have created a custom content type with a custom field “Field 1”.
>>>>>
>>>>>
>>>>>
>>>>> I have configured the job to include all meta data.
>>>>>
>>>>>
>>>>>
>>>>> I do not see “Field 1” being populated in the Elastic Search index.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Does anyone have any experience with this?
>>>>>
>>>>>
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> Mike H
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message