manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Running ManifoldCF on Resin
Date Mon, 07 Mar 2011 11:13:48 GMT
Hi Erlend,

I'd originally proposed 4/15 for 0.2 release.  And yes, please open a
new ticket if there are problems running under Resin; it would be
great to figure out what the issue was.  I'll triage this problem is
required for 0.2 release.

We do have to get the release candidate together by 3/22 or so if
we're going to make that proposed release schedule, because we need to
get people to sign off, and that takes time.  So I hope you'd be able
to do a little research on it between now and then.

Thanks,
Karl

On Mon, Mar 7, 2011 at 5:24 AM, Erlend Garåsen <e.f.garasen@usit.uio.no> wrote:
>
> I'm sorry about the late reply, I have been out of office.
>
> The reason why it didn't work is because I didn't read the documentation
> properly. I simply forgot to run the following command after I started
> Resin:
> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.AgentRun
>
> I thought it was sufficient to start my application server.
>
> And I'm sorry about the import expansions in the Java classes. I think I
> need a separate version of Eclipse for the Apache projects without all the
> settings I have for the other Java projects I'm working with at the
> university. It will not happen again.
>
> I think I will create another ticket if you don't mind. It can be discussed
> whether it is a bug, but the ManifoldCF pages cannot be displayed properly
> by Firefox if the application is deployed on Resin. Opera and Safari works.
> I have mentioned this before, and the problem is that the CSS file is
> probably not loaded. This is the only problem I have with ManifoldCF and
> Resin.
>
> BTW, do you think the 0.2-incubating version will be released before June
> this year?
>
> And thanks for the quick reply and for resolving this issue.
>
> Erlend
>
>
> On 04.03.11 16.32, Karl Wright wrote:
>>
>> I ran the patch against the standard postgresql tests, which do a
>> number of file-system crawls, and all was well.
>> So I am still very curious as to why you are seeing the hang.  Let me
>> know what I can do to help chase this down.
>>
>> Thanks!
>> Karl
>>
>> On Fri, Mar 4, 2011 at 9:40 AM, Karl Wright<daddywri@gmail.com>  wrote:
>>>
>>> Yes, please submit the patch.
>>>
>>> Can you give me more details of the start up issue?  I'd like to try
>>> it here; are these pages I can get at?
>>>
>>> The crawler leaves the "starting up" state when the seeds have been
>>> added to the queue.  The fact that this seeding process does not seem
>>> to be completing means that something is pretty badly broken
>>> somewhere. For the web connector (which from your history I am
>>> presuming you are using), the seeding phase doesn't do much else than
>>> write records to the database.
>>>
>>> I think you may be misinterpreting how to use the Simple History
>>> report.  You are supposed to select what activities you want to see
>>> from the pulldown in the upper right.  By default, all of them are
>>> selected, and the initial report displayed is based on that.  If it is
>>> empty (that is, you see column headers but nothing else), then NO
>>> activity at all has occurred in the time window given for that
>>> connection.
>>>
>>> Thanks,
>>> Karl
>>>
>>> On Fri, Mar 4, 2011 at 9:28 AM, Erlend Garåsen<e.f.garasen@usit.uio.no>
>>>  wrote:
>>>>
>>>> Maybe I should create the PostgreSQL patch without any more testing. I
>>>> have
>>>> some problems to run the jobs inside the application, but I doubt the
>>>> reason
>>>> is my PostgreSQL code changes since the same happens with Apache Derby.
>>>> The
>>>> status of the job says "Starting up" and nothing shows up in the
>>>> manifoldcf.log nor in the history logs available inside the application.
>>>>
>>>> If I select "Simple History", the list is empty, but in the activities
>>>> list
>>>> (in the upper right corner) I see that "robots parse" is the last
>>>> activity
>>>> run.
>>>>
>>>> This is not something Resin specific since the same happens under
>>>> Tomcat.
>>>>
>>>> My contribution is clean and I have only made a few changes in the
>>>> DBInterfacePostgreSQL and ManifoldCF class. Nothing is changed in the
>>>> Database super class.
>>>>
>>>> I haven't run "ant test-pg" since it seems to need a local Postgresql
>>>> server
>>>> running. But "ant test" reports no errors.
>>>>
>>>> I have done an svn up before I created the patch in order to include the
>>>> latest changes from trunk.
>>>>
>>>> The documentation needs to be updated and I can suggest what to add
>>>> later.
>>>>
>>>> BTW, it should be interesting to know what causes the problems described
>>>> above since I really want to run my jobs.
>>>>
>>>> PS: I do not have access to the Eclipse 3.2 codestyle.xml file, even if
>>>> I'm
>>>> logging in, so I had to turn of formatting in Eclipse in order to keep
>>>> the
>>>> existing formatting (I get the following error when I click on the link
>>>> to
>>>> the xml file: "You are not allowed to do AttachFile on this page.")
>>>>
>>>> Erlend
>>>>
>>>>
>>>>
>>>> I don't think it is something PG specific either,
>>>>
>>>> On 25.02.11 16.15, Karl Wright wrote:
>>>>>
>>>>> The DBCreate command requires privs necessary to create the database
>>>>> instance in the first place.  It uses the admin credentials that you
>>>>> supply to it as command args.  These are NOT the same credentials used
>>>>> to access the database instance, after it has been created.
>>>>>
>>>>> Thus, you can skip the DBCreate step if you have already created the
>>>>> database by hand.
>>>>>
>>>>> It does not surprise me that PostgreSQL's JDBC driver uses the global
>>>>> Java trust store.  Since there is only one database active for any
>>>>> installation, it is perfectly OK for the global trust store to be used
>>>>> for this purpose.  It will, however, probably require the addition of
>>>>> some documentation as to how to set it up properly on various
>>>>> platforms.
>>>>>
>>>>> Thanks,
>>>>> Karl
>>>>>
>>>>> On Fri, Feb 25, 2011 at 9:46 AM, Erlend
>>>>> Garåsen<e.f.garasen@usit.uio.no>
>>>>>  wrote:
>>>>>>
>>>>>> I think we need another parameter as well:
>>>>>> org.apache.manifoldcf.postgresql.port
>>>>>>
>>>>>> I was going to test my code today, but figured out that the following
>>>>>> command class needs admin privileges to the PostgreSQL database,
which
>>>>>> I
>>>>>> don't and should never have:
>>>>>> org.apache.manifoldcf.core.DBCreate
>>>>>>
>>>>>> So I skipped that part. Anyway, the DBCreate method is also supposed
>>>>>> to
>>>>>> work
>>>>>> with my contribution if I had the sufficient privileges.
>>>>>>
>>>>>> I went on by using the Install method class instead, but ended up
with
>>>>>> a
>>>>>> Handshake exception:
>>>>>> javax.net.ssl.SSLHandshakeException:
>>>>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable
to
>>>>>> find
>>>>>> valid certification path to requested target.
>>>>>>
>>>>>> After I added our PG server's certificate to my local keystore, things
>>>>>> began
>>>>>> to work as it should:
>>>>>> ->    Agent tables installed
>>>>>>
>>>>>> To be continued. I have a lot of things to do at the moment, and
I
>>>>>> cannot
>>>>>> test my contribution properly yet since I'm at home with a slower
>>>>>> bandwidth.
>>>>>>
>>>>>> To be continued. :)
>>>>>>
>>>>>> Erlend
>>>>>>
>>>>>>
>>>>>> On 04.02.11 16.30, Karl Wright wrote:
>>>>>>>
>>>>>>> Sure, feel free to create a ticket and try to come up with a
patch.
>>>>>>> Since the DBInterfacePostgreSQL class is responsible for any
remote
>>>>>>> communication, I would make those parameters local to the postgresql
>>>>>>> implementation.  So the parameters would be something like:
>>>>>>>
>>>>>>> org.apache.manifoldcf.postgresql.hostname
>>>>>>> org.apache.manifoldcf.postgresql.ssl
>>>>>>>
>>>>>>> or some such.  Then, you'd want to load the necessary parameters
in
>>>>>>> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with
MySQL
>>>>>>> since that's not officially supported at this time.)
>>>>>>>
>>>>>>> The only problem I see is that the constructor architecture of
>>>>>>> Database.java may make it a challenge to load the appropriate
>>>>>>> parameters in the DBInterfacePostgreSQL constructor.  But I
think you
>>>>>>> could make it work, and if not, we should probably fix this anyway
to
>>>>>>> make it easier.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Karl
>>>>>>>
>>>>>>> On Fri, Feb 4, 2011 at 10:17 AM, Erlend
>>>>>>> Garåsen<e.f.garasen@usit.uio.no>
>>>>>>>  wrote:
>>>>>>>>
>>>>>>>> You're right! I didn't read the command section in the documentation
>>>>>>>> properly. ManifoldCF does also run on Resin now. The only
problem
>>>>>>>> with
>>>>>>>> Resin
>>>>>>>> is that the necessary resources, such as CSS, are not loaded.
All
>>>>>>>> the
>>>>>>>> other
>>>>>>>> errors have disappeared, which means that it should be possible
to
>>>>>>>> run
>>>>>>>> the
>>>>>>>> application on Resin.
>>>>>>>>
>>>>>>>> Do you think I should create a ticket about running an external
>>>>>>>> PostgreSQL
>>>>>>>> database as well? I think I know which classes to modify.
I see
>>>>>>>> there
>>>>>>>> are
>>>>>>>> classes for MySQL as well, so perhaps the issue should be
generally,
>>>>>>>> i.e.
>>>>>>>> not PG specific.
>>>>>>>>
>>>>>>>> I was thinking about extending the property list like this:
>>>>>>>> org.apache.manifoldcf.database.hostname (not required, defaults
to
>>>>>>>> localhost)
>>>>>>>> org.apache.manifoldcf.database.ssl (not required, defaults
to false)
>>>>>>>>
>>>>>>>> For PostgreSQL I think it should be quite easy to implement
this,
>>>>>>>> but
>>>>>>>> I'm
>>>>>>>> not sure when it comes to MySQL. Haven't touched that database
>>>>>>>> system
>>>>>>>> for
>>>>>>>> years.
>>>>>>>>
>>>>>>>> Erlend
>>>>>>>>
>>>>>>>>
>>>>>>>> On 04.02.11 15.42, Karl Wright wrote:
>>>>>>>>>
>>>>>>>>> Ok, have you followed the instructions for a multi-process
>>>>>>>>> installation in how-to-build-and-deploy.html?
>>>>>>>>> You will need to create the database (using DBCreate),
and install
>>>>>>>>> the
>>>>>>>>> schema (using Install), and register the crawler agent
(using
>>>>>>>>> Register).  You will also need to create the properties.xml
and
>>>>>>>>> logging.ini files by hand.
>>>>>>>>>
>>>>>>>>> Let me know if this helps.
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend
>>>>>>>>> Garåsen<e.f.garasen@usit.uio.no>
>>>>>>>>>  wrote:
>>>>>>>>>>
>>>>>>>>>> I guess the problem is that the database does not
exist. When I
>>>>>>>>>> installed
>>>>>>>>>> it
>>>>>>>>>> on Tomcat, I got an error message telling me that
the schema TEST
>>>>>>>>>> does
>>>>>>>>>> not
>>>>>>>>>> exist. I tried to build the database with the username
and
>>>>>>>>>> password
>>>>>>>>>> "test"
>>>>>>>>>> by the following command:
>>>>>>>>>> ./processes/script/executecommand.sh
>>>>>>>>>> org.apache.manifoldcf.core.DBCreate
>>>>>>>>>> test test
>>>>>>>>>>
>>>>>>>>>> And I also configured these two lines in my properties.xml
file:
>>>>>>>>>>  <property name="org.apache.manifoldcf.database.username"
>>>>>>>>>> value="test"/>
>>>>>>>>>>  <property name="org.apache.manifoldcf.database.password"
>>>>>>>>>> value="test"/>
>>>>>>>>>>
>>>>>>>>>> I'm using the Derby database since I cannot see how
I could
>>>>>>>>>> contact
>>>>>>>>>> an
>>>>>>>>>> external PostgreSQL server on another host. PG is
one of several
>>>>>>>>>> database
>>>>>>>>>> systems the university supports. I guess there should
be an option
>>>>>>>>>> for
>>>>>>>>>> configuring external hosts and optionally turn on
SSL encryption.
>>>>>>>>>> Maybe
>>>>>>>>>> another ticket to create.
>>>>>>>>>>
>>>>>>>>>> But first I want to run ManifoldCF on Tomcat using
Derby, then
>>>>>>>>>> deploy
>>>>>>>>>> it
>>>>>>>>>> on
>>>>>>>>>> Resin and then try to configure an external PostgreSQL
server.
>>>>>>>>>>
>>>>>>>>>> Erlend
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>>>>>>>
>>>>>>>>>>> Sure, if you want, create a ticket, and see if
you can figure out
>>>>>>>>>>> what
>>>>>>>>>>> the problems are.  I'd start with not being
able to reach the
>>>>>>>>>>> stylesheet, actually.  The "redirection" error
sound like the
>>>>>>>>>>> crawler
>>>>>>>>>>> UI is trying to present an error and resin is
handling one of the
>>>>>>>>>>> java
>>>>>>>>>>> standard tags in a non-standard way.
>>>>>>>>>>>
>>>>>>>>>>> Karl
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
>>>>>>>>>>> Garåsen<e.f.garasen@usit.uio.no>
>>>>>>>>>>>  wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Resin is de facto application server at the
university, but we
>>>>>>>>>>>> also
>>>>>>>>>>>> support
>>>>>>>>>>>> Tomcat.
>>>>>>>>>>>>
>>>>>>>>>>>> I can try to deploy the war files on Tomcat
instead. The error
>>>>>>>>>>>> message
>>>>>>>>>>>> comes
>>>>>>>>>>>> from Resin, probably caused by a redirect
after the http header
>>>>>>>>>>>> is
>>>>>>>>>>>> written.
>>>>>>>>>>>> I can notify you, eventually create a patch,
if I manage to run
>>>>>>>>>>>> ManifoldCF
>>>>>>>>>>>> on Resin.
>>>>>>>>>>>>
>>>>>>>>>>>> Erlend
>>>>>>>>>>>>
>>>>>>>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have never tried this on Resin, just
on Tomcat and Jetty.
>>>>>>>>>>>>>  Clearly
>>>>>>>>>>>>> Resin as an application server is having
difficulty doing some
>>>>>>>>>>>>> very
>>>>>>>>>>>>> basic stuff.
>>>>>>>>>>>>>
>>>>>>>>>>>>> FWIW, when we first started the Jetty
development, I had to do
>>>>>>>>>>>>> nothing
>>>>>>>>>>>>> whatsoever to the crawler UI or the war
packaging to make it
>>>>>>>>>>>>> work.
>>>>>>>>>>>>>  It
>>>>>>>>>>>>> just worked.  Your experiences with
Resin are making me wonder
>>>>>>>>>>>>> if
>>>>>>>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>>>>>>>> Garåsen<e.f.garasen@usit.uio.no>
>>>>>>>>>>>>>  wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have placed the three war files
into Resin's webapps
>>>>>>>>>>>>>> directory
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> set
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> following system variable inside
resin.xml:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <system-property
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>>>>>>>>>>> />
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have also created my own home directory
for ManifoldCF and
>>>>>>>>>>>>>> copied
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> necessary files from the example
directory. When I start Resin
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> goes
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> ... mcf-crawler-ui/, the start page
shows up, but the page
>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>> ugly
>>>>>>>>>>>>>> (I
>>>>>>>>>>>>>> guess the path to CSS files etc.
are broken). And when I try
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> click
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> link, e.g. "List Output Connections",
I get the following
>>>>>>>>>>>>>> error:
>>>>>>>>>>>>>> java.lang.IllegalStateException:
can't forward after writing
>>>>>>>>>>>>>> HTTP
>>>>>>>>>>>>>> headers
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> BTW: PostgreSQL is not configured.
I will start using Derby
>>>>>>>>>>>>>> until
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> runs
>>>>>>>>>>>>>> stable on Resin.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> E
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Erlend Garåsen
>>>>>>>>>>>>>> Center for Information Technology
Services
>>>>>>>>>>>>>> University of Oslo
>>>>>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO,
Norway
>>>>>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970,
Mobile: (+47)
>>>>>>>>>>>>>> 91380968,
>>>>>>>>>>>>>> VIP:
>>>>>>>>>>>>>> 31050
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Erlend Garåsen
>>>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>>>> University of Oslo
>>>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970,
Mobile: (+47) 91380968,
>>>>>>>>>>>> VIP:
>>>>>>>>>>>> 31050
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Erlend Garåsen
>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>> University of Oslo
>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile:
(+47) 91380968,
>>>>>>>>>> VIP:
>>>>>>>>>> 31050
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Erlend Garåsen
>>>>>>>> Center for Information Technology Services
>>>>>>>> University of Oslo
>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
>>>>>>>> VIP:
>>>>>>>> 31050
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Erlend Garåsen
>>>>>> Center for Information Technology Services
>>>>>> University of Oslo
>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
VIP:
>>>>>> 31050
>>>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>> 31050
>>>>
>>>
>
>
> --
> Erlend Garåsen
> Center for Information Technology Services
> University of Oslo
> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>

Mime
View raw message