lucene-openrelevance-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Ingersoll <gsing...@apache.org>
Subject Re: ORP System Requirements
Date Mon, 08 Jul 2013 01:34:27 GMT
Yes, I think that makes sense, although I'd worry about authentication/authorization last,
if possible.

On Jul 6, 2013, at 4:24 PM, Fan Jiang <dcvan24@gmail.com> wrote:

> Hi Grant,
> 
> Thank you for your advices. I also come up with a new idea. The system
> should runs on several services - One for running evaluations and keeping
> track of the evaluation records, one for authentication and
> authorization(if the system is exposed to multi-users) and one works as a
> repository of test collections.  In this way, the system can be more
> scalable and the business logic can be more clear. Does this make sense?
> Thanks!
> 
> Sincerely,
> Fan
> 
> 
> 2013/7/3 Grant Ingersoll <gsingers@apache.org>
> 
>> Hi Fan,
>> 
>> On Jul 2, 2013, at 11:31 PM, Fan Jiang <dcvan24@gmail.com> wrote:
>> 
>>> Hi all,
>>> 
>>> I am Fan and I am new to Apache Lucene Open Relevance Project. Recently I
>>> have been working on developing a web tool for the project with a number
>> of
>>> collaborators and we are thinking of the system requirements now. The
>>> requirements are listed as following:
>>> 
>>> 1. The system is designed to used for evaluating the searching relevancy
>> of
>>> queries or search engines.
>> 
>> I think we'll want to support various types of evaluations, both
>> TREC-style as well as focus-group style evals.
>> 
>>> 2. The system can be used on various commercial or open-source search
>>> engines, such as Google, Bing, Lucene, Solr, etc.
>> 
>> Some evaluations won't make sense w/ some engines, but I like the idea
>> that they engines are pluggable.
>> 
>>> 3. The system should be able to accept and store test collections. The
>> test
>>> collections include a collection of queries and qrels.
>>> 4. The system should be able to run test collections and evaluate queries
>>> or search engines with various evaluators. The system should provide with
>>> certain standard evaluators, while user should be able to run customized
>>> evaluators.
>>> 5. The system should be able to keep track of timestamps, queries,
>> document
>>> set,  schema(including data model, analyzer, tokenizer, filter, etc.),
>>> scoring model, evaluation metrics, searching results and relevancy score.
>>> 
>>> Any ideas? Thanks!
>> 
>> A few things come to mind for now:
>> 
>> 1. Simple, easy to use UI that others can easily pick up and extend
>> 2. Good documentation on using and extending
>> 
>> Overall, I like this and glad to see someone picking back up on ORP.
>> 
>> -Grant
> 
> 
> 
> 
> -- 
> Sincerely,
> Fan Jiang
> 
> Master of Science in Information Science
> School of Library and Information Science
> University of North Carolina at Chapel Hill
> (919)-360-1047

--------------------------------------------
Grant Ingersoll | @gsingers
http://www.lucidworks.com






Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message