lucene-openrelevance-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Ingersoll <gsing...@apache.org>
Subject Re: ORP System Requirements
Date Wed, 03 Jul 2013 12:54:27 GMT
Hi Fan,

On Jul 2, 2013, at 11:31 PM, Fan Jiang <dcvan24@gmail.com> wrote:

> Hi all,
> 
> I am Fan and I am new to Apache Lucene Open Relevance Project. Recently I
> have been working on developing a web tool for the project with a number of
> collaborators and we are thinking of the system requirements now. The
> requirements are listed as following:
> 
> 1. The system is designed to used for evaluating the searching relevancy of
> queries or search engines.

I think we'll want to support various types of evaluations, both TREC-style as well as focus-group
style evals.

> 2. The system can be used on various commercial or open-source search
> engines, such as Google, Bing, Lucene, Solr, etc.

Some evaluations won't make sense w/ some engines, but I like the idea that they engines are
pluggable.

> 3. The system should be able to accept and store test collections. The test
> collections include a collection of queries and qrels.
> 4. The system should be able to run test collections and evaluate queries
> or search engines with various evaluators. The system should provide with
> certain standard evaluators, while user should be able to run customized
> evaluators.
> 5. The system should be able to keep track of timestamps, queries, document
> set,  schema(including data model, analyzer, tokenizer, filter, etc.),
> scoring model, evaluation metrics, searching results and relevancy score.
> 
> Any ideas? Thanks!

A few things come to mind for now:

1. Simple, easy to use UI that others can easily pick up and extend
2. Good documentation on using and extending

Overall, I like this and glad to see someone picking back up on ORP.

-Grant
Mime
View raw message