lucene-openrelevance-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fan Jiang <dcva...@gmail.com>
Subject Re: ORP System Requirements
Date Sat, 06 Jul 2013 20:24:51 GMT
Hi Grant,

Thank you for your advices. I also come up with a new idea. The system
should runs on several services - One for running evaluations and keeping
track of the evaluation records, one for authentication and
authorization(if the system is exposed to multi-users) and one works as a
repository of test collections.  In this way, the system can be more
scalable and the business logic can be more clear. Does this make sense?
Thanks!

Sincerely,
Fan


2013/7/3 Grant Ingersoll <gsingers@apache.org>

> Hi Fan,
>
> On Jul 2, 2013, at 11:31 PM, Fan Jiang <dcvan24@gmail.com> wrote:
>
> > Hi all,
> >
> > I am Fan and I am new to Apache Lucene Open Relevance Project. Recently I
> > have been working on developing a web tool for the project with a number
> of
> > collaborators and we are thinking of the system requirements now. The
> > requirements are listed as following:
> >
> > 1. The system is designed to used for evaluating the searching relevancy
> of
> > queries or search engines.
>
> I think we'll want to support various types of evaluations, both
> TREC-style as well as focus-group style evals.
>
> > 2. The system can be used on various commercial or open-source search
> > engines, such as Google, Bing, Lucene, Solr, etc.
>
> Some evaluations won't make sense w/ some engines, but I like the idea
> that they engines are pluggable.
>
> > 3. The system should be able to accept and store test collections. The
> test
> > collections include a collection of queries and qrels.
> > 4. The system should be able to run test collections and evaluate queries
> > or search engines with various evaluators. The system should provide with
> > certain standard evaluators, while user should be able to run customized
> > evaluators.
> > 5. The system should be able to keep track of timestamps, queries,
> document
> > set,  schema(including data model, analyzer, tokenizer, filter, etc.),
> > scoring model, evaluation metrics, searching results and relevancy score.
> >
> > Any ideas? Thanks!
>
> A few things come to mind for now:
>
> 1. Simple, easy to use UI that others can easily pick up and extend
> 2. Good documentation on using and extending
>
> Overall, I like this and glad to see someone picking back up on ORP.
>
> -Grant




-- 
Sincerely,
Fan Jiang

Master of Science in Information Science
School of Library and Information Science
University of North Carolina at Chapel Hill
(919)-360-1047

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message