lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Sun (JIRA)" <>
Subject [jira] [Commented] (SOLR-10317) Solr Nightly Benchmarks
Date Wed, 14 Jun 2017 21:02:00 GMT


Michael Sun commented on SOLR-10317:

Just uploaded the first cut of Solr benchmark I built during my work, as one more option for
community for benchmarking. There are a few good benchmarks in the community for different
use cases, using different frameworks. The goal of my benchmark, in short, is to design an
extensible, standardized benchmark that can be used for a variety of common performance use
cases. Nightly performance regression tests are very important. Meanwhile it would be good
if we can reuse the same benchmark for capacity planning, scalability study, troubleshooting,
etc., which has slightly different requirement to nightly tests. It would be a good saving
for everyone in community if he only needs to extend the benchmark, not rebuild one, for his
own use cases in near future.

In addition, the benchmark includes a variety of instruments to help understand why the performance
is, in addition to what the performance is. One obvious reason is that answering why is the
primary goal for some use cases, such as troubleshooting, scalability study. Meanwhile it
also helps to build 'correct' performance tests. For example, performance bottleneck discovered
in tests may not be a code defect but some setup issue. Being able to analyze a bit can make
sure the performance tests are testing the right thing. 

Designing a good benchmark is one of my primary jobs at work. So I will continue to elaborate
the framework and add new tests. There are a few good benchmarks for Solr. Also []
has done a great job in designing a few new test cases. I can help you in porting or adding
new test cases with my framework if you like.

The patch mainly includes object model and a sample test to demonstrate object model. More
components will follow. It's an option to community of course but I do think community can
benefit from this contribution. Any feedback is appreciated.

> Solr Nightly Benchmarks
> -----------------------
>                 Key: SOLR-10317
>                 URL:
>             Project: Solr
>          Issue Type: Task
>            Reporter: Ishan Chattopadhyaya
>              Labels: gsoc2017, mentor
>         Attachments: changes-lucene-20160907.json, changes-solr-20160907.json, managed-schema,
Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks.docx, Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks-FINAL-PROPOSAL.pdf,
> Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be found here,
> Preferably, we need:
> # A suite of benchmarks that build Solr from a commit point, start Solr nodes, both in
SolrCloud and standalone mode, and record timing information of various operations like indexing,
querying, faceting, grouping, replication etc.
> # It should be possible to run them either as an independent suite or as a Jenkins job,
and we should be able to report timings as graphs (Jenkins has some charting plugins).
> # The code should eventually be integrated in the Solr codebase, so that it never goes
out of date.
> There is some prior work / discussion:
> # (Shalin)
> # (Ishan/Vivek)
> # SOLR-2646 & SOLR-9863 (Mark Miller)
> # (Mike McCandless)
> # (Tim Potter)
> There is support for building, starting, indexing/querying and stopping Solr in some
of these frameworks above. However, the benchmarks run are very limited. Any of these can
be a starting point, or a new framework can as well be used. The motivation is to be able
to cover every functionality of Solr with a corresponding benchmark that is run every night.
> Proposing this as a GSoC 2017 project. I'm willing to mentor, and I'm sure [~shalinmangar]
and [] would help here.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message