lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 李 世明 <limingni...@live.com>
Subject Re: LTRScoringQuery. ModelWeight. Inside scorer featureWeight. Scorer is synchronous
Date Tue, 11 Jun 2019 13:36:57 GMT
Hi:
When I use LTR, there are 72 features, 500 models and 200 million data. During use, LTR execution
was found to be extremely slow, with each query above 5s.

In LTRScoringQuery. ModelWeight scorer method in class,

for (final Feature.FeatureWeight featureWeight : extractedFeatureWeights) {

  final Feature.FeatureWeight.FeatureScorer scorer = featureWeight.scorer(context);
  if (scorer != null) {
    featureScorers.add(scorer);
  }
}



Modified to a thread pool call. Please help to see if this can lead to performance optimization.Thank
you very much



List<Future<Feature.FeatureWeight.FeatureScorer> > futures = new ArrayList<>(extractedFeatureWeights.length);
for (final Feature.FeatureWeight featureWeight : extractedFeatureWeights) {
  MyCreateFeatureWeightCallable callable = new MyCreateFeatureWeightCallable(featureWeight,
context);
  RunnableFuture<Feature.FeatureWeight.FeatureScorer> runnableFuture = new FutureTask<>(callable);
  ltrThreadMgr.execute(runnableFuture);//releases semaphore when done
  futures.add(runnableFuture);
}
for (final Future<Feature.FeatureWeight.FeatureScorer> future : futures) {
  featureScorers.add(future.get()); // future.get() will block if the job is still running
}



public class MyCreateFeatureWeightCallable implements Callable<Feature.FeatureWeight.FeatureScorer>
{

    private Feature.FeatureWeight featureWeight;
    private LeafReaderContext context;

    public MyCreateFeatureWeightCallable(Feature.FeatureWeight featureWeight, LeafReaderContext
context) {
        this.featureWeight = featureWeight;
        this.context = context;
    }

    @Override
    public Feature.FeatureWeight.FeatureScorer call() throws Exception {
        return featureWeight.scorer(context);
    }
}




在 2019年6月9日,上午11:14,李世明 <limingnihao@live.com<mailto:limingnihao@live.com>>
写道:

Hi:
When I use LTR, there are 72 features, 500 models and 200 million data. During use, LTR execution
was found to be extremely slow, with each query above 5s.
Through the analysis found in LTRScoringQuery ModelWeight scorer method of class 72 when creating
FeatureScorer feature value in the use of content of the slowly, and here is the synchronization
of execution.
I tried to change this to multithreaded, which had some effect. The average query time was
600ms.
Is it too slow for payload or is there room for optimization here?

thank you

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message