lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Noble Paul (JIRA)" <>
Subject [jira] [Commented] (SOLR-7576) Implement RequestHandler in Javascript
Date Fri, 07 Aug 2015 04:14:45 GMT


Noble Paul commented on SOLR-7576:

bq. is the annoyance of having to update a file in ZooKeeper from one's file system 

I can see that pain point.

But the current model is totally broken and we MUST move away from from that
* we can't keep large amounts data in ZK
* We should not give a new feature out which requires a developer direct access to write to
ZK.We are talking about the same thing for a toy single node system to large clusters with
1000's of nodes. Any screw up in ZK will have huge impact . It is not like a solr node going
down, it is going to blow up a huge cluster 
* Editing files and picking them up immediately may sound ok if you don't cache stuff. Which
is terrible for performance. So there has to be a way to tell Solr that a particular resource
is changed
* In real production systems we need to rollback stuff if things are broken. This design lets
you do that. You always upload a new version of the same file like a version control system.
Uploading a new version will have no impact till you push it to production. Play with the
new version till you are happy and then push it to production. The point is , the developer
can't screw up the cluster by introducing a small syntax error 

I'm not against adding an easier dev sandbox for such features. This is just the first step.
First of all we need to get the feature right for a deployed large cluster. That is the story
that we will want to discuss first. This just one extra step for a developer in that phase.
Ease of use stuff for playing with things is a secondary thing and we should be able to do
it as a separate ticket. Understand that a standalone process and a huge cluster with 1000's
of nodes will have some difference in the way stuff is deployed

guys , editing ZK directly is fraught with huge risk. Please don't recommend it to users.
I understand that we have a lot of legacy stuff without proper APIs and we need to edit things
directly. We MUST have a safe way of achieving the same through APIs (eventually). 

So please do not introduce any new feature that needs ZK write.  

> Implement RequestHandler in Javascript
> --------------------------------------
>                 Key: SOLR-7576
>                 URL:
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Noble Paul
>         Attachments: SOLR-7576.patch, SOLR-7576.patch
> Solr now support dynamic loading (SOLR-7073) of components and it is secured in SOLR-7126
> We can extend the same functionality with JS as well
> the handler {{/js}} is implicitly registered
> To make this work
> * Solr should be started with {{-Denable.js.loading=true}}
> * The javascript must be loaded to the {{.system}} collection using the blob store API
> * Sign the javascript and pass the signature in a param called {{_sig}}
> The {{JSRequestHandler}} is implicitly defined and it can be accessed by hitting {{/js/<jsname>/<version>}}

> Steps for developing scripts
> # start the cluster with the {{enable.js.loading}} . If you are starting using our script
it would be {{bin/solr start -e cloud -a "-Denable.js.loading=true"}} . You would not need
security during development , so don't add the private keys to Solr
> # create {{.system}} collection {{bin/solr create -c .system}}
> # Write your javascript code . (say {{test.js}} )
> # post it to {{.system}} collection . {{curl -X POST -H 'Content-Type: application/octet-stream'
--data-binary @test.js http://localhost:8983/solr/.system/blob/test}}
> # run your script {{http://host:8983/solr/gettingstarted/js/test/1}}
> # Edit your script and repeat from step #4 . Keep in mind that the version would be bumped
up every time you post a new script . So, the second time the url would be {{http://host:8983/solr/gettingstarted/js/test/2}}
. So on and so forth
> sample programs
> 1) writes a val to output
> {code:javascript}
> //empty line
> $.response().add('testkey','Test Val');
> {code}
> 2)  manipulate the output to add an extra field to each doc 
> {code}
> //empty line
> var l = [];
> $.query({
>               q: '*:*',
>               qt: '/select',
>               start:0,
>           }).forEach('response', function(doc) {
>                                          doc.put('script', 'Added this value');
>                                          l.push(doc);
>           });
>  $.response().add('alldocs', l);
> {code}
> 3)  stream through all the docs
> {code:Javascript}
> //empty line
> $.query({
>               q: '*:*',
>               qt: '/select',
>               start:0,
>               distrib:'false'
>           }).pipe('response', 'docs', function(doc) { // the pipe function is executed
right before the response writer and right after the transformers   
>                                          if('IT'== doc.get('genre_s')) return null;
>                                          doc.put('script', 'Added this value');
>                                          return doc;
>           });
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message