lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <>
Subject Re: Solr TCP layer
Date Mon, 09 Mar 2015 03:15:57 GMT
On 3/8/2015 2:05 PM, Saumitra Srivastav wrote:
> I want to start working on adding a TCP layer for client to node and
> inter-node communication.
> I am not up to date on recent changes happening to Solr. So before I start
> looking into code, I would like to know if there is already some work done
> in this direction, which I can reuse. Are there any know
> challenges/complexities?
> I would appreciate any help to kick start this effort. Also, what would be
> the best way to discuss and get feedback on design from contributors? Open a
> JIRA??

What follows is my mostly my opinion, interspersed with a few things
that might be loosely called facts:

I personally do not mind at all that Solr uses HTTP.

Some people view it as an inefficient protocol, but the overhead is low
on all but the slowest connections.  On a LAN, it is probably not even
worth discussing.  Most of the time, there will be a LAN connection
between the Solr client and the Solr server.

I think there are two primary advantages to HTTP:

*) Testing can be carried out in a browser with hand-typed URLs.
*) We don't have to worry about writing the code to handle the
underlying protocol.

The first point makes testing throughout the Solr deployment process
VERY easy.

Expanding on the second point:  There are well-tested and mostly
bug-free HTTP client and server libraries available for us to use.  We
don't have to figure out how to write them, troubleshoot them, or
optimize them.  There are hundreds or thousands of other people using
those libraries that can find and report bugs and inefficiencies, often
including the fix in the bug report.

We might be able to make Solr a little more efficient if we use our own
TCP protocol instead of HTTP, but there are drawbacks.  It takes a lot
of experience to write a network protocol from scratch, and we are bound
to make mistakes, mistakes that will cause users a LOT of problems.  The
authors of those libraries that I mentioned have already gone through
that pain, often many times.

Solr must be a standalone application to realistically use a custom TCP
protocol directly.  Although there are already loose plans in place to
pull the HTTP and network layers into Solr and make it a standalone
application, we are not there yet.

If you want to work on a new protocol, feel free.  If there is anything
I can do to help in between my other duties, I will.  I do not know of
any existing work that you can re-use, although it's possible there
might be something.  You might look at the Zookeeper project for an
existing implementation of a custom network protocol in Java.

I suspect that it will take a very long time before any such work is as
stable as HttpClient and the Servlet API (implemented by Jetty).  Unless
you can demonstrate that stability, it will not become the default protocol.


View raw message