lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erick Erickson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-11198) downconfig downloads empty file as folder
Date Sat, 05 Aug 2017 17:40:00 GMT

    [ https://issues.apache.org/jira/browse/SOLR-11198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115470#comment-16115470
] 

Erick Erickson commented on SOLR-11198:
---------------------------------------


I think I see the problem and that this particular issue should be fixed, i.e. empty znodes
with no children should be files locally rather than directories.

I don't think I want to try anything fancier. If a user copies down from ZK and the ZK state
changes (i.e. a node with data gets children etc.) then erroring out and recommending that
they use a clean local directory seems reasonable.

The rest of this is details you can read through if you want to or wonder what the heck the
second point means.

I think the problem is in ZkMaintenanceUtils.downloadFromZk. This code
{code}
   if (children.size() == 0) {
        // If we didn't copy data down, then we also didn't create the file. But we still
need a marker on the local
        // disk so create a dir.
        if (copyDataDown(zkClient, zkPath, file.toFile()) == 0) {
          Files.createDirectories(file);
        }
{code}

copyDataDown returns, well, zero if there's no data in the znode. I changed this code a while
back because recursive copy wasn't working correctly (SOLR-10108) and I suspect it was introduced
then. And it went in Solr 6.6 so it fits Isabelle's testing, including the bit about putting
a comment in the stopwords.txt makes it a file rather than a directory..

zNodes can have both data and children. The logic says, in effect, "if the znode has no data
and no children it'll be mapped into an empty directory". There's logic in there that if a
znode has both data and children, it's made into a directory with a special file containing
its data (zknode.data).

The simple fix would be just to decide the other way, i.e. a znode with no data and no children
would become an empty file locally rather than a directory. That actually seems OK since I
don't think ZK cares. 

The more I think about this the more I think the correct behavior is the easy fix above. ZK
doesn't care; a znode can have data added and children added at will so if we make the local
node an empty text file it'd be copied back up as a znode just like any other, albeit one
without data or children, but that's OK as far as ZK is concerned. That doesn't preclude someone
adding children to the ZK node after pushing it back up via another mechanism.

There'll still be an edge case where 
- someone copies an empty znode from ZK and it becomes a file
- the ZK node gets children
- the person tries the copy again from ZK to local

Since the copy down now already has a text file for that znode and then tries to make a directory
there it'll probably error out. At least it better. I think we can live with that, it would
be a good thing to add a test though.

I'm torn about whether to try to "do the right thing" in the case above. First, it appears
to much of an edge case. One could write something like:
if (the local file is zero length) {
  remove the file
  create a directory in it's place
}

Which seems relatively safe. Except.... that doesn't handle the case where 
- a znode exists with data
- downconfig copies it locally as a text file
- the znode gets children through some other mechanism
- downconfig is run again

In that case "the right thing" would be to move the data to the zknode.data and make the local
node into a directory and continue. Probably could do this "en passant" by just deleting the
text file and continuing, the directory would be add it in its place and children would be
added. Deleting data on the local disk makes me nervous though.

I'll assign it to myself, if anyone else wants to take it feel free.

> downconfig downloads empty file as folder
> -----------------------------------------
>
>                 Key: SOLR-11198
>                 URL: https://issues.apache.org/jira/browse/SOLR-11198
>             Project: Solr
>          Issue Type: Bug
>      Security Level: Public(Default Security Level. Issues are Public) 
>    Affects Versions: 6.6
>         Environment: Windows 7
>            Reporter: Isabelle Giguere
>            Priority: Minor
>
> With Solr 6.6.0, when downloading a config from Zookeeper (3.4.10), if a file is empty,
it is downloaded as a folder (on Windows, at least).
> A Zookeeper browser (Eclipse: Zookeeper Explorer) shows the file as a file, however,
in ZK.
> Noticed because we keep an empty synonyms.txt file in the Solr config provided with our
product, in case a client would want to use it.
> The workaround is simple, since the file allows comments: just add a comment, so it is
not empty.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message