drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From arina-ielchiieva <...@git.apache.org>
Subject [GitHub] drill issue #701: DRILL-4963: Fix issues with dynamically loaded overloaded ...
Date Mon, 23 Jan 2017 13:54:04 GMT
Github user arina-ielchiieva commented on the issue:

    @paul-rogers as we discussed to have tried to find the way to preserve lazy-init approach.
    I have renamed PR to reflect latest changes. Please find new solution and description
of changes below:
    Lazy-init was performed only when function was not found during Calcite parsing but DRILL-4963
shows different cases when Calcite parsing can pass (usually during function overloading)
but still function is not found. To handle such cases, we need to enhance lazy-init process:
    1. Lazy-init process should be more light-weight. Currently when function is not found,
we load all jars from remote function registry and compare with jars from local function registry.
It's not optimal especially when both registries are in sync. To improve performance I have
introduced remove function registry version which can be used to check if we need to sync
remote and local registries prior to checking jars and functions.
    2. During parsing stage we were only catching Calcite parsing exception but function not
found can be also indicated by Drill function error and so on. Not to be engaged into enumerating
possible exceptions (which can added in the code later), we are checking if remote and local
function registries are in sync on any error. Such check may only affect on queries that will
fail anyway. So if failure time will take a little bit longer, it won't make significant difference.
    3. During execution stage Drill attempts to find matching function from the list of functions
with the same names. `DefaultFunctionResolver.getBestMatch()` does not do exact match, it
may return function with different input parameters types. Best match is found according to
rules described in `TypeCastRules.class`. Currently we attempt to sync remote and local function
registries only if best matching function was found but it is not correct since even if Drill
finds the best matching function among current functions but does not mean that remote function
registry does not hold even better matching function. To fix this issue we would first try
to find function using `ExactFunctionResolver.getBestMatch()` and if exactly matching function
is not found, we'll check if remote and local function registries are in sync and then use
`DefaultFunctionResolver.getBestMatch()` to find the best matching function. But if exactly
matching function is found, we'll return it right away without
  any registries sync checks.
    1. Add `consists` method to PersistentStore interface which can return true if key exists
in store, false otherwise. This method is needed to return only remote function registry version
without its content (unlike method `get`). We'll pull remote function registry content only
if versions are different.
    2. Added check if remote and local function registries are in sync on any failure during
planning stage
    and on exact matching function not found during execution stage.
    3. Added additional debug messages for `CreateFunctionHandler` and `DropFunctionHandler`.
    4. Updated unit tests to reflect new changes.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message