drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Givre <cgi...@gmail.com>
Subject Re: Deprecation of BaseTestQuery FYI
Date Thu, 05 Jul 2018 15:39:41 GMT
Hi Timothy, 
I am still having issues with the unit tests for this particular function.  The code is below.
 This function takes the soundex values of two strings and if they are equal is supposed to
return true, if not returns false.  I do this by setting the value of a BitHolder to either
-1 or 0.   

The code for the function appears to work, however, when I think what is causing the error
here was the sounds_like function itself.  
If I run the query:
select *
from 
WHERE soundex( ‘brian’ ) = soundex(`first_name` )

This returns results

However, if I run the query 
SELECT * 
FROM <Data>
WEHRE sounds_like( ‘brian’, `first_name` )
I get no results.  However, if I run this query:

SELECT sounds_like( ‘brian’, `first_name` )
FROM cp..

I get trues and falses.  I’m not sure why this isn’t working properly and in the interests
of not holding up the works, I’m removing the sounds_like from the PR, but I’d like to
understand how to fix this. 


@FunctionTemplate(name = "sounds_like", scope = FunctionTemplate.FunctionScope.SIMPLE, nulls
= FunctionTemplate.NullHandling.NULL_IF_NULL)
public static class SoundsLikeFunction implements DrillSimpleFunc {

  @Param
  VarCharHolder rawInput1;

  @Param
  VarCharHolder rawInput2;

  @Output
  BitHolder out;

  @Override
  public void setup() {
  }

  @Override
  public void eval() {

    String input1 = org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(rawInput1.start,
rawInput1.end, rawInput1.buffer);
    String input2 = org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(rawInput2.start,
rawInput2.end, rawInput2.buffer);

    String soundex1 = new org.apache.commons.codec.language.Soundex().soundex(input1);
    String soundex2 = new org.apache.commons.codec.language.Soundex().soundex(input2);

    boolean result = soundex1.equals(soundex2);
    if (result) {
      out.value = -1;
    } else {
      out.value = 0;
    }
  }
}




jdbc:drill:zk=local> SELECT typeof(sounds_like( 'Steven', 'Stephen' )) AS steve FROM (VALUES(1));
Error: SYSTEM ERROR: IllegalArgumentException: Can not set org.apache.drill.exec.vector.complex.reader.FieldReader
field org.apache.drill.exec.expr.fn.impl.UnionFunctions$GetType.input to org.apache.drill.exec.expr.holders.BitHolder


[Error Id: f15a91b9-ae19-481a-b145-1024709547e9 on 10.1.9.176:31010] (state=,code=0)
0: jdbc:drill:zk=local> SELECT typeof(sounds_like( 'Steven', 'Stephen' ))  FROM (VALUES(1));
Error: SYSTEM ERROR: IllegalArgumentException: Can not set org.apache.drill.exec.vector.complex.reader.FieldReader
field org.apache.drill.exec.expr.fn.impl.UnionFunctions$GetType.input to org.apache.drill.exec.expr.holders.BitHolder


[Error Id: 870a2471-8ab7-4a0f-ab7f-9ff873ba767a on 10.1.9.176:31010] (state=,code=0)
0: jdbc:drill:zk=local> SELECT sounds_like( 'Steven', 'Stephen' )  FROM (VALUES(1));
+---------+
| EXPR$0  |
+---------+
| true    |
+---------+
1 row selected (0.689 seconds)
0: jdbc:drill:zk=local> SELECT sounds_like( 'Steven', 'bob' )  FROM (VALUES(1));
+---------+
| EXPR$0  |
+---------+
| false   |
+---------+
1 row selected (0.175 seconds)




> On Jun 28, 2018, at 19:46, Timothy Farkas <tfarkas@mapr.com> wrote:
> 
> Hi Charles,
> 
> So it is actually supported. Drill's boolean vector is BitVector.
> Internally bits are stored efficiently, but when you fetch a bit from the
> vector it becomes an int, -1 for true and 0 for false. So currently you can
> check this by using singletonInt and comparing against -1 and 0.
> 
> Here is a test snippet that does it. We should probably add some
> convenience methods specifically for booleans so this is not so confusing.
> 
> public class SimpleQueryTest extends ClusterTest {
>  @Rule
>  public final BaseDirTestWatcher baseDirTestWatcher = new BaseDirTestWatcher();
> 
>  @Before
>  public void start() throws Exception {
>    startCluster(new ClusterFixtureBuilder(baseDirTestWatcher));
>  }
> 
>  @Test
>  public void testBoolean() throws RpcException {
>    RowSet rowSet = this.queryBuilder().sql("select position_id,
> cast(false as boolean) from cp.`employee.json`").rowSet();
>    RowSetReader reader = rowSet.reader();
>    reader.next();
>    System.out.println(reader.column(1).scalar().getInt());
>    rowSet.clear();
> 
>    rowSet = this.queryBuilder().sql("select position_id, cast(true as
> boolean) from cp.`employee.json`").rowSet();
>    reader = rowSet.reader();
>    reader.next();
>    System.out.println(reader.column(1).scalar().getInt());
>    rowSet.clear();
>  }
> }
> 
> 
> 
> On Thu, Jun 28, 2018 at 4:09 PM, Timothy Farkas <tfarkas@mapr.com> wrote:
> 
>> Hi Charles,
>> 
>> Currently the RowSetReader doesn't support booleans, but it can be added.
>> I'll try to add it now, and see if it could be done quickly. I'll update
>> you on my progress.
>> 
>> Tim
>> 
>> On Thu, Jun 28, 2018 at 2:38 PM, Charles Givre <cgivre@gmail.com> wrote:
>> 
>>> Hi Tim,
>>> Could post some sample code as to how to test a SQL query that returns a
>>> Boolean?
>>> —C
>>> 
>>>> On Jun 28, 2018, at 17:30, Timothy Farkas <tfarkas@mapr.com> wrote:
>>>> 
>>>> - We would have to add a boolean column reader to ColumnAccessors and
>>> wire
>>>> it in and add a getBoolean method to ScalarReader.
>>>> 
>>>> - Your example should work as is, ClusterTest has a testBuilder method
>>>> that allows you to use the traditional test builder. Is there something
>>> not
>>>> working with the test builder?
>>>> 
>>>> Tim
>>>> 
>>>> 
>>>> On Thu, Jun 28, 2018 at 12:39 PM, Arina Yelchiyeva <
>>>> arina.yelchiyeva@gmail.com> wrote:
>>>> 
>>>>> Hi Tim,
>>>>> 
>>>>> it looks like deprecating BaseTestQuery was a little bit pre-mature.
>>>>> For example, from in this PR - https://urldefense.proofpoint.
>>> com/v2/url?u=https-3A__urldefense.proofpoint&d=DwIFaQ&c=cskd
>>> kSMqhcnjZxdQVpwTXg&r=4eQVr8zB8ZBff-yxTimdOQ&m=D9i95wxPKk0aPH
>>> ip2Rj6Xju0J151UakaSe6OnMGTW5s&s=z5tYyOUI_Cyx1cwhMEMfnCb-
>>> BDFaZkRaKQKwt_zl2HA&e=.
>>>>> com/v2/url?u=https-3A__github.com_apache_drill_pull_1331&d=DwIBaQ&c=
>>>>> cskdkSMqhcnjZxdQVpwTXg&r=4eQVr8zB8ZBff-yxTimdOQ&m=
>>>>> zoNJPdWKxMX9-jbR2bftzwkX-CSrihIbpCirhHM1kA0&s=_uxhA_
>>>>> qiMBTjbit6DDw-DvZNRGesfeA5g-FQjkl7f10&e= -
>>>>> Charles is trying to re-work  BaseTestQuery usage to ClusterTest.
>>>>> First, it did not contain getSigletonDouble method which Charles has
>>>>> implemented. Now he has troubles with implementing getSigletonBoolean
>>>>> method which might be due to reader limitations.
>>>>> Also I am not quite clear how we can verify columns names and multiple
>>>>> columns in the result.
>>>>> For example:
>>>>> 
>>>>> testBuilder()
>>>>>         .sqlQuery("select (mi || lname) as CONCATOperator, mi, lname,
>>>>> concat(mi, lname) as CONCAT from concatNull")
>>>>>         .ordered()
>>>>>         .baselineColumns("CONCATOperator", "mi", "lname", "CONCAT")
>>>>>         .baselineValues("A.Nowmer", "A.", "Nowmer", "A.Nowmer")
>>>>>         .baselineValues("I.Whelply", "I.", "Whelply", "I.Whelply")
>>>>>         .baselineValues(null, null, "Derry", "Derry")
>>>>>         .baselineValues("J.Spence", "J.", "Spence", "J.Spence")
>>>>>         .build().run();
>>>>> 
>>>>> Can you please suggest how this example can be re-written?
>>>>> 
>>>>> Kind regards,
>>>>> Arina
>>>>> 
>>>>> On Mon, Jun 25, 2018 at 11:10 PM Timothy Farkas <tfarkas@mapr.com>
>>> wrote:
>>>>> 
>>>>>> Hi All,
>>>>>> 
>>>>>> BaseTestQuery was deprecated a while ago. Keeping it short and sweet
>>> :),
>>>>> if
>>>>>> you want to use BaseTestQuery directly, don't. Use ClusterTest
>>> instead.
>>>>> If
>>>>>> you are using PlanTestBase for planner tests, continue to do so.
>>>>> Eventually
>>>>>> PlanTestBase will be changed to extend ClusterTest instead. There
is a
>>>>> JIRA
>>>>>> to track that issue https://urldefense.proofpoint.
>>> com/v2/url?u=https-3A__urldefense.proofpoint&d=DwIFaQ&c=cskd
>>> kSMqhcnjZxdQVpwTXg&r=4eQVr8zB8ZBff-yxTimdOQ&m=D9i95wxPKk0aPH
>>> ip2Rj6Xju0J151UakaSe6OnMGTW5s&s=z5tYyOUI_Cyx1cwhMEMfnCb-
>>> BDFaZkRaKQKwt_zl2HA&e=.
>>>>> com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-
>>>>> 2D6536&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=4eQVr8zB8ZBff-yxTimdOQ&m=
>>>>> zoNJPdWKxMX9-jbR2bftzwkX-CSrihIbpCirhHM1kA0&s=
>>>>> BPAlenAq0k1kjAz7fUYXyEQMaJM1IPOgmdeySMMY84U&e=.
>>>>>> 
>>>>>> Thanks,
>>>>>> Tim
>>>>>> 
>>>>> 
>>> 
>>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message