spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <>
Subject [jira] [Assigned] (SPARK-26930) Tests in ParquetFilterSuite don't verify filter class
Date Fri, 22 Feb 2019 06:10:00 GMT


Hyukjin Kwon reassigned SPARK-26930:

    Assignee: Nandor Kollar

> Tests in ParquetFilterSuite don't verify filter class
> -----------------------------------------------------
>                 Key: SPARK-26930
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.4.0
>            Reporter: Nandor Kollar
>            Assignee: Nandor Kollar
>            Priority: Minor
> While investigating Parquet predicate pushdown test cases, I noticed that several tests
seems to be broken, they don't test what they were originally intended to. Most of the verification
ends up in one of the overloaded checkFilterPredicate functions, which supposed to test if
a given filter class is generated or not with this call: {{maybeFilter.exists(_.getClass ===
filterClass)}}, but on one side an assert is missing from here, on the other side, the filters
are more complicated, for example equality is checked with an 'and' wrapping not null check
along with an equality check for the given value. 'Exists' function call won't help with these
compounds filters, since they are not collection instances.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message