spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: spark git commit: [SPARK-15204][SQL] improve nullability inference for Aggregator
Date Tue, 05 Jul 2016 20:07:53 GMT
On Mon, Jul 4, 2016 at 6:14 AM,  <wenchen@apache.org> wrote:
> Repository: spark
> Updated Branches:
>   refs/heads/master 88134e736 -> 8cdb81fa8
>
>
> [SPARK-15204][SQL] improve nullability inference for Aggregator
>
> ## What changes were proposed in this pull request?
>
> TypedAggregateExpression sets nullable based on the schema of the outputEncoder
>
> ## How was this patch tested?
>
> Add test in DatasetAggregatorSuite
>
> Author: Koert Kuipers <koert@tresata.com>
...
> +    assert(ds1.select(typed.sum((i: Int) => i)).schema.head.nullable === false)
> +    val ds2 = Seq(AggData(1, "a"), AggData(2, "a")).toDS()
> +    assert(ds2.select(SeqAgg.toColumn).schema.head.nullable === true)
> +    val ds3 = sql("SELECT 'Some String' AS b, 1279869254 AS a").as[AggData]
> +    assert(ds3.select(NameAgg.toColumn).schema.head.nullable === true)

Why do we assert predicates? If it's true, it's true already (no need
to compare whether it's true or not). I'd vote to "fix" it.

Jacek

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message