spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: Spark 2.0 regression when querying very wide data frames
Date Sat, 20 Aug 2016 08:45:10 GMT
Yes, have a look through JIRA in cases like this.
https://issues.apache.org/jira/browse/SPARK-16664

On Sat, Aug 20, 2016 at 1:57 AM, mhornbech <morten@datasolvr.com> wrote:
> I did some extra digging. Running the query "select column1 from myTable" I
> can reproduce the problem on a frame with a single row - it occurs exactly
> when the frame has more than 200 columns, which smells a bit like a
> hardcoded limit.
>
> Interestingly the problem disappears when replacing the query with "select
> column1 from myTable limit N" where N is arbitrary. However it appears again
> when running "select * from myTable limit N" with sufficiently many columns
> (haven't determined the exact threshold here).
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-2-0-regression-when-querying-very-wide-data-frames-tp27567p27568.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message