spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anwar AliKhan <anwaralikhan...@gmail.com>
Subject Re: Hey good looking toPandas ()
Date Fri, 19 Jun 2020 07:15:02 GMT
I got an illegal argument error with 2.4.6.

I then pointed my Jupiter notebook  to 3.0 version and it worked as
expected.
Using same .ipnyb file.

I was following this machine learning example.
“Your First Apache Spark ML Model” by Favio Vázquez
https://towardsdatascience.com/your-first-apache-spark-ml-model-d2bb82b599dd


In the example he is using version 3.0 so I assumed I got the error because
I am using different version (2.4.6).



On Fri, 19 Jun 2020, 08:06 Stephen Boesch, <javadba@gmail.com> wrote:

> afaik It has been there since  Spark 2.0 in 2015.   Not certain about
> Spark 1.5/1.6
>
> On Thu, 18 Jun 2020 at 23:56, Anwar AliKhan <anwaralikhanuae@gmail.com>
> wrote:
>
>> I first ran the  command
>> df.show()
>>
>> For sanity check of my dataFrame.
>>
>> I wasn't impressed with the display.
>>
>> I then ran
>> df.toPandas() in Jupiter Notebook.
>>
>> Now the display is really good looking .
>>
>> Is toPandas() a new function which became available in Spark 3.0 ?
>>
>>
>>
>>
>>
>>

Mime
View raw message