spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ruslan Dautkhanov (JIRA)" <>
Subject [jira] [Commented] (SPARK-26019) pyspark/ "TypeError: object of type 'NoneType' has no len()" in authenticate_and_accum_updates()
Date Tue, 20 Nov 2018 06:12:00 GMT


Ruslan Dautkhanov commented on SPARK-26019:

[~viirya] exception stack reads that error happened in, BaseRequestHandler
class constructor, excerpt from the full exception stack above :

  File "/opt/cloudera/parcels/Anaconda/lib/python2.7/", line 652, in __init__

Notice constructor here calls `self.handle()`  -

`handle()` is defined in derived class _UpdateRequestHandler here
and expects `auth_token` to be set : - that's exactly
where exception happens. 

[~irashid] was right - those two lines have to be swapped.

[~hyukjin.kwon] that's odd you closed this jira, although I said it always reproduces for
me (100 % of times ), 
and even [posted reproducer here|!default.jspa?id=13197858&commentId=16692219].
[~saivarunvishal] also said it happens for him in SPARK-26113 and you closed that jira as
It seems not in line with - "Contributing Bug Reports".
Please let me know what I miss here.

I called out [~bersprockets] because we use Cloudera distribution of Spark and Cloudera has
a few patches on top of open-source Spark. 
I wanted to make sure it's not Cloudera distro specific. Also we worked with Bruce on several
other Spark issue and noticed here's in watchers list on this jira... Now I see that this
issue is not Cloudera specific though. 

> pyspark/ "TypeError: object of type 'NoneType' has no len()" in authenticate_and_accum_updates()
> ----------------------------------------------------------------------------------------------------------------
>                 Key: SPARK-26019
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 2.3.2, 2.4.0
>            Reporter: Ruslan Dautkhanov
>            Priority: Major
> Started happening after 2.3.1 -> 2.3.2 upgrade.
> {code:python}
> Exception happened during processing of request from ('', 43418)
> ----------------------------------------
> Traceback (most recent call last):
>   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/", line 290, in
>     self.process_request(request, client_address)
>   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/", line 318, in
>     self.finish_request(request, client_address)
>   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/", line 331, in
>     self.RequestHandlerClass(request, client_address, self)
>   File "/opt/cloudera/parcels/Anaconda/lib/python2.7/", line 652, in
>     self.handle()
>   File "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179/lib/spark2/python/lib/",
line 263, in handle
>     poll(authenticate_and_accum_updates)
>   File "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179/lib/spark2/python/lib/",
line 238, in poll
>     if func():
>   File "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179/lib/spark2/python/lib/",
line 251, in authenticate_and_accum_updates
>     received_token =
> TypeError: object of type 'NoneType' has no len()
> {code}
> Error happens here:
> The PySpark code was just running a simple pipeline of 
> binary_rdd = sc.binaryRecords(full_file_path, record_length).map(lambda .. )
> and then converting it to a dataframe and running a count on it.
> It seems error is flaky - on next rerun it didn't happen.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message