spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shannon Quinn <squ...@gatech.edu>
Subject Re: Pyspark Error
Date Tue, 18 Nov 2014 18:42:33 GMT
My best guess would be a networking issue--it looks like the Python 
socket library isn't able to connect to whatever hostname you're 
providing Spark in the configuration.

On 11/18/14 9:10 AM, amin mohebbi wrote:
> Hi there,
>
> *I have already downloaded Pre-built spark-1.1.0, I want to run 
> pyspark by try typing ./bin/pyspark but I got the following error:*
> *
> *
>
>
>
>
>
>
>
> *scala shell is up and working fine*
>
> hduser@master:~/Downloads/spark-1.1.0$ ./bin/spark-shell
> Java HotSpot(TM) Client VM warning: ignoring option MaxPermSize=128m; 
> support was removed in 8.0
> Using Spark's default log4j profile: 
> org/apache/spark/log4j-defaults.properties
> .....
> .....
> 14/11/18 04:33:13 INFO AkkaUtils: Connecting to HeartbeatReceiver: 
> akka.tcp://sparkDriver@master:34937/user/HeartbeatReceiver
> 14/11/18 04:33:13 INFO SparkILoop: Created spark context..
> Spark context available as sc.
>
> scala> hduser@master:~/Downloads/spark-1.1.0$
>
>
> *
> *
> *But python shell does not work:*
>
> hduser@master:~/Downloads/spark-1.1.0$
> hduser@master:~/Downloads/spark-1.1.0$
> hduser@master:~/Downloads/spark-1.1.0$ ./bin/pyspark
> Python 2.7.3 (default, Feb 27 2014, 20:00:17)
> [GCC 4.6.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> Java HotSpot(TM) Client VM warning: ignoring option MaxPermSize=128m; 
> support was removed in 8.0
> Using Spark's default log4j profile: 
> org/apache/spark/log4j-defaults.properties
> 14/11/18 04:36:06 INFO SecurityManager: Changing view acls to: hduser,
> 14/11/18 04:36:06 INFO SecurityManager: Changing modify acls to: hduser,
> 14/11/18 04:36:06 INFO SecurityManager: SecurityManager: 
> authentication disabled; ui acls disabled; users with view 
> permissions: Set(hduser, ); users with modify permissions: Set(hduser, )
> 14/11/18 04:36:06 INFO Slf4jLogger: Slf4jLogger started
> 14/11/18 04:36:06 INFO Remoting: Starting remoting
> 14/11/18 04:36:06 INFO Remoting: Remoting started; listening on 
> addresses :[akka.tcp://sparkDriver@master:52317]
> 14/11/18 04:36:06 INFO Remoting: Remoting now listens on addresses: 
> [akka.tcp://sparkDriver@master:52317]
> 14/11/18 04:36:06 INFO Utils: Successfully started service 
> 'sparkDriver' on port 52317.
> 14/11/18 04:36:06 INFO SparkEnv: Registering MapOutputTracker
> 14/11/18 04:36:06 INFO SparkEnv: Registering BlockManagerMaster
> 14/11/18 04:36:06 INFO DiskBlockManager: Created local directory at 
> /tmp/spark-local-20141118043606-c346
> 14/11/18 04:36:07 INFO Utils: Successfully started service 'Connection 
> manager for block manager' on port 47507.
> 14/11/18 04:36:07 INFO ConnectionManager: Bound socket to port 47507 
> with id = ConnectionManagerId(master,47507)
> 14/11/18 04:36:07 INFO MemoryStore: MemoryStore started with capacity 
> 267.3 MB
> 14/11/18 04:36:07 INFO BlockManagerMaster: Trying to register BlockManager
> 14/11/18 04:36:07 INFO BlockManagerMasterActor: Registering block 
> manager master:47507 with 267.3 MB RAM
> 14/11/18 04:36:07 INFO BlockManagerMaster: Registered BlockManager
> 14/11/18 04:36:07 INFO HttpFileServer: HTTP File server directory is 
> /tmp/spark-8b29544a-c74b-4a3e-88e0-13801c8dcc65
> 14/11/18 04:36:07 INFO HttpServer: Starting HTTP Server
> 14/11/18 04:36:07 INFO Utils: Successfully started service 'HTTP file 
> server' on port 40029.
> 14/11/18 04:36:12 INFO Utils: Successfully started service 'SparkUI' 
> on port 4040.
> 14/11/18 04:36:12 INFO SparkUI: Started SparkUI at http://master:4040 
> <http://master:4040/>
> 14/11/18 04:36:12 INFO AkkaUtils: Connecting to HeartbeatReceiver: 
> akka.tcp://sparkDriver@master:52317/user/HeartbeatReceiver
> 14/11/18 04:36:12 INFO SparkUI: Stopped Spark web UI at 
> http://master:4040 <http://master:4040/>
> 14/11/18 04:36:12 INFO DAGScheduler: Stopping DAGScheduler
> 14/11/18 04:36:13 INFO MapOutputTrackerMasterActor: 
> MapOutputTrackerActor stopped!
> 14/11/18 04:36:13 INFO ConnectionManager: Selector thread was interrupted!
> 14/11/18 04:36:13 INFO ConnectionManager: ConnectionManager stopped
> 14/11/18 04:36:13 INFO MemoryStore: MemoryStore cleared
> 14/11/18 04:36:13 INFO BlockManager: BlockManager stopped
> 14/11/18 04:36:13 INFO BlockManagerMaster: BlockManagerMaster stopped
> 14/11/18 04:36:13 INFO RemoteActorRefProvider$RemotingTerminator: 
> Shutting down remote daemon.
> 14/11/18 04:36:13 INFO SparkContext: Successfully stopped SparkContext
> 14/11/18 04:36:13 INFO RemoteActorRefProvider$RemotingTerminator: 
> Remote daemon shut down; proceeding with flushing remote transports.
> 14/11/18 04:36:13 INFO Remoting: Remoting shut down
> 14/11/18 04:36:13 INFO RemoteActorRefProvider$RemotingTerminator: 
> Remoting shut down.
> Traceback (most recent call last):
>   File "/home/hduser/Downloads/spark-1.1.0/python/pyspark/shell.py", 
> line 44, in <module>
>     sc = SparkContext(appName="PySparkShell", pyFiles=add_files)
>   File "/home/hduser/Downloads/spark-1.1.0/python/pyspark/context.py", 
> line 107, in __init__
>     conf)
>   File "/home/hduser/Downloads/spark-1.1.0/python/pyspark/context.py", 
> line 159, in _do_init
>     self._accumulatorServer = accumulators._start_update_server()
>   File 
> "/home/hduser/Downloads/spark-1.1.0/python/pyspark/accumulators.py", 
> line 251, in _start_update_server
>     server = AccumulatorServer(("localhost", 0), _UpdateRequestHandler)
>   File "/usr/lib/python2.7/SocketServer.py", line 408, in __init__
>     self.server_bind()
>   File "/usr/lib/python2.7/SocketServer.py", line 419, in server_bind
>     self.socket.bind(self.server_address)
>   File "/usr/lib/python2.7/socket.py", line 224, in meth
>     return getattr(self._sock,name)(*args)
> socket.gaierror: [Errno -5] No address associated with hostname
> >>> sc.parallelize(range(1000)).count()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> NameError: name 'sc' is not defined
> >>> sc
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> NameError: name 'sc' is not defined
> >>> spark
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> NameError: name 'spark' is not defined
> >>>
>
> Best Regards
>
> .......................................................
>
> Amin Mohebbi
>
> PhD candidate in Software Engineering
>  at university of Malaysia
>
> Tel : +60 18 2040 017
>
>
>
> E-Mail : TP025921@ex.apiit.edu.my
>
>               amin_524@me.com


Mime
View raw message