spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piotr Kołaczkowski (JIRA) <j...@apache.org>
Subject [jira] [Commented] (SPARK-1712) ParallelCollectionRDD operations hanging forever without any error messages
Date Tue, 06 May 2014 08:22:15 GMT

    [ https://issues.apache.org/jira/browse/SPARK-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13990417#comment-13990417
] 

Piotr Kołaczkowski commented on SPARK-1712:
-------------------------------------------

I modified log4j config and this is what I got in the spark.log:
{noformat}
14/05/06 10:17:29 INFO HttpServer: Starting HTTP Server
14/05/06 10:17:33 WARN Utils: Your hostname, m4600 resolves to a loopback address: 127.0.0.2;
using 192.168.122.1 instead (on interface virbr0)
14/05/06 10:17:33 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
14/05/06 10:17:34 INFO Slf4jLogger: Slf4jLogger started
14/05/06 10:17:34 INFO Remoting: Starting remoting
14/05/06 10:17:34 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@m4600.local:33012]
14/05/06 10:17:34 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@m4600.local:33012]
14/05/06 10:17:34 INFO SparkEnv: Registering BlockManagerMaster
14/05/06 10:17:34 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20140506101734-42f3
14/05/06 10:17:34 INFO MemoryStore: MemoryStore started with capacity 1178.1 MB.
14/05/06 10:17:34 INFO ConnectionManager: Bound socket to port 60842 with id = ConnectionManagerId(m4600.local,60842)
14/05/06 10:17:34 INFO BlockManagerMaster: Trying to register BlockManager
14/05/06 10:17:34 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager
m4600.local:60842 with 1178.1 MB RAM
14/05/06 10:17:34 INFO BlockManagerMaster: Registered BlockManager
14/05/06 10:17:34 INFO HttpServer: Starting HTTP Server
14/05/06 10:17:34 INFO HttpBroadcast: Broadcast server started at http://192.168.122.1:51030
14/05/06 10:17:35 INFO SparkEnv: Registering MapOutputTracker
14/05/06 10:17:35 INFO HttpFileServer: HTTP File server directory is /tmp/spark-5013023c-f851-4398-a344-5493e62edd26
14/05/06 10:17:35 INFO HttpServer: Starting HTTP Server
14/05/06 10:17:35 INFO SparkUI: Started Spark Web UI at http://m4600.local:4040
14/05/06 10:17:35 INFO SharkContext: Added JAR /home/pkolaczk/.spark/cassandra-context/spark-cassandra-context.jar
at http://192.168.122.1:49386/jars/spark-cassandra-context.jar with timestamp 1399364255576
14/05/06 10:17:35 INFO AppClient$ClientActor: Connecting to master spark://127.0.0.1:7077...
14/05/06 10:17:35 INFO Master: Registering app Spark shell
14/05/06 10:17:35 INFO Master: Registered app Spark shell with ID app-20140506101735-0001
14/05/06 10:17:35 INFO Master: Launching executor app-20140506101735-0001/0 on worker worker-20140506101633-127.0.0.1-44566
14/05/06 10:17:35 INFO Worker: Asked to launch executor app-20140506101735-0001/0 for Spark
shell
14/05/06 10:17:35 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID
app-20140506101735-0001
14/05/06 10:17:35 INFO AppClient$ClientActor: Executor added: app-20140506101735-0001/0 on
worker-20140506101633-127.0.0.1-44566 (127.0.0.1:44566) with 1 cores
14/05/06 10:17:35 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140506101735-0001/0
on hostPort 127.0.0.1:44566 with 1 cores, 2.0 GB RAM
14/05/06 10:17:35 INFO AppClient$ClientActor: Executor updated: app-20140506101735-0001/0
is now RUNNING
14/05/06 10:17:36 INFO ExecutorRunner: Launch command: "/opt/jdk/bin/java" "-cp" ":/home/pkolaczk/Projekty/datastax/bdp/build/dse-4.5.0-SNAPSHOT.jar:/home/pkolaczk/Projekty/datastax/bdp/build/maven-ant-tasks-2.1.3.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/cassandra-driver-core-2.0.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/commons-codec-1.6.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/commons-io-2.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/guava-15.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/HdrHistogram-1.0.9.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/java-uuid-generator-3.1.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/jbcrypt-0.3m.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/jline-1.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/jna-3.4.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/journalio-1.4.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/log4j-1.2.17.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/metrics-core-3.0.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/netty-3.9.0.Final.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/netty-all-4.0.13.Final.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/slf4j-api-1.7.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/slf4j-log4j12-1.7.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/conf:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/conf:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/conf:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/tools/lib/stress.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/antlr-2.7.7.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/antlr-3.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/antlr-runtime-3.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/cassandra-all-2.0.7.31.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/cassandra-clientutil-2.0.7.31.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/cassandra-thrift-2.0.7.31.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/commons-cli-1.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/commons-codec-1.6.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/commons-lang-2.6.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/commons-lang3-3.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/commons-logging-1.1.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/compress-lzf-0.8.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/disruptor-3.0.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/elephant-bird-hadoop-compat-4.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/guava-15.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/hibernate-validator-4.3.0.Final.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/high-scale-lib-1.1.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/httpclient-4.2.5.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/httpcore-4.2.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/jackson-core-asl-1.9.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/jamm-0.2.5.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/jbcrypt-0.3m.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/joda-time-1.6.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/json-simple-1.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/libthrift-0.9.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/log4j-1.2.16.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/lz4-1.2.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/metrics-core-2.2.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/netty-3.6.6.Final.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/reporter-config-2.1.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/slf4j-api-1.7.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/snakeyaml-1.11.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/snappy-java-1.0.5.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/snaptree-0.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/stringtemplate-3.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/super-csv-2.1.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/thrift-server-0.3.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/validation-api-1.0.0.GA.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/../driver/lib/cassandra-driver-core-2.0.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/../driver/lib/cassandra-driver-dse-2.0.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/../driver/lib/metrics-core-3.0.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/../driver/lib/netty-3.9.0.Final.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/../driver/lib/slf4j-api-1.7.5.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/slf4j-api-1.7.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/dse/lib/slf4j-log4j12-1.7.2.jar:::/home/pkolaczk/Projekty/datastax/bdp/resources/spark/conf:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/activation-1.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/akka-actor_2.10-2.2.3-shaded-protobuf.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/akka-remote_2.10-2.2.3-shaded-protobuf.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/akka-slf4j_2.10-2.2.3-shaded-protobuf.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/akka-zeromq_2.10-2.2.3-shaded-protobuf.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/algebird-core_2.10-0.1.11.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/asm-4.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/asm-commons-4.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/asm-tree-4.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/avro-1.7.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/avro-ipc-1.7.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/calliope_2.10-0.9.0-EA.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/chill_2.10-0.3.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/chill-java-0.3.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/colt-1.2.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/commons-beanutils-1.7.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/commons-beanutils-core-1.8.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/commons-cli-1.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/commons-codec-1.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/commons-collections-3.2.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/commons-compress-1.4.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/commons-configuration-1.6.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/commons-digester-1.8.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/commons-el-1.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/commons-httpclient-3.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/commons-io-2.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/commons-lang-2.5.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/commons-logging-1.1.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/compress-lzf-1.0.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/concurrent-1.3.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/config-1.0.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/core-3.1.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/fastutil-6.4.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/flume-ng-sdk-1.2.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/gmetric4j-1.0.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/guava-14.0.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/hadoop-client-1.0.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/hbase-0.94.6.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/high-scale-lib-1.1.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/httpclient-4.1.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/httpcore-4.1.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jackson-annotations-2.2.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jackson-core-2.2.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jackson-core-asl-1.8.8.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jackson-databind-2.2.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jackson-jaxrs-1.8.8.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jackson-mapper-asl-1.8.8.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jackson-xc-1.8.8.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jamon-runtime-2.3.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jansi-1.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jasper-compiler-5.5.23.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jasper-runtime-5.5.23.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/JavaEWAH-0.6.6.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/java-xmlbuilder-0.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/javax.servlet-2.5.0.v201103041518.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jaxb-api-2.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jaxb-impl-2.2.3-1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jblas-1.2.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jersey-core-1.8.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jersey-json-1.8.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jersey-server-1.8.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jets3t-0.9.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jettison-1.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jetty-6.1.26.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jetty-continuation-7.6.8.v20121106.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jetty-http-7.6.8.v20121106.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jetty-io-7.6.8.v20121106.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jetty-server-7.6.8.v20121106.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jetty-util-6.1.26.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jetty-util-7.6.8.v20121106.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jline-2.10.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jnr-constants-0.8.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jruby-complete-1.6.5.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jsp-2.1-6.1.14.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jsp-api-2.1-6.1.14.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jsr305-1.3.9.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/jul-to-slf4j-1.7.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/kafka_2.10-0.8.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/kryo-2.21.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/libthrift-0.7.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/lift-json_2.10-2.5.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/mesos-0.13.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/metrics-annotation-2.2.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/metrics-core-2.1.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/metrics-core-3.0.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/metrics-ganglia-3.0.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/metrics-graphite-3.0.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/metrics-json-3.0.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/metrics-jvm-3.0.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/minlog-1.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/mqtt-client-0.4.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/netty-3.5.9.Final.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/netty-all-4.0.13.Final.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/objenesis-1.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/oncrpc-1.0.7.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/paranamer-2.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/protobuf-java-2.4.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/protobuf-java-2.4.1-shaded.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/reflectasm-1.07-shaded.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/scala-compiler-2.10.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/scala-library-2.10.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/scala-reflect-2.10.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/servlet-api-2.5-20081211.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/servlet-api-2.5-6.1.14.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/spark-bagel_2.10-0.9.0-incubating.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/spark-core_2.10-0.9.0-incubating.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/spark-examples_2.10-0.9.0-incubating.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/spark-mllib_2.10-0.9.0-incubating.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/spark-repl_2.10-0.9.0-incubating.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/spark-streaming_2.10-0.9.0-incubating.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/spark-streaming-flume_2.10-0.9.0-incubating.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/spark-streaming-kafka_2.10-0.9.0-incubating.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/spark-streaming-mqtt_2.10-0.9.0-incubating.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/spark-streaming-twitter_2.10-0.9.0-incubating.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/spark-streaming-zeromq_2.10-0.9.0-incubating.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/stax-api-1.0.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/stream-2.4.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/twitter4j-core-3.0.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/twitter4j-stream-3.0.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/uncommons-maths-1.2.2a.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/velocity-1.7.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/xz-1.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/zeromq-scala-binding_2.10-0.0.7.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/zkclient-0.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/lib/zookeeper-3.4.5.jar:/home/pkolaczk/.spark/cassandra-context/spark-cassandra-context.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/spark/conf::/home/pkolaczk/Projekty/datastax/bdp/resources/shark/lib/commons-codec-1.6.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/shark/lib/commons-httpclient-3.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/shark/lib/commons-io-2.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/shark/lib/commons-logging-1.1.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/shark/lib/guava-14.0.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/shark/lib/hadoop-client-1.0.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/shark/lib/httpclient-4.3.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/shark/lib/httpcore-4.3.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/shark/lib/javax.servlet-2.5.0.v201103041518.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/shark/lib/jets3t-0.7.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/shark/lib/shark_2.10-0.9.0.1-DSP-3062-SNAPSHOT.jar::/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/conf:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/elephant-bird-hadoop-compat-4.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/hadoop-core-1.0.4.10.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/hadoop-examples-1.0.4.10.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/hadoop-fairscheduler-1.0.4.10.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/hadoop-streaming-1.0.4.10.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/hadoop-test-1.0.4.10.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/hadoop-tools-1.0.4.10.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/ant-1.6.5.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/automaton-1.11-8.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/commons-beanutils-1.7.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/commons-beanutils-core-1.8.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/commons-cli-1.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/commons-codec-1.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/commons-collections-3.2.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/commons-configuration-1.6.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/commons-digester-1.8.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/commons-el-1.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/commons-httpclient-3.0.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/commons-lang-2.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/commons-logging-1.1.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/commons-math-2.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/commons-net-1.4.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/core-3.1.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/ftplet-api-1.0.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/ftpserver-core-1.0.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/ftpserver-deprecated-1.0.0-M2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/hsqldb-1.8.0.10.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/httpclient-4.1.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/httpcore-4.1.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/jackson-core-asl-1.8.8.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/jasper-compiler-5.5.12.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/jasper-runtime-5.5.12.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/java-xmlbuilder-0.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/jets3t-0.9.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/jetty-6.1.26.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/jetty-util-6.1.26.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/jsp-2.1-6.1.14.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/jsp-api-2.1-6.1.14.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/kfs-0.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/mina-core-2.0.0-M5.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/oro-2.0.8.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/servlet-api-2.5-20081211.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/servlet-api-2.5-6.1.14.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/snappy-java-1.0.5.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/lib/xmlenc-0.52.jar::/home/pkolaczk/Projekty/datastax/bdp/build/dse-4.5.0-SNAPSHOT.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/antlr-runtime-3.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/conf::/home/pkolaczk/Projekty/datastax/bdp/build/dse-4.5.0-SNAPSHOT.jar:/home/pkolaczk/Projekty/datastax/bdp/build/dse-4.5.0-SNAPSHOT.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/cassandra/lib/antlr-runtime-3.2.jar::/home/pkolaczk/Projekty/datastax/bdp/resources/hive/conf:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/antlr-2.7.7.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/antlr-3.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/antlr-runtime-3.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/asm-4.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/avro-1.7.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/avro-ipc-1.7.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/avro-mapred-1.7.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/bonecp-0.7.1.RELEASE.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/commons-beanutils-1.7.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/commons-beanutils-core-1.8.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/commons-cli-1.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/commons-codec-1.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/commons-collections-3.2.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/commons-compress-1.4.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/commons-configuration-1.6.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/commons-digester-1.8.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/commons-io-2.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/commons-lang-2.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/commons-lang3-3.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/commons-logging-1.1.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/commons-logging-api-1.0.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/commons-pool-1.5.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/datanucleus-api-jdo-3.2.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/datanucleus-core-3.2.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/datanucleus-rdbms-3.2.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/derby-10.4.2.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/guava-15.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/hive-cli-0.12.0.3-20140319.091653-2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/hive-common-0.12.0.3-20140319.091659-2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/hive-exec-0.12.0.3-20140319.091716-2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/hive-hwi-0.12.0.3-20140319.091745-2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/hive-jdbc-0.12.0.3-20140319.091754-2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/hive-metastore-0.12.0.3-20140319.091801-2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/hive-serde-0.12.0.3-20140319.091811-2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/hive-service-0.12.0.3-20140319.091817-2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/hive-shims-0.12.0.3-20140319.091825-2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/httpclient-4.2.5.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/httpcore-4.2.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/jackson-core-asl-1.8.8.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/jackson-mapper-asl-1.8.8.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/JavaEWAH-0.3.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/java-xmlbuilder-0.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/javolution-5.5.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/jdo-api-3.0.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/jets3t-0.9.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/jetty-util-6.1.26.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/json-20090211.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/jta-1.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/libfb303-0.9.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/libthrift-0.9.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/log4j-1.2.16.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/netty-3.5.9.Final.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/paranamer-2.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/protobuf-java-2.4.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/servlet-api-2.5-20081211.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/slf4j-api-1.6.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/snappy-0.2.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/snappy-java-1.0.5.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/ST4-4.0.4.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/stringtemplate-3.2.1.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/velocity-1.7.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/xz-1.0.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/lib/zookeeper-3.4.3.jar:/home/pkolaczk/Projekty/datastax/bdp/resources/hive/conf"
"-Djava.library.path=:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/native/Linux-amd64-64/lib"
"-Djava.system.class.loader=com.datastax.bdp.loader.DseClientClassLoader" "-XX:MaxPermSize=256M"
"-Djava.library.path=:/home/pkolaczk/Projekty/datastax/bdp/resources/hadoop/native/Linux-amd64-64/lib"
"-Djava.system.class.loader=com.datastax.bdp.loader.DseClientClassLoader" "-XX:MaxPermSize=256M"
"-Xms2048M" "-Xmx2048M" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "akka.tcp://spark@m4600.local:33012/user/CoarseGrainedScheduler"
"0" "127.0.0.1" "1" "akka.tcp://sparkWorker@127.0.0.1:44566/user/Worker" "app-20140506101735-0001"
14/05/06 10:17:37 INFO Slf4jLogger: Slf4jLogger started
14/05/06 10:17:37 INFO Remoting: Starting remoting
14/05/06 10:17:38 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutor@127.0.0.1:39919]
14/05/06 10:17:38 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkExecutor@127.0.0.1:39919]
14/05/06 10:17:38 INFO CoarseGrainedExecutorBackend: Connecting to driver: akka.tcp://spark@m4600.local:33012/user/CoarseGrainedScheduler
14/05/06 10:17:38 INFO WorkerWatcher: Connecting to worker akka.tcp://sparkWorker@127.0.0.1:44566/user/Worker
14/05/06 10:17:38 INFO WorkerWatcher: Successfully connected to akka.tcp://sparkWorker@127.0.0.1:44566/user/Worker
14/05/06 10:17:38 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@127.0.0.1:39919/user/Executor#-1863004534]
with ID 0
14/05/06 10:17:38 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
14/05/06 10:17:38 INFO Executor: Using REPL class URI: http://192.168.122.1:58332
14/05/06 10:17:38 INFO Slf4jLogger: Slf4jLogger started
14/05/06 10:17:38 INFO Remoting: Starting remoting
14/05/06 10:17:38 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@127.0.0.1:49111]
14/05/06 10:17:38 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@127.0.0.1:49111]
14/05/06 10:17:38 INFO SparkEnv: Connecting to BlockManagerMaster: akka.tcp://spark@m4600.local:33012/user/BlockManagerMaster
14/05/06 10:17:38 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20140506101738-4884
14/05/06 10:17:38 INFO MemoryStore: MemoryStore started with capacity 1178.1 MB.
14/05/06 10:17:38 INFO ConnectionManager: Bound socket to port 32898 with id = ConnectionManagerId(127.0.0.1,32898)
14/05/06 10:17:38 INFO BlockManagerMaster: Trying to register BlockManager
14/05/06 10:17:38 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager
127.0.0.1:32898 with 1178.1 MB RAM
14/05/06 10:17:38 INFO BlockManagerMaster: Registered BlockManager
14/05/06 10:17:38 INFO SparkEnv: Connecting to MapOutputTracker: akka.tcp://spark@m4600.local:33012/user/MapOutputTracker
14/05/06 10:17:38 INFO HttpFileServer: HTTP File server directory is /tmp/spark-ed849647-1ea9-4446-9798-326ddf33c8da
14/05/06 10:17:38 INFO HttpServer: Starting HTTP Server
14/05/06 10:17:38 WARN Utils: Your hostname, m4600 resolves to a loopback address: 127.0.0.2;
using 192.168.122.1 instead (on interface virbr0)
14/05/06 10:17:38 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
14/05/06 10:17:57 INFO SharkContext: Starting job: sum at <console>:24
14/05/06 10:17:57 INFO DAGScheduler: Got job 0 (sum at <console>:24) with 2 output partitions
(allowLocal=false)
14/05/06 10:17:57 INFO DAGScheduler: Final stage: Stage 0 (sum at <console>:24)
14/05/06 10:17:57 INFO DAGScheduler: Parents of final stage: List()
14/05/06 10:17:57 INFO DAGScheduler: Missing parents: List()
14/05/06 10:17:57 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[2] at numericRDDToDoubleRDDFunctions
at <console>:24), which has no missing parents
14/05/06 10:17:58 INFO DAGScheduler: Submitting 2 missing tasks from Stage 0 (MappedRDD[2]
at numericRDDToDoubleRDDFunctions at <console>:24)
14/05/06 10:17:58 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
14/05/06 10:17:58 INFO TaskSetManager: Starting task 0.0:0 as TID 0 on executor 0: 127.0.0.1
(PROCESS_LOCAL)
14/05/06 10:17:59 INFO TaskSetManager: Serialized task 0.0:0 as 13890653 bytes in 615 ms
{noformat}


> ParallelCollectionRDD operations hanging forever without any error messages 
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-1712
>                 URL: https://issues.apache.org/jira/browse/SPARK-1712
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 0.9.0
>         Environment: Linux Ubuntu 14.04, a single spark node; standalone mode.
>            Reporter: Piotr Kołaczkowski
>            Priority: Blocker
>         Attachments: executor.jstack.txt, master.jstack.txt, repl.jstack.txt, spark-hang.png,
worker.jstack.txt
>
>
> {noformat}
> scala> val collection = (1 to 1000000).map(i => ("foo" + i, i)).toVector
> collection: Vector[(String, Int)] = Vector((foo1,1), (foo2,2), (foo3,3), (foo4,4), (foo5,5),
(foo6,6), (foo7,7), (foo8,8), (foo9,9), (foo10,10), (foo11,11), (foo12,12), (foo13,13), (foo14,14),
(foo15,15), (foo16,16), (foo17,17), (foo18,18), (foo19,19), (foo20,20), (foo21,21), (foo22,22),
(foo23,23), (foo24,24), (foo25,25), (foo26,26), (foo27,27), (foo28,28), (foo29,29), (foo30,30),
(foo31,31), (foo32,32), (foo33,33), (foo34,34), (foo35,35), (foo36,36), (foo37,37), (foo38,38),
(foo39,39), (foo40,40), (foo41,41), (foo42,42), (foo43,43), (foo44,44), (foo45,45), (foo46,46),
(foo47,47), (foo48,48), (foo49,49), (foo50,50), (foo51,51), (foo52,52), (foo53,53), (foo54,54),
(foo55,55), (foo56,56), (foo57,57), (foo58,58), (foo59,59), (foo60,60), (foo61,61), (foo62,62),
(foo63,63), (foo64,64), (foo...
> scala> val rdd = sc.parallelize(collection)
> rdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[0] at parallelize
at <console>:24
> scala> rdd.first
> res4: (String, Int) = (foo1,1)
> scala> rdd.map(_._2).sum
> // nothing happens
> {noformat}
> CPU and I/O idle. 
> Memory usage reported by JVM, after manually triggered GC:
> repl: 216 MB / 2 GB
> executor: 67 MB / 2 GB
> worker: 6 MB / 128 MB
> master: 6 MB / 128 MB
> No errors found in worker's stderr/stdout. 
> It works fine with 700,000 elements and then it takes about 1 second to process the request
and calculate the sum. With 700,000 items the spark executor memory doesn't even exceed 300
MB out of 2GB available. It fails with 800,000 items.
> Multiple parralelized collections of size 700,000 items at the same time in the same
session work fine.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message