spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mobcdi <>
Subject Apache Spark Video Processing from NFS Shared storage: Advise needed
Date Thu, 26 May 2016 13:53:04 GMT
Hi all,

Is it advisable to use nfs as shared storage for a small Spark cluster to
process video and images? I have a total of 20 vms (2vCPU, 6GB Ram, 20GB
Local Disk) connected to 500GB nfs shared storage (mounted the same in each
of the vms) at my disposal and I'm wondering if I can avoid the need for
hdfs and instead use the larger capacity nfs to work with my videos and
images in Spark?

I have spun up a master node (using maven not sbt) and connected 1 slave to
it but I haven't made any configuration changes to Spark. On
masternode:4040/ I don't see anything under storage. Is that to be expected
and if I do need to spin up hadoop can I double job the 20vms by running
both hadoop and spark on all 20 machines or would the recommendation be I
split them into separate hadoop and spark clusters 


View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message