spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Resolved] (SPARK-1526) Running spark driver program from my local machine
Date Fri, 23 Jan 2015 12:36:34 GMT


Sean Owen resolved SPARK-1526.
    Resolution: Not a Problem

> Running spark driver program from my local machine
> --------------------------------------------------
>                 Key: SPARK-1526
>                 URL:
>             Project: Spark
>          Issue Type: Wish
>          Components: Spark Core
>            Reporter: Idan Zalzberg
> Currently it seems that the design choice is that the driver program should be close
network-wise to the worker and allow connections to be created from either side.
> This makes using Spark somewhat harder since when I develop locally I not only to package
all my program, but also all it's local dependencies.
> let's say I have a local DB with names of files in HADOOP that I want to process with
spark, now I need my local DB to be accessible from the cluster so it can fetch the file names
in runtime.
> The driver program is an awesome thing, but it loses some of it's strength if you can't
really run it anywhere.
> It seems to me that the problem is with the DAGScheduler that needs to be close to the
worker, maybe it shouldn't be embedded in the driver then?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message