spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Imran Rashid (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-24541) TCP based shuffle
Date Fri, 01 Feb 2019 16:36:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16758482#comment-16758482
] 

Imran Rashid commented on SPARK-24541:
--------------------------------------

well, rpc is over tcp, so I'm still not really sure what this means.  Is the point sending
raw data directly over sockets?  I'd be interested in knowing what the purpose is.  I guess
to avoid the overhead associated w/ the extra headers etc from the rpc framework?  And if
this is really going to try to use raw sockets, not through netty, then you'd have to reimplement
encryption, manage your own buffers, etc.

> TCP based shuffle
> -----------------
>
>                 Key: SPARK-24541
>                 URL: https://issues.apache.org/jira/browse/SPARK-24541
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Structured Streaming
>    Affects Versions: 2.4.0
>            Reporter: Jose Torres
>            Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message