spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cyanny LIANG <>
Subject Fwd: Does pyspark worker really use pipe?
Date Thu, 20 Jul 2017 02:17:50 GMT
As pyspark internals wiki said,
pyspark worker use pipe to communicate, not socket.

I have checked the pyspark/ code:

if __name__ == '__main__':
    # Read a local port to connect to from stdin
    java_port = int(sys.stdin.readline())
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.connect(("", java_port))
    sock_file = sock.makefile("rwb", 65536)
    main(sock_file, sock_file)

it actually uses socket not pipe, I am wondering that is there anything I
why pyspark worker use socket not pipe? for performance reason?

Best & Regards
Cyanny LIANG

View raw message