From user-return-65784-apmail-spark-user-archive=spark.apache.org@spark.apache.org Fri Dec 2 01:23:13 2016 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 985F01912B for ; Fri, 2 Dec 2016 01:23:13 +0000 (UTC) Received: (qmail 91236 invoked by uid 500); 2 Dec 2016 01:23:09 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 91112 invoked by uid 500); 2 Dec 2016 01:23:09 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 91098 invoked by uid 99); 2 Dec 2016 01:23:09 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Dec 2016 01:23:09 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id D7DB7C1C52 for ; Fri, 2 Dec 2016 01:23:08 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -1.82 X-Spam-Level: X-Spam-Status: No, score=-1.82 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=hotmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 548tKagErPg8 for ; Fri, 2 Dec 2016 01:23:06 +0000 (UTC) Received: from SNT004-OMC4S9.hotmail.com (snt004-omc4s9.hotmail.com [65.55.90.212]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id EBD3F5F36F for ; Fri, 2 Dec 2016 01:23:05 +0000 (UTC) Received: from NAM04-SN1-obe.outbound.protection.outlook.com ([65.55.90.199]) by SNT004-OMC4S9.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.23008); Thu, 1 Dec 2016 17:22:59 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=gevGuiodc5UTBGeBM0Hm3SuamNdThhdxR/qI4/RQtNw=; b=BJ+D6+fE+v+HzZqLyU06caonoZG9izhstBo74E1YITKTJww0cJLo/WCZfugFtnlvooQ0avdIFVzEA8YahZu9zqUCr2Fq3hNb5BkUREvKjcbT4oZDLPzyaV24D6IuzU45551iQt16d0Q1FuyIe37GeyLZXtLcwCctpCKkMfSrCqhRNyvLbToD85QIHEwCC4dNvX2H4VmR9JJRi6Hp99x1HUD635rqGQEOgIIygRo0zknfkYFaVyw8gxAeRZUQ7WshtzZtFA0vBm/Hr/PD2FRRsJVRh1yKKHRsKefIH5pX1N6P41WD4aZcVD5UXUFpcAbGXUIaBHuFuvthBBk7IiIGMw== Received: from CO1NAM04FT046.eop-NAM04.prod.protection.outlook.com (10.152.90.60) by CO1NAM04HT100.eop-NAM04.prod.protection.outlook.com (10.152.91.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.734.4; Fri, 2 Dec 2016 01:22:58 +0000 Received: from BLUPR04MB772.namprd04.prod.outlook.com (10.152.90.54) by CO1NAM04FT046.mail.protection.outlook.com (10.152.91.117) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.734.4 via Frontend Transport; Fri, 2 Dec 2016 01:22:58 +0000 Received: from BLUPR04MB772.namprd04.prod.outlook.com ([10.141.208.26]) by BLUPR04MB772.namprd04.prod.outlook.com ([10.141.208.26]) with mapi id 15.01.0747.015; Fri, 2 Dec 2016 01:22:58 +0000 From: Felix Cheung To: Weiwei Zhang , user Subject: Re: [GraphFrame, Pyspark] Weighted Edge in PageRank Thread-Topic: [GraphFrame, Pyspark] Weighted Edge in PageRank Thread-Index: AQHSTCQUxfHAANuhkEGaaW9wxG0IlaDz3O/n Date: Fri, 2 Dec 2016 01:22:58 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dons.usfca.edu; dkim=none (message not signed) header.d=none;dons.usfca.edu; dmarc=none action=none header.from=hotmail.com; x-incomingtopheadermarker: OriginalChecksum:;UpperCasedChecksum:;SizeAsReceived:7456;Count:39 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [ITsVKnLlbkPc+jrXA2ySdlAOnbAMrBUJ] x-incomingheadercount: 39 x-eopattributedmessage: 0 x-microsoft-exchange-diagnostics: 1;CO1NAM04HT100;5:eocqvi7jwRuky9r7/6eWRqkobUvNc9c5M4LIU7TIRQMJqvDzFdZKMMQSYHi3I9M11fbsDN3CUnWPUMjAGEPvuYLMy5QFUXFIYP5jvzgwf7+K708u3oHIsoDeA8TvOsDBlYISieJD2T+2s147mmO2lQ==;24:tAakTW8hCJIoyGLNSi1o6Oal6mBGbY1hDKTZxnXlxi1Y0RAIepnysQ4qBJooDL9V1LRRtAs5Hs4PIKxchubv8kZ0HqIlegAXesi8U2SBrJA=;7:da2mrpIX79d7rqQevpm/ptyoFC9QIUCBdk1oUuTESekiTjwp8rbYCE2LTxaqD5IkuJA18wt62Y6gEJphRSpLPLla8uL3wGXJNNzPdoErzwgSALLFNn61E0XDRKSIdFb6aSSd81DAls1B7MwS1ELKDOWAK5IY9FcyUWLt6Ap2q9o/+66DKhDbcaQLYJLlWrQg0ZL41asNqOFmP4itMqlMrZO4MpbU55obcABPhCfdO3Pjxkw7gX3398lH6OglbFXQR4VNOKfh5OhEdpVL054bJMW9OMlQkXxn+tdqsoipeTBIiWiXbVJ56/qyuXRDH1CyjVcgAKjyoCA5o2kiDMRbkqpwgA06KVHaQ6cRHwrF5N4= x-forefront-antispam-report: EFV:NLI;SFV:NSPM;SFS:(10019020)(98900003);DIR:OUT;SFP:1102;SCL:1;SRVR:CO1NAM04HT100;H:BLUPR04MB772.namprd04.prod.outlook.com;FPR:;SPF:None;LANG:en; x-ms-office365-filtering-correlation-id: 942db114-d566-4d47-4d01-08d41a51c018 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(1601124038)(5061506293)(5061507293)(1603103113)(1603101340)(1601125047);SRVR:CO1NAM04HT100; x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(432015012)(82015046);SRVR:CO1NAM04HT100;BCL:0;PCL:0;RULEID:;SRVR:CO1NAM04HT100; x-forefront-prvs: 0144B30E41 spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: multipart/alternative; boundary="_000_BLUPR04MB7728A72CB3494987F21DFD0918E0BLUPR04MB772namprd_" MIME-Version: 1.0 X-OriginatorOrg: hotmail.com X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Dec 2016 01:22:58.1822 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1NAM04HT100 X-OriginalArrivalTime: 02 Dec 2016 01:22:59.0266 (UTC) FILETIME=[9E2BF220:01D24C3A] --_000_BLUPR04MB7728A72CB3494987F21DFD0918E0BLUPR04MB772namprd_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable That's correct - currently GraphFrame does not compute PageRank with weight= ed edges. _____________________________ From: Weiwei Zhang = > Sent: Thursday, December 1, 2016 2:41 PM Subject: [GraphFrame, Pyspark] Weighted Edge in PageRank To: user > Hi guys, I am trying to compute the pagerank for the locations in the following dumm= y dataframe, src des shared_gas_stations A B 2 A C 10 C E 3 D E 12 E G 5 ... I have tried the function graphframe.pageRank(resetProbability=3D0.01, maxI= ter=3D20) in GraphFrame but it seems like this function doesn't take weight= ed edges. Maybe I am not using it correctly. How can I pass the weighted ed= ges to this function? Also I am not sure if this function works for the und= irected graph. Thanks a lot! - Weiwei --_000_BLUPR04MB7728A72CB3494987F21DFD0918E0BLUPR04MB772namprd_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
That's correct - currently GraphFrame does not compute PageRank with w= eighted edges.


_____________________________
From: Weiwei Zhang <wzhang41@dons.usfca.edu>
Sent: Thursday, December 1, 2016 2:41 PM
Subject: [GraphFrame, Pyspark] Weighted Edge in PageRank
To: user <user@spark.apache.org>


Hi guys, 

I am trying to compute the pagerank for the locations in the following dumm= y dataframe, 

src    des      shared_gas_stations
 A       B           &nbs= p;   2
 A       C           &nbs= p;  10
 C       E           &nbs= p;   3
 D       E           &nbs= p;  12
 E       G           &nbs= p;   5
...

I have tried the function graphframe.pageRank(resetProbability=3D0.= 01, maxIter=3D20) in GraphFrame but it seems like this function do= esn't take weighted edges. Maybe I am not using it correctly. How can I pas= s the weighted edges to this function? Also I am not sure if this function works for the undirected graph. 


Thanks a lot!

- Weiwei


--_000_BLUPR04MB7728A72CB3494987F21DFD0918E0BLUPR04MB772namprd_--