spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aakash Basu <>
Subject Spark MLLib vs. SciKitLearn
Date Fri, 19 Jan 2018 13:42:49 GMT
Hi all,

I am totally new to ML APIs. Trying to get the *ROC_Curve* for Model
Evaluation on both *ScikitLearn* and *PySpark MLLib*. I do not find any API
for ROC_Curve calculation for BinaryClassification in SparkMLLib.

The codes below have a wrapper function which is creating the respective
dataframe from the source data with two columns which is as attached.

I want to achieve the same result as Python code in the Spark to get the
roc_curve. Is there any API from MLLib side to achieve the same?

Python sklearn Code -

def roc(self, y_true, y_pred):
    df_a = self._df.copy()
    values_1_tmp = df_a[y_true].values
    values_1_tmp2 = values_1_tmp[~np.isnan(values_1_tmp)]
    values_1 = values_1_tmp2.astype(int)
    values_2_tmp = df_a[y_pred].values
    values_2_tmp2 = values_2_tmp[~np.isnan(values_2_tmp)]
    values_2 = values_2_tmp2.astype(int)
    specificity, sensitivity, thresholds = metrics.roc_curve(values_1,
values_2, pos_label=2)
    # area_under_roc = metrics.roc_auc_score(values_1, values_2)
    print(sensitivity, specificity)
    return sensitivity, specificity


[ 0.          0.34138342  0.67412045  1.        ] [ 0.          0.33373458
0.67378875  1.        ]

PySpark Code -

def roc(self, y_true, y_pred):
    print('using pyspark df')
    df_a = self._df
    values_1 = list(df_a[y_true, y_pred].toPandas().values)
    new_list = [l.tolist() for l in values_1]

    double_list = []
    for myList in new_list:
        temp = []
        for item in myList:

    new_rdd = self._sc.parallelize(double_list)
    metrics = BinaryClassificationMetrics(new_rdd)
    roc_calc = metrics.areaUnderROC
    return 1

Please help.


View raw message