spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miles Crawford (JIRA)" <>
Subject [jira] [Created] (SPARK-17669) Strange UI behavior using Datasets
Date Mon, 26 Sep 2016 20:31:20 GMT
Miles Crawford created SPARK-17669:

             Summary: Strange UI behavior using Datasets
                 Key: SPARK-17669
             Project: Spark
          Issue Type: Bug
          Components: SQL, Web UI
    Affects Versions: 2.0.0
            Reporter: Miles Crawford

I recently migrated my application to Spark 2.0, and everything worked well, except for one
function that uses "toDS" and the ML libraries.

This stage used to complete in 15 minutes or so on 1.6.2, and now takes almost two hours.

The UI shows very strange behavior - completed stages still being worked on, concurrent work
on tons of stages, including ones from downstream jobs:

The only source change I made was changing "toDF" to "toDS()" before handing my RDDs to the
ML libraries.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message