spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jungtaek Lim (JIRA)" <>
Subject [jira] [Commented] (SPARK-24161) Enable debug package feature on structured streaming
Date Wed, 02 May 2018 23:07:00 GMT


Jungtaek Lim commented on SPARK-24161:

I have a working patch. Will raise a PR sooner.

> Enable debug package feature on structured streaming
> ----------------------------------------------------
>                 Key: SPARK-24161
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 2.3.0
>            Reporter: Jungtaek Lim
>            Priority: Major
> Currently, debug package has a implicit class which matches Dataset to provide debug
features on Dataset class. It doesn't work with structured streaming: it requires query is
already started, and the information can be retrieved from StreamingQuery, not Dataset. For
the same reason, "explain" had to be placed to StreamingQuery whereas it exists on Dataset.
> This issue tracks effort to enable debug package feature on structured streaming. Unlike
batch, it may have some restrictions.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message