flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maximilian Michels (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3133) Introduce collect()/count()/print() methods in DataStream API
Date Thu, 17 Nov 2016 21:47:58 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15674912#comment-15674912

Maximilian Michels commented on FLINK-3133:

Hi [~AlexanderShoshin]! I've adjusted the description a bit. In general, we face the problem
that we have potentially infinite output which we can't store in the same way as count/collect/print
works in the DataSet API. The idea here is to sample parts of the stream from the client while
the job is executing.

> Introduce collect()/count()/print() methods in DataStream API
> -------------------------------------------------------------
>                 Key: FLINK-3133
>                 URL: https://issues.apache.org/jira/browse/FLINK-3133
>             Project: Flink
>          Issue Type: Improvement
>          Components: DataStream API, Streaming
>    Affects Versions: 0.10.0, 1.0.0, 0.10.1
>            Reporter: Maximilian Michels
>            Assignee: Alexander Shoshin
>             Fix For: 1.0.0
> The DataSet API's methods {{collect()}}, {{count()}}, and {{print()}} should be mirrored
to the DataStream API. 
> The semantics of the calls are different. We need to be able to sample parts of a stream,
e.g. by supplying a time period in the arguments to the methods. Users should use the {{JobClient}}
to retrieve the results.
> {code:java}
> StreamExecutionEnvironment env = StramEnvironment.getStreamExecutionEnvironment();
> DataStream<DataType> streamData = env.addSource(..).map(..);
> JobClient jobClient = env.executeWithControl();
> Iterable<DataType> sampled = jobClient.sampleStream(streamData, Time.seconds(5));
> {code}

This message was sent by Atlassian JIRA

View raw message