flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: Retrieve written records of a sink after job
Date Wed, 14 Feb 2018 11:44:43 GMT
The only way to access this info from the client is the REST API 
<https://ci.apache.org/projects/flink/flink-docs-master/monitoring/rest_api.html#details-of-a-running-or-completed-job>

or the Metrics REST API 
<https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#rest-api-integration>.

On 14.02.2018 12:38, Flavio Pompermaier wrote:
> Actually I'd like to get this number from my Java class in order to 
> update some external dataset "catalog",
> so I'm asking if there's some programmatic way to access this info 
> (from JobExecutionResult for example).
>
> On Wed, Feb 14, 2018 at 12:25 PM, Chesnay Schepler <chesnay@apache.org 
> <mailto:chesnay@apache.org>> wrote:
>
>     Do you want to know how many records the sink received, or how
>     many the sink wrote to the DB?
>     If it's the first you're in luck because we measure that already,
>     check out the metrics documentation.
>     If it's the latter, then this issue is essentially covered by
>     FLINK-7286 which aims at allowing functions
>     to modify the numRecordsIn/numRecordsOut counts.
>
>
>     On 14.02.2018 12:22, Flavio Pompermaier wrote:
>>     Hi to all,
>>     I have a (batch) job that writes to 1 or more sinks.
>>     Is there a way to retrieve, once the job has terminated, the
>>     number of records written to each sink?
>>     Is there any better way than than using an accumulator for each sink?
>>     If that is the only way to do that, the Sink API could be
>>     enriched in order to automatically create an accumulator when
>>     required. E.g.
>>
>>     dataset.output(JDBCOutputFormat.buildJDBCOutputFormat()
>>                 .setDrivername(...)
>>                 .setDBUrl(...)
>>                 .setQuery(...)
>>     *.addRecordsCountAccumulator("some-name")*
>>                 .finish())
>>
>>     Best,
>>     Flavio
>
>
>
>
>
> -- 
> Flavio Pompermaier
> Development Department
>
> OKKAM S.r.l.
> Tel. +(39) 0461 041809



Mime
View raw message