flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: Retrieve written records of a sink after job
Date Wed, 14 Feb 2018 11:25:43 GMT
Do you want to know how many records the sink received, or how many the 
sink wrote to the DB?
If it's the first you're in luck because we measure that already, check 
out the metrics documentation.
If it's the latter, then this issue is essentially covered by FLINK-7286 
which aims at allowing functions
to modify the numRecordsIn/numRecordsOut counts.

On 14.02.2018 12:22, Flavio Pompermaier wrote:
> Hi to all,
> I have a (batch) job that writes to 1 or more sinks.
> Is there a way to retrieve, once the job has terminated, the number of 
> records written to each sink?
> Is there any better way than than using an accumulator for each sink?
> If that is the only way to do that, the Sink API could be enriched in 
> order to automatically create an accumulator when required. E.g.
> dataset.output(JDBCOutputFormat.buildJDBCOutputFormat()
>             .setDrivername(...)
>             .setDBUrl(...)
>             .setQuery(...)
> *.addRecordsCountAccumulator("some-name")*
>             .finish())
> Best,
> Flavio

View raw message