beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Kirpichov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-702) Simple pattern for per-bundle and per-DoFn Closeable resources
Date Tue, 04 Oct 2016 18:55:20 GMT

    [ https://issues.apache.org/jira/browse/BEAM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15546300#comment-15546300
] 

Eugene Kirpichov commented on BEAM-702:
---------------------------------------

Hmm, I didn't realize that we don't call finishBundle and teardown in case the bundle fails.
But yes, this makes a lot of sense.

I'm not sure whether I personally would prefer annotation-based closeables or explicit calls
(e.g. c.addCloseable(createDBWriter)). Explicit calls would make it easier to open a dynamic
set of resources (e.g. lazily open connections to different shards of a database depending
on the data). However that could be encapsulated into a single Closeable object, making these
styles equivalent.

> Simple pattern for per-bundle and per-DoFn Closeable resources
> --------------------------------------------------------------
>
>                 Key: BEAM-702
>                 URL: https://issues.apache.org/jira/browse/BEAM-702
>             Project: Beam
>          Issue Type: Improvement
>            Reporter: Eugene Kirpichov
>
> Dealing with Closeable resources inside a processElement call is easy: simply use try-with-resources.
> However, bundle- or DoFn-scoped resources, such as long-lived database connections, are
less convenient to deal with: you have to open them in startBundle and conditionally close
in finishBundle (likewise setup/teardown), taking special care if there's multiple resources
to close all of them.
> Perhaps we should provide something like Guava's Closer to DoFn's https://github.com/google/guava/wiki/ClosingResourcesExplained.
Ideally, the user would need to only write a startBundle() or setup() method, but not write
finishBundle() or teardown() - resources would be closed automatically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message