beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2276) make it easier to specify windowed filename policy
Date Fri, 26 May 2017 06:49:04 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025890#comment-16025890
] 

ASF GitHub Bot commented on BEAM-2276:
--------------------------------------

GitHub user reuvenlax opened a pull request:

    https://github.com/apache/beam/pull/3232

    [BEAM-2276] Cleanups to windowed DefaultFilenamePolicy

    Make window strings more concise, and provide more information about the pane more concisely
(most users care about whether a pane is the last one, and whether it is late).
    
    Cleanup complicated code around filename templates - simply store two default templates,
and pick the correct one based on whether windowed writes are enabled.
    
    Remove log statements that would be extremely noisy in practice. Users may have valid
reasons for not including W and P template variables - e.g. they may know they are in the
global window and not want every file to contain GlobalWindow. This is the same reason we
don't warn if S and N are missing from their template (they may be writing to a single shard).
    
    R: @jbonofre 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/reuvenlax/incubator-beam cleanup_filename_policy

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/3232.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3232
    
----
commit ea451387bbee9a1c9ed037751d35868a61fa9042
Author: Reuven Lax <relax@google.com>
Date:   2017-05-26T06:42:17Z

    Some cleanups.

----


> make it easier to specify windowed filename policy
> --------------------------------------------------
>
>                 Key: BEAM-2276
>                 URL: https://issues.apache.org/jira/browse/BEAM-2276
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-core
>            Reporter: Borisa Zivkovic
>            Assignee: Borisa Zivkovic
>             Fix For: 2.1.0
>
>
> It should be easier for Beam users to specify filename policy which understands windowing
concepts.
> Discussion here 
> https://lists.apache.org/thread.html/b53e437894eb511c9161d34fc77c657300b77a7be75f0fab6566b3d6@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message