beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From git-site-r...@apache.org
Subject [beam] branch asf-site updated: Publishing website 2020/02/06 01:18:56 at commit a8af0e1
Date Thu, 06 Feb 2020 01:19:03 GMT
This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
     new 6ad0c23  Publishing website 2020/02/06 01:18:56 at commit a8af0e1
6ad0c23 is described below

commit 6ad0c23c530bae840823f1266cf8c15163fa4ea3
Author: jenkins <builds@apache.org>
AuthorDate: Thu Feb 6 01:18:56 2020 +0000

    Publishing website 2020/02/06 01:18:56 at commit a8af0e1
---
 .../documentation/programming-guide/index.html     | 24 ++++++++++++++--------
 1 file changed, 15 insertions(+), 9 deletions(-)

diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html
index 178854b..7d1dfae 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -3263,10 +3263,6 @@ elements.</p>
 
 <h4 id="managing-late-data">7.4.1. Managing late data</h4>
 
-<blockquote>
-  <p><strong>Note:</strong> Managing late data is not supported in the
Beam SDK for Python.</p>
-</blockquote>
-
 <p>You can allow late data by invoking the <code class="highlighter-rouge">.withAllowedLateness</code>
operation when
 you set your <code class="highlighter-rouge">PCollection</code>’s windowing
strategy. The following code example
 demonstrates a windowing strategy that will allow late data up to two days after
@@ -3278,6 +3274,14 @@ the end of a window.</p>
               <span class="o">.</span><span class="na">withAllowedLateness</span><span
class="o">(</span><span class="n">Duration</span><span class="o">.</span><span
class="na">standardDays</span><span class="o">(</span><span class="mi">2</span><span
class="o">)));</span>
 </code></pre></div></div>
 
+<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
  <span class="n">pc</span> <span class="o">=</span> <span class="p">[</span><span
class="n">Initial</span> <span class="n">PCollection</span><span class="p">]</span>
+   <span class="n">pc</span> <span class="o">|</span> <span class="n">beam</span><span
class="o">.</span><span class="n">WindowInto</span><span class="p">(</span>
+              <span class="n">FixedWindows</span><span class="p">(</span><span
class="mi">60</span><span class="p">),</span>
+              <span class="n">trigger</span><span class="o">=</span><span
class="n">trigger_fn</span><span class="p">,</span>
+              <span class="n">accumulation_mode</span><span class="o">=</span><span
class="n">accumulation_mode</span><span class="p">,</span>
+              <span class="n">timestamp_combiner</span><span class="o">=</span><span
class="n">timestamp_combiner</span><span class="p">,</span>
+              <span class="n">allowed_lateness</span><span class="o">=</span><span
class="n">Duration</span><span class="p">(</span><span class="n">seconds</span><span
class="o">=</span><span class="mi">2</span><span class="o">*</span><span
class="mi">24</span><span class="o">*</span><span class="mi">60</span><span
class="o">*</span><span class="mi">60</span><span class="p">))</span>
<span class="c"># 2 days</span>
+</code></pre></div></div>
 <p>When you set <code class="highlighter-rouge">.withAllowedLateness</code>
on a <code class="highlighter-rouge">PCollection</code>, that allowed lateness
 propagates forward to any subsequent <code class="highlighter-rouge">PCollection</code>
derived from the first
 <code class="highlighter-rouge">PCollection</code> you applied allowed lateness
to. If you want to change the allowed
@@ -3555,10 +3559,6 @@ on each firing:</p>
 
 <h4 id="handling-late-data">8.4.2. Handling late data</h4>
 
-<blockquote>
-  <p>The Beam SDK for Python does not currently support allowed lateness.</p>
-</blockquote>
-
 <p>If you want your pipeline to process data that arrives after the watermark
 passes the end of the window, you can apply an <em>allowed lateness</em> when
you set
 your windowing configuration. This gives your trigger the opportunity to react
@@ -3574,7 +3574,13 @@ windowing function:</p>
                                                              <span class="o">.</span><span
class="na">plusDelayOf</span><span class="o">(</span><span class="n">Duration</span><span
class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span
class="mi">1</span><span class="o">)))</span>
                               <span class="o">.</span><span class="na">withAllowedLateness</span><span
class="o">(</span><span class="n">Duration</span><span class="o">.</span><span
class="na">standardMinutes</span><span class="o">(</span><span class="mi">30</span><span
class="o">));</span>
 </code></pre></div></div>
-<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
 <span class="c"># The Beam SDK for Python does not currently support allowed lateness.</span>
+<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
 <span class="n">pc</span> <span class="o">=</span> <span class="p">[</span><span
class="n">Initial</span> <span class="n">PCollection</span><span class="p">]</span>
+  <span class="n">pc</span> <span class="o">|</span> <span class="n">beam</span><span
class="o">.</span><span class="n">WindowInto</span><span class="p">(</span>
+            <span class="n">FixedWindows</span><span class="p">(</span><span
class="mi">60</span><span class="p">),</span>
+            <span class="n">trigger</span><span class="o">=</span><span
class="n">AfterProcessingTime</span><span class="p">(</span><span
class="mi">60</span><span class="p">),</span>
+            <span class="n">allowed_lateness</span><span class="o">=</span><span
class="mi">1800</span><span class="p">)</span> <span class="c">#
30 minutes</span>
+     <span class="o">|</span> <span class="o">...</span>
+  
 </code></pre></div></div>
 
 <p>This allowed lateness propagates to all <code class="highlighter-rouge">PCollection</code>s
derived as a result of


Mime
View raw message