From commits-return-27776-apmail-beam-commits-archive=beam.apache.org@beam.apache.org Thu Mar 30 09:20:46 2017 Return-Path: X-Original-To: apmail-beam-commits-archive@minotaur.apache.org Delivered-To: apmail-beam-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B039319ED6 for ; Thu, 30 Mar 2017 09:20:46 +0000 (UTC) Received: (qmail 17917 invoked by uid 500); 30 Mar 2017 09:20:46 -0000 Delivered-To: apmail-beam-commits-archive@beam.apache.org Received: (qmail 17845 invoked by uid 500); 30 Mar 2017 09:20:46 -0000 Mailing-List: contact commits-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list commits@beam.apache.org Received: (qmail 17810 invoked by uid 99); 30 Mar 2017 09:20:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Mar 2017 09:20:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id DD3CFCD2E0 for ; Thu, 30 Mar 2017 09:20:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id rjyqtc8-AlKZ for ; Thu, 30 Mar 2017 09:20:44 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id B2A265F5FD for ; Thu, 30 Mar 2017 09:20:43 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 5CDFCE0BD2 for ; Thu, 30 Mar 2017 09:20:42 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 97E5524172 for ; Thu, 30 Mar 2017 09:20:41 +0000 (UTC) Date: Thu, 30 Mar 2017 09:20:41 +0000 (UTC) From: "Etienne Chauchot (JIRA)" To: commits@beam.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (BEAM-1261) State API should allow state to be managed in different windows MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/BEAM-1261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15948727#comment-15948727 ] Etienne Chauchot commented on BEAM-1261: ---------------------------------------- I have one possible use case of this. I'm sure [~kenn] you know the one I'am talking about :) In nexmark query3 uses state to do an incremental join of the auctions and the people. Auctions and person events can arrive out of order and in different (fixed) windows - person element is stored in state in order to match future auctions by that person - auction elements are stored in state until we have seen the corresponding person record But state seem to become useless for this query because it is actually scoped to a window, so when the stored element will be needed in a future window, it will not be there anymore. > State API should allow state to be managed in different windows > --------------------------------------------------------------- > > Key: BEAM-1261 > URL: https://issues.apache.org/jira/browse/BEAM-1261 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-java-core > Reporter: Ben Chambers > > For example, even if the elements are being processed in fixed windows of an hour, it may be desirable for the state to "roll over" between windows (or be available to all windows). > It will also be necessary to figure out when this state should be deleted (TTL? maximum retention?) > Another problem is how to deal with out of order data. If data comes in from the 10:00 AM window, should its state changes be visible to the data in the 9:00 AM window? -- This message was sent by Atlassian JIRA (v6.3.15#6346)