samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lhaiesp <...@git.apache.org>
Subject [GitHub] samza pull request #633: SAMZA-1870: hdfs offset comparator to handle end of...
Date Tue, 11 Sep 2018 15:51:03 GMT
GitHub user lhaiesp opened a pull request:

    https://github.com/apache/samza/pull/633

    SAMZA-1870: hdfs offset comparator to handle end of stream offset

    This happens particularly when using HDFS as a bootstrap stream:
    
    org.apache.samza.SamzaException: Invalid offset for MultiFileHdfsReader: END_OF_STREAM
    at org.apache.samza.system.hdfs.reader.MultiFileHdfsReader.getCurFileIndex(MultiFileHdfsReader.java:64)
    at org.apache.samza.system.hdfs.HdfsSystemAdmin.offsetComparator(HdfsSystemAdmin.java:224)
    at org.apache.samza.system.chooser.BootstrappingChooser.org$apache$samza$system$chooser$BootstrappingChooser$$checkOffset(BootstrappingChooser.scala:274)
    at org.apache.samza.system.chooser.BootstrappingChooser.choose(BootstrappingChooser.scala:204)
    at org.apache.samza.system.chooser.DefaultChooser.choose(DefaultChooser.scala:294)
    at org.apache.samza.system.SystemConsumers.choose(SystemConsumers.scala:210)
    at org.apache.samza.task.AsyncRunLoop.chooseEnvelope(AsyncRunLoop.java:208)
    at org.apache.samza.task.AsyncRunLoop.run(AsyncRunLoop.java:156)
    at org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:787)
    at org.apache.samza.runtime.LocalContainerRunner.run(LocalContainerRunner.java:101)
    at org.apache.samza.runtime.LocalContainerRunner.main(LocalContainerRunner.java:148)

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/lhaiesp/samza master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/samza/pull/633.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #633
    
----
commit 42b80cc34a999955b79997494fe078f8024c9c2c
Author: Hai Lu <halu@...>
Date:   2018-09-11T15:49:51Z

    hdfs offset comparator to handle end of stream offset

----


---

Mime
View raw message