samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From voronenko-da <...@git.apache.org>
Subject [GitHub] samza pull request #631: Invalid partition calculation in KafkaSystemProduce...
Date Mon, 10 Sep 2018 07:38:07 GMT
GitHub user voronenko-da opened a pull request:

    https://github.com/apache/samza/pull/631

    Invalid partition calculation in KafkaSystemProducer 0.14.1

    If you use a byte array as the key, the key.hashCode() will be different all the time.
Proposal to remove the preliminary calculation of the partition.
    
    https://github.com/apache/samza/blob/master/samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala
    
    val partitionKey = if (envelope.getPartitionKey != null) KafkaUtil.getIntegerPartitionKey(envelope,
partitions) else null
    
    https://github.com/apache/samza/blob/master/samza-kafka/src/main/scala/org/apache/samza/util/KafkaUtil.scala
    
      def getIntegerPartitionKey(envelope: OutgoingMessageEnvelope, partitions: java.util.List[PartitionInfo]):
Integer = {
        val numPartitions = partitions.size
        abs(envelope.getPartitionKey.hashCode()) % numPartitions
      }

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/apache/samza 0.14.1

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/samza/pull/631.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #631
    
----
commit 3f2c0775bfc69e7464f45343503c5e51310d55dd
Author: xiliu <xiliu@...>
Date:   2018-05-07T22:25:35Z

    Trigger notification to github of 0.14.1 branch

commit 5ca7fac5767496a5c22ed6cc3096f366c1e304a9
Author: xiliu <xiliu@...>
Date:   2018-05-07T23:11:59Z

    SAMZA-1702: Prepare 0.14.1 release on the 0.14.1 branch
    
    Author: xiliu <xiliu@linkedin.com>
    
    Reviewers: Prateek M <prateekm@apache.org>
    
    Closes #505 from xinyuiscool/SAMZA-1702

commit 06039b789b8134c4dbbe7883b34722427b3f46c8
Author: xiliu <xiliu@...>
Date:   2018-05-08T00:06:35Z

    SAMZA-1703: Disable flaky test TestEmbeddedTaggedRateLimiter.testAcquireWithTimeout
    
    Author: xiliu <xiliu@linkedin.com>
    
    Reviewers: Boris S <sborya@apache.org>
    
    Closes #507 from xinyuiscool/SAMZA-1703
    
    (cherry picked from commit 9ac221e4e99d3e576c7fad03d0b83eeb489564e9)
    Signed-off-by: xiliu <xiliu@linkedin.com>

commit 4b19efefec78ec6d2f7352dfa1972a9d04181861
Author: xiliu <xiliu@...>
Date:   2018-05-08T01:14:29Z

    SAMZA-1704: Fix compatibility issues with scala 2.12
    
    Need to add override keyword for overriding a method in scala 2.12.
    
    Author: xiliu <xiliu@linkedin.com>
    
    Reviewers: Prateek M <prateekm@apache.org>
    
    Closes #508 from xinyuiscool/SAMZA-1704
    
    (cherry picked from commit 31a9e4aea8a8233853334b7ed70da8694f42bd52)
    Signed-off-by: xiliu <xiliu@linkedin.com>

commit 44d6f40f7d8d331e40752fc369a83fc69edf7867
Author: xinyuiscool <xiliu@...>
Date:   2018-05-09T00:48:55Z

    SAMZA-1705: Switch to use snapshot in iterable impl of RocksDb
    
    We should use rocksDb.snapshot() method to keep the snapshot and creates a new iterator
with it all the time. The perf shows a little bit more expensive but mostly on par with range
iterator query.
    
    Author: xinyuiscool <xiliu@linkedin.com>
    
    Reviewers: Jagadish V <vjagadish@apache.org>
    
    Closes #510 from xinyuiscool/SAMZA-1705
    
    (cherry picked from commit 89beb1fccb01c781a4de905d57a4bd99df25577a)
    Signed-off-by: xiliu <xiliu@linkedin.com>

commit 64ea33d7d89c5f7f934a0cc8290f3ad3f925e98a
Author: Hai Lu <halu@...>
Date:   2018-05-09T22:32:44Z

    SAMZA-1706: lazy initialization for eventhub system producer
    
    We are seeing slow shutdown issue for eventhub system producers for users who only use
eventhub consumer (but then Samza system creates both consumer and producer together no matter
what). As a workaround, add lazy initialization to the producer to avoid the slow shutdown
    
    Author: Hai Lu <halu@linkedin.com>
    
    Reviewers: Jagadish <jagadish@apache.org>
    
    Closes #511 from lhaiesp/master

commit adb514aaee046bf760e4f992c45e1f4582bb759d
Author: Prateek Maheshwari <pmaheshw@...>
Date:   2018-05-10T16:21:12Z

    Fixed test failure for TestRocksDbKeyValueStoreJava#testIterate
    
    RocksDB Snapshots and any iterators obtained from them need to be closed before the store
is closed.
    
    Otherwise the process aborts with the following message (at least on OSX):
    `Assertion failed: (is_last_reference), function ~ColumnFamilyData, file db/column_family.cc,
line 457.`
    
    Author: Prateek Maheshwari <pmaheshw@linkedin.com>
    
    Reviewers: Jagadish V <vjagadish1989@gmail.com>
    
    Closes #513 from prateekm/rocksdb-test-fi

commit 1e1330cefbcfc7f2ff8e20c8ff19c465f7df4b66
Author: Prateek Maheshwari <pmaheshw@...>
Date:   2018-05-10T19:16:33Z

    Fixed test failure for TestRocksDbKeyValueStoreJava#testPerf
    
    Iterators (incl. those obtained from snapshots) must be closed before store close.
    
    Author: Prateek Maheshwari <pmaheshw@linkedin.com>
    
    Reviewers: Xinyu Liu <xinyuliu.us@gmail.com>
    
    Closes #514 from prateekm/rocksdb-test-fi
    
    (cherry picked from commit 54c690e6fc52ac13d1c59fe1bd1c34f35f710164)
    Signed-off-by: xiliu <xiliu@linkedin.com>

commit b2ca67ea19becac1ea8c7a73d544438e75f94784
Author: xinyuiscool <xiliu@...>
Date:   2018-05-10T21:19:08Z

    Remove the iterable interface from KeyValueSnapshot
    
    The iterable interface makes it hard for the users to close it after using.
    
    Author: xinyuiscool <xiliu@linkedin.com>
    
    Reviewers: Prateek M <prateekm@apache.org>
    
    Closes #516 from xinyuiscool/kv-snapshot
    
    (cherry picked from commit e204d298e84f549ecf4d8d9d642304502f046c29)
    Signed-off-by: xiliu <xiliu@linkedin.com>

commit 973e0735e076101fa9c05e88df999cfdd289fd0a
Author: Yi Pan (Data Infrastructure) <yipan@...>
Date:   2018-05-17T05:19:37Z

    SAMZA-1715: Unit test for Kafka admin deletedMessagesCalled() fail
    
    Test locally and works.
    
    Author: Yi Pan (Data Infrastructure) <yipan@yipan-mn1.linkedin.biz>
    
    Reviewers: Jagadish <jagadish@apache.org>
    
    Closes #523 from nickpan47/fix-unittest-deleted-messages

commit 69e63d8153b89cf47c618b3f4e871609d7b18a02
Author: Shanthoosh Venkataraman <santhoshvenkat1988@...>
Date:   2018-05-18T19:29:24Z

    SAMZA-1720: Remove javafx.util dependency from samza-sql tests.
    
    In samza-sql module, currently few test classes(`TestSamzaSqlRelMessageSerde` and `TestSamzaSqlRelRecordSerde`)
are dependent upon `javafx.util.Pair` class(coming from `javafx` module). `javafx.util.Pair`
is not supported by default in all JDK builds(example; open-jdk java-8 doesn't support `javafx`
module) and it belongs to `javafx` package which is primarily used for developing GUI applications.
This dependency is removed and replaced with `Pair` class from `apache-commons`.
    
    Author: Shanthoosh Venkataraman <santhoshvenkat1988@gmail.com>
    
    Reviewers: Jagadish V <vjagadish@gmail.com>
    
    Closes #527 from shanthoosh/SAMZA-1720
    
    (cherry picked from commit 3b9e14be3aa5fc2b6003b7e244fba466db34236f)
    Signed-off-by: xiliu <xiliu@linkedin.com>

commit 9bc03f7ab32199a15af157cc515d897917694cb5
Author: Srinivasulu Punuru <spunuru@...>
Date:   2018-05-18T19:31:44Z

    Implementing the fetchSinkInfo in ConfigBasedIOResolver
    
    1. I think we missed implementing the fetchSinkInfo method in the ConfigBasedResolver
when the API was introduced which is breaking the samza sql console tool. This fixes it.
    2. latest release of mac removed realpath so the command line tools are broken. Removed
the usage of realpath to fix these tools.
    
    Thanks to nickpan47 for identifying these problems.
    
    Author: Srinivasulu Punuru <spunuru@linkedin.com>
    
    Reviewers: Yi Pan <nickpan47@gmail.com>
    
    Closes #528 from srinipunuru/release-fix.1
    
    (cherry picked from commit 171793b69b33081fc6277c9505b3055f79fcb4b7)
    Signed-off-by: xiliu <xiliu@linkedin.com>

commit a652e6f5230a297f3d991c6441311ef55e562d9a
Author: xiliu <xiliu@...>
Date:   2018-05-25T22:18:38Z

    Remove git checkout latest from the docs in 0.14.1 branch

----


---

Mime
View raw message