flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3679) DeserializationSchema should handle zero or more outputs for every input
Date Mon, 20 Feb 2017 16:03:44 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15874739#comment-15874739
] 

ASF GitHub Bot commented on FLINK-3679:
---------------------------------------

Github user rmetzger commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3314#discussion_r102048475
  
    --- Diff: flink-connectors/flink-connector-kafka-0.8/src/main/java/org/apache/flink/streaming/connectors/kafka/internals/SimpleConsumerThread.java
---
    @@ -373,16 +370,28 @@ else if (partitionsRemoved) {
     								keyPayload.get(keyBytes);
     							}
     
    -							final T value = deserializer.deserialize(keyBytes, valueBytes, 
    -									currentPartition.getTopic(), currentPartition.getPartition(), offset);
    -							
    -							if (deserializer.isEndOfStream(value)) {
    -								// remove partition from subscribed partitions.
    -								partitionsIterator.remove();
    -								continue partitionsLoop;
    -							}
    -							
    -							owner.emitRecord(value, currentPartition, offset);
    +							final Collector<T> collector = new Collector<T>() {
    --- End diff --
    
    I'm not sure of the performance implications for this. The JVM will create a Collector
instance for each record read from Kafka.
    I wonder if we can re-use one collector instance here.
    
    
    Also, I wonder if we need to use this `Collector` implementation, with a `close()` method
we are not using and an exception we are turning into a `RuntimeException`. Maybe we should
let the collect throw an exception?


> DeserializationSchema should handle zero or more outputs for every input
> ------------------------------------------------------------------------
>
>                 Key: FLINK-3679
>                 URL: https://issues.apache.org/jira/browse/FLINK-3679
>             Project: Flink
>          Issue Type: Bug
>          Components: DataStream API, Kafka Connector
>            Reporter: Jamie Grier
>            Assignee: Haohui Mai
>
> There are a couple of issues with the DeserializationSchema API that I think should be
improved.  This request has come to me via an existing Flink user.
> The main issue is simply that the API assumes that there is a one-to-one mapping between
input and outputs.  In reality there are scenarios where one input message (say from Kafka)
might actually map to zero or more logical elements in the pipeline.
> Particularly important here is the case where you receive a message from a source (such
as Kafka) and say the raw bytes don't deserialize properly.  Right now the only recourse is
to throw IOException and therefore fail the job.  
> This is definitely not good since bad data is a reality and failing the job is not the
right option.  If the job fails we'll just end up replaying the bad data and the whole thing
will start again.
> Instead in this case it would be best if the user could just return the empty set.
> The other case is where one input message should logically be multiple output messages.
 This case is probably less important since there are other ways to do this but in general
it might be good to make the DeserializationSchema.deserialize() method return a collection
rather than a single element.
> Maybe we need to support a DeserializationSchema variant that has semantics more like
that of FlatMap.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message