kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams
Date Thu, 18 Jan 2018 02:07:57 GMT
Hi,

I updated the KIP and added an `OutputVerifier` class. Looking forward
to your feedback.


I will start the Vote thread in parallel.


-Matthias



On 1/17/18 11:40 AM, Matthias J. Sax wrote:
> Saïd,
> 
> sorry for late reply. Avro records are basically supported. However,
> there are no plans atm to integrate the test driver with a schema
> registry. Note, that Apache Kafka does not provide a schema registry by
> itself. If we integrate one RS, we need to integrate all of them -- that
> is something Apache Kafka should not do from my point of view.
> 
> Thus, if you want to use ArvoSerdes that use a schema registry, those
> should mock be mocked accordingly. However, there is nothing this KIP
> can help with this regard IMHO.
> 
> I left a comment on the ticket to pointed out. :)
> 
> 
> -Matthias
> 
> On 1/17/18 1:29 AM, Saïd Bouras wrote:
>> Matthias,
>>
>> What about testing topology that use avro schema ? Have you read my
>> previous response ?
>>
>> Thanks.
>>
>> On Wed, Jan 17, 2018 at 3:34 AM Matthias J. Sax <matthias@confluent.io>
>> wrote:
>>
>>> Colin,
>>>
>>> the TopologyTestDriver does not connect to any broker and simulates
>>> processing of single-partitioned input topics purely in-memory (the
>>> driver is basically a mock for a StreamThread). This is sufficient to
>>> test basic business logic. For more complex topologies that are actually
>>> divided into sub-topologies and connected via topics, the driver detects
>>> this case and does an in-memory forward.
>>>
>>>
>>> -Matthias
>>>
>>> On 1/16/18 10:08 AM, Colin McCabe wrote:
>>>> Thanks, Matthias, this looks great.
>>>>
>>>> It seems like these APIs could either be used against mock objects, or
>>> against real brokers running in the same process.  Is there a way for the
>>> user to select which they want when using the API?  Sorry if it's in the
>>> KIP and I missed it.
>>>>
>>>> cheers,
>>>> Colin
>>>>
>>>>
>>>> On Thu, Jan 11, 2018, at 18:06, Matthias J. Sax wrote:
>>>>> Dear Kafka community,
>>>>>
>>>>> I want to propose KIP-247 to add public test utils to the Streams API.
>>>>> The goal is to simplify testing of Kafka Streams applications.
>>>>>
>>>>> Please find details in the wiki:
>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams
>>>>>
>>>>> This is an initial KIP, and we hope to add more utility functions later.
>>>>> Thus, this KIP is not comprehensive but a first step. Of course, we can
>>>>> enrich this initial KIP if we think it falls too short. But we should
>>>>> not aim to be comprehensive to keep the scope manageable.
>>>>>
>>>>> In fact, I think we should add some more helpers to simplify result
>>>>> verification. I will update the KIP with this asap. Just wanted to start
>>>>> the discussion early on.
>>>>>
>>>>> An initial WIP PR can be found here:
>>>>> https://github.com/apache/kafka/pull/4402
>>>>>
>>>>> I also included the user-list (please hit "reply-all" to include both
>>>>> lists in this KIP discussion).
>>>>>
>>>>> Thanks a lot.
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>>
>>>>> Email had 1 attachment:
>>>>> + signature.asc
>>>>>   1k (application/pgp-signature)
>>>
>>>
>>
> 


Mime
View raw message