kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Gainty <mgai...@hotmail.com>
Subject Re: First time building a streaming app and I need help understanding how to build out my use case
Date Mon, 10 Jun 2019 13:20:58 GMT
MG>below

________________________________
From: Simon Calvin <scalvin@hoganassessments.com>
Sent: Friday, June 7, 2019 3:39 PM
To: users@kafka.apache.org
Subject: First time building a streaming app and I need help understanding how to build out
my use case

Hello, everyone. I feel like I have a use case that it is well suited to the Kafka streaming
paradigm, but I'm having a difficult time understanding how certain aspects will work as I'm
prototyping.

So here's my use case: Service 1 assigns a job to a user which is published as an event to
Kafka. Service 2 is a domain service that owns the definition for all jobs. In this case,
the definition boils down to a bunch of form fields that need to be filled in. As changes
are made to the definitions, the updated versions are published by Service 2 to Kafka (I think
this is a KTable?). The job from Service 1 and the definition from Service 2 get joined together
to create a "bill of materials" that the user needs to fulfill.
 Service 3, a REST API,

MG>can you risk implementing a non-secured HTTP connection?... then go ahead
MG>if not you will need to look into some manner of PKI implementation for your Kafka Streams
(user_login or certs&keys)

needs to pull any unfulfilled bills for a given user. Ideally we want the bill to contain
the most current version of the job definition at the point it is retrieved (vs the version
at the point that the job assignment was published). Then, as the user fulfills the items,
we update the bill with their responses. Once the bill is complete it gets pushed on to the
one or more additional services (all basic consumers).

MG>for Ktable stream example please reference org.apache.kafka.streams.smoketest.SmokeTestClient
createKafkaStreams

The part I'm having the most trouble with is the retrieval of bills for a user in Service
3. I got this idea in my head that because Kafka is effectively a storage system there was
a(n at least fairly) straightforward way of querying out messages that were keyed/tagged a
certain way (i.e., with the user ID), but it's not clear to me if and how that works in practice.
I'm very new to the idea of streaming and so I think a lot of the issue is that I'm trying
to force foreign concepts (the non-streaming way I'm used to doing things) in to the streaming
paradigm. Any help is appreciated!

MG>assuming your ID is *NOT* generated for your table
MG>if implementing HTTPS request/response you might want to consider using identifier of
unique secured SESSION_ID
https://security.stackexchange.com/questions/87269/how-is-the-session-id-sent-securely
[https://cdn.sstatic.net/Sites/security/img/apple-touch-icon@2.png?v=497726d850f9]<https://security.stackexchange.com/questions/87269/how-is-the-session-id-sent-securely>
How is the session ID sent securely? - Stack Exchange<https://security.stackexchange.com/questions/87269/how-is-the-session-id-sent-securely>
Answer 1: if the server uses SSL/HTTPS(verified by third party-not self-signed certificate),
cookies and session IDs travel as cipher-text over the network, and if an attacker (Man in
the Middle) uses a packet sniffer, they can not obtain any information. They can not decrypt
data because the connection between client and server is secured by a verified third party.so
HTTPS without verified ...
security.stackexchange.com


Thanks very much for your kind attention!

Simon Calvin

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message