samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jagadish Venkatraman <jagadish1...@gmail.com>
Subject Re: Regarding my use case to explore with Samza-
Date Wed, 02 Mar 2016 06:43:15 GMT
Please take a look at the hello-world example. You can implement your
business logic in the process() callback.

What kind of transformation are you doing? Are you doing a group by/count
style aggregation to generate the report? If so, you could use the embedded
rocksdb store in Samza and potentially batch your writes to the database.

How many Qps do you process at peak? Do you expect to buffer any state per
message? What's the ratio of input to output messages on average?

There's nothing that stops you from using JDBC and Samza.

On Tue, Mar 1, 2016 at 8:58 PM, Manohar Reddy <
Manohar.Reddy@happiestminds.com> wrote:

> Hello Team,
>
> we are part of some service based company and trying to explore the
> available real time streaming technologies.so one of the first option we
> are trying is Samza.
> let me explain brief about my use case here:
>
> we are trying to build  real time reporting dashboard for e-learning
> domain.
> To build this dash board the input is RDBMS.so if any
> DML(inserts/updates/deletes)  into the source RDBMS,immediately some
> adapter will publish to kafka with RDBMS table name and primary keys as
> JSON format.
> Now Samza has to consume the kafka event and query back to source RDBMS
> table to get the whole data set of  RDBMS relation tables by using json
> event information.
> now do some transformation here as per business and load into
> Target(Reporting DB) RDBMS.
> more or less here we are handling with few  JDBC calls through Samza and
> here every day data load is small I can say max 2Gb of data but we need
> real time processing ecosystem in place.
> that's it brief about my usecase,so team please provide your inputs how we
> can approach with samza for this requirement.is there any utility API
> with Samza for JDBC calls.
>
> Thank you very much in Advance.
>
> ~~Manohar
> ________________________________
> Happiest Minds Disclaimer
>
> This message is for the sole use of the intended recipient(s) and may
> contain confidential, proprietary or legally privileged information. Any
> unauthorized review, use, disclosure or distribution is prohibited. If you
> are not the original intended recipient of the message, please contact the
> sender by reply email and destroy all copies of the original message.
>
> Happiest Minds Technologies <http://www.happiestminds.com>
>
> ________________________________
>



-- 
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message