samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jagadish Venkatraman <jagadish1...@gmail.com>
Subject Re: Samza + MongoDb Sharding
Date Sat, 08 Jul 2017 00:57:34 GMT
Thank you for clarifying.

Yes, You should partition your input streams as well.  Typically, it does
not hurt to over-partition them.

https://samza.apache.org/learn/documentation/0.13/container/samza-container.html

Thanks,
Jagdish

On Fri, Jul 7, 2017 at 10:55 PM Angelica Garcia-Gutierrez <
agarcia@transcendinsights.com> wrote:

> That is correct, I am augmenting each message in the input stream with
> further information from the MongoDB.
>
>
>
> On 7/6/17, 5:43 PM, "Jagadish Venkatraman" <jagadish1989@gmail.com> wrote:
>
> >What is your input source you are reading the stream from?  IIUC, you are
> >trying to make a remote call to mongo to decorate/ augment each message?
> >
> >Best,
> >Jagdish
> >
> >On Fri, Jul 7, 2017 at 12:51 AM Angelica Garcia-Gutierrez <
> >agarcia@transcendinsights.com> wrote:
> >
> >> Hello,
> >>
> >> I have a Samza job that currently makes remote calls to a MongoDB to get
> >> additional information about the input stream. For scalability, MongoDB
> >>was
> >> initially partitioned into 4 shards (more shards will be added as
> >>needed).
> >> The questions are:
> >>
> >>   *   Does it make sense to attempt to partition the input stream into
> >> multiple partitions such that a given task can consume it and expand the
> >> message with information retrieved from a specific MongoDB shard?
> >>
> >> Can someone please shed some lights?
> >>
> >> Thanks,
> >> Angelica.
> >>
> >>
> >> The information transmitted is intended only for the person or entity to
> >> which it is addressed
> >> and may contain CONFIDENTIAL material.  If you receive this
> >> material/information in error,
> >> please contact the sender and delete or destroy the
> >>material/information.
> >
> >--
> >Sent from my iphone.
>
> The information transmitted is intended only for the person or entity to
> which it is addressed
> and may contain CONFIDENTIAL material.  If you receive this
> material/information in error,
> please contact the sender and delete or destroy the material/information.
>
-- 
Sent from my iphone.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message