kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: "broadcast" tablet replication for kudu?
Date Wed, 24 Apr 2019 16:35:24 GMT
Hey Boris,

Sorry to say that the situation is still the same.

-Todd

On Wed, Apr 24, 2019 at 9:02 AM Boris Tyukin <boris@boristyukin.com> wrote:

> sorry to revive the old thread but curious if there is a better solution 1
> year after...We have a few small tables (under 300k rows) which are
> practically used with every single query and to make things worse joined
> more than once in the same query.
>
> Is there a way to replicate this table on every node to improve
> performance and avoid broadcasting this table every time?
>
> On Mon, Jul 23, 2018 at 10:52 AM Todd Lipcon <todd@cloudera.com> wrote:
>
>>
>>
>> On Mon, Jul 23, 2018, 7:21 AM Boris Tyukin <boris@boristyukin.com> wrote:
>>
>>> Hi Todd,
>>>
>>> Are you saying that your earlier comment below is not longer valid with
>>> Impala 2.11 and if I replicate a table to all our Kudu nodes Impala can
>>> benefit from this?
>>>
>>
>> No, the earlier comment is still valid. Just saying that in some cases
>> exchange can be faster in the new Impala version.
>>
>>
>>> "
>>> *It's worth noting that, even if your table is replicated, Impala's
>>> planner is unaware of this fact and it will give the same plan regardless.
>>> That is to say, rather than every node scanning its local copy, instead a
>>> single node will perform the whole scan (assuming it's a small table) and
>>> broadcast it from there within the scope of a single query. So, I don't
>>> think you'll see any performance improvements on Impala queries by
>>> attempting something like an extremely high replication count.*
>>>
>>> *I could see bumping the replication count to 5 for these tables since
>>> the extra storage cost is low and it will ensure higher availability of the
>>> important central tables, but I'd be surprised if there is any measurable
>>> perf impact.*
>>> "
>>>
>>> On Mon, Jul 23, 2018 at 9:46 AM Todd Lipcon <todd@cloudera.com> wrote:
>>>
>>>> Are you on the latest release of Impala? It switched from using Thrift
>>>> for RPC to a new implementation (actually borrowed from kudu) which might
>>>> help broadcast performance a bit.
>>>>
>>>> Todd
>>>>
>>>> On Mon, Jul 23, 2018, 6:43 AM Boris Tyukin <boris@boristyukin.com>
>>>> wrote:
>>>>
>>>>> sorry to revive the old thread but I am curious if there is a good way
>>>>> to speed up requests to frequently used tables in Kudu.
>>>>>
>>>>> On Thu, Apr 12, 2018 at 8:19 AM Boris Tyukin <boris@boristyukin.com>
>>>>> wrote:
>>>>>
>>>>>> bummer..After reading your guys conversation, I wish there was an
>>>>>> easier way...we will have the same issue as we have a few dozens
of tables
>>>>>> which are used very frequently in joins and I was hoping there was
an easy
>>>>>> way to replicate them on most of the nodes to avoid broadcasts every
time
>>>>>>
>>>>>> On Thu, Apr 12, 2018 at 7:26 AM, Clifford Resnick <
>>>>>> cresnick@mediamath.com> wrote:
>>>>>>
>>>>>>> The table in our case is 12x hashed and ranged by month, so the
>>>>>>> broadcasts were often to all (12) nodes.
>>>>>>>
>>>>>>> On Apr 12, 2018 12:58 AM, Mauricio Aristizabal <mauricio@impact.com>
>>>>>>> wrote:
>>>>>>> Sorry I left that out Cliff, FWIW it does seem to have been
>>>>>>> broadcast..
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Not sure though how a shuffle would be much different from a
>>>>>>> broadcast if entire table is 1 file/block in 1 node.
>>>>>>>
>>>>>>> On Wed, Apr 11, 2018 at 8:52 PM, Cliff Resnick <cresny@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> From the screenshot it does not look like there was a broadcast
of
>>>>>>>> the dimension table(s), so it could be the case here that
the multiple
>>>>>>>> smaller sends helps. Our dim tables are generally in the
single-digit
>>>>>>>> millions and Impala chooses to broadcast them. Since the
fact result
>>>>>>>> cardinality is always much smaller, we've found that forcing
a [shuffle]
>>>>>>>> dimension join is actually faster since it only sends dims
once rather than
>>>>>>>> all to all nodes. The degenerative performance of broadcast
is especially
>>>>>>>> obvious when the query returns zero results. I don't have
much experience
>>>>>>>> here, but it does seem that Kudu's efficient predicate scans
can sometimes
>>>>>>>> "break" Impala's query plan.
>>>>>>>>
>>>>>>>> -Cliff
>>>>>>>>
>>>>>>>> On Wed, Apr 11, 2018 at 5:41 PM, Mauricio Aristizabal <
>>>>>>>> mauricio@impact.com> wrote:
>>>>>>>>
>>>>>>>>> @Todd not to belabor the point, but when I suggested
breaking up
>>>>>>>>> small dim tables into multiple parquet files (and in
this thread's context
>>>>>>>>> perhaps partition kudu table, even if small, into multiple
tablets), it was
>>>>>>>>> to speed up joins/exchanges, not to parallelize the scan.
>>>>>>>>>
>>>>>>>>> For example recently we ran into this slow query where
the 14M
>>>>>>>>> record dimension fit into a single file & block,
so it got scanned on a
>>>>>>>>> single node though still pretty quickly (300ms), however
it caused the join
>>>>>>>>> to take 25+ seconds and bogged down the entire query.
 See highlighted
>>>>>>>>> fragment and its parent.
>>>>>>>>>
>>>>>>>>> So we broke it into several small files the way I described
in my
>>>>>>>>> previous post, and now join and query are fast (6s).
>>>>>>>>>
>>>>>>>>> -m
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Mar 16, 2018 at 3:55 PM, Todd Lipcon <todd@cloudera.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I suppose in the case that the dimension table scan
makes a
>>>>>>>>>> non-trivial portion of your workload time, then yea,
parallelizing the scan
>>>>>>>>>> as you suggest would be beneficial. That said, in
typical analytic queries,
>>>>>>>>>> scanning the dimension tables is very quick compared
to scanning the
>>>>>>>>>> much-larger fact tables, so the extra parallelism
on the dim table scan
>>>>>>>>>> isn't worth too much.
>>>>>>>>>>
>>>>>>>>>> -Todd
>>>>>>>>>>
>>>>>>>>>> On Fri, Mar 16, 2018 at 2:56 PM, Mauricio Aristizabal
<
>>>>>>>>>> mauricio@impactradius.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> @Todd I know working with parquet in the past
I've seen small
>>>>>>>>>>> dimensions that fit in 1 single file/block limit
parallelism of
>>>>>>>>>>> join/exchange/aggregation nodes, and I've forced
those dims to spread
>>>>>>>>>>> across 20 or so blocks by leveraging SET PARQUET_FILE_SIZE=8m;
or similar
>>>>>>>>>>> when doing INSERT OVERWRITE to load them, which
then allows these
>>>>>>>>>>> operations to parallelize across that many nodes.
>>>>>>>>>>>
>>>>>>>>>>> Wouldn't it be useful here for Cliff's small
dims to be
>>>>>>>>>>> partitioned into a couple tablets to similarly
improve parallelism?
>>>>>>>>>>>
>>>>>>>>>>> -m
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Mar 16, 2018 at 2:29 PM, Todd Lipcon
<todd@cloudera.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Mar 16, 2018 at 2:19 PM, Cliff Resnick
<
>>>>>>>>>>>> cresny@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey Todd,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for that explanation, as well
as all the great work
>>>>>>>>>>>>> you're doing  -- it's much appreciated!
I just have one last follow-up
>>>>>>>>>>>>> question. Reading about BROADCAST operations
(Kudu, Spark, Flink, etc. ) it
>>>>>>>>>>>>> seems the smaller table is always copied
in its entirety BEFORE the
>>>>>>>>>>>>> predicate is evaluated.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> That's not quite true. If you have a predicate
on a joined
>>>>>>>>>>>> column, or on one of the columns in the joined
table, it will be pushed
>>>>>>>>>>>> down to the "scan" operator, which happens
before the "exchange". In
>>>>>>>>>>>> addition, there is a feature called "runtime
filters" that can push
>>>>>>>>>>>> dynamically-generated filters from one side
of the exchange to the other.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> But since the Kudu client provides a
serialized scanner as
>>>>>>>>>>>>> part of the ScanToken API, why wouldn't
Impala use that instead if it knows
>>>>>>>>>>>>> that the table is Kudu and the query
has any type of predicate? Perhaps if
>>>>>>>>>>>>> I hash-partition the table I could maybe
force this (because that
>>>>>>>>>>>>> complicates a BROADCAST)? I guess this
is really a question for Impala but
>>>>>>>>>>>>> perhaps there is a more basic reason.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Impala could definitely be smarter, just
a matter of
>>>>>>>>>>>> programming Kudu-specific join strategies
into the optimizer. Today, the
>>>>>>>>>>>> optimizer isn't aware of the unique properties
of Kudu scans vs other
>>>>>>>>>>>> storage mechanisms.
>>>>>>>>>>>>
>>>>>>>>>>>> -Todd
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Cliff
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Mar 16, 2018 at 4:10 PM, Todd
Lipcon <
>>>>>>>>>>>>> todd@cloudera.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Mar 16, 2018 at 12:30 PM,
Clifford Resnick <
>>>>>>>>>>>>>> cresnick@mediamath.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I thought I had read that the
Kudu client can configure a
>>>>>>>>>>>>>>> scan for CLOSEST_REPLICA and
assumed this was a way to take advantage of
>>>>>>>>>>>>>>> data collocation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yea, when a client uses CLOSEST_REPLICA
it will read a local
>>>>>>>>>>>>>> one if available. However, that doesn't
influence the higher level
>>>>>>>>>>>>>> operation of the Impala (or Spark)
planner. The planner isn't aware of the
>>>>>>>>>>>>>> replication policy, so it will use
one of the existing supported JOIN
>>>>>>>>>>>>>> strategies. Given statistics, it
will choose to broadcast the small table,
>>>>>>>>>>>>>> which means that it will create a
plan that looks like:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                                 
  +-------------------------+
>>>>>>>>>>>>>>                                 
  |                         |
>>>>>>>>>>>>>>                         +---------->build
     JOIN          |
>>>>>>>>>>>>>>                         |       
  |                         |
>>>>>>>>>>>>>>                         |       
  |              probe      |
>>>>>>>>>>>>>>                  +--------------+
 +-------------------------+
>>>>>>>>>>>>>>                  |              |
                 |
>>>>>>>>>>>>>>                  | Exchange     |
                 |
>>>>>>>>>>>>>>             +----+ (broadcast   |
                 |
>>>>>>>>>>>>>>             |    |              |
                 |
>>>>>>>>>>>>>>             |    +--------------+
                 |
>>>>>>>>>>>>>>             |                   
                  |
>>>>>>>>>>>>>>       +---------+               
                  |
>>>>>>>>>>>>>>       |         |
>>>>>>>>>>>>>> +-----------------------+
>>>>>>>>>>>>>>       |  SCAN   |               
        |
>>>>>>>>>>>>>>    |
>>>>>>>>>>>>>>       |  KUDU   |               
        |   SCAN (other
>>>>>>>>>>>>>> side)   |
>>>>>>>>>>>>>>       |         |               
        |
>>>>>>>>>>>>>>    |
>>>>>>>>>>>>>>       +---------+
>>>>>>>>>>>>>> +-----------------------+
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (hopefully the ASCII art comes through)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In other words, the "scan kudu" operator
scans the table
>>>>>>>>>>>>>> once, and then replicates the results
of that scan into the JOIN operator.
>>>>>>>>>>>>>> The "scan kudu" operator of course
will read its local copy, but it will
>>>>>>>>>>>>>> still go through the exchange process.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For the use case you're talking about,
where the join is just
>>>>>>>>>>>>>> looking up a single row by PK in
a dimension table, ideally we'd be using
>>>>>>>>>>>>>> an altogether different join strategy
such as nested-loop join, with the
>>>>>>>>>>>>>> inner "loop" actually being a Kudu
PK lookup, but that strategy isn't
>>>>>>>>>>>>>> implemented by Impala.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  If this exists then how far
out of context is my
>>>>>>>>>>>>>>> understanding of it? Reading
about HDFS cache replication, I do know that
>>>>>>>>>>>>>>> Impala will choose a random replica
there to more evenly distribute load.
>>>>>>>>>>>>>>> But especially compared to Kudu
upsert, managing mutable data using Parquet
>>>>>>>>>>>>>>> is painful. So, perhaps to sum
thing up, if nearly 100% of my metadata scan
>>>>>>>>>>>>>>> are single Primary Key lookups
followed by a tiny broadcast then am I
>>>>>>>>>>>>>>> really just splitting hairs performance-wise
between Kudu and HDFS-cached
>>>>>>>>>>>>>>> parquet?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> From:  Todd Lipcon <todd@cloudera.com>
>>>>>>>>>>>>>>> Reply-To: "user@kudu.apache.org"
<user@kudu.apache.org>
>>>>>>>>>>>>>>> Date: Friday, March 16, 2018
at 2:51 PM
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To: "user@kudu.apache.org" <user@kudu.apache.org>
>>>>>>>>>>>>>>> Subject: Re: "broadcast" tablet
replication for kudu?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It's worth noting that, even
if your table is replicated,
>>>>>>>>>>>>>>> Impala's planner is unaware of
this fact and it will give the same plan
>>>>>>>>>>>>>>> regardless. That is to say, rather
than every node scanning its local copy,
>>>>>>>>>>>>>>> instead a single node will perform
the whole scan (assuming it's a small
>>>>>>>>>>>>>>> table) and broadcast it from
there within the scope of a single query. So,
>>>>>>>>>>>>>>> I don't think you'll see any
performance improvements on Impala queries by
>>>>>>>>>>>>>>> attempting something like an
extremely high replication count.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I could see bumping the replication
count to 5 for these
>>>>>>>>>>>>>>> tables since the extra storage
cost is low and it will ensure higher
>>>>>>>>>>>>>>> availability of the important
central tables, but I'd be surprised if there
>>>>>>>>>>>>>>> is any measurable perf impact.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Todd
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Mar 16, 2018 at 11:35
AM, Clifford Resnick <
>>>>>>>>>>>>>>> cresnick@mediamath.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for that, glad I was
wrong there! Aside from
>>>>>>>>>>>>>>>> replication considerations,
is it also recommended the number of tablet
>>>>>>>>>>>>>>>> servers be odd?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I will check forums as you
suggested, but from what I read
>>>>>>>>>>>>>>>> after searching is that Impala
relies on user configured caching strategies
>>>>>>>>>>>>>>>> using HDFS cache.  The workload
for these tables is very light write, maybe
>>>>>>>>>>>>>>>> a dozen or so records per
hour across 6 or 7 tables. The size of the tables
>>>>>>>>>>>>>>>> ranges from thousands to
low millions of rows so so sub-partitioning would
>>>>>>>>>>>>>>>> not be required. So perhaps
this is not a typical use-case but I think it
>>>>>>>>>>>>>>>> could work quite well with
kudu.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> From: Dan Burkert <danburkert@apache.org>
>>>>>>>>>>>>>>>> Reply-To: "user@kudu.apache.org"
<user@kudu.apache.org>
>>>>>>>>>>>>>>>> Date: Friday, March 16, 2018
at 2:09 PM
>>>>>>>>>>>>>>>> To: "user@kudu.apache.org"
<user@kudu.apache.org>
>>>>>>>>>>>>>>>> Subject: Re: "broadcast"
tablet replication for kudu?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The replication count is
the number of tablet servers which
>>>>>>>>>>>>>>>> Kudu will host copies on.
 So if you set the replication level to 5, Kudu
>>>>>>>>>>>>>>>> will put the data on 5 separate
tablet servers.  There's no built-in
>>>>>>>>>>>>>>>> broadcast table feature;
upping the replication factor is the closest
>>>>>>>>>>>>>>>> thing.  A couple of things
to keep in mind:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Always use an odd replication
count.  This is important
>>>>>>>>>>>>>>>> due to how the Raft algorithm
works.  Recent versions of Kudu won't even
>>>>>>>>>>>>>>>> let you specify an even number
without flipping some flags.
>>>>>>>>>>>>>>>> - We don't test much much
beyond 5 replicas.  It *should*
>>>>>>>>>>>>>>>> work, but you may run in
to issues since it's a relatively rare
>>>>>>>>>>>>>>>> configuration.  With a heavy
write workload and many replicas you are even
>>>>>>>>>>>>>>>> more likely to encounter
issues.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It's also worth checking
in an Impala forum whether it has
>>>>>>>>>>>>>>>> features that make joins
against small broadcast tables better?  Perhaps
>>>>>>>>>>>>>>>> Impala can cache small tables
locally when doing joins.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Dan
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Mar 16, 2018 at 10:55
AM, Clifford Resnick <
>>>>>>>>>>>>>>>> cresnick@mediamath.com>
wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The problem is, AFIK,
that replication count is not
>>>>>>>>>>>>>>>>> necessarily the distribution
count, so you can't guarantee all tablet
>>>>>>>>>>>>>>>>> servers will have a copy.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mar 16, 2018 1:41
PM, Boris Tyukin <
>>>>>>>>>>>>>>>>> boris@boristyukin.com>
wrote:
>>>>>>>>>>>>>>>>> I'm new to Kudu but we
are also going to use Impala mostly
>>>>>>>>>>>>>>>>> with Kudu. We have a
few tables that are small but used a lot. My plan is
>>>>>>>>>>>>>>>>> replicate them more than
3 times. When you create a kudu table, you can
>>>>>>>>>>>>>>>>> specify number of replicated
copies (3 by default) and I guess you can put
>>>>>>>>>>>>>>>>> there a number, corresponding
to your node count in cluster. The downside,
>>>>>>>>>>>>>>>>> you cannot change that
number unless you recreate a table.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Mar 16, 2018
at 10:42 AM, Cliff Resnick <
>>>>>>>>>>>>>>>>> cresny@gmail.com>
wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We will soon be moving
our analytics from AWS Redshift to
>>>>>>>>>>>>>>>>>> Impala/Kudu. One
Redshift feature that we will miss is its ALL
>>>>>>>>>>>>>>>>>> Distribution, where
a copy of a table is maintained on each server. We
>>>>>>>>>>>>>>>>>> define a number of
metadata tables this way since they are used in nearly
>>>>>>>>>>>>>>>>>> every query. We are
considering using parquet in HDFS cache for these, and
>>>>>>>>>>>>>>>>>> Kudu would be a much
better fit for the update semantics but we are worried
>>>>>>>>>>>>>>>>>> about the additional
contention.  I'm wondering if having a Broadcast, or
>>>>>>>>>>>>>>>>>> ALL, tablet replication
might be an easy feature to add to Kudu?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Cliff
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Todd Lipcon
>>>>>>>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Todd Lipcon
>>>>>>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Todd Lipcon
>>>>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *MAURICIO ARISTIZABAL*
>>>>>>>>>>> Architect - Business Intelligence + Data Science
>>>>>>>>>>> mauricio@impactradius.com(m)+1 323 309 4260 <(323)%20309-4260>
>>>>>>>>>>> 223 E. De La Guerra St. | Santa Barbara, CA 93101
>>>>>>>>>>> <https://maps.google.com/?q=223+E.+De+La+Guerra+St.+%7C+Santa+Barbara,+CA+93101&entry=gmail&source=g>
>>>>>>>>>>>
>>>>>>>>>>> Overview <http://www.impactradius.com/?src=slsap>
| Twitter
>>>>>>>>>>> <https://twitter.com/impactradius> | Facebook
>>>>>>>>>>> <https://www.facebook.com/pages/Impact-Radius/153376411365183>
>>>>>>>>>>>  | LinkedIn
>>>>>>>>>>> <https://www.linkedin.com/company/impact-radius-inc->
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Todd Lipcon
>>>>>>>>>> Software Engineer, Cloudera
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Mauricio Aristizabal
>>>>>>>>> Architect - Data Pipeline
>>>>>>>>> *M * 323 309 4260
>>>>>>>>> *E  *mauricio@impact.com  |  *W * https://impact.com
>>>>>>>>> <https://www.linkedin.com/company/608678/>
>>>>>>>>> <https://www.facebook.com/ImpactMarTech/>
>>>>>>>>> <https://twitter.com/impactmartech>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Mauricio Aristizabal
>>>>>>> Architect - Data Pipeline
>>>>>>> *M * 323 309 4260
>>>>>>> *E  *mauricio@impact.com  |  *W * https://impact.com
>>>>>>> <https://www.linkedin.com/company/608678/>
>>>>>>> <https://www.facebook.com/ImpactMarTech/>
>>>>>>> <https://twitter.com/impactmartech>
>>>>>>>
>>>>>>
>>>>>>

-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
View raw message