kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Tyukin <bo...@boristyukin.com>
Subject Re: first and second run 2x query time difference
Date Wed, 03 Jan 2018 18:38:54 GMT
so it was the page cache that makes this difference. we did a series of
tests either restarting Kudu only, Impala only or both and resetting or not
touching page cache.

as for Kudu failures after restart, it was a sequence of services that need
to be started before Kudu. If we start Kudu after HDFS, everything is fine.
Data is intact

thanks again for your help, J-D

On Sat, Dec 16, 2017 at 4:05 PM, Jean-Daniel Cryans <jdcryans@apache.org>
wrote:

> I'm more thinking in terms of the startup IO having some impact on the
> co-located services, but we really need to know what "went down" means.
>
> On Sat, Dec 16, 2017 at 12:50 PM, Boris Tyukin <boris@boristyukin.com>
> wrote:
>
>> yep it is really weird since Kudu does not use neither one. I'll get with
>> him on Monday to gather more details
>>
>> On Sat, Dec 16, 2017 at 3:28 PM, Jean-Daniel Cryans <jdcryans@apache.org>
>> wrote:
>>
>>> Hi Boris,
>>>
>>> How exactly did HDFS and ZK go down? A Kudu restart is fairly
>>> IO-intensive but I don't know how that can cause things like DataNodes to
>>> fail.
>>>
>>> J-D
>>>
>>> On Sat, Dec 16, 2017 at 11:45 AM, Boris Tyukin <boris@boristyukin.com>
>>> wrote:
>>>
>>>> well our admin had fun two days - it was the first time we restarted
>>>> Kudu on our DEV cluster and it did not go well. He is still troubleshooting
>>>> what happened but after Kudu restart zookeeper and HDFS went down after 3-4
>>>> minutes. If we disable Kudu, all is well. No error in Kudu logs...I will
>>>> have more details next week so not asking for help as I do not know all the
>>>> details. What is obvious thought is that it has to do something with Kudu
:)
>>>>
>>>> On Thu, Dec 14, 2017 at 9:40 AM, Boris Tyukin <boris@boristyukin.com>
>>>> wrote:
>>>>
>>>>> thanks for your suggestions, J-D, I am sure you are right more often
>>>>> than that! :))
>>>>>
>>>>> I will report back with our results. So far I am really impressed with
>>>>> Kudu - we have been benchmarking ingest and egress throughput and our
>>>>> typical queries runtime. The biggest pain so far is lack of support for
>>>>> decimals
>>>>>
>>>>> On Wed, Dec 13, 2017 at 5:07 PM, Jean-Daniel Cryans <
>>>>> jdcryans@apache.org> wrote:
>>>>>
>>>>>> On Wed, Dec 13, 2017 at 11:30 AM, Boris Tyukin <boris@boristyukin.com
>>>>>> > wrote:
>>>>>>
>>>>>>> thanks J-D! we are going to try that and see how it impacts the
>>>>>>> runtime.
>>>>>>>
>>>>>>> is there any way to load this metadata upfront? a lot of our
queries
>>>>>>> are adhoc in nature but they will be hitting the same tables
with different
>>>>>>> predicates and join patterns though.
>>>>>>>
>>>>>>
>>>>>> You could use Impala to compute all the stats of all the tables after
>>>>>> each Kudu restart. Actually, do try that, restart Kudu then compute
stats
>>>>>> and see how fast it scans.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> I am curious why this metadata does not survive restarts though.
We
>>>>>>> are going to run our benchmarks again and this time restart Kudu
and Impala.
>>>>>>>
>>>>>>
>>>>>> It's in the tserver memory, it can't survive a restart.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> I just ran another query first time which hits 2 large tables
and
>>>>>>> these tables have been scanned by the previous query and this
time I do not
>>>>>>> see any difference in query time before the first and second
time - I guess
>>>>>>> this confirms your statement about " first time ever scanning
the
>>>>>>> table since a Kudu restart" and collecting metadata.
>>>>>>>
>>>>>>
>>>>>> Maybe, I've been known to be right once or twice a year :)
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Dec 13, 2017 at 11:18 AM, Jean-Daniel Cryans <
>>>>>>> jdcryans@apache.org> wrote:
>>>>>>>
>>>>>>>> Hi Boris,
>>>>>>>>
>>>>>>>> Given that we don't have much data we can use here, I'll
have to
>>>>>>>> extrapolate. As an aside though, this is yet another example
where we need
>>>>>>>> more Kudu-side metrics in the query profile.
>>>>>>>>
>>>>>>>> So, Kudu lazily loads a bunch of metadata and that can really
>>>>>>>> affect scan times. If this was your first time ever scanning
the table
>>>>>>>> since a Kudu restart, it's very possible that that's where
that time was
>>>>>>>> spent. There's also the page cache in the OS that might now
be populated.
>>>>>>>> You could do something like "sync; echo 3 > /proc/sys/vm/drop_caches"
on
>>>>>>>> all the machines and run the query 2 times again, without
restarting Kudu,
>>>>>>>> to understand the effect of the page cache itself. There's
currently now
>>>>>>>> way to purge the cached metadata in Kudu though.
>>>>>>>>
>>>>>>>> Hope this helps a bit,
>>>>>>>>
>>>>>>>> J-D
>>>>>>>>
>>>>>>>> On Wed, Dec 13, 2017 at 8:07 AM, Boris Tyukin <
>>>>>>>> boris@boristyukin.com> wrote:
>>>>>>>>
>>>>>>>>> Hi guys,
>>>>>>>>>
>>>>>>>>> I am doing some benchmarks with Kudu and Impala/Parquet
and hope
>>>>>>>>> to share it soon but there is one thing that bugs me.
This is perhaps
>>>>>>>>> Impala question but since I am using Kudu with Impala
I am going to try and
>>>>>>>>> ask anyway.
>>>>>>>>>
>>>>>>>>> One of my queries takes 120 seconds to run the very first
time. It
>>>>>>>>> joins one large 5B row table with a bunch of smaller
tables and then stores
>>>>>>>>> result in Impala/parquet (not Kudu).
>>>>>>>>>
>>>>>>>>> Now if I run it second and third time, it only takes
60 seconds.
>>>>>>>>> Can someone explain why? Is there any settings to decrease
this gap?
>>>>>>>>>
>>>>>>>>> I've compared query profiles in CM and the only thing
that was
>>>>>>>>> very different is scan against Kudu table (the large
one):
>>>>>>>>>
>>>>>>>>> ***************************
>>>>>>>>> first time:
>>>>>>>>> ***************************
>>>>>>>>> KUDU_SCAN_NODE (id=0) (47.68s)
>>>>>>>>> <https://lkmaorabd103.multihosp.net:7183/cmf/impala/queryDetails?queryId=5143f7165be82819%3Ae00a103500000000&serviceName=impala#>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>    - BytesRead: *0 B*
>>>>>>>>>    - InactiveTotalTime: *0ns*
>>>>>>>>>    - KuduRemoteScanTokens: *0*
>>>>>>>>>    - NumScannerThreadsStarted: *20*
>>>>>>>>>    - PeakMemoryUsage: *35.8 MiB*
>>>>>>>>>    - RowsRead: *693,502,241*
>>>>>>>>>    - RowsReturned: *693,502,241*
>>>>>>>>>    - RowsReturnedRate: *14643448 per second*
>>>>>>>>>    - ScanRangesComplete: *20*
>>>>>>>>>    - ScannerThreadsInvoluntaryContextSwitches: *1,341*
>>>>>>>>>    - ScannerThreadsTotalWallClockTime: *36.2m*
>>>>>>>>>       - MaterializeTupleTime(*): *47.57s*
>>>>>>>>>       - ScannerThreadsSysTime: *31.42s*
>>>>>>>>>       - ScannerThreadsUserTime: *1.7m*
>>>>>>>>>    - ScannerThreadsVoluntaryContextSwitches: *96,855*
>>>>>>>>>    - TotalKuduScanRoundTrips: *52,308*
>>>>>>>>>    - TotalReadThroughput: *0 B/s*
>>>>>>>>>    - TotalTime: *47.68s*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ***************************
>>>>>>>>> second time:
>>>>>>>>> ***************************
>>>>>>>>> KUDU_SCAN_NODE (id=0) (4.28s)
>>>>>>>>> <https://lkmaorabd103.multihosp.net:7183/cmf/impala/queryDetails?queryId=53497a308f860837%3A243772e000000000&serviceName=impala#>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>    - BytesRead: *0 B*
>>>>>>>>>    - InactiveTotalTime: *0ns*
>>>>>>>>>    - KuduRemoteScanTokens: *0*
>>>>>>>>>    - NumScannerThreadsStarted: *20*
>>>>>>>>>    - PeakMemoryUsage: *37.9 MiB*
>>>>>>>>>    - RowsRead: *693,502,241*
>>>>>>>>>    - RowsReturned: *693,502,241*
>>>>>>>>>    - RowsReturnedRate: *173481534 per second*
>>>>>>>>>    - ScanRangesComplete: *20*
>>>>>>>>>    - ScannerThreadsInvoluntaryContextSwitches: *1,451*
>>>>>>>>>    - ScannerThreadsTotalWallClockTime: *19.5m*
>>>>>>>>>       - MaterializeTupleTime(*): *4.20s*
>>>>>>>>>       - ScannerThreadsSysTime: *38.22s*
>>>>>>>>>       - ScannerThreadsUserTime: *1.7m*
>>>>>>>>>    - ScannerThreadsVoluntaryContextSwitches: *480,870*
>>>>>>>>>    - TotalKuduScanRoundTrips: *52,142*
>>>>>>>>>    - TotalReadThroughput: *0 B/s*
>>>>>>>>>    - TotalTime: *4.28s*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message