kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Tyukin <bo...@boristyukin.com>
Subject Re: first and second run 2x query time difference
Date Wed, 03 Jan 2018 20:20:22 GMT
it is possible but I thought Kudu keeps its stuff in its own folders

On Wed, Jan 3, 2018 at 1:45 PM, Jean-Daniel Cryans <jdcryans@apache.org>
wrote:

> Hey Boris,
>
> Thanks for reporting back with results!
>
> On Wed, Jan 3, 2018 at 10:38 AM, Boris Tyukin <boris@boristyukin.com>
> wrote:
>
>> so it was the page cache that makes this difference. we did a series of
>> tests either restarting Kudu only, Impala only or both and resetting or not
>> touching page cache.
>>
>> as for Kudu failures after restart, it was a sequence of services that
>> need to be started before Kudu. If we start Kudu after HDFS, everything is
>> fine. Data is intact
>>
>
> Is it possible that Kudu is sharing disks with ZK?
>
>
>>
>> thanks again for your help, J-D
>>
>> On Sat, Dec 16, 2017 at 4:05 PM, Jean-Daniel Cryans <jdcryans@apache.org>
>> wrote:
>>
>>> I'm more thinking in terms of the startup IO having some impact on the
>>> co-located services, but we really need to know what "went down" means.
>>>
>>> On Sat, Dec 16, 2017 at 12:50 PM, Boris Tyukin <boris@boristyukin.com>
>>> wrote:
>>>
>>>> yep it is really weird since Kudu does not use neither one. I'll get
>>>> with him on Monday to gather more details
>>>>
>>>> On Sat, Dec 16, 2017 at 3:28 PM, Jean-Daniel Cryans <
>>>> jdcryans@apache.org> wrote:
>>>>
>>>>> Hi Boris,
>>>>>
>>>>> How exactly did HDFS and ZK go down? A Kudu restart is fairly
>>>>> IO-intensive but I don't know how that can cause things like DataNodes
to
>>>>> fail.
>>>>>
>>>>> J-D
>>>>>
>>>>> On Sat, Dec 16, 2017 at 11:45 AM, Boris Tyukin <boris@boristyukin.com>
>>>>> wrote:
>>>>>
>>>>>> well our admin had fun two days - it was the first time we restarted
>>>>>> Kudu on our DEV cluster and it did not go well. He is still troubleshooting
>>>>>> what happened but after Kudu restart zookeeper and HDFS went down
after 3-4
>>>>>> minutes. If we disable Kudu, all is well. No error in Kudu logs...I
will
>>>>>> have more details next week so not asking for help as I do not know
all the
>>>>>> details. What is obvious thought is that it has to do something with
Kudu :)
>>>>>>
>>>>>> On Thu, Dec 14, 2017 at 9:40 AM, Boris Tyukin <boris@boristyukin.com>
>>>>>> wrote:
>>>>>>
>>>>>>> thanks for your suggestions, J-D, I am sure you are right more
often
>>>>>>> than that! :))
>>>>>>>
>>>>>>> I will report back with our results. So far I am really impressed
>>>>>>> with Kudu - we have been benchmarking ingest and egress throughput
and our
>>>>>>> typical queries runtime. The biggest pain so far is lack of support
for
>>>>>>> decimals
>>>>>>>
>>>>>>> On Wed, Dec 13, 2017 at 5:07 PM, Jean-Daniel Cryans <
>>>>>>> jdcryans@apache.org> wrote:
>>>>>>>
>>>>>>>> On Wed, Dec 13, 2017 at 11:30 AM, Boris Tyukin <
>>>>>>>> boris@boristyukin.com> wrote:
>>>>>>>>
>>>>>>>>> thanks J-D! we are going to try that and see how it impacts
the
>>>>>>>>> runtime.
>>>>>>>>>
>>>>>>>>> is there any way to load this metadata upfront? a lot
of our
>>>>>>>>> queries are adhoc in nature but they will be hitting
the same tables with
>>>>>>>>> different predicates and join patterns though.
>>>>>>>>>
>>>>>>>>
>>>>>>>> You could use Impala to compute all the stats of all the
tables
>>>>>>>> after each Kudu restart. Actually, do try that, restart Kudu
then compute
>>>>>>>> stats and see how fast it scans.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I am curious why this metadata does not survive restarts
though.
>>>>>>>>> We are going to run our benchmarks again and this time
restart Kudu and
>>>>>>>>> Impala.
>>>>>>>>>
>>>>>>>>
>>>>>>>> It's in the tserver memory, it can't survive a restart.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I just ran another query first time which hits 2 large
tables and
>>>>>>>>> these tables have been scanned by the previous query
and this time I do not
>>>>>>>>> see any difference in query time before the first and
second time - I guess
>>>>>>>>> this confirms your statement about " first time ever
scanning the
>>>>>>>>> table since a Kudu restart" and collecting metadata.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Maybe, I've been known to be right once or twice a year :)
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Dec 13, 2017 at 11:18 AM, Jean-Daniel Cryans
<
>>>>>>>>> jdcryans@apache.org> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Boris,
>>>>>>>>>>
>>>>>>>>>> Given that we don't have much data we can use here,
I'll have to
>>>>>>>>>> extrapolate. As an aside though, this is yet another
example where we need
>>>>>>>>>> more Kudu-side metrics in the query profile.
>>>>>>>>>>
>>>>>>>>>> So, Kudu lazily loads a bunch of metadata and that
can really
>>>>>>>>>> affect scan times. If this was your first time ever
scanning the table
>>>>>>>>>> since a Kudu restart, it's very possible that that's
where that time was
>>>>>>>>>> spent. There's also the page cache in the OS that
might now be populated.
>>>>>>>>>> You could do something like "sync; echo 3 > /proc/sys/vm/drop_caches"
on
>>>>>>>>>> all the machines and run the query 2 times again,
without restarting Kudu,
>>>>>>>>>> to understand the effect of the page cache itself.
There's currently now
>>>>>>>>>> way to purge the cached metadata in Kudu though.
>>>>>>>>>>
>>>>>>>>>> Hope this helps a bit,
>>>>>>>>>>
>>>>>>>>>> J-D
>>>>>>>>>>
>>>>>>>>>> On Wed, Dec 13, 2017 at 8:07 AM, Boris Tyukin <
>>>>>>>>>> boris@boristyukin.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi guys,
>>>>>>>>>>>
>>>>>>>>>>> I am doing some benchmarks with Kudu and Impala/Parquet
and hope
>>>>>>>>>>> to share it soon but there is one thing that
bugs me. This is perhaps
>>>>>>>>>>> Impala question but since I am using Kudu with
Impala I am going to try and
>>>>>>>>>>> ask anyway.
>>>>>>>>>>>
>>>>>>>>>>> One of my queries takes 120 seconds to run the
very first time.
>>>>>>>>>>> It joins one large 5B row table with a bunch
of smaller tables and then
>>>>>>>>>>> stores result in Impala/parquet (not Kudu).
>>>>>>>>>>>
>>>>>>>>>>> Now if I run it second and third time, it only
takes 60 seconds.
>>>>>>>>>>> Can someone explain why? Is there any settings
to decrease this gap?
>>>>>>>>>>>
>>>>>>>>>>> I've compared query profiles in CM and the only
thing that was
>>>>>>>>>>> very different is scan against Kudu table (the
large one):
>>>>>>>>>>>
>>>>>>>>>>> ***************************
>>>>>>>>>>> first time:
>>>>>>>>>>> ***************************
>>>>>>>>>>> KUDU_SCAN_NODE (id=0) (47.68s)
>>>>>>>>>>> <https://lkmaorabd103.multihosp.net:7183/cmf/impala/queryDetails?queryId=5143f7165be82819%3Ae00a103500000000&serviceName=impala#>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>    - BytesRead: *0 B*
>>>>>>>>>>>    - InactiveTotalTime: *0ns*
>>>>>>>>>>>    - KuduRemoteScanTokens: *0*
>>>>>>>>>>>    - NumScannerThreadsStarted: *20*
>>>>>>>>>>>    - PeakMemoryUsage: *35.8 MiB*
>>>>>>>>>>>    - RowsRead: *693,502,241*
>>>>>>>>>>>    - RowsReturned: *693,502,241*
>>>>>>>>>>>    - RowsReturnedRate: *14643448 per second*
>>>>>>>>>>>    - ScanRangesComplete: *20*
>>>>>>>>>>>    - ScannerThreadsInvoluntaryContextSwitches:
*1,341*
>>>>>>>>>>>    - ScannerThreadsTotalWallClockTime: *36.2m*
>>>>>>>>>>>       - MaterializeTupleTime(*): *47.57s*
>>>>>>>>>>>       - ScannerThreadsSysTime: *31.42s*
>>>>>>>>>>>       - ScannerThreadsUserTime: *1.7m*
>>>>>>>>>>>    - ScannerThreadsVoluntaryContextSwitches:
*96,855*
>>>>>>>>>>>    - TotalKuduScanRoundTrips: *52,308*
>>>>>>>>>>>    - TotalReadThroughput: *0 B/s*
>>>>>>>>>>>    - TotalTime: *47.68s*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ***************************
>>>>>>>>>>> second time:
>>>>>>>>>>> ***************************
>>>>>>>>>>> KUDU_SCAN_NODE (id=0) (4.28s)
>>>>>>>>>>> <https://lkmaorabd103.multihosp.net:7183/cmf/impala/queryDetails?queryId=53497a308f860837%3A243772e000000000&serviceName=impala#>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>    - BytesRead: *0 B*
>>>>>>>>>>>    - InactiveTotalTime: *0ns*
>>>>>>>>>>>    - KuduRemoteScanTokens: *0*
>>>>>>>>>>>    - NumScannerThreadsStarted: *20*
>>>>>>>>>>>    - PeakMemoryUsage: *37.9 MiB*
>>>>>>>>>>>    - RowsRead: *693,502,241*
>>>>>>>>>>>    - RowsReturned: *693,502,241*
>>>>>>>>>>>    - RowsReturnedRate: *173481534 per second*
>>>>>>>>>>>    - ScanRangesComplete: *20*
>>>>>>>>>>>    - ScannerThreadsInvoluntaryContextSwitches:
*1,451*
>>>>>>>>>>>    - ScannerThreadsTotalWallClockTime: *19.5m*
>>>>>>>>>>>       - MaterializeTupleTime(*): *4.20s*
>>>>>>>>>>>       - ScannerThreadsSysTime: *38.22s*
>>>>>>>>>>>       - ScannerThreadsUserTime: *1.7m*
>>>>>>>>>>>    - ScannerThreadsVoluntaryContextSwitches:
*480,870*
>>>>>>>>>>>    - TotalKuduScanRoundTrips: *52,142*
>>>>>>>>>>>    - TotalReadThroughput: *0 B/s*
>>>>>>>>>>>    - TotalTime: *4.28s*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message