kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adar Lieber-Dembo <a...@cloudera.com>
Subject Re: tablet server not using all resources availables
Date Sun, 21 Apr 2019 20:56:56 GMT
500 MB RAM per VM is very low. Is this a typo? If not, it's the first
thing you should tackle.

As for your questions:
- You can look into increasing --maintenance_manager_num_threads. We
typically recommend a 1:3 ratio of threads to number of disks, but
that's when using spinning disks; with fast nvme disks, you can
explore 1:1, 2:1, or higher. If each VM has multiple disks, you should
let Kudu stripe its data across all of the disks (via --fs_data_dirs).
You should also make sure that your tables are sufficiently
partitioned so as to take advantage of every machine in the cluster. I
recommend you read through
https://kudu.apache.org/docs/schema_design.html if you haven't
already.
- You should definitely NOT run multiple tservers on the same VM.
Besides crippling your fault tolerance (unless you go out of your way
to set up location awareness), it's just not necessary: each tserver
is architected to take advantage of all of the machine's resources if
you let it).

On Fri, Apr 19, 2019 at 4:37 AM Manuel Sopena Ballesteros
<manuel.sb@garvan.org.au> wrote:
>
> Dear Kudu community,
>
> We have a test kudu cluster with 3 vms and each one of them has a master and a tablet
server.
>
> Each vm has:
> x50 vcpus
> 500 MB RAM
> 4TB virtual disks with xfs on local hypervisor nvme drives
>
> We started an ETL process using apache spark and realised that the vms are not fully
utilised (~15 vcpus are used and the iowait time is 0). That tell us that kudu is not using
all the resources available on the machine and the system is not fully optimised.
>
> Question:
>
> How could we configure the tablet server to fully utilize the resources available?
> Is it a good practice to run multiple tablet server on the same vm?
>
> thank you very much
>
>
>
>
> NOTICE
> Please consider the environment before printing this email. This message and any attachments
are intended for the addressee named and may contain legally privileged/confidential/copyright
information. If you are not the intended recipient, you should not read, use, disclose, copy
or distribute this communication. If you have received this message in error please notify
us at once by return email and then delete both messages. We accept no liability for the distribution
of viruses or similar in electronic communications. This notice should not be removed.

Mime
View raw message