manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Many sleep process in MySQL while crawling files using Window share connection
Date Wed, 12 Dec 2012 09:27:02 GMT
The MySQL threads correspond to handles in the ManifoldCF handle pool.
 Since a worker thread can use only one handle at a time, one expects
that at best the number of MySQL processes that are active during a
crawl are about equal to the number of ManifoldCF worker threads.  If
this is not true it indicates low database use - which may be OK,
depending on your crawl, because of throttle settings.  For example,
if you are crawling only N domains and you have more than N worker
threads, some of these threads will have to wait.

However, if your CPU is 100%, and that is all going into ONE MySQL
process, it means that one query is blocking all the rest.  This would
usually be the stuffing query, which is the one we have been looking
at over the last couple of days.  This query must be fast for
ManifoldCF to use its resources well; if it takes a long time to run,
the rest of the worker threads get nothing to do.

A good way of assessing the state of ManifoldCF under these conditions
is to get a thread dump (which can be gotten with kill -QUIT on Linux
systems).  Look at the worker threads and see what they are doing.  If
you send me a dump, I will interpret it for you.

Thanks,
Karl


On Wed, Dec 12, 2012 at 2:52 AM, Shigeki Kobayashi
<shigeki.kobayashi3@g.softbank.co.jp> wrote:
>
>
> Hi
>
> I run MCF1.1dev downloaded at Dec. 11th with Mysql5.5
> While crawling, I listed process in MySQL and realized there are so many
> process that are sleeping.
> I set org.apache.manifoldcf.database.maxhandles to 100.
>
> Does this mean that MCF does not handle MySQL process appropriately?
> I feel strange that even though there are many process created, they are not
> used much.
> I see CPU usages 100% in mysql but the cpu state is shown as sleep. Do you
> think this is related to the sleep process in MySQL?
> Is this a correct behavior?
>
> mysql> show processlist;
> +----+------------+-----------------+------------+---------+------+--------+------------------------------------------------------------------------------------------------------+
> | Id | User       | Host            | db         | Command | Time | State  |
> Info
> |
> +----+------------+-----------------+------------+---------+------+--------+------------------------------------------------------------------------------------------------------+
> |  1 | manifoldcf | localhost:37683 | manifoldcf | Sleep   |  279 |        |
> NULL
> |
> |  2 | manifoldcf | localhost:37684 | manifoldcf | Query   |    0 | update |
> INSERT INTO ingeststatus
> (id,changecount,dockey,firstingest,connectionname,authorityname,urihash,las
> |
> |  3 | manifoldcf | localhost:37685 | manifoldcf | Sleep   |  279 |        |
> NULL
> |
> |  4 | manifoldcf | localhost:37686 | manifoldcf | Sleep   |   24 |        |
> NULL
> |
> |  5 | manifoldcf | localhost:37687 | manifoldcf | Sleep   |   24 |        |
> NULL
> |
> |  6 | manifoldcf | localhost:37688 | manifoldcf | Sleep   |  217 |        |
> NULL
> |
> |  7 | manifoldcf | localhost:37689 | manifoldcf | Sleep   |  279 |        |
> NULL
> |
> |  8 | manifoldcf | localhost:37690 | manifoldcf | Sleep   |   12 |        |
> NULL
> |
> |  9 | manifoldcf | localhost:37694 | manifoldcf | Sleep   |  279 |        |
> NULL
> |
> | 10 | manifoldcf | localhost:37695 | manifoldcf | Sleep   |    0 |        |
> NULL
> |
> | 11 | manifoldcf | localhost:37696 | manifoldcf | Sleep   |   24 |        |
> NULL
> |
> | 12 | manifoldcf | localhost:37697 | manifoldcf | Sleep   |  279 |        |
> NULL
> |
> | 13 | manifoldcf | localhost:37698 | manifoldcf | Sleep   |  279 |        |
> NULL
> |
> | 14 | manifoldcf | localhost:37699 | manifoldcf | Sleep   |  279 |        |
> NULL
> |
> | 15 | manifoldcf | localhost:37700 | manifoldcf | Sleep   |  217 |        |
> NULL
> |
> | 16 | manifoldcf | localhost:37701 | manifoldcf | Sleep   |   24 |        |
> NULL
> |
> | 17 | manifoldcf | localhost:37703 | manifoldcf | Sleep   |   24 |        |
> NULL
> |
> | 18 | manifoldcf | localhost:37732 | manifoldcf | Sleep   |  217 |        |
> NULL
> |
> | 19 | manifoldcf | localhost:37733 | manifoldcf | Sleep   |    5 |        |
> NULL
> |
> | 20 | manifoldcf | localhost:37734 | manifoldcf | Sleep   |   24 |        |
> NULL
> |
> | 21 | manifoldcf | localhost:37735 | manifoldcf | Sleep   |  217 |        |
> NULL
> |
> | 22 | manifoldcf | localhost:37736 | manifoldcf | Sleep   |    0 |        |
> NULL
> |
> | 23 | manifoldcf | localhost:37737 | manifoldcf | Sleep   |  217 |        |
> NULL
> |
> | 24 | manifoldcf | localhost:37738 | manifoldcf | Sleep   |  217 |        |
> NULL
> |
> | 25 | manifoldcf | localhost:37739 | manifoldcf | Sleep   |   24 |        |
> NULL
> |
> | 26 | manifoldcf | localhost:37740 | manifoldcf | Sleep   |   24 |        |
> NULL
> |
> | 27 | manifoldcf | localhost:39340 | manifoldcf | Sleep   |  279 |        |
> NULL
> |
> | 28 | manifoldcf | localhost:39341 | manifoldcf | Sleep   |    3 |        |
> NULL
> |
> | 29 | manifoldcf | localhost:39342 | manifoldcf | Sleep   |    0 |        |
> NULL
> |
> | 30 | manifoldcf | localhost:39343 | manifoldcf | Query   |    0 | NULL   |
> COMMIT
> |
> | 31 | manifoldcf | localhost:39739 | manifoldcf | Sleep   |   24 |        |
> NULL
> |
> | 32 | manifoldcf | localhost:41759 | manifoldcf | Sleep   |  279 |        |
> NULL
> |
> | 33 | manifoldcf | localhost:41760 | manifoldcf | Sleep   |  279 |        |
> NULL
> |
> | 34 | manifoldcf | localhost:41761 | manifoldcf | Sleep   |   24 |        |
> NULL
> |
> | 35 | manifoldcf | localhost:41762 | manifoldcf | Sleep   |  279 |        |
> NULL
> |
> | 36 | manifoldcf | localhost:41763 | manifoldcf | Sleep   |  279 |        |
> NULL
> |
> | 37 | manifoldcf | localhost:41764 | manifoldcf | Sleep   |    3 |        |
> NULL
> |
> | 38 | manifoldcf | localhost:41765 | manifoldcf | Sleep   |   24 |        |
> NULL
> |
> | 39 | manifoldcf | localhost:41766 | manifoldcf | Sleep   |  279 |        |
> NULL
> |
> | 40 | manifoldcf | localhost:41767 | manifoldcf | Sleep   |  279 |        |
> NULL
> |
> | 41 | manifoldcf | localhost:41768 | manifoldcf | Sleep   |  217 |        |
> NULL
> |
> | 42 | manifoldcf | localhost:41769 | manifoldcf | Sleep   |   23 |        |
> NULL
> |
> | 43 | manifoldcf | localhost:41770 | manifoldcf | Sleep   |   24 |        |
> NULL
> |
> | 44 | manifoldcf | localhost:41772 | manifoldcf | Sleep   |    0 |        |
> NULL
> |
> | 45 | root       | localhost       | NULL       | Query   |    0 | NULL   |
> show processlist
> |
> | 46 | root       | localhost       | NULL       | Sleep   |   53 |        |
> NULL
> |
> +----+------------+-----------------+------------+---------+------+--------+------------------------------------------------------------------------------------------------------+
> 46 rows in set (0.00 sec)

Mime
View raw message