etch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Fitzner <fitzner.mich...@googlemail.com>
Subject Re: Etch/C Memory Consumption
Date Fri, 23 Sep 2011 18:12:12 GMT
Hi Thomas,
Thanks for further investigations. I will check your patch and give
feedback. It would be still great if you could attach the patch file
to Jira https://issues.apache.org/jira/browse/ETCH-176

Cheers
Michael

2011/9/20 Thomas Marsh <ThomasMarsh@gamestop.com>:
> Hello Michael and all,
>
> I believe I have located the source of the memory leak. I implemented a fix in my source
tree, but due to my lack of familiarity with APR, I am not sure whether there are any issues
with my change. The problem lies in the fact that the etch_queue mutexes are created in the
context of the g_etch_main_pool, the global apr_pool, and therefore will not be deallocated
until etch_runtime_shutdown() is called.
>
> The patch from svn diff is shown below. If there are no issues with this change, I would
like to check it in and will close the JIRA issue I submitted.
>
> Thanks, and all the best,
>
> --thomas
>
> Index: common/etch_mutex.c
> ===================================================================
> --- common/etch_mutex.c (revision 1173358)
> +++ common/etch_mutex.c (working copy)
> @@ -68,7 +68,7 @@
>     ETCH_ASSERT(newmutex != NULL);
>
>     apr_thread_mutex_lock(g_etch_main_pool_mutex);
> -    apr_status = apr_thread_mutex_create(&apr_mutex, flags, g_etch_main_pool);
> +    apr_status = apr_thread_mutex_create(&apr_mutex, flags, pool);
>     apr_thread_mutex_unlock(g_etch_main_pool_mutex);
>     if(apr_status != APR_SUCCESS) {
>         char temp[1024];
> Index: transport/etch_plain_mailbox.c
> ===================================================================
> --- transport/etch_plain_mailbox.c      (revision 1173351)
> +++ transport/etch_plain_mailbox.c      (working copy)
> @@ -110,13 +110,14 @@
>
>     do
>     {
> +        if (NULL == (queue  = new_queue(capacity))) break;
> +
>         // TODO: pool
> -        status = etch_mutex_create(&mutex, ETCH_MUTEX_UNNESTED, NULL);
> +        status = etch_mutex_create(&mutex, ETCH_MUTEX_UNNESTED, queue->subpool);
>         if(status != ETCH_SUCCESS) {
>             // error
>             break;
>         }
> -        if (NULL == (queue  = new_queue(capacity))) break;
>
>         /* - - - - - - - - - - - - - - -
>          * i_mailbox
>
> -----Original Message-----
> From: Thomas Marsh
> Sent: Thursday, September 15, 2011 2:41 PM
> To: 'etch-dev@incubator.apache.org'
> Subject: RE: Etch/C Memory Consumption
>
> Hello Michael,
>
> I have started to investigate the memory consumption issue in more detail, and can confirm
it is leaking around 32 bytes per call of say_hello(). That is about the size of the User
helloworld_object and the user->name wchar_t which is allocated in the client loop, but
clearly must be a copy of that data if this is the issue. The leak traceback is below (for
a run with a loop length of 1000):
>
> 32,768 bytes in 4 blocks are possibly lost in loss record 567 of 567
>    at  malloc (vg_replace_malloc.c:195)
>    by apr_pool_create_ex (apr_pools.c:344)
>    by new_queue (etch_queue.c:59)
>    by new_mailbox_a (etch_plain_mailbox.c:119)
>    by new_mailbox (etch_plain_mailbox.c:82)
>    by pmboxmgr_transport_call (etch_plain_mailbox_manager.c:543)
>    by tcpdelsvc_begincall (etch_transport.c:525)
>    by etchremote_begincall (etch_remote.c:129)
>    by helloworld_remote_begin_server_say_hello (helloworld_remote_server.c:217)
>    by helloworld_remote_server_say_hello (helloworld_remote_server.c:276)
>    by main (helloworld_client_main.c:183)
>
> The leak size at this location varies depending on the number of times I iterate over
the say_hello() invocation. Here are the leak sizes from this routine at various loop durations:
>
> 500 -> 16,384 bytes lost
> 1,000 -> 32,768 bytes lost
> 5,000 -> 155,648 bytes lost
> 10,000 -> 319,488 bytes lost
>
> However, this memory is released upon a call to etch_runtime_shutdown(), so you cannot
see the leak unless you interrupt the program execution (or watch the memory grow...).
>
> As for my platform:
> - Linux CentOS 5.4 (32 bit i686)
> - libapr-1.4.5
> - libapr-util-1.3.12
> - libapr-iconv-1.2.1
> - etch (latest from SVN)
> - gcc 4.1.2
>
> I hope that the traceback may give you some clue as to the source of the leak. Otherwise,
I will dig deeper when I find more time to focus on this again.
>
> Viele Gruesse,
>
> --thomas
>
> -----Original Message-----
> From: Michael Fitzner [mailto:Michael.Fitzner@bmw-carit.de]
> Sent: Wednesday, September 14, 2011 6:34 AM
> To: etch-dev@incubator.apache.org
> Subject: AW: Etch/C Memory Consumption
>
> Hi Thomas,
> Your code should be correct and no memory leaks in this part. Could you provide me with
some more information about what systems do you use (Linux, Windows) and a call stack of you
memory leak if available. I will also try to reproduce your test.
>
> Thanks
> Michael
>
> -----Ursprüngliche Nachricht-----
> Von: Thomas Marsh [mailto:ThomasMarsh@gamestop.com]
> Gesendet: Dienstag, 13. September 2011 16:47
> An: etch-dev@incubator.apache.org
> Betreff: RE: Etch/C Memory Consumption
>
> Hello Martijn,
>
> Thanks for your response. However, this is not the source of the leak. There is a clear
etch_object_destroy in the helloworld_remote_begin_server_say_hello() which deallocates any
parameters to the methods (meaning you will get a segfault if you try to reuse the parameters
which now no longer point to valid memory). This usage semantic is also clearly stated in
the C Binding notes
>
> From http://incubator.apache.org/etch/c-binding-tips-tricks.html:
>
>        The C Binding for Etch has the following memory management rules:
>
>        Implementation side: Parameters of functions have to be destroyed by the function
implementation using etch_object_destroy.
>        Caller side: Result Objects have to be freed by the caller using etch_object_destroy.
Parameters of calls will be freed by the runtime automatically.
>
> To reiterate, program memory usage in this very simple usage scenario _must_ be stable.
I suspect an ever growing hash table, or some other similar culprit within the runtime. Can
any of the Etch/C binding developers comment?
>
> Thanks, and best regards,
>
> --thomas
>
> -----Original Message-----
> From: Martijn Dashorst [mailto:martijn.dashorst@gmail.com]
> Sent: Tuesday, September 13, 2011 7:23 AM
> To: etch-dev@incubator.apache.org
> Subject: Re: Etch/C Memory Consumption
>
> And  user->name = new_stringw(L"User"); will do so as well
>
> Martijn
>
> On Tue, Sep 13, 2011 at 1:52 PM, Martijn Dashorst <martijn.dashorst@gmail.com>
wrote:
>> user = new_helloworld_user()
>>
>> will allocate memory for each pass through the loop.
>>
>> Martijn
>>
>> On Mon, Sep 12, 2011 at 11:19 PM, Thomas Marsh <ThomasMarsh@gamestop.com> wrote:
>>> Hello all,
>>>
>>> I have a question about memory consumption in the Etch/C runtime based on behavior
we are seeing within our C client. I have modified the C implementation of the HelloWorld
example in the distribution to run an infinite loop of requests:
>>>
>>> In the main() routine of helloworld_client_main.c:
>>>
>>>                ...
>>>                while (1) {
>>>                                user = new_helloworld_user();
>>>                                user->id = 5;
>>>                                user->name = new_stringw(L"User");
>>>                                result = remote->say_hello(remote,
>>> user);
>>>                                if (is_etch_exception(result))
{
>>>                                                ...
>>>                                }
>>>                                printf("%S\n", result->v.valw);
>>>                                etch_object_destroy(result);
>>>                }
>>>                ...
>>>
>>> While running this, I see that the memory consumption of the client continually
grows. (In this example, it grows by about 1 mB every 5 seconds.) I cannot see the memory
leak when testing with valgrind, so it would suggest that the leak is in Etch managed memory
which is cleared at exit.
>>>
>>> My understanding is that the call to remote->say_hello() should delegate responsibility
of deallocation of the parameters to the Etch runtime, and that the client code is only responsible
for deallocating the result object. The memory use should be stable within this tight loop.
Can anyone comment on the potential cause of the memory consumption?
>>>
>>> Thanks, and best regards,
>>>
>>> --thomas
>>>
>>
>>
>>
>> --
>> Become a Wicket expert, learn from the best: http://wicketinaction.com
>>
>
>
>
> --
> Become a Wicket expert, learn from the best: http://wicketinaction.com
>

Mime
View raw message