airavata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcus Christie (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRAVATA-2692) Django: hitting a 15 second timeout of some sort with large file uploads
Date Thu, 10 Jan 2019 16:25:00 GMT

    [ https://issues.apache.org/jira/browse/AIRAVATA-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739579#comment-16739579
] 

Marcus Christie commented on AIRAVATA-2692:
-------------------------------------------

I've been able to reproduce this problem locally. So that rules out it being a timeout in
Apache or mod_wsgi. It appears to be a timeout in Thrift.

The Thrift connection to the API server is opened at the beginning of the request. It appears
that after 15 seconds of nothing being written to it the connection closes and the first thrift
API call after that 15 seconds fails because the connection is closed.

A better approach would be to use a connection pool for thrift connections and get connections
on demand, as needed. I found the following Thrift connection pool library: https://github.com/Thriftpy/thrift_connector.
I threw together a quick test and with the {{time.sleep(15)}} the upload still completes.

I'll shift gears on this issue and work on integrating this thrift connection pool library.

> Django: hitting a 15 second timeout of some sort with large file uploads
> ------------------------------------------------------------------------
>
>                 Key: AIRAVATA-2692
>                 URL: https://issues.apache.org/jira/browse/AIRAVATA-2692
>             Project: Airavata
>          Issue Type: Bug
>            Reporter: Marcus Christie
>            Assignee: Marcus Christie
>            Priority: Major
>
> For some reason it fails to get the project when doing a file upload
> {noformat}
> [Fri Feb 23 03:46:37.783086 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
[2018-02-23 03:46:37,782 django_airavata.apps.api.views:652 ERROR] Failed to upload file
> [Fri Feb 23 03:46:37.783108 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
Traceback (most recent call last):
> [Fri Feb 23 03:46:37.783113 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
  File "/var/www/portals/django-seagrid/django-airavata-gateway/django_airavata/apps/api/views.py",
line 642, in upload_input_file
> [Fri Feb 23 03:46:37.783118 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
    request.authz_token, project_id)
> [Fri Feb 23 03:46:37.783122 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
  File "/var/www/portals/django-seagrid/django-airavata-gateway/airavata/api/Airavata.py",
line 4733, in getProject
> [Fri Feb 23 03:46:37.783126 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
    return self.recv_getProject()
> [Fri Feb 23 03:46:37.783130 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
  File "/var/www/portals/django-seagrid/django-airavata-gateway/airavata/api/Airavata.py",
line 4746, in recv_getProject
> [Fri Feb 23 03:46:37.783134 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
    (fname, mtype, rseqid) = iprot.readMessageBegin()
> [Fri Feb 23 03:46:37.783138 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
  File "/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/protocol/TBinaryProtocol.py",
line 134, in readMessageBegin
> [Fri Feb 23 03:46:37.783152 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
    sz = self.readI32()
> [Fri Feb 23 03:46:37.783185 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
  File "/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/protocol/TBinaryProtocol.py",
line 217, in readI32
> [Fri Feb 23 03:46:37.783190 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
    buff = self.trans.readAll(4)
> [Fri Feb 23 03:46:37.783194 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
  File "/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/transport/TTransport.py",
line 60, in readAll
> [Fri Feb 23 03:46:37.783198 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
    chunk = self.read(sz - have)
> [Fri Feb 23 03:46:37.783202 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
  File "/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/transport/TTransport.py",
line 161, in read
> [Fri Feb 23 03:46:37.783206 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
    self.__rbuf = BufferIO(self.__trans.read(max(sz, self.__rbuf_size)))
> [Fri Feb 23 03:46:37.783210 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
  File "/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/transport/TSocket.py",
line 132, in read
> [Fri Feb 23 03:46:37.783239 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
    message='TSocket read 0 bytes')
> [Fri Feb 23 03:46:37.783251 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
thrift.transport.TTransport.TTransportException: TSocket read 0 bytes
> [Fri Feb 23 03:46:37.783263 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]

> {noformat}
> This is hitting a 15 second timeout. Not sure where it is at.
> Maybe in mod_wsgi?
> https://groups.google.com/forum/#!topic/modwsgi/zuhFREjOE8M
> http://modwsgi.readthedocs.io/en/master/configuration-directives/WSGIDaemonProcess.html?highlight=timeout
> I know it is a 15 second timeout thing and not related to the size of the file. If I
put {{time.sleep(15)}} in my Django view handler and upload a very small file it triggers
the same Thrift error. Weird that it causes Thrift to generate this error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message