airavata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcus Christie (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRAVATA-2692) Django: hitting a Thrift 15 second timeout with large file uploads
Date Thu, 10 Jan 2019 16:26:01 GMT

     [ https://issues.apache.org/jira/browse/AIRAVATA-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Marcus Christie updated AIRAVATA-2692:
--------------------------------------
    Description: 
For some reason it fails to get the project when doing a file upload
{noformat}
[Fri Feb 23 03:46:37.783086 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341] [2018-02-23
03:46:37,782 django_airavata.apps.api.views:652 ERROR] Failed to upload file
[Fri Feb 23 03:46:37.783108 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341] Traceback
(most recent call last):
[Fri Feb 23 03:46:37.783113 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]   File
"/var/www/portals/django-seagrid/django-airavata-gateway/django_airavata/apps/api/views.py",
line 642, in upload_input_file
[Fri Feb 23 03:46:37.783118 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]    
request.authz_token, project_id)
[Fri Feb 23 03:46:37.783122 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]   File
"/var/www/portals/django-seagrid/django-airavata-gateway/airavata/api/Airavata.py", line 4733,
in getProject
[Fri Feb 23 03:46:37.783126 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]    
return self.recv_getProject()
[Fri Feb 23 03:46:37.783130 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]   File
"/var/www/portals/django-seagrid/django-airavata-gateway/airavata/api/Airavata.py", line 4746,
in recv_getProject
[Fri Feb 23 03:46:37.783134 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]    
(fname, mtype, rseqid) = iprot.readMessageBegin()
[Fri Feb 23 03:46:37.783138 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]   File
"/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/protocol/TBinaryProtocol.py",
line 134, in readMessageBegin
[Fri Feb 23 03:46:37.783152 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]    
sz = self.readI32()
[Fri Feb 23 03:46:37.783185 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]   File
"/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/protocol/TBinaryProtocol.py",
line 217, in readI32
[Fri Feb 23 03:46:37.783190 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]    
buff = self.trans.readAll(4)
[Fri Feb 23 03:46:37.783194 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]   File
"/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/transport/TTransport.py",
line 60, in readAll
[Fri Feb 23 03:46:37.783198 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]    
chunk = self.read(sz - have)
[Fri Feb 23 03:46:37.783202 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]   File
"/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/transport/TTransport.py",
line 161, in read
[Fri Feb 23 03:46:37.783206 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]    
self.__rbuf = BufferIO(self.__trans.read(max(sz, self.__rbuf_size)))
[Fri Feb 23 03:46:37.783210 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]   File
"/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/transport/TSocket.py",
line 132, in read
[Fri Feb 23 03:46:37.783239 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]    
message='TSocket read 0 bytes')
[Fri Feb 23 03:46:37.783251 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341] thrift.transport.TTransport.TTransportException:
TSocket read 0 bytes
[Fri Feb 23 03:46:37.783263 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341] 
{noformat}

This is hitting a 15 second timeout. Not sure where it is at.

Maybe in mod_wsgi?

https://groups.google.com/forum/#!topic/modwsgi/zuhFREjOE8M
http://modwsgi.readthedocs.io/en/master/configuration-directives/WSGIDaemonProcess.html?highlight=timeout

I know it is a 15 second timeout thing and not related to the size of the file. If I put {{time.sleep(15)}}
in my Django view handler and upload a very small file it triggers the same Thrift error.
Weird that it causes Thrift to generate this error.

h5. TODO
* [ ] integrate Thrift connection pool (thrift_connector)



  was:
For some reason it fails to get the project when doing a file upload
{noformat}
[Fri Feb 23 03:46:37.783086 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341] [2018-02-23
03:46:37,782 django_airavata.apps.api.views:652 ERROR] Failed to upload file
[Fri Feb 23 03:46:37.783108 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341] Traceback
(most recent call last):
[Fri Feb 23 03:46:37.783113 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]   File
"/var/www/portals/django-seagrid/django-airavata-gateway/django_airavata/apps/api/views.py",
line 642, in upload_input_file
[Fri Feb 23 03:46:37.783118 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]    
request.authz_token, project_id)
[Fri Feb 23 03:46:37.783122 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]   File
"/var/www/portals/django-seagrid/django-airavata-gateway/airavata/api/Airavata.py", line 4733,
in getProject
[Fri Feb 23 03:46:37.783126 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]    
return self.recv_getProject()
[Fri Feb 23 03:46:37.783130 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]   File
"/var/www/portals/django-seagrid/django-airavata-gateway/airavata/api/Airavata.py", line 4746,
in recv_getProject
[Fri Feb 23 03:46:37.783134 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]    
(fname, mtype, rseqid) = iprot.readMessageBegin()
[Fri Feb 23 03:46:37.783138 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]   File
"/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/protocol/TBinaryProtocol.py",
line 134, in readMessageBegin
[Fri Feb 23 03:46:37.783152 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]    
sz = self.readI32()
[Fri Feb 23 03:46:37.783185 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]   File
"/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/protocol/TBinaryProtocol.py",
line 217, in readI32
[Fri Feb 23 03:46:37.783190 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]    
buff = self.trans.readAll(4)
[Fri Feb 23 03:46:37.783194 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]   File
"/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/transport/TTransport.py",
line 60, in readAll
[Fri Feb 23 03:46:37.783198 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]    
chunk = self.read(sz - have)
[Fri Feb 23 03:46:37.783202 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]   File
"/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/transport/TTransport.py",
line 161, in read
[Fri Feb 23 03:46:37.783206 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]    
self.__rbuf = BufferIO(self.__trans.read(max(sz, self.__rbuf_size)))
[Fri Feb 23 03:46:37.783210 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]   File
"/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/transport/TSocket.py",
line 132, in read
[Fri Feb 23 03:46:37.783239 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]    
message='TSocket read 0 bytes')
[Fri Feb 23 03:46:37.783251 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341] thrift.transport.TTransport.TTransportException:
TSocket read 0 bytes
[Fri Feb 23 03:46:37.783263 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341] 
{noformat}

This is hitting a 15 second timeout. Not sure where it is at.

Maybe in mod_wsgi?

https://groups.google.com/forum/#!topic/modwsgi/zuhFREjOE8M
http://modwsgi.readthedocs.io/en/master/configuration-directives/WSGIDaemonProcess.html?highlight=timeout

I know it is a 15 second timeout thing and not related to the size of the file. If I put {{time.sleep(15)}}
in my Django view handler and upload a very small file it triggers the same Thrift error.
Weird that it causes Thrift to generate this error.




> Django: hitting a Thrift 15 second timeout with large file uploads
> ------------------------------------------------------------------
>
>                 Key: AIRAVATA-2692
>                 URL: https://issues.apache.org/jira/browse/AIRAVATA-2692
>             Project: Airavata
>          Issue Type: Bug
>            Reporter: Marcus Christie
>            Assignee: Marcus Christie
>            Priority: Major
>
> For some reason it fails to get the project when doing a file upload
> {noformat}
> [Fri Feb 23 03:46:37.783086 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
[2018-02-23 03:46:37,782 django_airavata.apps.api.views:652 ERROR] Failed to upload file
> [Fri Feb 23 03:46:37.783108 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
Traceback (most recent call last):
> [Fri Feb 23 03:46:37.783113 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
  File "/var/www/portals/django-seagrid/django-airavata-gateway/django_airavata/apps/api/views.py",
line 642, in upload_input_file
> [Fri Feb 23 03:46:37.783118 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
    request.authz_token, project_id)
> [Fri Feb 23 03:46:37.783122 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
  File "/var/www/portals/django-seagrid/django-airavata-gateway/airavata/api/Airavata.py",
line 4733, in getProject
> [Fri Feb 23 03:46:37.783126 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
    return self.recv_getProject()
> [Fri Feb 23 03:46:37.783130 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
  File "/var/www/portals/django-seagrid/django-airavata-gateway/airavata/api/Airavata.py",
line 4746, in recv_getProject
> [Fri Feb 23 03:46:37.783134 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
    (fname, mtype, rseqid) = iprot.readMessageBegin()
> [Fri Feb 23 03:46:37.783138 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
  File "/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/protocol/TBinaryProtocol.py",
line 134, in readMessageBegin
> [Fri Feb 23 03:46:37.783152 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
    sz = self.readI32()
> [Fri Feb 23 03:46:37.783185 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
  File "/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/protocol/TBinaryProtocol.py",
line 217, in readI32
> [Fri Feb 23 03:46:37.783190 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
    buff = self.trans.readAll(4)
> [Fri Feb 23 03:46:37.783194 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
  File "/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/transport/TTransport.py",
line 60, in readAll
> [Fri Feb 23 03:46:37.783198 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
    chunk = self.read(sz - have)
> [Fri Feb 23 03:46:37.783202 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
  File "/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/transport/TTransport.py",
line 161, in read
> [Fri Feb 23 03:46:37.783206 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
    self.__rbuf = BufferIO(self.__trans.read(max(sz, self.__rbuf_size)))
> [Fri Feb 23 03:46:37.783210 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
  File "/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/transport/TSocket.py",
line 132, in read
> [Fri Feb 23 03:46:37.783239 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
    message='TSocket read 0 bytes')
> [Fri Feb 23 03:46:37.783251 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]
thrift.transport.TTransport.TTransportException: TSocket read 0 bytes
> [Fri Feb 23 03:46:37.783263 2018] [wsgi:error] [pid 5145] [remote 68.45.52.105:56341]

> {noformat}
> This is hitting a 15 second timeout. Not sure where it is at.
> Maybe in mod_wsgi?
> https://groups.google.com/forum/#!topic/modwsgi/zuhFREjOE8M
> http://modwsgi.readthedocs.io/en/master/configuration-directives/WSGIDaemonProcess.html?highlight=timeout
> I know it is a 15 second timeout thing and not related to the size of the file. If I
put {{time.sleep(15)}} in my Django view handler and upload a very small file it triggers
the same Thrift error. Weird that it causes Thrift to generate this error.
> h5. TODO
> * [ ] integrate Thrift connection pool (thrift_connector)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message