jclouds-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Calcote <john.calc...@gmail.com>
Subject multi-part upload writing temp files
Date Thu, 21 May 2020 16:56:17 GMT
Hi Andrew,

I've got code that's uploading using the jclouds multi-part abstraction. Here are the log
trace messages for a portion of an upload of a 1G file:

2020-05-21 16:52:07.203 +0000,36570333402959 {} DEBUG o.j.r.i.InvokeHttpMethod [cloud-normalpri-4]
>> invoking PutObject
2020-05-21 16:52:07.205 +0000,36570335895211 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[cloud-normalpri-4] Sending request -527754056: POST http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?uploads
HTTP/1.1
2020-05-21 16:52:07.206 +0000,36570336221046 {} DEBUG j.headers [cloud-normalpri-4] >>
POST http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?uploads
HTTP/1.1
2020-05-21 16:52:07.206 +0000,36570336410769 {} DEBUG j.headers [cloud-normalpri-4] >>
x-amz-meta-packedsize: 536870896
2020-05-21 16:52:07.206 +0000,36570336571487 {} DEBUG j.headers [cloud-normalpri-4] >>
x-amz-meta-checksum: 27B425F8F9450EF701D3161C7923064A
2020-05-21 16:52:07.206 +0000,36570336727425 {} DEBUG j.headers [cloud-normalpri-4] >>
Content-Disposition: Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0
2020-05-21 16:52:07.206 +0000,36570336906947 {} DEBUG j.headers [cloud-normalpri-4] >>
Content-Type: application/octet-stream
2020-05-21 16:52:07.206 +0000,36570337050453 {} DEBUG j.headers [cloud-normalpri-4] >>
Date: Thu, 21 May 2020 16:52:07 GMT
2020-05-21 16:52:07.206 +0000,36570337147190 {} DEBUG j.headers [cloud-normalpri-4] >>
Authorization: AWS 14FP9F8ISZPXYCVYVLB1:F6tyKFeT79IUeXNmSFAVzs0NPT8=
2020-05-21 16:52:07.250 +0000,36570380776500 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[cloud-normalpri-4] Receiving response -527754056: HTTP/1.1 200 OK
2020-05-21 16:52:07.251 +0000,36570381572643 {} DEBUG j.headers [cloud-normalpri-4] <<
HTTP/1.1 200 OK
2020-05-21 16:52:07.251 +0000,36570382074919 {} DEBUG j.headers [cloud-normalpri-4] <<
Transfer-Encoding: chunked
2020-05-21 16:52:07.251 +0000,36570382210690 {} DEBUG j.headers [cloud-normalpri-4] <<
Accept-Ranges: bytes
2020-05-21 16:52:07.252 +0000,36570382315489 {} DEBUG j.headers [cloud-normalpri-4] <<
Server: Minio/RELEASE.2018-08-02T23-11-36Z (linux; amd64)
2020-05-21 16:52:07.252 +0000,36570382436254 {} DEBUG j.headers [cloud-normalpri-4] <<
X-Xss-Protection: 1; mode=block
2020-05-21 16:52:07.252 +0000,36570382531435 {} DEBUG j.headers [cloud-normalpri-4] <<
Content-Security-Policy: block-all-mixed-content
2020-05-21 16:52:07.252 +0000,36570382623669 {} DEBUG j.headers [cloud-normalpri-4] <<
Vary: Origin
2020-05-21 16:52:07.252 +0000,36570382716393 {} DEBUG j.headers [cloud-normalpri-4] <<
Date: Thu, 21 May 2020 16:52:07 GMT
2020-05-21 16:52:07.252 +0000,36570383135581 {} DEBUG j.headers [cloud-normalpri-4] <<
X-Amz-Request-Id: 16111944E9E38DFA
2020-05-21 16:52:07.253 +0000,36570383234336 {} DEBUG j.headers [cloud-normalpri-4] <<
Content-Type: application/xml
2020-05-21 16:52:07.264 +0000,36570394322320 {} DEBUG o.j.r.i.InvokeHttpMethod [user thread
3] >> invoking PutObject
2020-05-21 16:52:07.265 +0000,36570395342332 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[user thread 3] Sending request 425611548: PUT http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=1&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.266 +0000,36570396738315 {} DEBUG o.j.r.i.InvokeHttpMethod [user thread
4] >> invoking PutObject
2020-05-21 16:52:07.267 +0000,36570397622955 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[user thread 4] Sending request 1704123656: PUT http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=2&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.277 +0000,36570407929930 {} DEBUG o.j.r.i.InvokeHttpMethod [user thread
5] >> invoking PutObject
2020-05-21 16:52:07.279 +0000,36570409298611 {} DEBUG o.j.r.i.InvokeHttpMethod [user thread
6] >> invoking PutObject
2020-05-21 16:52:07.281 +0000,36570411898907 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[user thread 6] Sending request 340915254: PUT http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=4&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.284 +0000,36570414474806 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[user thread 5] Sending request -678166897: PUT http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=3&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.296 +0000,36570426258943 {} DEBUG o.j.r.i.InvokeHttpMethod [user thread
7] >> invoking PutObject
2020-05-21 16:52:07.297 +0000,36570427297262 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[user thread 7] Sending request 1947228191: PUT http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=5&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.300 +0000,36570431072679 {} DEBUG o.j.r.i.InvokeHttpMethod [user thread
8] >> invoking PutObject
2020-05-21 16:52:07.302 +0000,36570432756663 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[user thread 8] Sending request 1338175229: PUT http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=6&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.308 +0000,36570438251042 {} DEBUG o.j.r.i.InvokeHttpMethod [user thread
9] >> invoking PutObject
2020-05-21 16:52:07.309 +0000,36570439257615 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[user thread 9] Sending request -170507915: PUT http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=7&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.326 +0000,36570456516784 {} DEBUG o.j.r.i.InvokeHttpMethod [user thread
10] >> invoking PutObject
2020-05-21 16:52:07.334 +0000,36570464872503 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[user thread 10] Sending request 1648226958: PUT http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=8&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.340 +0000,36570470751445 {} DEBUG o.j.r.i.InvokeHttpMethod [user thread
11] >> invoking PutObject
2020-05-21 16:52:07.344 +0000,36570474255437 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[user thread 11] Sending request -318660758: PUT http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=9&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.356 +0000,36570486541555 {} DEBUG o.j.r.i.InvokeHttpMethod [user thread
12] >> invoking PutObject
2020-05-21 16:52:07.357 +0000,36570488032689 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[user thread 12] Sending request -704157967: PUT http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=10&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.360 +0000,36570490561292 {} DEBUG o.j.r.i.InvokeHttpMethod [user thread
14] >> invoking PutObject
2020-05-21 16:52:07.360 +0000,36570491192156 {} DEBUG o.j.r.i.InvokeHttpMethod [user thread
13] >> invoking PutObject
2020-05-21 16:52:07.361 +0000,36570491949833 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[user thread 13] Sending request -787144195: PUT http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=11&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.365 +0000,36570496147151 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[user thread 14] Sending request 245837237: PUT http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=12&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.377 +0000,36570508079079 {} DEBUG o.j.r.i.InvokeHttpMethod [user thread
15] >> invoking PutObject
2020-05-21 16:52:07.378 +0000,36570509031993 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[user thread 15] Sending request -1250633896: PUT http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=13&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.386 +0000,36570516221740 {} DEBUG o.j.r.i.InvokeHttpMethod [user thread
16] >> invoking PutObject
2020-05-21 16:52:07.386 +0000,36570517146028 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[user thread 16] Sending request -814695581: PUT http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=14&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.397 +0000,36570527331368 {} DEBUG o.j.r.i.InvokeHttpMethod [user thread
17] >> invoking PutObject
2020-05-21 16:52:07.398 +0000,36570528861611 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[user thread 17] Sending request 942770423: PUT http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=15&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.428 +0000,36570558288743 {} DEBUG o.j.r.i.InvokeHttpMethod [user thread
18] >> invoking PutObject
2020-05-21 16:52:07.429 +0000,36570559287323 {} DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
[user thread 18] Sending request 1053643314: PUT http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=16&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:11.657 +0000,36574788135328 {} DEBUG o.j.h.i.HttpWire [user thread 3] over
limit 33554432/262144: wrote temp file
2020-05-21 16:52:15.571 +0000,36578701711351 {} DEBUG o.j.h.i.HttpWire [user thread 4] over
limit 33554432/262144: wrote temp file
2020-05-21 16:52:18.784 +0000,36581914617380 {} DEBUG o.j.h.i.HttpWire [user thread 5] over
limit 33554432/262144: wrote temp file
2020-05-21 16:52:23.365 +0000,36586495218895 {} DEBUG o.j.h.i.HttpWire [user thread 6] over
limit 33554432/262144: wrote temp file
2020-05-21 16:52:29.448 +0000,36592579053941 {} DEBUG o.j.h.i.HttpWire [user thread 7] over
limit 33554432/262144: wrote temp file
2020-05-21 16:52:35.268 +0000,36598398683431 {} DEBUG o.j.h.i.HttpWire [user thread 8] over
limit 33554432/262144: wrote temp file
2020-05-21 16:52:41.503 +0000,36604633555238 {} DEBUG o.j.h.i.HttpWire [user thread 9] over
limit 33554432/262144: wrote temp file
2020-05-21 16:52:45.577 +0000,36608707438866 {} DEBUG o.j.h.i.HttpWire [user thread 10] over
limit 33554432/262144: wrote temp file
2020-05-21 16:52:50.963 +0000,36614093381218 {} DEBUG o.j.h.i.HttpWire [user thread 11] over
limit 33554432/262144: wrote temp file
2020-05-21 16:52:55.288 +0000,36618418403439 {} DEBUG o.j.h.i.HttpWire [user thread 12] over
limit 33554432/262144: wrote temp file
2020-05-21 16:53:00.235 +0000,36623365314879 {} DEBUG o.j.h.i.HttpWire [user thread 13] over
limit 33554432/262144: wrote temp file
2020-05-21 16:53:02.495 +0000,36625625255182 {} DEBUG o.j.h.i.HttpWire [user thread 14] over
limit 33554432/262144: wrote temp file
2020-05-21 16:53:05.601 +0000,36628731966110 {} DEBUG o.j.h.i.HttpWire [user thread 15] over
limit 33554432/262144: wrote temp file
2020-05-21 16:53:08.236 +0000,36631366445393 {} DEBUG o.j.h.i.HttpWire [user thread 16] over
limit 33554432/262144: wrote temp file
2020-05-21 16:53:10.854 +0000,36633984956169 {} DEBUG o.j.h.i.HttpWire [user thread 17] over
limit 33554432/262144: wrote temp file
2020-05-21 16:53:12.020 +0000,36635150704660 {} DEBUG o.j.h.i.HttpWire [user thread 18] over
limit 33554416/262144: wrote temp file

I'm concerned about the trailing messages - "over limit ... wrote temp file" - what does this
mean?

Thanks,
John

Mime
View raw message