flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] Fokko commented on issue #7519: FLINK-11378 - Allow writing to Hadoop compatible Filesystems via Hado…
Date Fri, 01 Mar 2019 15:10:30 GMT
Fokko commented on issue #7519: FLINK-11378 - Allow writing to Hadoop compatible Filesystems
via Hado…
URL: https://github.com/apache/flink/pull/7519#issuecomment-468696725
 
 
   We're running the GCS connector using the HDFS interface. We're having some issues with
performance, after firing up the logging we seem some renaming going on. Would this be something
that we can get rid of, or should we implement the StreamingFileSink with GCS ourselves? Any
guidance?
   
   A snipper from the logs:
   ```
   Successfully copied gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-239-6.inprogress.648a93c1-61f5-4a0e-aa50-cc6685c62050
to gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/part-239-6
   deleteObjects([gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-239-6.inprogress.648a93c1-61f5-4a0e-aa50-cc6685c62050])
   getItemInfo: gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=15/part-219-7/: exists: no
   listObjectNames(fokko-events-raw, yyyymmdd=2019-03-01/hh=15/part-219-7/, /, 1)
   listStorageObjectsAndPrefixes(fokko-events-raw, yyyymmdd=2019-03-01/hh=15/part-219-7/,
/, false, 1)
   createListRequest(fokko-events-raw, yyyymmdd=2019-03-01/hh=15/part-219-7/, /, false, 1)
   listStorageObjectsAndPrefixesPage({bucket=fokko-events-raw, delimiter=/, includeTrailingDelimiter=false,
maxResults=2, prefix=yyyymmdd=2019-03-01/hh=15/part-219-7/}, 1)
   Successfully deleted gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-239-6.inprogress.648a93c1-61f5-4a0e-aa50-cc6685c62050
at generation 1551452605129306
   Successfully copied gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-294-6.inprogress.806455c4-9a36-49f8-9ae9-6e00b90bbdd2
to gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/part-294-6
   deleteObjects([gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-294-6.inprogress.806455c4-9a36-49f8-9ae9-6e00b90bbdd2])
   Successfully copied gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=15/.part-33-7.inprogress.fec75b00-723c-452e-8041-d8608c9ad07a
to gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=15/part-33-7
   deleteObjects([gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=15/.part-33-7.inprogress.fec75b00-723c-452e-8041-d8608c9ad07a])
   Successfully deleted gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-294-6.inprogress.806455c4-9a36-49f8-9ae9-6e00b90bbdd2
at generation 1551452603846557
   Successfully deleted gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=15/.part-33-7.inprogress.fec75b00-723c-452e-8041-d8608c9ad07a
at generation 1551452602868674
   getItemInfo(gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-33-6.inprogress.27ca9b20-6f8a-476c-9a0d-0f97852b595d)
   getObject(gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-33-6.inprogress.27ca9b20-6f8a-476c-9a0d-0f97852b595d)
   getItemInfo(gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-33-6.inprogress.27ca9b20-6f8a-476c-9a0d-0f97852b595d/)
   getObject(gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-33-6.inprogress.27ca9b20-6f8a-476c-9a0d-0f97852b595d/)
   Successfully copied gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=15/.part-219-7.inprogress.79dbbc8c-8c89-49c7-989f-4a6460eb3b25
to gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=15/part-219-7
   deleteObjects([gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=15/.part-219-7.inprogress.79dbbc8c-8c89-49c7-989f-4a6460eb3b25])
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message