hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
Date Fri, 20 Jul 2018 04:35:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550230#comment-16550230

Steve Loughran commented on HADOOP-15616:

bq. I noticed most previous work like s3, adl, oss, etc. are all put under hadoop-tools. Do
we have plan to move them all to under hadoop-cloud-storage-project?

usual issue: makes backporting patches to older versions harder, and cherrypicking is all
too common.

One thing you will want to do is make sure that the hadoop-cloud-storage POM picks you your
artifacts, *excluding hadoop-common*, which you can downgrade to provided. But do make sure
that you sort our your declared dependencies properly first (always the pain). Why this? Spark
and Tez now depend on this artifact, so will automatically get the new stuff when built with
the right profile. But you will still need to carefully work with the dependencies there.

See also https://github.com/hortonworks-spark/cloud-integration for some integration tests
downstream of spark itself

> Incorporate Tencent Cloud COS File System Implementation
> --------------------------------------------------------
>                 Key: HADOOP-15616
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15616
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/cos
>            Reporter: Junping Du
>            Assignee: YangY
>            Priority: Major
> Tencent cloud is top 2 cloud vendors in China market and the object store COS (https://intl.cloud.tencent.com/product/cos)
is widely used among China’s cloud users but now it is hard for hadoop user to access data
laid on COS storage as no native support for COS in Hadoop.
> This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just like what
we do before for S3, ADL, OSS, etc. With simple configuration, Hadoop applications can read/write
data from COS without any code change.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message