flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8267) update Kinesis Producer example for setting Region key
Date Wed, 31 Jan 2018 14:00:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346869#comment-16346869

Tzu-Li (Gordon) Tai commented on FLINK-8267:

I want to reconfirm a few points here.

First of all, isn't the root cause of [~dyanarose] issue when trying to use {{AWSConfigConstants.AWS_REGION}}
caused by FLINK-8249 (which had been merged to 1.4.1, btw)? The described situation would
no have occurred at all, if the bug of FLINK-8249 wasn't in the picture.

Second of all, while I generally agree that we should try getting rid of {{AWSConfigConstants}}
in the future, completely relying on KCL / KPL constants could also be confusing on the consumer
side, because we don't actually use KCL there. There was even discussion to _not_ use KPL
anymore on the producer due to licensing issues.

Lastly, I'm kind of against "partially updating" the document, i.e. the PR is updating only
the "Region" key, while others still use {{AWSConfigConstants}} keys.
IMHO, that confuses users even more. I think we should either leave it as is (which is fine
because FLINK-8249 is fixed), or update it all at once once {{AWSConfigConstants}} is really

[~phoenixjiangnan] [~dyanarose] what do you think?

> update Kinesis Producer example for setting Region key
> ------------------------------------------------------
>                 Key: FLINK-8267
>                 URL: https://issues.apache.org/jira/browse/FLINK-8267
>             Project: Flink
>          Issue Type: Bug
>          Components: Kinesis Connector
>            Reporter: Dyana Rose
>            Assignee: Bowen Li
>            Priority: Minor
> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/connectors/kinesis.html#kinesis-producer
> In the example code for the kinesis producer the region key is set like:
> {code:java}
> producerConfig.put(AWSConfigConstants.AWS_REGION, "us-east-1");
> {code}
> However, the AWS Kinesis Producer Library requires that the region key be Region (https://github.com/awslabs/amazon-kinesis-producer/blob/94789ff4bb2f5dfa05aafe2d8437d3889293f264/java/amazon-kinesis-producer-sample/default_config.properties#L269)
so the setting at this point should be:
> {code:java}
> producerConfig.put(AWSConfigConstants.AWS_REGION, "us-east-1");
> producerConfig.put("Region", "us-east-1");
> {code}
> When you run the Kinesis Producer you can see the effect of not setting the Region key
by a log line
> {noformat}
> org.apache.flink.streaming.connectors.kinesis.FlinkKinesisProducer  - Started Kinesis
producer instance for region ''
> {noformat}
> The KPL also then assumes it's running on EC2 and attempts to determine it's own region,
which fails.
> {noformat}
> (EC2MetadataClient)Http request to Ec2MetadataService failed.
> [error] [main.cc:266] Could not configure the region. It was not given in the config
and we were unable to retrieve it from EC2 metadata
> {noformat}
> At the least I'd say the documentation should mention the difference between these two
keys and when you are required to also set the Region key.
> On the other hand, is this even the intended behaviour of this connector? Was it intended
that the AWSConfigConstants.AWS_REGION key also set the region of the of the kinesis stream?
The documentation for the example states 
> {noformat}
> The example demonstrates producing a single Kinesis stream in the AWS region “us-east-1”.
> {noformat}

This message was sent by Atlassian JIRA

View raw message