spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Miller <cmiller11...@gmail.com>
Subject Re: reading file from S3
Date Wed, 16 Mar 2016 07:45:28 GMT
+1 for Sab's thoughtful answer...

Yasemin: As Gourav said, using IAM roles is considered best practice and
generally will give you fewer headaches in the end... but you may have a
reason for doing it the way you are, and certainly the way you posted
should be supported and not cause the error you described.

--
Chris Miller

On Tue, Mar 15, 2016 at 11:22 PM, Sabarish Sasidharan <
sabarish.sasidharan@manthan.com> wrote:

> There are many solutions to a problem.
>
> Also understand that sometimes your situation might be such. For ex what
> if you are accessing S3 from your Spark job running in your continuous
> integration server sitting in your data center or may be a box under your
> desk. And sometimes you are just trying something.
>
> Also understand that sometimes you want answers to solve your problem at
> hand without redirecting you to something else. Understand what you
> suggested is an appropriate way of doing it, which I myself have proposed
> before, but that doesn't solve the OP's problem at hand.
>
> Regards
> Sab
> On 15-Mar-2016 8:27 pm, "Gourav Sengupta" <gourav.sengupta@gmail.com>
> wrote:
>
>> Oh!!! What the hell!!!!
>>
>> Please never use the URI
>>
>> *s3n://AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY.*That is a major cause of
>> pain, security issues, code maintenance issues and ofcourse something that
>> Amazon strongly suggests that we do not use. Please use roles and you will
>> not have to worry about security.
>>
>> Regards,
>> Gourav Sengupta
>>
>> On Tue, Mar 15, 2016 at 2:38 PM, Sabarish Sasidharan <
>> sabarish.spk@gmail.com> wrote:
>>
>>> You have a slash before the bucket name. It should be @<bucket name>.
>>>
>>> Regards
>>> Sab
>>> On 15-Mar-2016 4:03 pm, "Yasemin Kaya" <godot85@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am using Spark 1.6.0 standalone and I want to read a txt file from S3
>>>> bucket named yasemindeneme and my file name is deneme.txt. But I am getting
>>>> this error. Here is the simple code
>>>> <https://gist.github.com/anonymous/6d174f8587f0f3fd2334>
>>>> Exception in thread "main" java.lang.IllegalArgumentException: Invalid
>>>> hostname in URI s3n://AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY@
>>>> /yasemindeneme/deneme.txt
>>>> at
>>>> org.apache.hadoop.fs.s3.S3Credentials.initialize(S3Credentials.java:45)
>>>> at
>>>> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.java:55)
>>>>
>>>>
>>>> I try 2 options
>>>> *sc.hadoopConfiguration() *and
>>>> *sc.textFile("s3n://AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY@/yasemindeneme/deneme.txt/");*
>>>>
>>>> Also I did export AWS_ACCESS_KEY_ID= .....
>>>>  export AWS_SECRET_ACCESS_KEY=
>>>> But there is no change about error.
>>>>
>>>> Could you please help me about this issue?
>>>>
>>>>
>>>> --
>>>> hiç ender hiç
>>>>
>>>
>>

Mime
View raw message