nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Wing <jvw...@gmail.com>
Subject Re: PutS3Object error
Date Fri, 04 Nov 2016 18:39:36 GMT
John,

If I understand correctly, your ListS3/PutS3Object processors are
configured with the local cntlm settings?

Proxy Host: localhost
Proxy Host Port: 3128

And have you tried hitting the us-east-1 endpoint by comparison (Google
searching suggests variance in CA cert acceptance)?

Thanks,

James


On Fri, Nov 4, 2016 at 1:41 AM, John Burns <jzburns@gmail.com> wrote:

> Hi James
>
> Yes, happy to share the configuration we use:
>
> We have an institute-wide proxy server that requires user credentials for
> each request (domain, uname, passwd, port 80 and 443 only). We run NiFi on
> Linux hosts using cntlm as the local proxy. Users provide their domain,
> uname and passwd to cntlm, and point their applications to localhost:3128
> as the proxy, and cntlm sends on the proper credentials to the actual proxy
> for each request (if that is clear). We point GetHTTP processors etc to
> cntlm and it works fine, even for https web pages.
>
> We have one cert that is imported into browsers, and again, all browsers
> point to localhost:3128 as the proxy. This seems to work fine, we just
> export http_proxy=localhost:3128 and https_proxy=localhost:3128 at the bash
> shell.
>
> AWS endpoints are https and unfortunately aws command line tools now only
> work when we specify --no-verify-ssl option, otherwise we get the
> following error:
>
> [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
>
> So I was wondering what further configuration steps I need to take to get
> S3/SQS working behind our proxy.
>
> Many thanks
>
> John
>
>
> On Thu, Nov 3, 2016 at 6:42 PM, James Wing <jvwing@gmail.com> wrote:
>
>> The short answer is no, PutS3Object does not currently support a direct
>> equivalent of the AWS CLI's --no-verify-ssl option.  There is an option to
>> provide your own SSLContextService, if you need to establish trust with
>> your proxy server (maybe, I'm not sure).
>>
>> https://nifi.apache.org/docs/nifi-docs/components/org.apache
>> .nifi.ssl.StandardSSLContextService/index.html
>>
>> Can you share a bit more about your use case and proxy setup?  I know
>> there are other NiFi installations using proxy servers against S3, and I do
>> not believe they have had this problem.
>>
>> Last, I believe I foolishly stated in an earlier email that the AWS CLI
>> was a good comparison tool, but I might have to flip-flop now that we're
>> bringing proxy settings and SSL verification into the picture.  Are you
>> sure the CLI is using your proxy similarly?
>>
>> Thanks,
>>
>> James
>>
>> On Thu, Nov 3, 2016 at 5:58 AM, John Burns <jzburns@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a workflow that compresses an file then invokes PutS3Object to
>>> store in an S3 bucket. This processor works fine in a non-proxy
>>> environment,  where PutS3Object is parameterised correctly with the proxy
>>> settings, but in a proxy environment I get the following error shown in the
>>> stack trace.
>>>
>>> Testing from the AWS cli tools, I need to use the --no-verify-ssl
>>> parameter:
>>>
>>> aws s3 ls --no-verify-ssl s3://nifibucket/
>>>
>>> Is there an equivalent "--no-verify-ss"for the PutS3Object processor?
>>>
>>> Thanks
>>>
>>> John
>>>
>>>
>>> ERROR [Timer-Driven Process Thread-10] o.a.nifi.processors.aws.s3.PutS3Object
>>> PutS3Object[id=26ea1644-0158-1000-be29-271b59722ea4] Failed to put
>>> StandardFlowFileRecord[uuid=72488dde-07c8-4236-8116-bd8b34d9
>>> 3716,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1478122984174-68,
>>> container=default, section=68], offset=233361,
>>> length=34033],offset=0,name=bbctext.gz,size=34033] to Amazon S3 due to
>>> com.amazonaws.AmazonClientException: Unable to execute HTTP request:
>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to
>>> find valid certification path to requested target:
>>> com.amazonaws.AmazonClientException: Unable to execute HTTP request:
>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to
>>> find valid certification path to requested target
>>> 2016-11-03 12:49:50,876 ERROR [Timer-Driven Process Thread-10]
>>> o.a.nifi.processors.aws.s3.PutS3Object
>>> com.amazonaws.AmazonClientException: Unable to execute HTTP request:
>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to
>>> find valid certification path to requested target
>>>         at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:706)
>>> ~[aws-java-sdk-core-1.11.8.jar:na]
>>>         at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:447)
>>> ~[aws-java-sdk-core-1.11.8.jar:na]
>>>         at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:409)
>>> ~[aws-java-sdk-core-1.11.8.jar:na]
>>>         at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:358)
>>> ~[aws-java-sdk-core-1.11.8.jar:na]
>>>         at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3787)
>>> ~[aws-java-sdk-s3-1.11.8.jar:na]
>>>         at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1399)
>>> ~[aws-java-sdk-s3-1.11.8.jar:na]
>>>         at org.apache.nifi.processors.aws.s3.PutS3Object$1.process(PutS3Object.java:451)
>>> ~[nifi-aws-processors-1.0.0.jar:1.0.0]
>>>         at org.apache.nifi.controller.repository.StandardProcessSession
>>> .read(StandardProcessSession.java:1880) ~[na:na]
>>>         at org.apache.nifi.controller.repository.StandardProcessSession
>>> .read(StandardProcessSession.java:1851) ~[na:na]
>>>         at org.apache.nifi.processors.aws.s3.PutS3Object.onTrigger(PutS3Object.java:401)
>>> ~[nifi-aws-processors-1.0.0.jar:1.0.0]
>>>         at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>> [nifi-api-1.0.0.jar:1.0.0]
>>>         at org.apache.nifi.controller.StandardProcessorNode.onTrigger(S
>>> tandardProcessorNode.java:1064) [nifi-framework-core-1.0.0.jar:1.0.0]
>>>         at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>>> .call(ContinuallyRunProcessorTask.java:136)
>>> [nifi-framework-core-1.0.0.jar:1.0.0]
>>>         at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>>> .call(ContinuallyRunProcessorTask.java:47)
>>> [nifi-framework-core-1.0.0.jar:1.0.0]
>>>         at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA
>>> gent$1.run(TimerDrivenSchedulingAgent.java:132)
>>> [nifi-framework-core-1.0.0.jar:1.0.0]
>>>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> [na:1.8.0_60]
>>>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>> [na:1.8.0_60]
>>>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>> tureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_60]
>>>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>> tureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_60]
>>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> [na:1.8.0_60]
>>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>> lExecutor.ja
>>>
>>
>>
>

Mime
View raw message