flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juho Autio <juho.au...@rovio.com>
Subject Re: REST API "broken" on YARN because POST is not allowed via YARN proxy
Date Thu, 05 Apr 2018 14:45:28 GMT
Thanks for the answer. Wrapping with GET sounds good to me. You said next
version; do you mean that Flink 1.5 would already include this improvement
when it's released?

On Thu, Apr 5, 2018 at 2:40 PM, Till Rohrmann <trohrmann@apache.org> wrote:

> Hi Juho,
>
> you are right that due to a limitation in the Yarn proxy [1] we cannot
> directly contact the cluster through the Yarn proxy.
>
> The way it works at the moment is that the Flink client retrieves the AM's
> hostname through the ApplicationReport and then directly talks to the AM.
> This of course requires that one can reach the respective Yarn container.
> This is a limitation, though not a regression, which we want to improve
> with the next version of Flink. An idea would be to wrap the REST calls in
> a GET call to make them pass through the Yarn proxy.
>
> [1] https://issues.apache.org/jira/browse/YARN-2084
>
> Cheers,
> Till
>
> On Wed, Apr 4, 2018 at 4:31 PM, Fabian Hueske <fhueske@gmail.com> wrote:
>
>> Hi Juho,
>>
>> Thanks for raising this point!
>>
>> I'll add Chesnay and Till to the thread who contributed to the REST API.
>>
>> Best, Fabian
>>
>> 2018-04-04 15:02 GMT+02:00 Juho Autio <juho.autio@rovio.com>:
>>
>>> I just learned that Flink savepoints API was refactored to require using
>>> HTTP POST.
>>>
>>> That's fine otherwise, but makes life harder when Flink is run on top of
>>> YARN.
>>>
>>> I've added example calls below to show how POST is declined by
>>> the hadoop-yarn-server-web-proxy*, which only supports GET and PUT.
>>>
>>> Can you think of any solution to this? If I would be able to determine
>>> the actual host & port for Flink UI, I could use that instead of the proxy
>>> address. But that would probably require opening at least one more port, so
>>> it's not the optimal solution either. Ideally I would have Flink REST API
>>> completely accessible with GET and PUT methods.
>>>
>>> To me it seems like AWS EMR will also hit this issue as soon as they
>>> start supporting Flink 1.5, because they seem to run Flink as a YARN app.
>>>
>>>
>>> *) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-p
>>> roject/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web
>>> -proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/
>>> WebAppProxyServlet.java#L296-L306
>>>
>>>
>>> $ http POST http://10.0.10.71:20888/proxy/application_1522844153347_0001
>>> /jobs/652e207f8578574d4a322e23d4f8b908/checkpoints
>>>
>>> HTTP/1.1 405 HTTP method POST is not supported by this URL
>>> Cache-Control: must-revalidate,no-cache,no-store
>>> Content-Length: 1523
>>> Content-Type: text/html; charset=iso-8859-1
>>> Date: Wed, 04 Apr 2018 12:48:05 GMT
>>> Date: Wed, 04 Apr 2018 12:48:05 GMT
>>> Pragma: no-cache
>>> Pragma: no-cache
>>>
>>> <html>
>>> <head>
>>> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
>>> <title>Error 405 HTTP method POST is not supported by this URL</title>
>>> </head>
>>> <body><h2>HTTP ERROR 405</h2>
>>> <p>Problem accessing /proxy/application_15228441533
>>> 47_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason:
>>> <pre>    HTTP method POST is not supported by this URL</pre></p><hr
>>> /><i><small>Powered by Jetty://</small></i><br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>> <br/>
>>>
>>> </body>
>>> </html>
>>>
>>>
>>> $ http PUT http://10.0.10.71:20888/proxy/application_1522844153347_0001
>>> /jobs/652e207f8578574d4a322e23d4f8b908/checkpoints
>>>
>>> HTTP/1.1 404 Not Found
>>> Access-Control-Allow-Origin: *
>>> Cache-Control: no-cache
>>> Content-Length: 25
>>> Content-Type: application/json; charset=UTF-8
>>> Date: Wed, 04 Apr 2018 12:48:09 GMT
>>> Date: Wed, 04 Apr 2018 12:48:09 GMT
>>> Expires: Wed, 04 Apr 2018 12:48:09 GMT
>>> Expires: Wed, 04 Apr 2018 12:48:09 GMT
>>> Pragma: no-cache
>>> Pragma: no-cache
>>>
>>> {
>>>     "errors": [
>>>         "Not found."
>>>     ]
>>> }
>>>
>>> ^ expected to get this from flink because there's no PUT /checkpoints.
>>>
>>
>>
>

Mime
View raw message