hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@cloudera.com>
Subject Re: Changing the JSON Serializer
Date Fri, 17 Nov 2017 06:01:54 GMT
The REST APIs are covered under the compatibility guidelines[1]. Presuming
these are under a new API version number, it's not clear to me from the
existing guidelines if adding one is okay in a maintenance release. It
sounds surprising to me.

[1]:
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html#REST_APIs

On Wed, Nov 15, 2017 at 9:23 PM, Daniel Templeton <daniel@cloudera.com>
wrote:

> Looks like our REST endpoints return malformed JSON for any DAO that
> includes a Map.  That includes:
>
> * the resourceSecondsMap and preemptedResourceSecondsMap entries in all
> the GET /apps/* endpoints,
> * the operationsInfo entry in the GET /scheduler endpoint for capacity
> scheduler,
> * the local_resources, environment, and acls entries in the POST /apps
> endpoint, and
> * the labelsToNodes entry in the GET /label-mappings endpoint.
>
> The issue is that each entry in the map is included with a duplicate key
> ("entry").  Some JSON parsers will choke on the error, and some will
> quietly drop the duplicates.  I've filed YARN-7505 to address the issue.
>
> The solution is to replace the Jersey JSON serializer with the Jackson
> JSON serializer.  This change fixes the issue, but it changes the structure
> of the resulting JSON.  For example, without YARN-7505, hitting /apps might
> yield JSON that contains something like:
>
> "resourceSecondsMap":{
>   "entry":{"key":"memory-mb","value":"11225"},
>   "entry":{"key":"vcores","value":"5"}
>   "entry":{"key":"test","value":"0"}
>   "entry":{"key":"test2","value":"0"}
> }
>
> With YARN-7505, we get:
>
> "resourceSecondsMap": {
>   "test2":0,
>   "test":0,
>   "memory-mb":11225,
>   "vcores":5
> }
>
> The first example is obviously broken, so the second one is clearly
> better, but it's structurally different.
>
> For the GET /label-mappings endpoint, the keys of the map also have to be
> changed to simple strings because JSON doesn't allow for complex map keys.
> So this:
>
> "labelsToNodes":{
>   "entry":{
>     "key":{"name":"label1","exclusivity":"true"},
>     "value":{"nodes":"localhost:63261"}
>   }
> }
>
> becomes this:
>
> "labelsToNodes":{
>   "label1":{
>     "nodes":["dhcp-10-16-0-181.pa.cloudera.com:63261"]
>   }
> }
>
> The first one sucks and is invalid, but changing to the second one will
> break clients that are parsing the first one, especially if they're
> expecting to get the label exclusivity from this endpoint.
>
> Before I try to get YARN-7505 committed, I want to give the community a
> chance to voice any concerns about the change.  It's too late to get into
> 3.0.0, so we'd be looking at 3.0.1 and 3.1.0.
>
> Feel free to comment here or on the JIRA directly.
>
> Thanks,
> Daniel
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>


-- 
busbey

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message