tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Grandl <rgra...@yahoo.com.INVALID>
Subject Re: Non blocking RPC service
Date Sun, 05 Mar 2017 00:14:46 GMT
Thanks a lot Sid. That was very helpful. 
- Robert 

    On Friday, March 3, 2017 9:40 PM, Siddharth Seth <sseth@apache.org> wrote:

 Hi Robert
Hadoop RPC has typically been blocking. There were some changes made to
make the client side asynchronous -
https://issues.apache.org/jira/browse/HADOOP-12909. I believe this is used
in some HDFS interfaces.
https://issues.apache.org/jira/browse/HADOOP-11552 added an option for
server side handoff. For this one, I'd look at the example. I don't know if
anyone uses this yet. There were some plans of using it for YARN-1503. Not
sure if that has happened yet though.  Hopefully for a lot more. Planning
on using this in Tez as well at some point, for task to AM communication.
Otherwise there's alternates like grpc, thrift.


On Thu, Mar 2, 2017 at 1:33 PM, Robert Grandl <rgrandl@yahoo.com.invalid>

> Hi all,
> This might be a bit out of topic, but I don't know where to ask.
> I am trying to create a non blocking RPC service in Yarn and Tez. However,
> I observed that all the communication is blocking.
> I was trying to implement a non blocking service inheriting from
> .Interface instead of .BlockingInterface, however I have a really hard time
> to make it work.
> Is there any example in Tez or Yarn codebase I can look at. Or do you have
> any other good pointers on how I can build such a service atop?
> Otherwise, how can I do a non blocking service using protobuf in any of
> the codebases? Is any such mechanism in Tez or Yarn?
> Thanks in advance for any advice,Robert

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message