hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer ...@effectivemachines.com>
Subject Re: [VOTE] Release Apache Hadoop 3.0.0 RC0
Date Sat, 25 Nov 2017 21:58:14 GMT

> On Nov 21, 2017, at 2:16 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org> wrote:
> 
>>> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even work.
Not just deprecated in favor of timelineserver as was advertised.
>> 
>> 	This works for me in trunk and the bash code doesn’t appear to have changed in
a very long time.  Probably something local to your install.  (I do notice that the deprecation
message says “starting” which is awkward when the stop command is given though.)  Also:
is the deprecation message even true at this point?
> 
> 
> Sorry, I mischaracterized the problem.
> 
> The real issue is that I cannot use this command line when the MapReduce JobHistoryServer
is already started on the same machine.

	The specific string is:

	hadoop-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}.pid

	More specifically, the pid handling code will conflict if the following are true:

		* same machine (obviously)
		* same subcommand name
		* same HADOOP_IDENT_USER: which by default is the user name of whatever starts it… but
was designed to be overridden way back in hadoop 0.X.

	… which means for most production setups, this is probably not real a problem.
	

> So, it looks like in shell-scripts, there can ever be only one daemon of a given name,
irrespective of which daemon scripts are invoked.

	Correct.  Naming multiple, different daemons the same thing is extremely anti-user.   In
fact, I thought this was originally about the “other” history server.

> 
> We need to figure out two things here
>  (a) The behavior of this command. Clearly, it will conflict with the MapReduce JHS -
only one of them can be started on the same node.

	… by the same user, by default.  Started by a different user or different HADOOP_IDENT_USER,
it will come up just fine.

>  (b) We need to figure out if this V1 TimelineService should even be support given ATSv2.

	If ATSv1 isn’t replaced by ATSv2, then why is it marked deprecated?

> On Nov 22, 2017, at 9:45 AM, Brahma Reddy Battula <brahma@apache.org> wrote:
> 
> 1) Change the name
> 2) Create PID based on the CLASS Name, here applicationhistoryserver and jobhistoryserver
> 3) Use same as branch-2.9..i.e suffixing with mapred or yarn
> 
> 
> @allen, any thoughts on this..?

	Using the classname works in this instance, but just as we saw with the router daemons, people
tend to use the same class names when building different components. It also means that if
different daemons can be started in different ways from the same class dependent upon options,
this conflict will still exist.  Also, with dynamic commands, it is very possible to run the
same daemon from multiple start points.

	As part of this discussion, I think it’s important to recognize:

a) This is likely to be primarily impacting developers.
b) We’re talking about two daemons where one has been deprecated.
c) Calling two different daemons “history server” is just awful from an end user perspective.
d) There is already a work around in place if one absolutely needs to run both on the same
node as the same user, just as people do with datanode and nodemanager today.



---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org


Mime
View raw message