From yarn-dev-return-36397-apmail-hadoop-yarn-dev-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 20 15:52:01 2020 Return-Path: X-Original-To: apmail-hadoop-yarn-dev-archive@locus.apache.org Delivered-To: apmail-hadoop-yarn-dev-archive@locus.apache.org Received: from mailroute1-lw-us.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by minotaur.apache.org (Postfix) with ESMTP id 9CBD61A700 for ; Tue, 20 Oct 2020 15:52:01 +0000 (UTC) Received: from mail.apache.org (localhost [127.0.0.1]) by mailroute1-lw-us.apache.org (ASF Mail Server at mailroute1-lw-us.apache.org) with SMTP id 4DA1B123BE8 for ; Tue, 20 Oct 2020 15:52:01 +0000 (UTC) Received: (qmail 77215 invoked by uid 500); 20 Oct 2020 15:52:00 -0000 Delivered-To: apmail-hadoop-yarn-dev-archive@hadoop.apache.org Received: (qmail 77139 invoked by uid 500); 20 Oct 2020 15:52:00 -0000 Mailing-List: contact yarn-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list yarn-dev@hadoop.apache.org Received: (qmail 77120 invoked by uid 99); 20 Oct 2020 15:52:00 -0000 Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) (95.217.134.168) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Oct 2020 15:52:00 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-fi.apache.org (ASF Mail Server at spamproc1-he-fi.apache.org) with ESMTP id 1795CBFE11 for ; Tue, 20 Oct 2020 15:51:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=cloudera.com Received: from mx1-ec2-va.apache.org ([116.203.227.195]) by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) (amavisd-new, port 10024) with ESMTP id OPmAUnZHos0s for ; Tue, 20 Oct 2020 15:51:55 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.175; helo=mail-lj1-f175.google.com; envelope-from=stevel@cloudera.com; receiver= Received: from mail-lj1-f175.google.com (mail-lj1-f175.google.com [209.85.208.175]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id D90E6BC703 for ; Tue, 20 Oct 2020 15:51:54 +0000 (UTC) Received: by mail-lj1-f175.google.com with SMTP id a28so2579157ljn.3 for ; Tue, 20 Oct 2020 08:51:54 -0700 (PDT) X-ASF-DKIM-Sig: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudera.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=plAr/f9gyn3mOqUWz1DR0NTSCXPIQTKXMQVPz6Vr9qo=; b=tiRfOb7ebiQZArzMWFBA+QHuG649py+aBHsQnxszy3CGCl7/cGG+d2mf4Mt8KbDujd KUDPEEuAJLvsoQIt2/jBjorhOMrBpEfQCzDLJcOddZDbPT2G+Ww9nzyaCoB/AiTEQ2Lv 8PYCna5ab71/x4bPWgFMkCSkWHcEKpYuUqShHxI7tXVtAX/LVU0oSIDkinlAsg68XD5k NAOx3ycS9QKXAz53wkMX3FR6FzZE+b83p1JZ0TtA/2GpaOOAmNjZ6Yy6P/JDr//Anp5h ZJ4CHMqmRrncyUC91jdZTl8heQsQVHJhluFhT6m8T9SSFGwxhJnJVhxGYwQkk8X3bY5F mhwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=plAr/f9gyn3mOqUWz1DR0NTSCXPIQTKXMQVPz6Vr9qo=; b=qNE9A72YRRB/Qu233R8ZPQuQDft2zuXxUGCALjhwDytZzgMdsPXb2uIc4Fb3djy/1G HOTE5ntSWfhFvoUA5DOMxu8kr0wYJ9uYP0HkzPu2hUadPWMn+b/V8I/C6ga8Vfktwbfn z9HASPUA2o2O4hVPpDFDhLpWBIJKMrzmhITZ00QbA52WYyZ4/ee1ODy2dGAWKD9UnTxj GEw5A9zyUBdvgPp8O8CT+jlWTVsbjOOU/ntOVLFPtsupo6W7UX1/NHUM/Z3L98XXXzoF xUiojeOaanFcTY8pL6wP9bEQdCJ1N0JJgaSDynyBVo2ewfNqawH53dV7G9k5XbM67L9m qceg== X-Gm-Message-State: AOAM533aNivw42K6PW6blPATMeL/3LfFPqActjoXZWaoNqF9UkgwFzZ9 8P0Fko3jpw6OUig/yO1SjFV1Oc3YMzifzIEYsYpTZQ== X-Google-Smtp-Source: ABdhPJzc/8wlEv/Y89o+mXT5Ba0z/5bJiZVnkTOaX8q2ZuAlR4SIwbGigxpMZ1BegqFDxVnKqudm2I9WqaFzjS/oE70= X-Received: by 2002:a2e:a315:: with SMTP id l21mr1352011lje.31.1603209108494; Tue, 20 Oct 2020 08:51:48 -0700 (PDT) MIME-Version: 1.0 References: <356E0903-F19D-4845-A5E7-27B19042E617@vipshop.com> <16AAD636-3A98-43EF-9F4A-5A217C4C7FDA@vipshop.com> In-Reply-To: From: Steve Loughran Date: Tue, 20 Oct 2020 16:51:12 +0100 Message-ID: Subject: Re: Wire compatibility between Hadoop 3.x client and 2.x server To: Chao Sun Cc: "Wu,Jianliang(vip.com)" , Wei-Chiu Chuang , Hdfs-dev , yarn-dev Content-Type: multipart/alternative; boundary="00000000000069bf7405b21c350e" --00000000000069bf7405b21c350e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I have this belief that webhdfs:// is better for cross-version compatibility. That right? On Fri, 16 Oct 2020 at 19:59, Chao Sun wrote: > Thanks for the replies all! > > > But in the opposite case, it might have problem, because 2.x server may > not understand new client calls added in 3.x > > Yes not expecting this to work. I'm thinking about the case where one > upgrades existing 2.x clients to 3.x and expects it to still work against > 2.x server, which should not involve those new APIs. > > > Another backcompat issue was HDFS-15191, in 3.2.1. > > Thanks for pointing this out! In fact this looks like a serious bug in > 3.2.1. Glad to see it is fixed in 3.2.2. > > > But aside from that, we've been using 3x client libraries against both = 2x > and 3x clusters without issue. > > Great! thanks. > > IMO it will be great if the community maintains official compatibility do= c > w.r.t 2.x/3.x, which can help the migration easier. > > Chao > > On Tue, Oct 13, 2020 at 11:57 PM Wu,Jianliang(vip.com) < > jianliang.wu@vipshop.com> wrote: > > > Ok, I will file a HDFS jira to report this issue. > > > > > 2020=E5=B9=B410=E6=9C=8813=E6=97=A5 20:43=EF=BC=8CWei-Chiu Chuang =E5=86=99=E9=81=93=EF=BC=9A > > > > > > Thanks Jialiang for reporting the issue. > > > That sounds bad and should've not happened. Could you file a HDFS jir= a > > and > > > fill in more details? > > > > > > On Mon, Oct 12, 2020 at 8:59 PM Wu,Jianliang(vip.com) < > > > jianliang.wu@vipshop.com> wrote: > > > > > >> In our case, when nn has upgraded to 3.1.3 and dn=E2=80=99s version = was still > > >> 2.6, we found hive to call getContentSummary method , the client an= d > > >> server was not compatible because of hadoop3 added new PROVIDED > storage > > >> type. > > >> > > >> 2020=E5=B9=B410=E6=9C=8813=E6=97=A5 06:41=EF=BC=8CChao Sun > sunchao@apache.org>> > > >> =E5=86=99=E9=81=93=EF=BC=9A > > >> > > >> > > >> > > >> > > > =E6=9C=AC=E7=94=B5=E5=AD=90=E9=82=AE=E4=BB=B6=E5=8F=AF=E8=83=BD=E4=B8=BA= =E4=BF=9D=E5=AF=86=E6=96=87=E4=BB=B6=E3=80=82=E5=A6=82=E6=9E=9C=E9=98=81=E4= =B8=8B=E9=9D=9E=E7=94=B5=E5=AD=90=E9=82=AE=E4=BB=B6=E6=89=80=E6=8C=87=E5=AE= =9A=E4=B9=8B=E6=94=B6=E4=BB=B6=E4=BA=BA=EF=BC=8C=E8=B0=A8=E8=AF=B7=E7=AB=8B= =E5=8D=B3=E9=80=9A=E7=9F=A5=E6=9C=AC=E4=BA=BA=E3=80=82=E6=95=AC=E8=AF=B7=E9= =98=81=E4=B8=8B=E4=B8=8D=E8=A6=81=E4=BD=BF=E7=94=A8=E3=80=81=E4=BF=9D=E5=AD= =98=E3=80=81=E5=A4=8D=E5=8D=B0=E3=80=81=E6=89=93=E5=8D=B0=E3=80=81=E6=95=A3= =E5=B8=83=E6=9C=AC=E7=94=B5=E5=AD=90=E9=82=AE=E4=BB=B6=E5=8F=8A=E5=85=B6=E5= =86=85=E5=AE=B9=EF=BC=8C=E6=88=96=E5=B0=86=E5=85=B6=E7=94=A8=E4=BA=8E=E5=85= =B6=E4=BB=96=E4=BB=BB=E4=BD=95=E7=9B=AE=E7=9A=84=E6=88=96=E5=90=91=E4=BB=BB= =E4=BD=95=E4=BA=BA=E6=8A=AB=E9=9C=B2=E3=80=82=E8=B0=A2=E8=B0=A2=E6=82=A8=E7= =9A=84=E5=90=88=E4=BD=9C=EF=BC=81 > > >> This communication is intended only for the addressee(s) and may > contain > > >> information that is privileged and confidential. You are hereby > notified > > >> that, if you are not an intended recipient listed above, or an > > authorized > > >> employee or agent of an addressee of this communication responsible > for > > >> delivering e-mail messages to an intended recipient, any > dissemination, > > >> distribution or reproduction of this communication (including any > > >> attachments hereto) is strictly prohibited. If you have received thi= s > > >> communication in error, please notify us immediately by a reply e-ma= il > > >> addressed to the sender and permanently delete the original e-mail > > >> communication and any attachments from all storage devices without > > making > > >> or otherwise retaining a copy. > > >> > > > > > =E6=9C=AC=E7=94=B5=E5=AD=90=E9=82=AE=E4=BB=B6=E5=8F=AF=E8=83=BD=E4=B8=BA= =E4=BF=9D=E5=AF=86=E6=96=87=E4=BB=B6=E3=80=82=E5=A6=82=E6=9E=9C=E9=98=81=E4= =B8=8B=E9=9D=9E=E7=94=B5=E5=AD=90=E9=82=AE=E4=BB=B6=E6=89=80=E6=8C=87=E5=AE= =9A=E4=B9=8B=E6=94=B6=E4=BB=B6=E4=BA=BA=EF=BC=8C=E8=B0=A8=E8=AF=B7=E7=AB=8B= =E5=8D=B3=E9=80=9A=E7=9F=A5=E6=9C=AC=E4=BA=BA=E3=80=82=E6=95=AC=E8=AF=B7=E9= =98=81=E4=B8=8B=E4=B8=8D=E8=A6=81=E4=BD=BF=E7=94=A8=E3=80=81=E4=BF=9D=E5=AD= =98=E3=80=81=E5=A4=8D=E5=8D=B0=E3=80=81=E6=89=93=E5=8D=B0=E3=80=81=E6=95=A3= =E5=B8=83=E6=9C=AC=E7=94=B5=E5=AD=90=E9=82=AE=E4=BB=B6=E5=8F=8A=E5=85=B6=E5= =86=85=E5=AE=B9=EF=BC=8C=E6=88=96=E5=B0=86=E5=85=B6=E7=94=A8=E4=BA=8E=E5=85= =B6=E4=BB=96=E4=BB=BB=E4=BD=95=E7=9B=AE=E7=9A=84=E6=88=96=E5=90=91=E4=BB=BB= =E4=BD=95=E4=BA=BA=E6=8A=AB=E9=9C=B2=E3=80=82=E8=B0=A2=E8=B0=A2=E6=82=A8=E7= =9A=84=E5=90=88=E4=BD=9C=EF=BC=81 > > This communication is intended only for the addressee(s) and may contai= n > > information that is privileged and confidential. You are hereby notifie= d > > that, if you are not an intended recipient listed above, or an authoriz= ed > > employee or agent of an addressee of this communication responsible for > > delivering e-mail messages to an intended recipient, any dissemination, > > distribution or reproduction of this communication (including any > > attachments hereto) is strictly prohibited. If you have received this > > communication in error, please notify us immediately by a reply e-mail > > addressed to the sender and permanently delete the original e-mail > > communication and any attachments from all storage devices without maki= ng > > or otherwise retaining a copy. > > > --00000000000069bf7405b21c350e--