From dev-return-31205-apmail-spark-dev-archive=spark.apache.org@spark.apache.org Tue Sep 22 03:06:39 2020 Return-Path: X-Original-To: apmail-spark-dev-archive@locus.apache.org Delivered-To: apmail-spark-dev-archive@locus.apache.org Received: from mxout1-ec2-va.apache.org (mxout1-ec2-va.apache.org [3.227.148.255]) by minotaur.apache.org (Postfix) with ESMTP id 1CE7B1AD6F for ; Tue, 22 Sep 2020 03:06:39 +0000 (UTC) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-ec2-va.apache.org (ASF Mail Server at mxout1-ec2-va.apache.org) with SMTP id D20C542A0F for ; Tue, 22 Sep 2020 03:06:38 +0000 (UTC) Received: (qmail 25364 invoked by uid 500); 22 Sep 2020 03:06:31 -0000 Delivered-To: apmail-spark-dev-archive@spark.apache.org Received: (qmail 24669 invoked by uid 500); 22 Sep 2020 03:06:28 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 24194 invoked by uid 99); 22 Sep 2020 03:06:26 -0000 Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) (116.203.196.100) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Sep 2020 03:06:26 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org) with ESMTP id 462C61FF3A8 for ; Tue, 22 Sep 2020 03:06:26 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org X-Spam-Flag: NO X-Spam-Score: 0.001 X-Spam-Level: X-Spam-Status: No, score=0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([116.203.227.195]) by localhost (spamproc1-he-de.apache.org [116.203.196.100]) (amavisd-new, port 10024) with ESMTP id sGfzv_dQThDG for ; Tue, 22 Sep 2020 03:06:25 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.216.45; helo=mail-pj1-f45.google.com; envelope-from=dongjoon.hyun@gmail.com; receiver= Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id E6043BC29A for ; Tue, 22 Sep 2020 02:58:47 +0000 (UTC) Received: by mail-pj1-f45.google.com with SMTP id kk9so758699pjb.2 for ; Mon, 21 Sep 2020 19:58:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=xiD54lQnFyzkFeU7hw6tbo62EGgusgkF/HYfSqtVa5w=; b=qOt5HPulFGoZn1ONa1G80jRfACKg2BnYLnPFKjc36YlauJa/J0a4e6BiQ4V2EEt0uQ +KadIUmuO1OcOnuXJ+MblKjSEAZOFOt3sN+JtVd8sgm15laGYt8whoMiIBh9d+J32Zki +SLkk7rA22rWXON56s7VE8nle9jeEbg8ZcBAkDbxCAZ5244BrfVcmrDgk+C7ilmxSH/2 fwYWFY9xLAwHdItn1bsxUza+hfyVBLpc3aam3lFwgnYuE8KURx81yQ/wReP/iGMUWs+R YNGHBxPdoKF9oI6cocxRLuoHpB8kddjGfSTXSObsITkmc5irtbv4+U/fvhmRXXqr2u1x /BPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=xiD54lQnFyzkFeU7hw6tbo62EGgusgkF/HYfSqtVa5w=; b=cAn2bXOsYILDDPaYO7/+5gNdjZfrFfkOMm6Oy8Li18AGeikrIB9dVjO0F+G+LSLuMB vBMkU3wWG3f3DJBEKquWqwlK5+FlP1+nTz94btd8/MI4zr4uVuvbcRuk2Hm9kE8L4709 BCHzCBOZ/NqkXBSwK56j7DiO7orTbEn5xVJ6cvdMn2CWkGU8NvOGnTuYzQVL+CmD57hf r2MF/5RWUidbq4x0SQeGl/PpYfjxKWYEteH1ZEyHWdeOXGsYGtbXISYFikkHb1GSNXT7 eY/pEgokGRL7Tzn3ATL6Xq6qVGCBscrs298QC9ur85VdOKQRn+W6FvWVKl1ssGYqpb2b LKKQ== X-Gm-Message-State: AOAM531ysRebWToRi9peMuMSzrDgMv5FvZXw+14XQTw2I3KWwIRDpDw7 B6UhZ+7PXfplAZF9gqlxUvNQqN+AFcJi9ODOCL4= X-Google-Smtp-Source: ABdhPJzLmcsXRcWjaR5hjlcfXpWNjfyJU0jj73vj1GSEQdcJC7zRFFwg05bJujGbjPJx5CPgJOqWTIijsXMBMAHnwVQ= X-Received: by 2002:a17:90a:e093:: with SMTP id q19mr1922274pjy.98.1600743520705; Mon, 21 Sep 2020 19:58:40 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Dongjoon Hyun Date: Mon, 21 Sep 2020 19:58:29 -0700 Message-ID: Subject: Re: A common naming policy for third-party packages/modules under org.apache.spark? To: Steve Loughran Cc: Apache Spark Dev Content-Type: multipart/alternative; boundary="000000000000ed9fc805afde2495" --000000000000ed9fc805afde2495 Content-Type: text/plain; charset="UTF-8" Hi, Steve. Sure, you can suggest, but I'm wondering how the suggested namespaces are able to satisfy the existing visibility rules. Could you give us some examples specifically? > Can I suggest some common prefix for third-party-classes put into the spark package tree, just to make clear that they are external contributions? Bests, Dongjoon. On Mon, Sep 21, 2020 at 6:29 AM Steve Loughran wrote: > > I've just been stack-trace-chasing the 404-in-task-commit code: > > https://issues.apache.org/jira/browse/HADOOP-17216 > > And although it's got an org.apache.spark. prefix, it's > actually org.apache.spark.sql.delta, which lives in github, so the > code/issue tracker lives elsewhere. > > I understand why they've done this -I've done it myself- it's to get a > classes package-scoped to spark ( > https://github.com/hortonworks-spark/cloud-integration/blob/master/spark-cloud-integration/src/main/scala/org/apache/spark/cloudera/ParallelizedWithLocalityRDD.scala > ) > > however, it can be confusing and time wasting > > Can I suggest some common prefix for third-party-classes put into the > spark package tree, just to make clear that they are external > contributions? It will set expectations up all round > > -Steve > > (*) Side node: Could whoever maintains that code do retries, which have to > have sleeps of >10-15s? We ended up having to do exponental backoff of > > 90s to make sure the load balancers were clean. The time for a 404 to clear > is not "time since file was added", it is "time since last HEAD/GET/COPY > request". thx > --000000000000ed9fc805afde2495 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi, Steve.

Sure, you c= an suggest, but I'm wondering how the=C2=A0suggested namespaces are abl= e to satisfy the existing visibility rules. Could you give us some examples= specifically?

> Can I suggest some common pref= ix for third-party-classes put into the spark package tree, just to make cl= ear that they are external contributions?

Bests,
Dongjoon.


On Mon, Sep 21, 2020 at 6:= 29 AM Steve Loughran <stevel@cloudera.com.invalid> wrote:

I've just been stack-trace-cha= sing the 404-in-task-commit code:


And although it's got an org.apache.spark. prefix, it's actually= =C2=A0org.apache.spark.sql.delta, which lives in github, so the code/issue = tracker lives elsewhere.


however, it can be confusing and time wast= ing

Can I suggest some common prefix for third-par= ty-classes put into the spark package tree, just to make clear that they ar= e external contributions? It will set expectations up all round
<= br>
-Steve

(*) Side node: Could whoever = maintains that code do retries, which=C2=A0have to have sleeps of >10-15= s? We ended up having to do exponental backoff of > 90s to make sure the= load balancers were clean. The time for a 404 to clear is not "time s= ince file was added", it is "time since last HEAD/GET/COPY reques= t". thx
--000000000000ed9fc805afde2495--