From issues-return-226205-apmail-flink-issues-archive=flink.apache.org@flink.apache.org Tue Apr 16 15:06:03 2019 Return-Path: X-Original-To: apmail-flink-issues-archive@minotaur.apache.org Delivered-To: apmail-flink-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by minotaur.apache.org (Postfix) with SMTP id 7565219B48 for ; Tue, 16 Apr 2019 15:06:03 +0000 (UTC) Received: (qmail 22409 invoked by uid 500); 16 Apr 2019 15:06:02 -0000 Delivered-To: apmail-flink-issues-archive@flink.apache.org Received: (qmail 22379 invoked by uid 500); 16 Apr 2019 15:06:02 -0000 Mailing-List: contact issues-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.apache.org Delivered-To: mailing list issues@flink.apache.org Received: (qmail 22366 invoked by uid 99); 16 Apr 2019 15:06:02 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Apr 2019 15:06:02 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 4E7C0E28DD for ; Tue, 16 Apr 2019 15:06:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id B266D2459E for ; Tue, 16 Apr 2019 15:06:00 +0000 (UTC) Date: Tue, 16 Apr 2019 15:06:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: issues@flink.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (FLINK-12195) Incorrect resource time setting causes flink to fail to submit MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/FLINK-12195?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-12195: ----------------------------------- Labels: pull-request-available (was: ) > Incorrect resource time setting causes flink to fail to submit > -------------------------------------------------------------- > > Key: FLINK-12195 > URL: https://issues.apache.org/jira/browse/FLINK-12195 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN > Affects Versions: 1.6.3 > Reporter: tangshangwen > Assignee: tangshangwen > Priority: Major > Labels: pull-request-available > > We used Tencent COS(object storage is similar to s3) as defaultFS, and wh= en we submitted the job, we ran into a YARN checking resource time mismatch= that prevented the job from being submitted > =C2=A0 > {code:java} > 2019-04-15 14:45:47,683 DEBUG org.apache.hadoop.security.UserGroupInforma= tion: PrivilegedActionException as:hadoop (auth:SIMPLE) cause:java.io.IOExc= eption: Resource cosn://xxx-xxx/user/hadoop/.flink/application_155507859611= 3_0014/logback.xml changed on src filesystem (expected 1555259286000, was 1= 555310742000=C2=A0 > {code} > I found that flink uses the lastModified of the local file, and why is it= not the latest time for the remote file system? > {code:java} > LOG.debug("Copying from {} to {}", localSrcPath, dst); > fs.copyFromLocalFile(false, true, localSrcPath, dst); > // Note: If we used registerLocalResource(FileSystem, Path) here, we woul= d access the remote > // file once again which has problems with eventually consistent read-aft= er-write file > // systems. Instead, we decide to preserve the modification time at the r= emote > // location because this and the size of the resource will be checked by = YARN based on > // the values we provide to #registerLocalResource() below. > fs.setTimes(dst, localFile.lastModified(), -1); > // now create the resource instance > LocalResource resource =3D registerLocalResource(dst, localFile.length(),= localFile.lastModified()); > return Tuple2.of(dst, resource);{code} > Maybe it should be > {code:java} > // now create the resource instance > LocalResource resource =3D registerLocalResource(dst, localFile.length(),= fs.getFileStatus(dst).getModificationTime());{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)