spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <>
Subject [jira] [Assigned] (SPARK-19715) Option to Strip Paths in FileSource
Date Wed, 01 Mar 2017 13:49:45 GMT


Apache Spark reassigned SPARK-19715:

    Assignee: Apache Spark

> Option to Strip Paths in FileSource
> -----------------------------------
>                 Key: SPARK-19715
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: Structured Streaming
>    Affects Versions: 2.1.0
>            Reporter: Michael Armbrust
>            Assignee: Apache Spark
> Today, we compare the whole path when deciding if a file is new in the FileSource for
structured streaming.  However, this cause cause false negatives in the case where the path
has changed in a cosmetic way (i.e. changing s3n to s3a).  We should add an option {{fileNameOnly}}
that causes the new file check to be based only on the filename (but still store the whole
path in the log).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message