spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pradeep Nayak <pradeep1...@gmail.com>
Subject how does sc.textFile translate regex in the input.
Date Wed, 13 Apr 2016 18:56:58 GMT
I am trying to understand on how spark's sc.textFile() works. I
specifically have the question on how it translates the paths with regex in
it.

For example:

files = sc.textFile("hdfs://<server>:<port>/file1/*/*/*/*.txt")

How does it find all the sub-directories and recurses to all the leaf
files. ? Is there any documentation on how this happens ?

Pradeep

Mime
View raw message