spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Նարեկ Գալստեան <ngalsty...@gmail.com>
Subject get directory names that are affected by sc.textFile("path/to/dir/*/*/*.js")
Date Tue, 27 Oct 2015 13:48:33 GMT
Dear Spark users,

I am reading a set of json files to compile them to Parquet data format.
I am willing to mark the folders in some way after having read their
contents so that I do not read it again(e.g. I can changed the name of the
folder).

I use .textFile("path/to*/dir/*/*/*.js") *technique to* automatically
*detect
the files.
I cannot however, use the same notation* to rename them.*

Could you suggest how I can *get the names of these folders* so that I can
rename them using native hadoop libraries.

I am using Apache Spark 1.4.1

I look forward to hearing suggestions!!

yours,

Narek

Նարեկ Գալստյան

Mime
View raw message