spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aditya" <aditya.calangut...@augmentiq.co.in>
Subject Re: How to specify file
Date Fri, 23 Sep 2016 07:14:35 GMT
Hi Sea,

For using Spark SQL you will need to create DataFrame from the file and 
then execute select * on dataframe.
In your case you will need to do something like this

         JavaRDD<String> DF = context.textFile("path");
         JavaRDD<Row> rowRDD3 = DF.map(new Function<String, Row>() {
             public Row call(String record) throws Exception {
                 String[] fields = record.split("\001");
                 Row createRow = createRow(fields);
                 return createRow;
             }
         });
         DataFrame ResultDf3 = hiveContext.createDataFrame(rowRDD3, schema);
         ResultDf3.registerTempTable("test")
         hiveContext.sql("select * from test");

You will need to create schema for the file first just like how you have 
created for csv file.




On Friday 23 September 2016 12:26 PM, Sea wrote:
> Hi, I want to run sql directly on files, I find that spark has 
> supported sql like select * from csv.`/path/to/file`, but files may 
> not be split by ','. Maybe it is split by '\001', how can I specify 
> delimiter?
>
> Thank you!
>
>




Mime
View raw message