spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Markovitz <Dudu.Markov...@microsoft.com.INVALID>
Subject RE: Run SQL on files directly
Date Sat, 08 Dec 2018 21:55:59 GMT
Thanks Subhash
I am familiar with the other APIs but I am curios about this specific one and I could not
figure it out from the git repository.

Best regards,

David (דודו) Markovitz
Technology Solutions Professional, Data Platform
Microsoft Israel

Mobile: +972-525-834-304
Office: +972-747-119-274

[cid:image002.png@01D166A7.36DE1270]

From: Subhash Sriram <subhash.sriram@gmail.com>
Sent: Saturday, December 8, 2018 10:38 PM
To: David Markovitz <Dudu.Markovitz@microsoft.com.INVALID>
Cc: user@spark.apache.org
Subject: Re: Run SQL on files directly

Hi David,

I’m not sure if that is possible, but why not just read the CSV file using the Scala API,
specifying those options, and then query it using SQL by creating a temp view?

Thanks,
Subhash
Sent from my iPhone

On Dec 8, 2018, at 12:39 PM, David Markovitz <Dudu.Markovitz@microsoft.com.INVALID<mailto:Dudu.Markovitz@microsoft.com.INVALID>>
wrote:
Hi
Spark SQL supports direct querying on files (here<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspark.apache.org%2Fdocs%2Flatest%2Fsql-data-sources-load-save-functions.html%23run-sql-on-files-directly&data=02%7C01%7CDudu.Markovitz%40microsoft.com%7C2ad116ba1f064540fdbe08d65d4d18c1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C636798983044111111&sdata=Xp9f21QYbUHBZcoGzPN8XXoCfcc%2BJ5gnx5DIzkZ4TEc%3D&reserved=0>),
e.g. –


select * from csv.`/my/path/myfile.csv`

Does anybody know if it possible to pass options (sep, header, encoding etc.) with this syntax?

Thanks


Best regards,

David (דודו) Markovitz
Technology Solutions Professional, Data Platform
Microsoft Israel

Mobile: +972-525-834-304
Office: +972-747-119-274

<image001.png>

Mime
View raw message