spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <PSw...@in.imshealth.com>
Subject Not able to remove header from a text file while creating a data frame .
Date Sat, 04 Mar 2017 14:42:57 GMT
Hi All,

I am reading a text file to create a dataframe . While I am trying to exclude header form
the text file I am not able r to do it .
Now my concern is how to know what all options are there that I can use while reading from
a source , I checked the API , there the
Arguments in option method are string .

Could any one of you suggests  how can I find all possible options ?


Code :-
scala> val df1 =hc.read.format("text").option("header","false").load("/development/data/swain/people.txt")
df1: org.apache.spark.sql.DataFrame = [value: string]

scala> df1.show()
+--------------------+
|               value|
+--------------------+
|name|age|address|...|
|       N1|27|A1|1201|
|       N2|28|A2|1202|
|       N3|29|A3|1203|
|       N4|30|A4|1204|



Thanks,

********************** IMPORTANT--PLEASE READ ************************
This electronic message, including its attachments, is CONFIDENTIAL and may contain PROPRIETARY
or LEGALLY PRIVILEGED or PROTECTED information and is intended for the authorized recipient
of the sender.
If you are not the intended recipient, you are hereby notified that any use, disclosure, copying,
or distribution of this message or any of the information included in it is unauthorized and
strictly prohibited.
If you have received this message in error, please immediately notify the sender by reply
e-mail and permanently delete this message and its attachments, along with any copies thereof,
from all locations received (e.g., computer, mobile device, etc.).
Thank you.
********************************************************************

Mime
View raw message