spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From KhajaAsmath Mohammed <mdkhajaasm...@gmail.com>
Subject Re: Not able to remove header from a text file while creating a data frame .
Date Sat, 04 Mar 2017 15:43:51 GMT
You have to use below. Include the com.databricks.spark.csv in sbt or maven
before doing it. If doing on spark submit, then you need to add as --jars
or --packages ..

sqlContext.read.format("com.databricks.spark.csv").option("header",
"true").option("delimiter",
",").load("data/datapoint_raw/BatteryVoltage.csv"

On Sat, Mar 4, 2017 at 8:42 AM, <PSwain@in.imshealth.com> wrote:

> Hi All,
>
>
>
> I am reading a text file to create a dataframe . While I am trying to
> exclude header form the text file I am not able r to do it .
>
> Now my concern is how to know what all options are there that I can use
> while reading from a source , I checked the API , there the
>
> Arguments in option method are string .
>
>
>
> Could any one of you suggests  how can I find all possible options ?
>
>
>
>
>
> Code :-
>
> scala> val df1 =hc.read.format("text").option("header","false").load("/
> development/data/swain/people.txt")
>
> df1: org.apache.spark.sql.DataFrame = [value: string]
>
>
>
> scala> df1.show()
>
> +--------------------+
>
> |               value|
>
> +--------------------+
>
> |name|age|address|...|
>
> |       N1|27|A1|1201|
>
> |       N2|28|A2|1202|
>
> |       N3|29|A3|1203|
>
> |       N4|30|A4|1204|
>
>
>
>
>
>
>
> Thanks,
>
> ********************** IMPORTANT--PLEASE READ ************************
> This electronic message, including its attachments, is CONFIDENTIAL and
> may contain PROPRIETARY or LEGALLY PRIVILEGED or PROTECTED information and
> is intended for the authorized recipient of the sender.
> If you are not the intended recipient, you are hereby notified that any
> use, disclosure, copying, or distribution of this message or any of the
> information included in it is unauthorized and strictly prohibited.
> If you have received this message in error, please immediately notify the
> sender by reply e-mail and permanently delete this message and its
> attachments, along with any copies thereof, from all locations received
> (e.g., computer, mobile device, etc.).
> Thank you.
> ********************************************************************
>

Mime
View raw message