spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hemant singh <>
Subject Re: Read text file row by row and apply conditions
Date Mon, 30 Sep 2019 07:03:57 GMT
You can use csv reader with delimiter as '|' to split data and create a
dataframe on top of the file data.
Second step, filter the dataframe on column value like indicator type=A,D
etc and put it in tables. Saving to tables you can use dataframewriter(not
sure what is you destination db type here).

You can keep this part where based on column type you have to filter data
generic and read the configurations like say - col value | tableName from
config file or metadata table and write the data to destination.


On Mon, Sep 30, 2019 at 8:56 AM swetha kadiyala <>

> dear friends,
> I am new to spark. can you please help me to read the below text file
> using spark and scala.
> Sample data
> bret|lee|A|12345|ae545|gfddfg|86786786
> 142343345|aaaa|D|ae342
> 67567|67777|U|aadfsd|34k4|84304|020|sdnfsdfn|3243|benej|32432|jsfsdf|3423
> 67564|67747|U|aad434|3435|843454|203|sdn454dfn|3233|gdfg|34325|sdfsddf|7657
> I am receiving indicator type with 3 rd column of each row. if my
> indicator type=A, then i need to store that particular row data into a
> table called Table1.
> if indicator type=D then i have to store data into seperate table called
> TableB and same as indicator type=U then i have to store all rows data into
> a separate table called Table3.
> Can anyone help me how to read row by row and split the columns and apply
> the condition based on indicator type and store columns data into
> respective tables.
> Thanks,
> Swetha

View raw message