spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rabin Banerjee <dev.rabin.baner...@gmail.com>
Subject Re: DataSet creation not working Spark 1.6.0 , populating wrong data CDH 5.7.1
Date Thu, 03 Aug 2017 17:10:50 GMT
Unfortunately I cant use scala/python. Any solution in Java ?

On Thu, Aug 3, 2017 at 6:04 PM, Gourav Sengupta <gourav.sengupta@gmail.com>
wrote:

> Guru,
>
> Anyways you can pick up SCALA or Python. Makes things way easier. The
> perfomance, maintainability, visibility, and minimum translation loss makes
> things better.
>
>
> Regards,
> Gourav
>
> On Thu, Aug 3, 2017 at 11:09 AM, Rabin Banerjee <
> dev.rabin.banerjee@gmail.com> wrote:
>
>> Hi All,
>>
>> I am trying to create a DataSet from DataFrame, where dataframe has been
>> created successfully, and using the same bean I am trying to create dataset.
>>
>> But when I am running it, Dataframe is created as expected. I am able to
>> print the content as well. But not the dataset. The DataSet is having only
>> one column populated with Person object blob and rest all are null.
>>
>> I am using Java Api not scala
>>
>> List<Person> data = new ArrayList<>();
>>     Calendar cal = Calendar.getInstance();
>>     long time = cal.getTime().getTime();
>>     data.add(new Person( "0",0,100L, new Timestamp(time)));
>>     data.add(new Person( "1",1,101L, new Timestamp(time)));
>>
>>     JavaRDD<Person> rdd = (JavaRDD<Person>)jsc().parallelize(data);
>>
>>     SQLContext sqlContext = SQLContext.getOrCreate(jsc().sc());
>>
>>     DataFrame df = sqlContext.createDataFrame(rdd,Person.class);
>>
>>
>>     df.show();
>>     Encoder<Person> es = Encoders.bean(Person.class);
>>
>>     Dataset<Person> dsi = new Dataset<>(sqlContext,df.logicalPlan(),es);
>>
>>     //Dataset<Person> dsi = df.as(es);
>>
>>
>>     dsi.printSchema();
>>     dsi.show()
>>
>> And
>>
>> public class Person  implements Serializable{
>>
>> String id;
>> Integer partition;
>> Long offset;
>> Timestamp eventTime;
>>
>> public Person() {
>> }
>>
>> public Person(String id, Integer partition, Long offset, Timestamp eventTime) {
>>     this.id = id;
>>     this.partition = partition;
>>     this.offset = offset;
>>     this.eventTime = eventTime;
>> }
>>
>> public String getId() {
>>     return id;
>> }
>>
>> public void setId(String id) {
>>     this.id = id;
>> }
>>
>> public Integer getPartition() {
>>     return partition;
>> }
>>
>> public void setPartition(Integer partition) {
>>     this.partition = partition;
>> }
>>
>> public Long getOffset() {
>>     return offset;
>> }
>>
>> public void setOffset(Long offset) {
>>
>>
>>     this.offset = offset;
>> }
>>
>> public Timestamp getEventTime() {
>>     return eventTime;
>> }
>>
>> public void setEventTime(Timestamp eventTime) {
>>     this.eventTime = eventTime;
>> }
>> }
>>
>>
>

Mime
View raw message