No, Sometime when you have table with column int and you insert in this column string job will be fail some times

 

 

Best Regards,

Mostafa Alaa Mohamed,

Technical Expert Big Data,

M: +971506450787

Email: mohamedamostafa@etisalat.ae

 

From: Michael Segel [mailto:msegel_hadoop@hotmail.com]
Sent: Thursday, September 29, 2016 04:26 PM
To: Mostafa Alaa Mohamed
Cc: user@spark.apache.org
Subject: Re: Spark Hive Rejection

 

Correct me if I’m wrong but isn’t hive schema on read and not on write? 

So you shouldn’t fail on write. 

 

 

On Sep 29, 2016, at 1:25 AM, Mostafa Alaa Mohamed <mohamedamostafa@etisalat.ae> wrote:

 

Dears,

I want to ask

·         What will happened if there are rejections rows when inserting dataframe into hive?

o   Rejection will be for example table required integer into column and dataframe include string.

o   Duplication rejection restriction from the table itself?

·         How can we specify the rejection directory?

If not avaiable do you recommend to open Jira issue?

 

Best Regards,

Mostafa Alaa Mohamed,

Technical Expert Big Data,

M: +971506450787

 


The content of this email together with any attachments, statements and opinions expressed herein contains information that is private and confidential are intended for the named addressee(s) only. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. If you have received this message in error please notify postmaster@etisalat.ae by email immediately and delete the message without making any copies.

 


The content of this email together with any attachments, statements and opinions expressed herein contains information that is private and confidential are intended for the named addressee(s) only. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. If you have received this message in error please notify postmaster@etisalat.ae by email immediately and delete the message without making any copies.