sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sambit Tripathy (RBEI/PJ-NBS)" <Sambit.Tripa...@in.bosch.com>
Subject RE: Joins in Sqoop
Date Tue, 21 Jan 2014 16:47:40 GMT
I have tried it but no luck. Got the same error again. Errcode: 28 - No space left on device
i.e MySQL server is running out of space. Looks like the temporary table created also takes
up a lot of storage space.

1. I created a temp table

sqoop eval  --connect jdbc:mysql://10.xx.xx.xx:3306/db_name --username user1 --password ******
 --query "CREATE TEMPORARY TABLE IF NOT EXISTS LIFECYCLE_DAT_TEMP AS (SELECT * FROM VIEW_JOIN)"

where VIEW_JOIN --> the view that contains the JOINs

2.  Then the import command

sqoop import --connect jdbc:mysql://10.xx.xx.xx:3306/db_name --table LIFECYCLE_DAT_TEMP --username
user1 --password ******* --split-by timestamp


@ Jarcec: Is this what you have suggested?

Cloudera is providing the Teradata Connector for Sqoop which does the same thing as default
but could not find anything for MySQL.



Regards,
Sambit

-----Original Message-----
From: Sambit Tripathy (RBEI/PJ-NBS) [mailto:Sambit.Tripathy@in.bosch.com] 
Sent: Saturday, January 18, 2014 12:30 AM
To: user@sqoop.apache.org
Subject: RE: Joins in Sqoop

That sounds good and in the past I have used temp tables for some other stuff and they work.
I will try it out and post my observations.

-----Original Message-----
From: Jarek Jarcec Cecho [mailto:jarcec@apache.org] 
Sent: Friday, January 17, 2014 10:25 PM
To: user@sqoop.apache.org
Subject: Re: Joins in Sqoop

Large joins are better performed on the database side and stored in temporary table (CREATE
TABLE temp_tbl AS SELECT ...) that can be subsequently imported by Sqoop without creating
large temp files.

Jarcec

On Thu, Jan 16, 2014 at 09:20:04PM +0800, Sambit Tripathy (RBEI/PJ-NBS) wrote:
> Hi,
> 
> I have written query which has 5 Join clauses and I am passing this query in Sqoop import.
> 
> Problem: This produces a large temp file in the MySQL server temp directory and throws
back an error saying No Space left on the device. Yes this can be fixed by increasing the
size of the temp directory in the MySQL server, but what if you actually don't have any space
left on MySQL server. Are there any workarounds for this? I mean something like a batch import
which does not create a big temp file in the server.
> 
> 
> Regards,
> Sambit.
> 

Mime
View raw message