sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Veena Basavaraj" <vbasava...@cloudera.com>
Subject Re: Review Request 28595: Date and DateTime is not encoded in Single Quotes
Date Tue, 02 Dec 2014 17:16:09 GMT


> On Dec. 1, 2014, 10:04 p.m., Jarek Cecho wrote:
> > The patch looks good to me.
> > 
> > Integration test PartitionerTest is failing on my local computer, so I have to investigate
whether it's environmental or not before I can commit it myself.
> 
> Veena Basavaraj wrote:
>     This is one test I usually skip since it eats away my CPU ..!
>     
>     looks like the tests used date time so I missed changing the expected. . I am updating
the tests and a new patch
> 
> Veena Basavaraj wrote:
>     Looks like this is more convuluted change that I expected. The generic JDBC connector
needs to encode dates in quotes I suppose.
> 
> Veena Basavaraj wrote:
>     Oh well lost my last comment.
>     
>     so traced this from start
>     
>     we do this in the test call to load test data
>         provider.insertRow(tableBaseName,  1, "Warty Warthog",    4.10,  "2004-10-20",
false);
>     
>     Then GenericJdbcExtractor
>             context.getDataWriter().writeArrayRecord(array);
>     it calls the SqoopMapDataWriter
>      @Override
>         public void writeArrayRecord(Object[] array) {
>           fromDataFormat.setObjectData(array);
>           writeContent();
>         }
>     it then calls CSVIDF wich encodes the date in single quotes
>     @Override
>       public void setObjectData(Object[] data) {
>         Column[] columnArray = schema.getColumns().toArray(new Column[data.length]);
>         encodeCSVStringElements(data, columnArray);
>         this.data = StringUtils.join(data, SEPARATOR_CHARACTER);
>       }
>       
>       on the TO side HDFSLoader
>       
>             while ((csv = reader.readTextRecord()) != null) {
>             filewriter.write(csv);
>        which calls SqoopOutputFormatDataReader
>        
>         public String readTextRecord() throws InterruptedException {
>           acquireSema();
>           // If the writer has finished, there is definitely no data remaining
>           if (writerFinished) {
>             return null;
>           }
>           try {
>             return dataFormat.getTextData();
>           } finally {
>             releaseSema();
>           }
>         }
>         
>         
>       which then calls CSVIDF
>       
>         */
>       @Override
>       public String getTextData() {
>         return data;
>       }
>     
>     the output should be have dates in single quotes, but it does not.
>     
>         <failure message="expected:&lt;3&gt; but was:&lt;10&gt;" type="java.lang.AssertionError">java.lang.AssertionError:
expected:&lt;3&gt; but was:&lt;10&gt;
> 
> Jarek Cecho wrote:
>     Nice work investigating Veena, I'm running the integration tests and I'll commit
them if they passes.

I have to update the patch again, the last one did not fix it. my bad.


- Veena


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28595/#review63493
-----------------------------------------------------------


On Dec. 2, 2014, 7 a.m., Veena Basavaraj wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28595/
> -----------------------------------------------------------
> 
> (Updated Dec. 2, 2014, 7 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1815
>     https://issues.apache.org/jira/browse/SQOOP-1815
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java
bd0fbf0 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/idf/TestCSVIntermediateDataFormat.java
75fe429 
>   test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/PartitionerTest.java
824a51d 
> 
> Diff: https://reviews.apache.org/r/28595/diff/
> 
> 
> Testing
> -------
> 
> yes
> 
> 
> Thanks,
> 
> Veena Basavaraj
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message