flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From greghogan <...@git.apache.org>
Subject [GitHub] flink pull request #2060: [FLINK-3921] StringParser encoding
Date Fri, 09 Sep 2016 15:50:11 GMT
Github user greghogan commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2060#discussion_r78203358
  
    --- Diff: flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
    @@ -75,8 +76,30 @@
     		/** Invalid Boolean value **/
     		BOOLEAN_INVALID
     	}
    -	
    +
    +	private Charset charset = Charset.forName("US-ASCII");
    +
     	private ParseErrorState errorState = ParseErrorState.NONE;
    +
    +	/**
    +	 * Parses the value of a field from the byte array.
    +	 * The start position within the byte array and the array's valid length is given.
    +	 * The content of the value is delimited by a field delimiter.
    +	 *
    +	 * @param bytes The byte array that holds the value.
    +	 * @param startPos The index where the field starts
    +	 * @param limit The limit unto which the byte contents is valid for the parser. The
limit is the
    +	 *              position one after the last valid byte.
    +	 * @param delim The field delimiter character
    +	 * @param reuse An optional reusable field to hold the value
    +	 * @param charset The charset to parse with
    +	 *
    +	 * @return The index of the next delimiter, if the field was parsed correctly. A value
less than 0 otherwise.
    +	 */
    +	public int parseField(byte[] bytes, int startPos, int limit, byte[] delim, T reuse,
Charset charset){
    +		this.charset = charset;
    --- End diff --
    
    Is this method needed? `GenericCsvInputFormat.open` can `setCharset` on each newly instantiated
`FieldParser`, and in the case where a user decided to change charset on an open file `GenericCsvInputFormat.setCharset`
could go through and `setCharset` on the list of `FieldParser`s.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message