kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henry Kim <henry....@resonate.com>
Subject Re: HDFS Connector Compression?
Date Mon, 07 Nov 2016 17:06:08 GMT
Nevermind, it seems snappy is supported by default for ParquetFormat. It was not obvious with
the fixed extension.

private final static String EXTENSION = ".parquet";

final Schema avroSchema = avroData.fromConnectSchema(record.valueSchema());
CompressionCodecName compressionCodecName = CompressionCodecName.SNAPPY;

final ParquetWriter<GenericRecord> writer =
    new AvroParquetWriter<>(path, avroSchema, compressionCodecName, blockSize, pageSize);

From: Henry Kim
Sent: Wednesday, November 2, 2016 2:46:27 PM
To: users@kafka.apache.org
Subject: HDFS Connector Compression?

Is it possible to add compression to the HDFS Connector out of the box? Or does it require
code change?


Henry Kim

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message