nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toivo Adams <toivo.ad...@gmail.com>
Subject Re: Common data exchange formats and tabular data
Date Sun, 08 Nov 2015 14:54:22 GMT
Matt,

Good overview.

>Avro is similar, but the schema must always be provided with the data. In 
>the case of NiFi DataFlows, it's likely more efficient to send the schema 
>once as an initialization packet (I can't remember the real term in NiFi), 
>then the rows can be streamed individually, in batches of user-defined
size, 
>sampled, etc. 

Do you mean "Initial Information Packet" or "IIP" ?
Mr. Morrison classical FBP includes such functionality, used often for
configuration.

As far I know NiFi don't have such concept.

But NiFi ExecuteSQL uses Avro with schema for query result.
Result is one big FlowFile which includes both schema and all rows.
Processor just creates schema from JDBC metadata, writes it to Avro
container
and next all rows are written to Avro container.
Writing and reading such file is done using streaming, so result can very
big.

Thanks
Toivo




--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Common-data-exchange-formats-and-tabular-data-tp3508p4271.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Mime
View raw message