nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toivo Adams <toivo.ad...@gmail.com>
Subject Re: Common data exchange formats and tabular data
Date Sun, 08 Nov 2015 15:40:30 GMT
All,

Benjamin has already done a lot of good work and it would very helpful we
can agree how to move on.
https://issues.apache.org/jira/browse/NIFI-901

My first post was naive, there are much more things to consider.

It is probably impossible to select only one “correct data exchange format”
what all processors should use.

But can we agree one or two preferred data format what SQL and NoSQL
processors should support.
And all other other formats are supported using converter processors.

I my opinion preferred data exchange format should:

1. Support schema in the way or another.

2. Support streaming.

3. Support different data types (String, numeric types, date/time, binary)

4. Serialization/deserialization should be fast and efficient.

5. Widely used and has strong supporters.

6. Can be used in transformations, filtering, join, split, etc.

7. Can be converted to/and from other formats relatively easily.

Nice to have:

1. Nested data structures. For example Orders can contain order rows.


Or maybe we recommend all SQL and NoSQL processors should support two or
more input/output formats and user can select format using configuration?
Or separate sets of processors for different formats?


Thanks
Toivo




--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Common-data-exchange-formats-and-tabular-data-tp3508p4337.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Mime
View raw message