pulsar-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Pulsar Slack" <apache.pulsar.sl...@gmail.com>
Subject Slack digest for #general - 2018-12-18
Date Tue, 18 Dec 2018 09:11:02 GMT
2018-12-17 09:27:17 UTC - Lena Gregor: I’m really sorry to bother you again, but it doesn’t
have to be the “one and only perfect answer”. Just a few classes that you would personally
recommend to a new developer to look at first would already help me a lot! So if somebody
of you could offer a few minutes of their time, that would be great. (You can also send your
answer to <mailto:lena.gregor01@gmail.com|lena.gregor01@gmail.com> if you’re afraid
that your answer could be judged by the others). So thank you very much in advance.
2018-12-17 09:30:51 UTC - Ali Ahmed: @Lena Gregor there is no simple answer to your question
it depends what areas you are interested in (storage , compute, co-ordination etc) and what
your level of understanding of Pulsar’s architecture is
2018-12-17 09:30:59 UTC - Ali Ahmed: How much do you know about Pulsar ?
2018-12-17 09:48:46 UTC - Lena Gregor: @Ali Ahmed Close to nothing, to be honest. I was just
looking for projects that are big enough so my computation would make sense/ could be useful.
But if it’s hard to name classes because the parts are so independent and you don’t really
need a “general overlook” but it’s better for a new developer to start working in their
specific region, than that’s an answer too. We could also do it the other way around: I
could compute the most central classes and you could tell me if you would recommend some of
2018-12-17 09:59:10 UTC - Ali Ahmed: You should read all blogs / video talks relevant to pulsar’s
architecture, then you ask for areas of interest and we can guide you through the code
2018-12-17 10:28:13 UTC - linxin: Why Pulsar schema is topic dimension, unlike Kafka, each
message gets a schema id?
What happens if the schema makes incompatible changes (such as delete a Topic schema and then
re-creating one) and then register a new consumer with new schema trying to consume the old
message with old schema?
1. Topic#setSchema(SchemaA); 2. send a messageA under SchemaA; 3. delete SchemaA and Topic#setSchema(SchemaB);
4. Consumer#subscribeTopicWith(SchemaB); How the consumer handles the messageA?
Is there anyone can solve my doubts?
2018-12-17 10:58:24 UTC - Victor Li: @Victor Li has joined the channel
2018-12-17 11:02:23 UTC - Victor Li: Hello Everyone! :slightly_smiling_face: I am a newbie
at Pulsar.  We are using NiFi + Kafka+ Storm(HDF by hortonworks) for our business streaming
2018-12-17 11:03:03 UTC - Victor Li: So is there any nifi connector for pulsar?
2018-12-17 11:03:43 UTC - Victor Li: we don't want to use storm any more
2018-12-17 11:07:31 UTC - Ali Ahmed: @Victor Li yes there is
2018-12-17 11:07:44 UTC - Ali Ahmed: @David Kjerrumgaard can help you with it
2018-12-17 11:52:03 UTC - Guangzhong Yao: @linxin Currently, pulsar may not support schema
evolution, so your steps will cause exceptions. I think you can ask @Sijie Guo, @Matteo Merli,
@Ivan Kelly for the pulsar schema design considerations.
2018-12-17 11:53:28 UTC - linxin: @Guangzhong Yao thanks.
2018-12-17 11:56:33 UTC - linxin: Maybe it’s jet lag. They probably haven’t seen the message
2018-12-17 12:53:56 UTC - Sijie Guo: the schema is stored in topic level. each message has
a field for schema version : <https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/proto/PulsarApi.proto#L100>

however as what @Guangzhong Yao pointed out, we are not fully leveraging the schema version
yet. there are still some work items for fully leveraging the schema version
2018-12-17 12:59:14 UTC - linxin: so at current stage, Pulsar can not handle the situation
I proposed?
2018-12-17 13:00:24 UTC - Sijie Guo: correct. that’s the missing piece at this moment
2018-12-17 13:00:40 UTC - Sijie Guo: feel free to contribute :slightly_smiling_face: since
we already have the schema version :slightly_smiling_face:
2018-12-17 13:06:17 UTC - linxin: :ok_hand:
2018-12-17 13:42:30 UTC - Sijie Guo: <https://github.com/apache/nifi/pull/3178>
2018-12-17 13:44:28 UTC - Sijie Guo: @David Kjerrumgaard I think it might be making sense
to port this connector back to Pulsar repo. if nifi is moving slowly at merging this.
2018-12-17 14:35:25 UTC - Ivan Kelly: @linxin generally you don't delete the schema before
adding a new one. when you add a new one you can set a compatibility level (backward, forward,
2018-12-17 16:57:20 UTC - David Kjerrumgaard: @Sijie Guo Agreed
2018-12-17 16:57:58 UTC - David Kjerrumgaard: They are starting to really irk me with their
lack of effort on getting this code merged.  I have been at it since the 1.6 release back
in March
2018-12-17 17:02:03 UTC - David Kjerrumgaard: Given the message retention periods afforded
by our tiered-storage, this may become an issue on longer-lived topics.
2018-12-17 17:12:17 UTC - David Kjerrumgaard: @Victor Li What version of NiFi are you currently
2018-12-18 00:58:25 UTC - Balamurugan Muthusamy: @Balamurugan Muthusamy has joined the channel
2018-12-18 01:44:36 UTC - 高天赐: @高天赐 has joined the channel
2018-12-18 05:04:25 UTC - Victor Li: @David Kjerrumgaard @Ali Ahmed Thanks.  our NiFi Cluster
using 1.5.  Edge side using 1.7
2018-12-18 05:15:21 UTC - Victor Li: @David Kjerrumgaard thanks for your information about
nifi pulsar processor.
2018-12-18 05:20:02 UTC - Roger Yates: @Roger Yates has joined the channel
View raw message