pulsar-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Pulsar Slack" <apache.pulsar.sl...@gmail.com>
Subject Slack digest for #general - 2019-01-24
Date Thu, 24 Jan 2019 09:11:04 GMT
2019-01-23 14:50:45 UTC - Pratik Narode: Hello, I am using pulsar go client. I am trying to
create a pulsar subscriber using regex for topic name. Ex. I have created a topic `testtopic`
and am trying to subscribe to it using regex `t.*`. The producer and consumer are created
successfully but I don't receive any msg on the consumer, that is produced on the `testtopic`.
could anyone please help me understand what am I doing wrong. My producer and consumer code
are as follow:
----
2019-01-23 14:51:17 UTC - Pratik Narode: 
----
2019-01-23 14:51:51 UTC - Pratik Narode: 
----
2019-01-23 15:01:05 UTC - jia zhai: If not support, would you please open an issue for it?
----
2019-01-23 15:03:40 UTC - jia zhai: @Pratik Narode I am not familiar with  go client. But
in cpp/java, we provide a topic pattern like this:
```
 String patternString = "<persistent://my-property/my-ns/pattern-topic.*>";
```
----
2019-01-23 15:04:12 UTC - jia zhai: ```
    std::string pattern = "<persistent://public/default/patternMultiTopicsConsumer.*>";
```
----
2019-01-23 15:05:33 UTC - Pratik Narode: @jia zhai: passing above topic actually gets parsed
as same what you showed i.e `<persistent://public/default/t.*>`
----
2019-01-23 15:06:03 UTC - Pratik Narode: Also I tried passing it as `<persistent://my-property/my-ns/t.*>`
----
2019-01-23 15:06:13 UTC - Pratik Narode: but no luck still same result
----
2019-01-23 15:07:23 UTC - jia zhai: oh
----
2019-01-23 15:07:54 UTC - jia zhai: @Pratik Narode checked the code, seems it is still not
supported in go client.
----
2019-01-23 15:08:25 UTC - jia zhai: I remember that some one is trying to do this.
----
2019-01-23 15:08:37 UTC - jia zhai: It should be soon.
----
2019-01-23 15:10:54 UTC - Pratik Narode: ok Thanks
----
2019-01-23 15:11:12 UTC - Pratik Narode: @jia zhai: Can you please tell me in what all client
this feature is available ?
----
2019-01-23 15:13:21 UTC - jia zhai: cpp and java
----
2019-01-23 15:13:29 UTC - jia zhai: @Pratik Narode
----
2019-01-23 15:13:52 UTC - Pratik Narode: @jia zhai: Thanks for you quick help
----
2019-01-23 15:14:35 UTC - jia zhai: welcome
----
2019-01-23 15:26:27 UTC - Brian: anyone have an idea of when 2.3 will be released?
----
2019-01-23 15:31:10 UTC - jia zhai: @Brian  From what I know, it is depends on bookkeeper
4.9 release, which is in voting status. After bk 4.9, it should be soon.
@Sijie Guo and @Matteo Merli may have more information.
+1 : Brian
----
2019-01-23 16:14:15 UTC - Matteo Merli: @jia zhai @Pratik Narode The feature is already supported
in 2.2 release.

Take a look at <https://godoc.org/github.com/apache/pulsar/pulsar-client-go/pulsar#ConsumerOptions>

There is `TopicsPattern` field that needs to be specified in alternative to `Topic` (or `Topics`)
----
2019-01-23 16:15:52 UTC - Matteo Merli: The reason is a different option is that topic names
are themselves allowed to contain special characters
+1 : jia zhai
----
2019-01-23 16:55:00 UTC - Matteo Merli: @jia zhai @bossbaby The reason cumulative ack it’s
not supported across partitions is that a particular `MessageId` is relative to only a partition.
Therefore each message would have to carry information of the message ids from other partitions
(as received by a particular consumer).

For that reason it’s not currently supported (in any of the client libraries). It’s not
impossible to support, it would just require to keep track of the overall ordering of messages
received across partitions.

Regarding seek(), it’s actually supported in C++: <http://pulsar.apache.org/api/cpp/classpulsar_1_1_consumer.html#a949eda4ed6de3ecc2e4f793cce1dc06c>
----
2019-01-23 16:57:07 UTC - Matteo Merli: @Brian we’re shooting for ASAP. Realistically it
would be a 1-2 week timeframe. We’re waiting for bk-4.9 to be out and then we’ll do few
adjustments in Pulsar (as enabled by new features in bk-4.9) and it will be ready.
----
2019-01-23 17:01:31 UTC - Brian: awesome, looking forward to it
----
2019-01-23 17:02:11 UTC - Grant Wu: @Jerry Peng
----
2019-01-23 18:56:22 UTC - Grant Wu: I’m seeing errors of this sort:
----
2019-01-23 18:56:32 UTC - Grant Wu: 
----
2019-01-23 18:56:49 UTC - Grant Wu: And this is causing all sorts of fun like this:
----
2019-01-23 18:56:53 UTC - Grant Wu: ```
18:48:25.442 [prometheus-stats-42-1] WARN  org.apache.pulsar.functions.worker.FunctionsStatsGenerator
- Failed to collect metrics for function instance public/grantwu/adminNotifications:0
```
----
2019-01-23 18:57:08 UTC - Grant Wu: Is this related to the `className` vs `classname` change
----
2019-01-23 19:51:58 UTC - Grant Wu: Okay, no, it doesn’t seem related to that
----
2019-01-23 19:52:10 UTC - Prasanna kumar Krishnamurthy: @Prasanna kumar Krishnamurthy has
joined the channel
----
2019-01-23 21:59:29 UTC - Sanjeev Kulkarni: @Grant Wu context.publish is essentially inside
the function logic. So if the function is being run in atmost_once mode, you may or may not
see the effects of context.publish on every incming message. In atleast_once mode, you will
see &gt;=1 effects of publish
----
2019-01-23 22:00:36 UTC - Grant Wu: Is there any difference between `AT_LEAST_ONCE` and `EFFECTIVELY_ONCE`
for the special case of a function which uses `context.publish` exclusively and does not publish
to an output topic?
----
2019-01-23 22:09:13 UTC - Sanjeev Kulkarni: context.publish is a ‘side-effect’ of the
function for the case of effectively once function.
----
2019-01-23 22:09:26 UTC - Sanjeev Kulkarni: One way to have effectigely once and publish inside
function would be to use sequence ids
----
2019-01-23 22:11:41 UTC - Sanjeev Kulkarni: We might have to enhance the publish interface
to allow that capability
----
2019-01-23 22:12:10 UTC - Grant Wu: So the answer is “no” :stuck_out_tongue: ?
----
2019-01-23 22:12:16 UTC - Grant Wu: Because it’s a side-effect, essentially
----
2019-01-23 22:12:25 UTC - Grant Wu: And the model assumes pure functions?
----
2019-01-23 22:12:33 UTC - Grant Wu: It is alright if the answer is “no”
----
2019-01-23 22:12:54 UTC - Grant Wu: It would be pretty magical if your system could somehow
rewind side effects of Pulsar exceptions
----
2019-01-23 22:13:35 UTC - Sanjeev Kulkarni: So currently its a no
----
2019-01-23 22:13:40 UTC - Grant Wu: Okay, got it!
----
2019-01-23 22:14:24 UTC - Sanjeev Kulkarni: But its a capability that we could easily add
----
2019-01-23 22:14:33 UTC - Sanjeev Kulkarni: maybe create a github issue?
----
2019-01-23 22:14:51 UTC - Grant Wu: Okay, I can do that.
----
2019-01-23 22:15:55 UTC - Grant Wu: Personally, getting this documented is higher on my list
of wants.
----
2019-01-23 22:16:49 UTC - Grant Wu: Appreciate that amount of time all of you spend answering
questions for free here for sure :slightly_smiling_face:, but it’s not very scalable
----
2019-01-23 22:16:53 UTC - Grant Wu: Maybe I can write the docs
----
2019-01-23 23:12:46 UTC - Grant Wu: Hey, @Sanjeev Kulkarni, I saw you touched the Python import
logic for Pulsar functions recently
----
2019-01-23 23:13:53 UTC - Grant Wu: I keep on running into this line <https://github.com/apache/pulsar/blob/63a0491ec86a4123062eb2835d6f385bf79e2552/pulsar-functions/instance/src/main/python/python_instance.py#L171>
----
2019-01-23 23:15:57 UTC - Grant Wu: ~My current working assumption is that something is wrong
with what I’m putting into the `classname` parameter~
----
2019-01-23 23:17:53 UTC - Grant Wu: ~Can you explain what exactly is supposed to go there?~
----
2019-01-24 00:50:57 UTC - Grant Wu: Okay, I figured it out.
----
2019-01-24 00:51:15 UTC - Grant Wu: It turns out that I was using a Python 3 only library
(pathlib) in my Pulsar function, causing an exception to be thrown when attempting to import
them
----
2019-01-24 00:53:43 UTC - Grant Wu: Where are the logging messages, in, for example, <https://github.com/apache/pulsar/blob/fa212bc8bb0cbd81729b498b6041cde794be9684/pulsar-functions/instance/src/main/python/util.py>
actually supposed to show up?
----
2019-01-24 00:53:59 UTC - Grant Wu: They don’t appear to show up in the stdout logs in pulsar
standalone, nor in the Pulsar broker logs when we deploy Pulsar in k8s
----
2019-01-24 00:55:17 UTC - Grant Wu: If I had gotten access to those logs earlier - I eventually
got access to them by inserting a `logging.basicConfig(level=logging.DEBUG)` statement at
the top of `log.py` - I would have saved many many hours of debugging
----
2019-01-24 01:00:47 UTC - Sanjeev Kulkarni: depending on where the function is running. If
running in pulsar cluster, they will appear on one of the function worker machines. If running
in k8, you can use kubectl logs to fetch them
----
2019-01-24 01:01:39 UTC - Grant Wu: Fairly sure they are not showing up
----
2019-01-24 01:03:07 UTC - Grant Wu: If you want I can try adding the logging.basicConfig statement
into `log.py` and then running my standalone cluster again
----
2019-01-24 01:03:19 UTC - jia zhai: Thanks @Matteo Merli :+1:
----
2019-01-24 01:03:24 UTC - Grant Wu: To see if that makes a difference
----
2019-01-24 01:12:37 UTC - Grant Wu: That doesn’t really seem to make a difference
----
2019-01-24 01:19:09 UTC - Grant Wu: I don’t have much to go on, but I can say I have never
seen the logging output in `util.py` ever show up in the stdout I see in `bin/pulsar standalone`
----
2019-01-24 01:27:45 UTC - Grant Wu: I can file a bug tomorrow.
----
2019-01-24 02:02:56 UTC - bossbaby: I mean here is seek earliest
----
2019-01-24 02:18:26 UTC - Matteo Merli: You can pass `MessageId::earliest()` <http://pulsar.apache.org/api/cpp/classpulsar_1_1_message_id.html#a6c5231ab632b49863c4a227e9cff353d>
----
2019-01-24 03:02:19 UTC - bossbaby: but i was testing and its not work, maybe partition not
support earliest
----
2019-01-24 03:06:39 UTC - bossbaby: i have a question that "message send by producer will
go to broker and broker will write it on bookie" or " broker will say message that you should
write in bookie"?
----
2019-01-24 05:37:26 UTC - Matteo Merli: seek is not supported on partitioned topics
----
2019-01-24 05:37:58 UTC - Matteo Merli: (Because it could be an arbitrary message id)
+1 : bossbaby
----
2019-01-24 06:29:32 UTC - Simon: @Simon has joined the channel
----
2019-01-24 07:01:13 UTC - Sijie Guo: sorry I am not very clear about your question here?
----
2019-01-24 07:14:50 UTC - bossbaby: Does broker transport messages from producer to bookie?
----
2019-01-24 07:15:11 UTC - Sijie Guo: yes
----
2019-01-24 07:22:10 UTC - bossbaby: 
----
2019-01-24 07:23:03 UTC - bossbaby: why performance in pulsar is slower than I expected with
4 broker and 4 bookie
----
Mime
View raw message