pulsar-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Pulsar Slack" <apache.pulsar.sl...@gmail.com>
Subject Slack digest for #general - 2019-01-18
Date Fri, 18 Jan 2019 09:11:02 GMT
2019-01-17 09:25:25 UTC - Vincent Ngan: What is the maximum size of a message allowed by Pulsar?
----
2019-01-17 09:25:48 UTC - Ali Ahmed: 5 mega bytes
----
2019-01-17 09:26:48 UTC - Vincent Ngan: Is this a hard limit? Can I configure it to a larger
value?
----
2019-01-17 10:47:42 UTC - Ganga Lakshmanasamy: @Karthik Ramasamy @Ravi Said that you have
some reference for real-time consumption of logs from a https endpoint and ingesting into
S3/ elastic using streamlio. Can you please share the details.
----
2019-01-17 11:08:54 UTC - jia zhai: This is limited by BookKeeper. and it is hard coded in
pulsar currently.
----
2019-01-17 11:51:27 UTC - Harald Gustafsson: When deleting a namespace all topics need to
be deleted first, that can be done with force even when producers or consumers are connected.
The problem is that producers will reconnect after 0.1 seconds, and then the namespace is
still not empty when trying to delete it. Is there a way of informing producers and consumers
that when forcefully disconnected they should not reconnect or alternatively is there some
broker admin interface to inform all clients on a topic to disconnect?
----
2019-01-17 12:17:33 UTC - jia zhai: @Harald Gustafsson Currently, seems there is no such interface.
----
2019-01-17 12:20:15 UTC - Harald Gustafsson: @jia zhai Was thinking if I removed access right
from namespace first, would that prevent clients from reconnecting?
----
2019-01-17 12:25:15 UTC - jia zhai: :+1: you could have a try.
----
2019-01-17 12:25:47 UTC - jia zhai: In the code, when do reconnection, it will verify the
authorize
----
2019-01-17 13:00:54 UTC - Vincent Ngan: Thx
----
2019-01-17 14:26:33 UTC - Gofu: @Gofu has joined the channel
----
2019-01-17 14:29:39 UTC - Gofu: Helo guys, can someone tell me if there is some documentation
of the REST admin api, that shows the schema of the http body? Here <https://pulsar.apache.org/docs/latest/reference/RestApi/#/admin/persistent/:property/:cluster/:namespace/:destination/partitions>,
I can't see the body schema. I am assuming that's missing because when I do the request the
response message is: The request entity cannot be empty.
----
2019-01-17 14:52:33 UTC - Paul van der Linden: I might be missing something
----
2019-01-17 14:52:43 UTC - Paul van der Linden: But I deployed a pulsar cluster to kubernetes
----
2019-01-17 14:52:54 UTC - Paul van der Linden: in a different namespace then "default"
----
2019-01-17 14:53:36 UTC - Paul van der Linden: somehow the broker reports it's url as if it
is running in the default namespace: `broker.default.svc.cluster.local` instead of `broker.test-pulsar.svc.cluster.local`
----
2019-01-17 15:10:22 UTC - Paul van der Linden: NVM, found it in the metadata init manifest
+1 : jia zhai
----
2019-01-17 15:24:50 UTC - Sijie Guo: @Gofu you are accessing the old website. you can find
the rest api documentation here : <https://pulsar.apache.org/en/admin-rest-api/>
----
2019-01-17 15:44:43 UTC - Matteo Merli: Yes, removing the permissions will immediately disconnect
existing producers/consumers that have no more access
----
2019-01-17 15:55:24 UTC - Gofu: thank you so much
----
2019-01-17 16:27:48 UTC - Gofu: @Sijie Guo even in <https://pulsar.apache.org/en/admin-rest-api/>
the endpoint doesnt have body specification in the endpoint "Create a partitioned topic" 
per example.
----
2019-01-17 16:32:10 UTC - Ganga Lakshmanasamy: We are having problem in sending and receiving
JSONObjects. The json values are converted into objects from string/array. is it something
expected? Here is an example. This is the data sent {"id":"Thgkkg"} , data received {"id":"java.lang.Object@69cccc31"}
Here is the consumer creation code
	       consumer = client.newConsumer(Schema.AVRO(JSONObject.class))
	                .topic(TOPIC_NAME)
	                .subscriptionType(SubscriptionType.Shared)
	                .subscriptionName(SUBSCRIPTION_NAME)
	                .subscribe();
----
2019-01-17 16:34:23 UTC - Sijie Guo: ah, i see. I think swagger annotation doesn’t render
the rest api page correctly.

the post body should be an integer - the number of the partition.
----
2019-01-17 16:35:36 UTC - Sijie Guo: if your payloads are json, you should use json schema
 not avro schema, no?
----
2019-01-17 16:36:52 UTC - Gofu: @Sijie Guo I have tried without success

{
  "partitions": 2
}
----
2019-01-17 17:02:38 UTC - Sijie Guo: @Gofu just post with ‘2’
----
2019-01-17 17:08:54 UTC - Gofu: @Sijie Guo thank you very much, the only way I can see it
is looking into pulsar api code?
----
2019-01-17 17:14:25 UTC - Sijie Guo: @Gofu - most of the rest apis might already have documented
the post body. however I think swagger annotation doesn’t handle well (or we misuse it),
this rest api doesn’t seem to have primitive post payload documented. for such case, pulsar
api code might be an easier way for you to figure out. or ping us on slack channel :slightly_smiling_face:

for this particular issue, do you mind creating a github issue for us? we will follow up to
address the documentation issue.
----
2019-01-17 17:15:24 UTC - Gofu: Yes sure @Sijie Guo, thanks for the support
----
2019-01-17 18:52:17 UTC - Ravi: any luck on the GUI for pulsar in a container ?
----
2019-01-17 18:59:49 UTC - chris: @Ravi there is an image with the pulsar gui in it (apachepulsar/pulsar-dashboard).
here is an example docker compose file to run it locally <https://github.com/apache/pulsar/blob/master/docker-compose/standalone-dashboard/docker-compose.yml>
----
2019-01-17 19:32:08 UTC - Louis Lapierre: @Louis Lapierre has joined the channel
----
2019-01-17 20:41:15 UTC - Emma Pollum: Currently there are some topic stats that are not available
through prometheus, only the command line. Is there anything in progress to combine these?
----
2019-01-17 20:51:56 UTC - Ivan Kelly: @Emma Pollum which stats in particular? some stats changes
have gone in recently
----
2019-01-17 20:55:45 UTC - Emma Pollum: I'm looking for average message size and message rate
expired
----
2019-01-17 21:07:12 UTC - Ivan Kelly: hmm, looks like the recent changes were all around functions
----
2019-01-17 22:00:30 UTC - Emma Pollum: @Ivan Kelly Are there any plans to put these into prometheus?
----
2019-01-17 23:38:32 UTC - Jerry Peng: @Grant Wu after this PR gets merged, the issue you described
in regards to getting the actual topic name of a message when subscribing to a regex topic
will be fix:
<https://github.com/apache/pulsar/pull/3384>
----
2019-01-17 23:41:07 UTC - Grant Wu: I saw that, great! :clap:
----
2019-01-18 00:16:13 UTC - Grant Wu: So I’m reading <https://pulsar.apache.org/docs/en/functions-guarantees/>
and I was wondering if I could get a little more clarity on how exactly we know when a function
has successfully processed an input
----
2019-01-18 00:17:58 UTC - Grant Wu: @Jerry Peng I’ve skimmed through <https://streaml.io/blog/exactly-once>
and it seems like a great overview of the concepts of effectively once processing
----
2019-01-18 00:18:17 UTC - Grant Wu: But I think “how do we know a Pulsar Function succeeded”
isn’t really described here
----
2019-01-18 00:18:24 UTC - Grant Wu: Is it that no exception was raised?
----
2019-01-18 00:18:28 UTC - Grant Wu: (For Python, at least)
----
2019-01-18 00:18:32 UTC - Grant Wu: cc @Jonathan Budd
----
2019-01-18 00:30:48 UTC - Emma Pollum: When setting cluster properties, what is the difference
between --url and --broker-service-url
----
2019-01-18 00:41:32 UTC - Emma Pollum: And what is a peer cluster?
----
2019-01-18 01:09:27 UTC - Sijie Guo: you can check this out - <https://github.com/apache/pulsar/wiki/PIP-8:-Pulsar-beyond-1M-topics>
----
2019-01-18 01:09:55 UTC - Sijie Guo: --broker-service-url is the pulsar binary protocol :
pulsar://&lt;dns&gt;:6650
----
2019-01-18 01:10:08 UTC - Sijie Guo: --url is the http rest endpoint: http://&lt;dns&gt;:8080
----
2019-01-18 04:51:32 UTC - bossbaby: how prevent pulsar client subcribe subcription that they
not own it. I think i use cert but im not found document about it
----
2019-01-18 05:59:07 UTC - jia zhai: @bossbaby FYI: <http://pulsar.apache.org/docs/en/security-overview.html#role-tokens>
----
2019-01-18 07:21:29 UTC - Ganga Lakshmanasamy: Thanks. Will check this. @Ravi FYI
----
2019-01-18 08:09:42 UTC - xiaolong.ran: @xiaolong.ran has joined the channel
----
Mime
View raw message