pulsar-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Pulsar Slack" <apache.pulsar.sl...@gmail.com>
Subject Slack digest for #general - 2019-01-17
Date Thu, 17 Jan 2019 09:11:02 GMT
2019-01-16 09:17:48 UTC - Nguyễn Hữu Phong: @jia zhai segment is like ledger, Each Ledger
consisting of one or more Fragments. So each segment consisting of one or more Fragments,
true or false
+1 : jia zhai
----
2019-01-16 09:24:54 UTC - Nguyễn Hữu Phong: @jia zhai bookie save segment or save entry
----
2019-01-16 09:26:45 UTC - Nguyễn Hữu Phong: Can you explain it ? Thanks
----
2019-01-16 09:33:51 UTC - bossbaby: pulsar is kidding me, a day ago, it workded at local then
i deploy it in my vps, sadly that it not successful until night, it worked while im not edit
anything, im go home and now, it not work.
joy : jia zhai
----
2019-01-16 09:35:38 UTC - bossbaby: i think entry like segment
slightly_smiling_face : jia zhai
----
2019-01-16 10:04:53 UTC - Guangzhong Yao: Entry is an array of bytes, as the basic element
in ledger/segment.
+1 : jia zhai
----
2019-01-16 11:12:36 UTC - naga: @naga has joined the channel
----
2019-01-16 13:32:10 UTC - Harald Gustafsson: I use TLS as authentication, now I tried also
sending a few curl commands in, to see if I could delete a topic and namespace. I managed
to delete the topic but not the namespace. I use the same key/cert for topic as for namespace
deletion, but the namespace deletetion gives a HTTP 401 Authentication required. I use my
super user key/cert. any ideas?
----
2019-01-16 13:32:45 UTC - Harald Gustafsson: curl --cacert certs/cacert.pem --cert certs/superUser.bundle.pem
<https://136.225.152.185:30754/admin/v2/persistent/test-20632f25-e674-46c1-a25e-e7bcb74e6d85/testns1/>
["<persistent://test-20632f25-e674-46c1-a25e-e7bcb74e6d85/testns1/t1\\data>"]
----
2019-01-16 13:33:01 UTC - Harald Gustafsson: curl --cacert certs/cacert.pem --cert certs/superUser.bundle.pem
-X DELETE <https://136.225.152.185:30754/admin/v2/persistent/test-20632f25-e674-46c1-a25e-e7bcb74e6d85/testns1/t1%5cdata>
----
2019-01-16 13:33:17 UTC - Harald Gustafsson: curl --cacert certs/cacert.pem --cert certs/superUser.bundle.pem
<https://136.225.152.185:30754/admin/v2/persistent/test-20632f25-e674-46c1-a25e-e7bcb74e6d85/testns1/>
[]
----
2019-01-16 13:33:31 UTC - Harald Gustafsson: curl --cacert certs/cacert.pem --cert certs/superUser.bundle.pem
-X DELETE <https://136.225.152.185:30754/admin/v2/namespaces/test-20632f25-e674-46c1-a25e-e7bcb74e6d85/testns1/>
{"reason":"HTTP 401 Authentication required"}
----
2019-01-16 13:33:41 UTC - Harald Gustafsson: Any ideas
----
2019-01-16 13:57:18 UTC - jia zhai: in this pic, you could treat entry as each message produced
by pulsar
----
2019-01-16 13:58:05 UTC - jia zhai: true
----
2019-01-16 13:59:53 UTC - jia zhai: all entries in this pic is belongs to one segment
----
2019-01-16 14:32:38 UTC - Bogdan BUNECI: Can I use complex types (arrays of Objects) with
Pulsar/Presto ?
----
2019-01-16 14:33:13 UTC - Bogdan BUNECI: Basic types are ok.
----
2019-01-16 14:49:08 UTC - jia zhai: in this pic, you could treat entry as each message produced
by pulsar
----
2019-01-16 14:49:21 UTC - jia zhai: all entries in this pic is belongs to one segment
----
2019-01-16 15:05:26 UTC - Sijie Guo: currently broker hasn’t enabled tls settings in its
usage of bookkeeper. so it might require some code changes. do you mind creating a github
issue for us?
----
2019-01-16 15:24:48 UTC - Brian: sure, will do
----
2019-01-16 15:39:25 UTC - Bogdan BUNECI: I’m using Java producer, pulsar version 2.2.1.
----
2019-01-16 15:50:29 UTC - Grant Wu: <https://pulsar.apache.org/docs/en/concepts-schema-registry/#supported-schema-formats>
----
2019-01-16 15:50:38 UTC - Grant Wu: So not exactly, no
----
2019-01-16 16:06:47 UTC - Bogdan BUNECI: So, if I attach a schema to a topic I can use only
basic types (int, ... string) ?
----
2019-01-16 16:07:05 UTC - Grant Wu: No, there’s a list of types there
----
2019-01-16 16:07:26 UTC - Bogdan BUNECI: Sorry, I will read again
----
2019-01-16 16:07:45 UTC - Grant Wu: There’s no support as far as I can see for arrays of
Objects.  You need to serialize the arrays of objects to a Protobuf/JSON object/Avro thing
first.
----
2019-01-16 16:08:31 UTC - Bogdan BUNECI: Objects are in Avro format
----
2019-01-16 16:08:36 UTC - Bogdan BUNECI: Of course
----
2019-01-16 16:08:53 UTC - Grant Wu: Ah, okay.
----
2019-01-16 16:08:54 UTC - Bogdan BUNECI: With AvroSchemaOf class
----
2019-01-16 16:10:00 UTC - Bogdan BUNECI: I’m producing avro records
----
2019-01-16 16:10:30 UTC - Grant Wu: Okay, I’ve never actually worked with this :sweat_smile:
----
2019-01-16 16:10:35 UTC - Grant Wu: <https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/Schema.html#AVRO-java.lang.Class->
seems to imply it takes a Class
----
2019-01-16 16:10:38 UTC - Bogdan BUNECI: The fields with basic types are ok ... 
----
2019-01-16 16:11:00 UTC - Grant Wu: I would wait for @Matteo Merli or @Sijie Guo to respond
----
2019-01-16 16:11:53 UTC - Bogdan BUNECI: Ok, Thanks !!!!
----
2019-01-16 16:29:19 UTC - Matteo Merli: I think that array/map type should work in Presto.
I can double check in a bit (and definitely this should be clarified in the docs)
----
2019-01-16 16:33:48 UTC - Ivan Kelly: one pulsar "segment". I think the confusion is that
we have different terminology in pulsar and bookkeeper, but sometimes we use them interchangeably
+1 : jia zhai
----
2019-01-16 17:20:28 UTC - Bogdan BUNECI: Please !
----
2019-01-16 17:20:46 UTC - Bogdan BUNECI: I can send you the code
----
2019-01-16 17:21:01 UTC - Bogdan BUNECI: Thanks ;)
----
2019-01-16 17:50:23 UTC - Matteo Merli: @Bogdan BUNECI I actually take my statement back.
Array fields are not being supported right now. I think it shouldn’t be (hard) to add to
support though
----
2019-01-16 17:51:39 UTC - Bogdan BUNECI: ;)
----
2019-01-16 17:51:47 UTC - Bogdan BUNECI: I saw 
----
2019-01-16 17:53:06 UTC - Bogdan BUNECI: But just in pulsar topic (with avro schema) should
work ?
----
2019-01-16 17:53:33 UTC - Matteo Merli: Yes, the avro ser/de and validation are ok
----
2019-01-16 17:53:56 UTC - Matteo Merli: the missing part is bridging an “avro” array into
Presto
----
2019-01-16 17:54:17 UTC - Bogdan BUNECI: We tried, but the nested objects were empty
----
2019-01-16 17:54:28 UTC - Bogdan BUNECI: In pulsar 
----
2019-01-16 17:54:54 UTC - Matteo Merli: What do you mean? When querying in Presto?
----
2019-01-16 17:55:00 UTC - Bogdan BUNECI: Nope
----
2019-01-16 17:55:49 UTC - Bogdan BUNECI: When logging from console consumer
----
2019-01-16 17:56:14 UTC - Bogdan BUNECI: I can send you all the details to spare some time
----
2019-01-16 17:56:35 UTC - Matteo Merli: Yes, please, can you open an issue in github with
all the details?
----
2019-01-16 17:56:52 UTC - Bogdan BUNECI: Ok ! 
----
2019-01-16 17:57:24 UTC - Matteo Merli: In general, we’re just passing the object to avro
to get us back a byte[] and same (the reverse) in consumer to deserialize it back.
----
2019-01-16 17:57:46 UTC - Matteo Merli: We shouldn’t be breaking up the internals there
----
2019-01-16 18:01:05 UTC - Bogdan BUNECI: Than we will double check and if necessary we will
open an issue 
----
2019-01-16 19:24:50 UTC - Emma Pollum: I'm having trouble getting my replication to connect.
Where does pulsar get the IP for inboundConnection to replicate to
----
2019-01-16 20:51:00 UTC - Ivan Kelly: @Emma Pollum it's part of the cluster configuration
i think
----
2019-01-16 20:51:56 UTC - Ivan Kelly: <http://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Clusters.html#updateCluster-java.lang.String-org.apache.pulsar.common.policies.data.ClusterData->
----
2019-01-16 20:52:24 UTC - Ivan Kelly: brokerServiceUrl in the ClusterData
----
2019-01-16 20:53:13 UTC - Ivan Kelly: the host in the url should be a dns name backed by multiple
ips or a VIP, for fault tolerance
----
2019-01-16 22:56:38 UTC - Grant Wu: @Jerry Peng Hey, follow-up on this - is that the same
for when I put it into a function config YAML?
----
2019-01-16 23:30:06 UTC - Stepan: Is there any way to send delayed messages in pulsar? Either
directly by specifying "I want this message to be delivered in 10 minutes" or with a workaround
like specifying TTL on a namespace and then instead of deleting expired messages send them
to a specified topic or something like that.
----
2019-01-16 23:51:53 UTC - Grant Wu: Would it be possible to adjust the semantics of `pulsar-admin
functions update` to more closely match that of the HTTP PUT verb that it uses?
----
2019-01-16 23:52:35 UTC - Grant Wu: HTTP PUT is supposed to “creates a new resource *or*
replaces a representation of the target resource with the request payload.” (<https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/PUT>,
emphasis mine)
----
2019-01-16 23:53:24 UTC - Grant Wu: However right now if you try to use `pulsar-admin functions
update` on a function that doesn’t exist it fails
----
2019-01-17 00:38:32 UTC - Emma Pollum: I recently deployed a function to pulsar, then deleted
it. The subscription remained, however.
When I do 'pulsar-admin topics subscriptions TOPICNAME' the function's subscription shows
up.
When I do 'pulsar-admin topics unsubscribe' it says it can't find the subscription that it
JUST listed.
Any ideas on how I can get rid of this dangling subscription?
----
2019-01-17 00:40:51 UTC - Sijie Guo: currently it is not supported yet. there are two outstanding
efforts on supporting this feature:

- <https://github.com/apache/pulsar/pull/3155>
- <https://github.com/apache/pulsar/issues/2375>
----
2019-01-17 01:13:23 UTC - Ivan Kelly: @Emma Pollum sounds like a bug. maybe the broker is
holding onto the subscription object even though the managed-ledger is deleted
----
2019-01-17 01:50:36 UTC - Emma Pollum: Unfortunately, the subscription is still filling with
messages, can you think of another way to remove the subscription?
----
2019-01-17 01:51:29 UTC - Matteo Merli: I believe that should have been fixed in the 2.2.1
release (for the pulsar-admin command)
----
2019-01-17 01:51:47 UTC - Matteo Merli: (issue was around the `/` in the subscription name
----
2019-01-17 01:53:41 UTC - Emma Pollum: I'm in 2.2.1
----
2019-01-17 01:56:18 UTC - Matteo Merli: ok, sorry. that wasn’t actually merged: <https://github.com/apache/pulsar/pull/2910>
----
2019-01-17 01:57:05 UTC - Matteo Merli: alternative is to make a REST call directly and do
the URLencode of the subscription name
----
2019-01-17 01:58:45 UTC - Matteo Merli: that would be something like:

`curl -XDELETE '<http://localhost:8080/admin/v2/persistent/public/default/my-topic/subscription/TENANT%2FNAMESPACE%2FFUNCTION>'`
----
2019-01-17 04:01:50 UTC - naga: can some one guide me to a doc or video on use case for getting
data from pulsar. can pulsar call apis?
----
2019-01-17 04:37:19 UTC - bossbaby: i found a bug and fixed it:
i used  tlsCiphers:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

and important, if you have a many server broker, you must setup 1 cert in all server
+1 : jia zhai
----
2019-01-17 04:47:42 UTC - jia zhai: @bossbaby you are welcome to make a PR to improve the
document.
----
2019-01-17 04:49:23 UTC - jia zhai: @naga Do you mean to have a try for Pulsar to produce
and consume data?
<http://pulsar.apache.org/docs/en/standalone/>
----
2019-01-17 04:51:26 UTC - jia zhai: Here above is a simple one.
to consume data:
`bin/pulsar-client consume -s "my-sub"  my-topic -n 0`
to produce data:
`bin/pulsar-client produce my-topic --messages "hello-pulsar"`
----
2019-01-17 05:02:08 UTC - naga: I have some APIs to send data to salesforce and I need call
the APIs once the data is in Pulsar. is it possible?
----
2019-01-17 05:02:53 UTC - naga: Produce part , I am clear, I need to understand subscribe
part
----
2019-01-17 05:03:06 UTC - naga: can we trigger some APis already availble?
----
2019-01-17 05:07:37 UTC - Nguyễn Hữu Phong: fragment is belongs to ledger or segment ?
@jia zhai
----
2019-01-17 05:09:01 UTC - jia zhai: 
----
2019-01-17 05:09:17 UTC - jia zhai: at the bottom, it is entries.
----
2019-01-17 05:13:03 UTC - Nguyễn Hữu Phong: So, segment is like ledger, right ?
----
2019-01-17 05:13:50 UTC - jia zhai: yes, in bookkeeper it is called ledger.
----
2019-01-17 05:22:45 UTC - Nguyễn Hữu Phong: segment is like ledger, so, in this pic, fragment
is belongs to one segment ?
----
2019-01-17 05:28:56 UTC - naga: or can Pulsar forward the data to subscribers?
----
2019-01-17 05:33:54 UTC - naga: any suggestions to achive this?
----
2019-01-17 05:37:42 UTC - Matteo Merli: @naga take a look at consumer in the API: <http://pulsar.apache.org/docs/en/client-libraries-java/#consumers>
----
2019-01-17 05:39:29 UTC - naga: :+1:
----
2019-01-17 05:43:03 UTC - naga: I Have gone through and I understand that it is pull from
subscriber. which needs to listen and pull whenever message available.
----
2019-01-17 05:43:45 UTC - naga: Is there any tool /service which I can utilise to call other
services to insert and update data
----
2019-01-17 05:48:12 UTC - Matteo Merli: You can set a message listener to be invoked whenever
a message is available: <http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#messageListener-org.apache.pulsar.client.api.MessageListener->
----
2019-01-17 05:58:39 UTC - naga: When a MessageListener is set, application will receive messages
through it. Calls to Consumer.receive() will not be allowed.
----
2019-01-17 05:58:46 UTC - naga: thanks it is helpful
----
2019-01-17 06:01:54 UTC - jia zhai: yes
----
2019-01-17 06:57:46 UTC - bossbaby: Pulsar not support set retention in different topic in
the same namespace, its right?
----
2019-01-17 06:58:40 UTC - Sijie Guo: currently most of the polices are available at namespace
level. there is an ongoing work to add topic-level policy.
+1 : bossbaby
----
2019-01-17 07:15:44 UTC - bossbaby: thanks you bro
----
2019-01-17 08:58:32 UTC - Yifan: Hi, all, I am having problem seeing my tenant in Pulsar Dashboard.
When I use pulsar-admin to list the tenants, it is there in the list, but not in Dashboard
----
2019-01-17 09:07:16 UTC - Yifan: Mmm... it appeared in dashboard now...
----
2019-01-17 09:08:36 UTC - Sijie Guo: @Yifan it works based on stats. If it doesn’t have
traffic, there is no stats and then the topics won’t show up 
----
2019-01-17 09:09:04 UTC - Yifan: Thanks @Sijie Guo Make sense.
----
Mime
View raw message