pulsar-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Pulsar Slack" <apache.pulsar.sl...@gmail.com>
Subject Slack digest for #general - 2019-01-10
Date Thu, 10 Jan 2019 09:11:02 GMT
2019-01-09 09:33:12 UTC - Romain Castagnet: Hi. When I want to connect with websocket to my
pulsar, I get a 500 error with TooLongFrameException ("Adjusted frame length exceeds 5242880:
369295620 - discarded"). I don't understand why... Do you have some ideas ?
----
2019-01-09 11:18:17 UTC - Samuel Sun: Hi, one question(could be very simple) , do we have
rest api for create non-partition persistent topic ? only found the interface for partitioned
topic. Thanks
----
2019-01-09 11:33:30 UTC - Karthik Palanivelu: Sure let me try and post here
----
2019-01-09 12:21:50 UTC - Jack: @Jack has joined the channel
----
2019-01-09 15:24:29 UTC - Grant Wu: Are you sending it to the right port?
+1 : Matteo Merli
----
2019-01-09 15:24:45 UTC - Grant Wu: I think I have had this happen before when I mixed up
the websocket port and the regular pulsar protocol port
----
2019-01-09 15:40:17 UTC - Matteo Merli: @Romain Castagnet Websocket service url should go
on the web-socket proxy HTTP port at 8080 (or HTTPS at 8443)
----
2019-01-09 15:57:22 UTC - Grant Wu: Aren’t those topics created on-demand?
----
2019-01-09 16:07:51 UTC - Romain Castagnet: I use this ports, 8080 and 8443 doesn't work for
me, I'm on 2.3.0 snapshot. I need region aware policies.
----
2019-01-09 17:15:20 UTC - Matteo Merli: Can you print the full exception text in the server
side?
----
2019-01-09 17:22:45 UTC - Emma Pollum: What is the best way for me to see how many messages
are nearing expiry as set by my retention policies?
----
2019-01-09 17:31:25 UTC - Emma Pollum: Specifically, to see the oldest unacked message?
----
2019-01-09 17:35:30 UTC - Matteo Merli: You can “peek” at the first unacked message: `pulsar-admin
topics peek-messages $MY_TOPIC -s $MY_SUBSCRIPTION -n 1`
----
2019-01-09 17:35:40 UTC - Matteo Merli: that should include the publish time
----
2019-01-09 17:37:19 UTC - Emma Pollum: awesome thank you
----
2019-01-09 19:15:24 UTC - Emma Pollum: Is there a good guide or starting point if I want to
add this to the exported prometheus metrics?
----
2019-01-09 21:44:06 UTC - Emma Pollum: are all these stats <https://pulsar.apache.org/docs/latest/admin/Stats/>
output in prometheus format?
----
2019-01-09 21:45:10 UTC - Matteo Merli: These are the stats you can fetch through REST API,
in JSON format for specific topics
----
2019-01-09 21:45:42 UTC - Matteo Merli: Brokers also have a /metrics endpoint to collect metrics
in Prometheus format
----
2019-01-09 21:48:03 UTC - Emma Pollum: But not all of these metrics are at that endpoint?
----
2019-01-09 21:49:04 UTC - Matteo Merli: It’s possible. I don’t remember the specific differences
right now :slightly_smiling_face:
----
2019-01-09 21:49:40 UTC - Emma Pollum: Ok!
----
2019-01-09 22:00:42 UTC - Grant Wu: @Jonathan Budd
----
2019-01-10 00:14:46 UTC - Emma Pollum: I'm having trouble gettting the msgRateIn and averageMsgSize
metrics ported to my collector (i can collect with either telegraf or prometheus, and open
to using other methods). It seems I my only option is to parse through the long json created
by pulsar-admin broker-stats topics .

Does anyone have any other ideas on how to do this effeciently?
----
2019-01-10 00:42:27 UTC - Grant Wu: Can we please, please make an effort to move away from
Slack?
----
2019-01-10 00:42:59 UTC - Grant Wu: As I have now been in this Slack for a while, the rate
at which I end up trying to find a message which I know existed, but which has been lost to
the 10k message limit continues to increase.
----
2019-01-10 00:43:45 UTC - Grant Wu: Alternatively… if anyone has a way to search the Slack
digests…
----
2019-01-10 00:44:43 UTC - Matteo Merli: Yes, the preferred form of communication, from Apache
perspective, would be the <mailto:user@pulsar.apache.org|user@pulsar.apache.org> mailing
list
----
2019-01-10 00:45:04 UTC - Matteo Merli: (That’s also where the slack is being archived)
----
2019-01-10 00:45:20 UTC - Grant Wu: Indeed… but there doesn’t seem to be any way to search
these archives as far as I can tell
----
2019-01-10 00:45:32 UTC - Matteo Merli: <https://lists.apache.org/list.html?users@pulsar.apache.org>
----
2019-01-10 00:45:45 UTC - Matteo Merli: That should work, no?
----
2019-01-10 00:45:50 UTC - Grant Wu: Ah, okay
----
2019-01-10 00:45:57 UTC - Grant Wu: I was looking at <http://mail-archives.apache.org/mod_mbox/pulsar-users/201812.mbox/browser>
----
2019-01-10 00:47:27 UTC - Matteo Merli: In any case, the lists are kind of more suited for
more in depth discussion, though people seem to be more confident in asking questions here
:)
----
2019-01-10 00:51:00 UTC - Grant Wu: Yeah… I am not particularly a fan of mailing lists :sweat:
My understanding is that for them to be usable at all you need to have a tricked out mail
client, and I don’t really have the time to deal with that right now
----
2019-01-10 00:54:26 UTC - Grant Wu: Anyways my immediate needs are satisfied by this better
search box…
----
2019-01-10 03:03:36 UTC - Shalin: I have deployed pulsar on a k8s cluster using the default
helm chart and exposed it using `aws-alb-ingress-controller` to create load balancers for
`pulsar-proxy` service. My ingress resource looks like this :
```
- http:
        paths:
          - path: /metrics/*
            backend:
              serviceName: "womping-newt-pulsar-proxy"
              servicePort: 8080
          - path: /*
            backend:
              serviceName: "womping-newt-pulsar-proxy"
              servicePort: 6650
```
How do I connect my pulsar client to the broker server. I tried `pulsar.Client('pulsar://&lt;alb-dns&gt;:6650')`
but its hangs when I try to create a producer or consumer.
Isn't that how the clients connect to the broker via the proxy?
----
2019-01-10 03:08:17 UTC - Ali Ahmed: check the logs of the proxy
----
2019-01-10 03:11:48 UTC - Shalin: There are errors `TooLongFrameException: Adjusted frame
length exceeds 5242880: 1195725860 - discarded` but not while making the connections. could
be from the liviness check from kubernetes or health check from alb.
----
2019-01-10 03:12:13 UTC - Shalin: `curl &lt;alb-dns&gt;/metrics/` works though
----
2019-01-10 03:12:45 UTC - Matteo Merli: I think the load balancer is proxying HTTP
----
2019-01-10 03:13:03 UTC - Matteo Merli: You’d have to point to port 8080 on brokers
----
2019-01-10 03:13:20 UTC - Matteo Merli: Or use a different config for the load balancer
----
2019-01-10 03:14:09 UTC - Shalin: 8080 for clients to connect to brokers? :thinking_face:
----
2019-01-10 03:14:56 UTC - Matteo Merli: Service discovery is also possible on HTTP
----
2019-01-10 03:15:23 UTC - Matteo Merli: For data, you’d have to expose 6650 in proxy in
a TCP load balancer 
----
2019-01-10 03:35:23 UTC - Shalin: If I am not wrong load balancer is forwarding all requests(except
"metrics/") to `pulsar-proxy` at port 6650 which in turn forwards it to 6650 of the broker.
----
2019-01-10 04:09:13 UTC - Matteo Merli: I think the problem is that it’s defined as HTTP
layer load balancer 
----
2019-01-10 04:10:30 UTC - Matteo Merli: Eg, take a look at <https://github.com/apache/pulsar/blob/99f05b5c6acca8dc73fe02a73bde2ca343349cd2/deployment/kubernetes/google-kubernetes-engine/proxy.yaml#L66>
----
2019-01-10 04:47:45 UTC - Shalin: Gotcha. I am using the helm chart version of the `pulsar-proxy`
in the repo which is just a service of type NodePort and not LoadBalancer.  I am exposing
the ports using an AWS Application load balancer in front of the proxy. But I get what you
mean. AWS ALB only supports HTTP and not TCP.
----
2019-01-10 05:23:14 UTC - Shalin: ALB is not an option.  Any ideas other than creating your
own ELB ingress controller?
----
2019-01-10 08:20:03 UTC - Romain Castagnet: This is the full stack with the first warning
until 500 http response.
----
Mime
View raw message