pulsar-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Pulsar Slack" <apache.pulsar.sl...@gmail.com>
Subject Slack digest for #general - 2019-04-30
Date Tue, 30 Apr 2019 09:11:04 GMT
2019-04-29 11:11:51 UTC - Poule: I follow the `functions` docs and when I create a function,
server standalone tells me `ImportError: No module named _pulsar`
----
2019-04-29 11:12:23 UTC - Poule: comes from `apache-pulsar-2.3.1/instances/python-instance/pulsar/__init__.py",
line 99`
----
2019-04-29 11:14:40 UTC - Alexandre DUVAL: Hi, where can I find the last archive of the master
branch, even snapshots versions?
----
2019-04-29 11:26:56 UTC - Poule: ok fixed, I started the standalone in a `pipenv` env with
python 2.7.15 and it works now
----
2019-04-29 11:29:07 UTC - jia zhai: @Alexandre DUVAL <https://repository.apache.org/content/repositories/snapshots/org/apache/pulsar/>
hope this will hep you.
----
2019-04-29 11:30:17 UTC - Alexandre DUVAL: @jia zhai it does, thank you.
----
2019-04-29 11:30:37 UTC - jia zhai: welcome:+1:
----
2019-04-29 11:32:54 UTC - Alexandre DUVAL: @jia zhai oh, nope, it's not the built one. I need
it as bin archive :stuck_out_tongue:.
----
2019-04-29 11:37:58 UTC - jia zhai: @Sijie Guo Is there a binary for nightly build?
----
2019-04-29 11:40:21 UTC - Sijie Guo: @jia zhai: it used to publish here - <https://repository.apache.org/content/repositories/snapshots/org/apache/pulsar/distribution/2.4.0-SNAPSHOT/>
----
2019-04-29 11:40:36 UTC - Sijie Guo: but it seems that it stopped publishing
----
2019-04-29 11:41:38 UTC - Sijie Guo: I am not sure if it is related to <https://github.com/apache/pulsar/commit/8365808ae66eafc2bc15c2dfdd4c1e98315c09ec>
----
2019-04-29 11:43:05 UTC - jia zhai: seems in 2.2.0 there is also no binary left:<https://repository.apache.org/content/repositories/snapshots/org/apache/pulsar/distribution/2.2.0-incubating-SNAPSHOT/>
----
2019-04-29 11:44:39 UTC - Sijie Guo: <https://repository.apache.org/content/repositories/snapshots/org/apache/pulsar/pulsar-server-distribution/2.4.0-SNAPSHOT/>
----
2019-04-29 11:44:40 UTC - Sijie Guo: nvm here you go @Alexandre DUVAL
----
2019-04-29 11:45:09 UTC - Sijie Guo: I checked the wrong location
----
2019-04-29 11:45:21 UTC - Alexandre DUVAL: Cool! Ty.
----
2019-04-29 11:46:32 UTC - jia zhai: seems this parameter “skipBuildDistribution” is not
used
----
2019-04-29 11:48:45 UTC - Sijie Guo: yes because it is not defined. it only skips deploying
the package when `-DskipBuildDistribution` is provided
----
2019-04-29 12:13:11 UTC - Alexandre DUVAL: For HTTP requests on admin rest api, how the auth
is provided on the http request (which headers, which format), I don't find these informations
on <https://pulsar.apache.org/admin-rest-api/>. More, why there is no HTTP 200 response
possibility on <https://pulsar.apache.org/admin-rest-api/#operation/createTenant>?
----
2019-04-29 12:16:55 UTC - Alexandre DUVAL: (I need to setup a tenant/namespace creation if
they don't exist during context.publish in pulsar function). What's the better way to do it?
I'm trying stuff like this: ```        context.publish(
          new StringBuilder("persistent://")
            .append(ownerId)
            .append("/")
            .append(appId)
            .append("/")
            .append("logs")
            .toString(),
          input
        ).exceptionally(exception -&gt; {
          if (exception instanceof PreconditionFailedException) { // tenant/namespace couple
does not exists (HTTP 412)
            createTenantAndNamespaceFor(ownerId, appId);
            // then retry to publish
          }
        })```
----
2019-04-29 12:25:55 UTC - Alexandre DUVAL: Or maybe the best way is to reuse pulsarAdmin methods
directly from pulsar function?
----
2019-04-29 12:30:13 UTC - Sijie Guo: &gt; how the auth is provided on the http request
(which headers, which format), I don’t find these informations on <https://pulsar.apache.org/admin-rest-api/>

ah yeah. the documentation around authentication is missing for rest api.

&gt;  maybe the best way is to reuse pulsarAdmin methods directly from pulsar function?

if you are using java to implement a pulsar function, it might be worth just using the pulsar-admin
library.
----
2019-04-29 12:39:07 UTC - Poule: in 2.3.1 is it possible that `pulsar-admin function trigger`calls
the wrong function?
----
2019-04-29 12:41:24 UTC - Poule: i created `reverse` and `exclamation` and calling `exclamation`
gives me the `reverse` result.
----
2019-04-29 12:42:03 UTC - Poule: I delete the `reverse` function then I get the correct result
when calling `exclamation`
----
2019-04-29 12:50:43 UTC - Poule: i will investigate more later
----
2019-04-29 13:41:24 UTC - Shivji Kumar Jha: I am trying to route the prometheus metrics pulsar
exposes at broker-url:8080/metrics to telegraf/ Telegraf errors out saying prometheus metrics
output is not in correct format.

Error:
2019-04-29T13:38:40Z E! Error in plugin [inputs.prometheus]: error reading metrics for &lt;broker-host&gt;:8080/metrics:
reading text format failed: text format parsing error in line 189: second TYPE line for metric
name “pulsar_subscriptions_count”, or TYPE reported after samples

Here s what the output of /metric line 189 where the error is pointing

# TYPE pulsar_subscriptions_count gauge
pulsar_subscriptions_count{cluster=“stage”,namespace=“beam/sc-realtime”,topic=“<persistent://beam/test-ns/test-topic4%22|persistent://beam/test-ns/test-topic4">}
1.0 1556541781642
----
2019-04-29 13:44:50 UTC - Shivji Kumar Jha: seems like <https://github.com/apache/pulsar/issues/3112>
----
2019-04-29 13:44:56 UTC - Shivji Kumar Jha: Anyone else faced this? how did you fix this?
----
2019-04-29 14:05:36 UTC - Alexandre DUVAL: ```# Enable Functions Worker Service in Broker
functionsWorkerEnabled=false``` this field in broker.conf is related to activate broker as
function worker or it should be enabled for the functions worker node to work?
----
2019-04-29 14:05:57 UTC - Alexandre DUVAL: Because I have the following issue ```bin/pulsar-admin
functions list                                                                           
                                    
Function worker service is not done initializing. Please try again in a little while.

Reason: HTTP 503 Service Unavailable```
----
2019-04-29 14:06:26 UTC - Alexandre DUVAL: But my functions worker node is running: ```13:55:10.274
[pulsar-client-io-7-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [<persistent://yo/functions/assignments>][reader-37f12b46a5]
Subscribing to topic on cnx [id: 0xf51b9e1a, L:/10.2.1.4:33364 - R:<http://c1-pulsar-yo-customers.services.clever-cloud.com/185.42.117.147:2001|c1-pulsar-yo-customers.services.clever-cloud.com/185.42.117.147:2001>]
13:55:10.279 [pulsar-client-io-7-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [<persistent://yo/functions/assignments>][reader-37f12b46a5]
Subscribed to topic on <http://c1-pulsar-yo-customers.services.clever-cloud.com/123.213.123.123:2001|c1-pulsar-yo-customers.services.clever-cloud.com/123.213.123.123:2001>
-- consumer: 2
13:55:10.280 [main] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [<persistent://yo/functions/assignments>][reader-37f12b46a5]
Get topic last message Id
13:55:10.284 [pulsar-client-io-7-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [<persistent://yo/functions/assignments>][reader-37f12b46a5]
Successfully getLastMessageId 120:-1
13:55:10.293 [main] INFO  org.apache.pulsar.broker.authentication.AuthenticationService -
Authentication is disabled
13:55:10.294 [main] INFO  org.apache.pulsar.functions.worker.WorkerService - Start cluster
services...
13:55:10.295 [main] INFO  org.apache.pulsar.functions.worker.FunctionRuntimeManager - /**
Starting Function Runtime Manager **/
13:55:10.296 [main] INFO  org.apache.pulsar.functions.worker.FunctionRuntimeManager - Initialize
metrics sink...
13:55:10.296 [main] INFO  org.apache.pulsar.functions.worker.FunctionRuntimeManager - Starting
function assignment tailer...
13:55:10.296 [main] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Searching
for connectors in /opt/apache-pulsar-2.3.1/./connectors
13:55:10.296 [main] WARN  org.apache.pulsar.functions.utils.io.ConnectorUtils - Connectors
archive directory not found
13:55:10.310 [main] INFO  org.eclipse.jetty.util.log - Logging initialized @4296ms to org.eclipse.jetty.util.log.Slf4jLog
13:55:10.452 [main] INFO  org.eclipse.jetty.server.Server - jetty-9.4.12.v20180830; built:
2018-08-30T13:59:14.071Z; git: 27208684755d94a92186989f695db2d7b21ebc51; jvm 1.8.0_192-b12
13:55:10.720 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@77ad2767{/admin,null,AVAILABLE}
13:55:10.789 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@e460ca1{/admin/v2,null,AVAILABLE}
13:55:10.908 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@411ae0d7{/admin/v3,null,AVAILABLE}
13:55:10.943 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@608c2042{/,null,AVAILABLE}
13:55:10.954 [main] INFO  org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@4f571c0e{HTTP/1.1,[http/1.1]}{0.0.0.0:2000}
13:55:10.958 [main] INFO  org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@3f919230{SSL,[ssl,
http/1.1]}{0.0.0.0:2001}
13:55:10.965 [main] INFO  org.eclipse.jetty.server.Server - Started @4952ms
13:55:10.965 [main] INFO  org.apache.pulsar.functions.worker.rest.WorkerServer - Worker Server
started at <http://192.168.10.16:2000/admin>
13:55:10.965 [main] INFO  org.apache.pulsar.functions.worker.Worker - Start worker server
on port 2000...
````
----
2019-04-29 14:06:44 UTC - Alexandre DUVAL: Or it's another problem? :smile:
----
2019-04-29 14:07:02 UTC - Alexandre DUVAL: (@Sijie Guo maybe :))
----
2019-04-29 14:09:32 UTC - Sijie Guo: @Alexandre DUVAL :

`functionsWorkerEnabled=false` is for enabling funcation worker on broker.

if you have run a separated function worker, specifying your function worker location when
you using `bin/pulsar-admin functions`.

`bin/pulsar-admin --admin-url http://&lt;function-worker&gt;:6750 functions`
----
2019-04-29 14:09:51 UTC - Alexandre DUVAL: Oh, ok :slightly_smiling_face:.
----
2019-04-29 14:28:18 UTC - Alexandre DUVAL: Ok, I still have problems on authentication configuration
on functions worker node.
----
2019-04-29 14:28:20 UTC - Alexandre DUVAL: ```############################################
# security settings for worker service
############################################

# Enforce authentication
authenticationEnabled: true
# Enforce authorization on accessing functions api
authorizationEnabled: true
# Set of autentication provider name list, which is a list of class names
authenticationProviders:
  - org.apache.pulsar.broker.authentication.AuthenticationProviderToken
# Set of role names that are treated as "super-user", meaning they will be able to access
any admin-api
superUserRoles:
  - super-user

tokenPublicKey: file:///pulsar/conf/keys/my-public.key```
----
2019-04-29 14:28:32 UTC - Alexandre DUVAL: I don't know how to write them.
----
2019-04-29 14:28:44 UTC - Alexandre DUVAL: ```14:24:51.797 [main] ERROR org.apache.pulsar.functions.worker.WorkerService
- Error Starting up in worker
org.apache.pulsar.broker.PulsarServerException: Failed to load an authentication provider.
        at org.apache.pulsar.broker.authentication.AuthenticationService.&lt;init&gt;(AuthenticationService.java:64)
~[org.apache.pulsar-pulsar-broker-common-2.3.1.jar:2.3.1]
        at org.apache.pulsar.functions.worker.WorkerService.start(WorkerService.java:179)
[org.apache.pulsar-pulsar-functions-worker-2.3.1.jar:2.3.1]
        at org.apache.pulsar.functions.worker.Worker.start(Worker.java:51) [org.apache.pulsar-pulsar-functions-worker-2.3.1.jar:2.3.1]
        at org.apache.pulsar.functions.worker.FunctionWorkerStarter.main(FunctionWorkerStarter.java:67)
[org.apache.pulsar-pulsar-functions-worker-2.3.1.jar:2.3.1]
Caused by: java.io.IOException: No secret key was provided for token authentication
        at org.apache.pulsar.broker.authentication.AuthenticationProviderToken.getValidationKey(AuthenticationProviderToken.java:135)
~[org.apache.pulsar-pulsar-broker-common-2.3.1.jar:2.3.1]
        at org.apache.pulsar.broker.authentication.AuthenticationProviderToken.initialize(AuthenticationProviderToken.java:61)
~[org.apache.pulsar-pulsar-broker-common-2.3.1.jar:2.3.1]
        at org.apache.pulsar.broker.authentication.AuthenticationService.&lt;init&gt;(AuthenticationService.java:56)
~[org.apache.pulsar-pulsar-broker-common-2.3.1.jar:2.3.1]
        ... 3 more                                                                       
       ```
----
2019-04-29 14:30:03 UTC - Alexandre DUVAL: (@Sijie Guo :p)
----
2019-04-29 15:12:26 UTC - Dan C: @Dan C has joined the channel
----
2019-04-29 15:12:59 UTC - Laurent Chriqui: Hello, everyone, we are experiencing TimeOut errors
when publishing messages to topics using the python client with async producers.
We have the option batching enabled and block if queue full enabled with a batching max publish
delay at 10 ms.
Has anyone encountered this problem yet ?
----
2019-04-29 15:17:53 UTC - Chris Bartholomew: @Shivji Kumar Jha Yes, I have seen this. I was
able to workaround it by disabling topic-level metrics. You do that by setting ```exposeTopicLevelMetricsInPrometheus=false```
in broker.conf. I am working on a fix for this. Just doing some final testing. I should be
submitting a PR today.
----
2019-04-29 15:22:38 UTC - Shivji Kumar Jha: Great, i will do that and try it out. Thank you
:blush: 
----
2019-04-29 15:31:13 UTC - Alexandre DUVAL: or @David Kjerrumgaard? or @Matteo Merli?
----
2019-04-29 15:46:35 UTC - David Kjerrumgaard: According to the error message, "No secret key
was provided for token authentication", It looks like you didn't specify a private key for
the token.
----
2019-04-29 15:46:51 UTC - David Kjerrumgaard: And the config file you shared only listed a
public key.
----
2019-04-29 15:50:37 UTC - Alexandre DUVAL: But if I'm using public key as auth, why do I need
secret?
----
2019-04-29 15:55:46 UTC - Alexandre DUVAL: I mean i use token generated from private key,
and i give the public to the node.
----
2019-04-29 16:00:34 UTC - David Kjerrumgaard: You will need a private key on your broker.
----
2019-04-29 16:00:35 UTC - David Kjerrumgaard: <http://pulsar.apache.org/docs/en/security-token-admin/>
----
2019-04-29 16:04:01 UTC - Alexandre DUVAL: It's write public or secret on this doc, no?
----
2019-04-29 16:05:43 UTC - Marc Le Labourier: We don't know yet if the batching is not working
properly, but it seems that our brokers are running at full capacity (Memory and CPU). Should
it not be regulated before it happens ?
----
2019-04-29 16:07:12 UTC - David Kjerrumgaard: When you configure token authentication, you
need to specify the 'tokenSecretKey' property in your broker config, but it looks like you
have 'tokenPublicKey' instead
----
2019-04-29 16:08:10 UTC - Alexandre DUVAL: No, I use asym tokens generated not using secret.
That's my point.
----
2019-04-29 16:08:25 UTC - Alexandre DUVAL: ````# If using secret key
tokenSecretKey=file:///path/to/secret.key
# The key can also be passed inline:
# tokenSecretKey=data:base64,FLFyW0oLJ2Fi22KKCm21J18mbAdztfSHN/lAT5ucEKU=

# If using public/private
# tokenPublicKey=file:///path/to/public.key``````
----
2019-04-29 16:08:27 UTC - Alexandre DUVAL: if*
----
2019-04-29 16:15:32 UTC - David Kjerrumgaard: With JWT Symmetric tokens there is a single
Secret key that is used both to generate and validate. So it has to be labelled as "secret"
in the broker config file AFAIK
----
2019-04-29 16:16:38 UTC - Alexandre DUVAL: Yes, but there are asymetric tokens that im using
:stuck_out_tongue:.
----
2019-04-29 16:17:09 UTC - Alexandre DUVAL: And not symetrics ones
----
2019-04-29 16:19:33 UTC - David Kjerrumgaard: Either way, the broker config expects a property
named "tokenSecretKey=file:///path/to/secret.key"  and you have tokenPublicKey=file:///path/to/public.key```
----
2019-04-29 16:20:14 UTC - David Kjerrumgaard: So the error is complaining that you configured
token based authentication but did NOT provide a required property, namely "tokenSecretKey"
----
2019-04-29 16:20:45 UTC - David Kjerrumgaard: you provided "tokenPublicKey", which the broker
doesn't recognize
----
2019-04-29 16:21:03 UTC - Alexandre DUVAL: In the documentation it is write:
----
2019-04-29 16:21:11 UTC - Alexandre DUVAL: ``# If using secret key
tokenSecretKey=file:///path/to/secret.key````
----
2019-04-29 16:21:18 UTC - Alexandre DUVAL: and
----
2019-04-29 16:21:20 UTC - Alexandre DUVAL: ``# If using public/private
# tokenPublicKey=file:///path/to/public.key```````
----
2019-04-29 16:21:31 UTC - Alexandre DUVAL: sorry windows char for ` is bad^^^.
----
2019-04-29 16:22:04 UTC - David Kjerrumgaard: "Caused by: java.io.IOException: No secret key
was provided for token authentication
        at org.apache.pulsar.broker.authentication.AuthenticationProviderToken.getValidationKey(AuthenticationProviderToken.java:135)
"
----
2019-04-29 16:22:22 UTC - Alexandre DUVAL: yes, I read this, but it's on functions worker
only.
----
2019-04-29 16:22:36 UTC - Alexandre DUVAL: Broker have publictokenkey only and it's work perfectly
----
2019-04-29 16:22:40 UTC - Alexandre DUVAL: you get the point?
----
2019-04-29 16:22:50 UTC - David Kjerrumgaard: ^^^ Above is complaining about the key.  Is
the path to the key correct?  Are file permissions correct nd allow read access?
----
2019-04-29 16:23:15 UTC - David Kjerrumgaard: So the error is in the function worker?
----
2019-04-29 16:23:17 UTC - Alexandre DUVAL: That's my point, I don't know what to write the
configuration using yaml, because functions worker conf uses yaml
----
2019-04-29 16:24:21 UTC - Alexandre DUVAL: There are no documentation on the way to write
the configuration field using file in yaml.
----
2019-04-29 16:27:25 UTC - Devin G. Bost: Is there a particular reason that `FunctionConfigBuilder`
is not public?
----
2019-04-29 16:39:55 UTC - Dan C: Hi all. I’m attempting to use the websocket API (broker
embedded) on a local minikube. When running a test consumer, I receive “Handshake status
404 Not Found”. I’ve confirmed that the API is enabled on the broker config. I am using
the pulsar proxy.
----
2019-04-29 16:41:48 UTC - David Kjerrumgaard: @Dan C Can you confirm that you have configured
port forwarding between the machine where the websocket client is running and the broker inside
the minikube container?
----
2019-04-29 16:44:06 UTC - Dan C: I have the proxy service running and the port forwarding
appears correct (30002 -&gt; 8080 on the container)
----
2019-04-29 16:44:58 UTC - David Kjerrumgaard: So you have completed both of these steps?
----
2019-04-29 16:46:13 UTC - David Kjerrumgaard: I would try using the standard auth settings
in the broker.conf file FIRST. if that doesn't work, then we can go to the function worker.yaml
file.  I suspect that the auth properties are picked up from the borker.conf file
----
2019-04-29 16:48:15 UTC - David Kjerrumgaard: Can you telnet to that address and port? (confirm
networking, etc first).
----
2019-04-29 17:14:24 UTC - Shivji Kumar Jha: That didnt work out :disappointed: Because you
are already working on this part can you run your test for my prometheus output maybe? Let
me know if you need anything else from my prometheus output.
----
2019-04-29 17:59:31 UTC - Devin G. Bost: Nevermind. I figured out how to use it.
----
2019-04-29 18:06:10 UTC - Alexandre DUVAL: Okay, but no @Sijie Guo told that functionworkerenabled=true
is only to run function on broker but im running a separated node for functions only.
----
2019-04-29 18:06:51 UTC - Alexandre DUVAL: And function cluster is correctly configured yes
----
2019-04-29 18:07:34 UTC - Dan C: I am able to telnet and also reach the admin REST endpoint.
It seems that it is just the websocket endpoint that isn’t working via the proxy, although
it is configured to run on the broker web service.
----
2019-04-29 18:09:05 UTC - Alexandre DUVAL: And it uses the yaml file cause it's work when
i disable auth.
----
2019-04-29 18:09:50 UTC - Matteo Merli: Each authentication plugin will have a different way
to pass the credentials. eg:
 * TLS --&gt; client certificate
 * Tokens &gt; `Authorization: Bearer xxxxxxxx`
----
2019-04-29 18:10:43 UTC - Dan C: If I ssh to a broker pod, I can access it though. So it appears
to just not work via the proxy service.
----
2019-04-29 18:11:04 UTC - Matteo Merli: @Shivji Kumar Jha This PR fixes that: <https://github.com/apache/pulsar/pull/4136>
----
2019-04-29 18:11:22 UTC - Alexandre DUVAL: But without auth, i cant create function on it.
----
2019-04-29 18:13:38 UTC - Alexandre DUVAL: WDYT @Matteo Merli (sorry there is no TL; DR :/)
----
2019-04-29 18:14:46 UTC - Alexandre DUVAL: Thanks for the answer
----
2019-04-29 18:40:33 UTC - Dan C: Changing the broker service to a NodePort resolved this.
----
2019-04-29 18:42:47 UTC - Dan C: I was connecting to the proxy service webservice as opposed
to the broker webservice which has the websocket api running.
+1 : David Kjerrumgaard
----
2019-04-29 19:08:56 UTC - Alexandre DUVAL: @Matteo Merli here is the configuration of a separated
functions worker
----
2019-04-29 19:09:22 UTC - Alexandre DUVAL: @Matteo Merli and here is the error, i can't understand
why it requires secret because im using asym with publickey
----
2019-04-29 21:36:08 UTC - Ryan Dietrich: @Ryan Dietrich has joined the channel
----
2019-04-29 21:37:23 UTC - Ryan Dietrich: Hi, quick questions using node.js and pulsar.  1.
Is the websocket API the best option right now?  2. Using websockets, can you subscribe to
multiple topics on a single connection, or is it one connection per topic?
----
2019-04-29 21:38:26 UTC - Ali Ahmed: @Ryan Dietrich you can try the binary protocol node js
client , it’s still under development . <https://github.com/apache/pulsar-client-node>
----
2019-04-29 21:38:39 UTC - Ryan Dietrich: Woah, cool!  I had no idea that was in the works!
----
2019-04-29 21:39:51 UTC - Guy Feldman: @Ryan Dietrich it's fairly new. There's a <#CFTUA9AHK|dev-node>
for discussion of the library
----
2019-04-29 21:41:13 UTC - Ryan Dietrich: that's really great news.. thank you both for the
quick reply!
----
2019-04-29 21:42:17 UTC - Ryan Dietrich: My place of work is having a bake-off between Pulsar
and Kafka, and after reading about embedded functions and non-persistent storage I became
a fan, just gotta deliver a solid implementation now.
----
2019-04-29 21:43:15 UTC - Guy Feldman: we chose pulsar over kafka, mostly for security and
multi-tenancy
----
2019-04-29 21:43:40 UTC - Guy Feldman: we're using the websocket api cause the node client
wasn't around yet
----
2019-04-29 21:44:11 UTC - Guy Feldman: and the performance has been good too
----
2019-04-29 21:45:06 UTC - Ryan Dietrich: do you happen to know the speed difference between
persistent and non-persistent storage?
----
2019-04-29 21:45:11 UTC - Ali Ahmed: @Ryan Dietrich I am guessing you are in finance , you
should know pulsar offers much stronger durability guarantees then kafka, that alone should
a reason to switch for mission critical applications
----
2019-04-29 21:45:28 UTC - Guy Feldman: i do not. we haven't used non-persistent yet
----
2019-04-29 21:45:29 UTC - Ryan Dietrich: @Ali Ahmed uh oh, you looked me up didn't you :wink:
----
2019-04-29 21:45:48 UTC - Guy Feldman: it's been fast enough with persistent storage
----
2019-04-29 21:45:49 UTC - Ryan Dietrich: I just have always had a thing for non-critical messages
not having to go to disk.
----
2019-04-29 21:46:21 UTC - Ryan Dietrich: The idea that "not all messages are created equal"
is something I think is valuable.
----
2019-04-29 21:46:43 UTC - Ryan Dietrich: just my opinion
----
2019-04-29 21:46:46 UTC - Guy Feldman: personally what sold me is the s3 offloading.
+1 : Matteo Merli
----
2019-04-29 21:47:04 UTC - Guy Feldman: you can always put those messages in a short lived
topic and offload it
----
2019-04-29 21:47:12 UTC - Ali Ahmed: I don’t think anyone has benchmarked non-persistent
topics. so don’t know how big is the perf delta.
----
2019-04-29 21:47:54 UTC - Ryan Dietrich: well, I will, in places where we don't send "deltas"
the latest messages is only the most important, so non-persistent would be better in that
case
----
2019-04-29 21:48:01 UTC - Ryan Dietrich: also, I'm ignorant of what s3 offloading does?
----
2019-04-29 21:48:34 UTC - Guy Feldman: you can basically set a size or time where messages
past a certain time are stored in s3
----
2019-04-29 21:48:37 UTC - Guy Feldman: rather than disk
----
2019-04-29 21:48:51 UTC - Guy Feldman: so you have infinite retention if you want but only
recent messages are on disk
----
2019-04-29 21:49:33 UTC - Ryan Dietrich: I assume you treat it as a cold-storage, as the access
time is a tad slower, right?
----
2019-04-29 21:49:37 UTC - Guy Feldman: yeah
----
2019-04-29 21:49:39 UTC - Ali Ahmed: it’s a transparent tiered storage layer so you can
build a cost efficient event store
----
2019-04-29 21:50:03 UTC - Guy Feldman: yeah the benefit is you can connect tools that use
pulsar and still query the data
----
2019-04-29 21:50:04 UTC - Guy Feldman: like presto
----
2019-04-29 21:50:34 UTC - Ryan Dietrich: Our devops team has complained that dealing with
zookeeper was their biggest fear for either project, has that been an issue for you?
----
2019-04-29 21:51:09 UTC - Guy Feldman: not for us
+1 : Ryan Dietrich
----
2019-04-29 21:52:56 UTC - Guy Feldman: we're using zookeeper for other things though too
----
2019-04-29 21:53:00 UTC - Ali Ahmed: pulsar depends upon but doesn’t excessively put load
on zk, it use bookkeeper for a lot heavy lifting so zk does not become a bottleneck. It can
therefore support millions of topics
----
2019-04-29 21:53:36 UTC - Guy Feldman: we're using zookeeper for flink ha too
----
2019-04-29 21:55:01 UTC - Ryan Dietrich: last question (you guys have been super awesome btw).
 can you give me any details on how replication between nodes works?  If you have a cluster
of pulsars, does it end up being having multiple points of failure, or do you get increased
performance and stability? (that question may sound weird, but please trust me I am asking
it for the right reasons).
----
2019-04-29 21:56:10 UTC - Ali Ahmed: pulsar has two layers stateless brokers for serving data
and distributed storage layer bookies for persistence.
----
2019-04-29 21:56:38 UTC - Ali Ahmed: bookies transparent keep the required number of copies
distributed among them in custer
----
2019-04-29 21:57:10 UTC - Ryan Dietrich: at a technical level, does it use Akka by chance?
----
2019-04-29 21:57:16 UTC - Ali Ahmed: you can scale brokers and bookies independently of each
other depending on the workload
----
2019-04-29 21:57:44 UTC - Ali Ahmed: no akka is not used anywhere
----
2019-04-29 21:58:54 UTC - Ryan Dietrich: Phew.. Ok.. I had some incredibly bad experiences
with OpenDaylight recently (which uses Akka, and was hoping to avoid debugging issues in there
again)
----
2019-04-29 23:08:41 UTC - Devin G. Bost: How much of a change would it be for us to be able
to get a hash (MD5 or SHA1) of the Jar for a given component (e.g. Function) in Pulsar? It
would be helpful for determining if the Jar version matches the version we have in our build
repo.
----
2019-04-30 02:44:02 UTC - Ali Ahmed: @Devin G. Bost I think it should not be that hard but
the wider issue is how to manage function versioning
----
2019-04-30 03:56:11 UTC - Matt Grosso: @Matt Grosso has joined the channel
----
2019-04-30 05:57:11 UTC - Shivji Kumar Jha: thanks @Matteo Merli I will cherry pick and this
and give it a shot.
----
2019-04-30 08:01:42 UTC - Shivji Kumar Jha: @Chris Bartholomew disabling topic level metrics
works. telegraf can parse the prometheus output in that case.
----
Mime
View raw message