pulsar-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Pulsar Slack" <apache.pulsar.sl...@gmail.com>
Subject Slack digest for #general - 2019-04-18
Date Thu, 18 Apr 2019 09:11:02 GMT
2019-04-17 11:59:56 UTC - Ramesh Bobba: @Ramesh Bobba has joined the channel
----
2019-04-17 12:59:23 UTC - Laurent Chriqui: Hello. I have tried running my pulsar function
on Mac Os X with the latest pulsar-client 2.3.1 and  I get the following error :
Traceback (most recent call last):
  File “/Users/laurentchriqui/Downloads/apache-pulsar-2.3.1/instances/python-instance/python_instance_main.py”,
line 36, in &lt;module&gt;
    import pulsar
  File “/Users/laurentchriqui/Downloads/apache-pulsar-2.3.1/instances/python-instance/pulsar/__init__.py”,
line 101, in &lt;module&gt;
    from _pulsar import Result, CompressionType, ConsumerType, InitialPosition, PartitionsRoutingMode
 # noqa: F401
ImportError: cannot import name InitialPosition

Indeed, the class is not present in my _pulsar.cpython-37m-darwin.so
@Matteo Merli
----
2019-04-17 13:02:53 UTC - Matteo Merli: It’s seems like you have the older client in your
mac. Did you upgrade the client lib? `pip3 install pulsar-client==2.3.1`
----
2019-04-17 13:04:43 UTC - Alexandre DUVAL: Hi, considering this quote, there is a missing
field on functions_worker.yml, something like tokenPublicKey?
----
2019-04-17 13:05:05 UTC - Alexandre DUVAL: And considering yaml, could it be `tokenPublicKey:
file:///pulsar/conf/keys/my-public.key` ?
----
2019-04-17 13:09:21 UTC - Alexandre DUVAL: here
----
2019-04-17 13:15:15 UTC - Alexandre DUVAL: or maybe i dont get the right informations from
the error logs :stuck_out_tongue:
----
2019-04-17 13:15:49 UTC - Laurent Chriqui: yes I did
----
2019-04-17 13:16:24 UTC - Laurent Chriqui: I think there may be a problem with the compiled
files for Mac Os
----
2019-04-17 13:19:06 UTC - Matteo Merli: definitely it’s missing the public key.. though
I’m not sure how to pass that in that yaml file
----
2019-04-17 13:19:08 UTC - Matteo Merli: :smile:
----
2019-04-17 13:19:32 UTC - Matteo Merli: let me try to pull them from PyPI
----
2019-04-17 13:21:17 UTC - Alexandre DUVAL: `tokenPublicKey: file:///pulsar/conf/keys/my-public.key`
this doesn't work, exactly the same error
----
2019-04-17 13:21:25 UTC - Matteo Merli: Seems to work here:

```
$ pip3 install pulsar-client
Collecting pulsar-client
  Downloading <https://files.pythonhosted.org/packages/38/74/01281eedf5134fc26b9763611b364f90be36b2f3f6d601cf6e31ce988924/pulsar_client-2.3.1-cp37-cp37m-macosx_10_14_x86_64.whl>
(15.7MB)
    100% |████████████████████████████████|
15.7MB 1.8MB/s
Requirement already satisfied: grpcio in /usr/local/lib/python3.7/site-packages/grpcio-1.18.0rc1-py3.7-macosx-10.14-x86_64.egg
(from pulsar-client) (1.18.0rc1)
Requirement already satisfied: six in /usr/local/Cellar/protobuf/3.6.1.3_1/libexec/lib/python3.7/site-packages
(from pulsar-client) (1.11.0)
Requirement already satisfied: ratelimit in /usr/local/lib/python3.7/site-packages/ratelimit-2.2.1-py3.7.egg
(from pulsar-client) (2.2.1)
Requirement already satisfied: fastavro in /usr/local/lib/python3.7/site-packages (from pulsar-client)
(0.21.16)
Requirement already satisfied: protobuf&gt;=3.6.1 in /usr/local/Cellar/protobuf/3.6.1.3_1/libexec/lib/python3.7/site-packages
(from pulsar-client) (3.6.1)
Requirement already satisfied: apache-bookkeeper-client in /usr/local/lib/python3.7/site-packages
(from pulsar-client) (4.9.1)
Requirement already satisfied: prometheus-client in /usr/local/lib/python3.7/site-packages/prometheus_client-0.5.0-py3.7.egg
(from pulsar-client) (0.5.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from
protobuf&gt;=3.6.1-&gt;pulsar-client) (40.8.0)
Requirement already satisfied: requests&lt;3.0.0dev,&gt;=2.18.0 in /usr/local/lib/python3.7/site-packages
(from apache-bookkeeper-client-&gt;pulsar-client) (2.21.0)
Requirement already satisfied: pytz in /usr/local/lib/python3.7/site-packages (from apache-bookkeeper-client-&gt;pulsar-client)
(2019.1)
Requirement already satisfied: mmh3&gt;=2.5.1 in /usr/local/lib/python3.7/site-packages
(from apache-bookkeeper-client-&gt;pulsar-client) (2.5.1)
Requirement already satisfied: idna&lt;2.9,&gt;=2.5 in /usr/local/lib/python3.7/site-packages
(from requests&lt;3.0.0dev,&gt;=2.18.0-&gt;apache-bookkeeper-client-&gt;pulsar-client)
(2.8)
Requirement already satisfied: chardet&lt;3.1.0,&gt;=3.0.2 in /usr/local/lib/python3.7/site-packages
(from requests&lt;3.0.0dev,&gt;=2.18.0-&gt;apache-bookkeeper-client-&gt;pulsar-client)
(3.0.4)
Requirement already satisfied: urllib3&lt;1.25,&gt;=1.21.1 in /usr/local/lib/python3.7/site-packages
(from requests&lt;3.0.0dev,&gt;=2.18.0-&gt;apache-bookkeeper-client-&gt;pulsar-client)
(1.24.1)
Requirement already satisfied: certifi&gt;=2017.4.17 in /usr/local/lib/python3.7/site-packages
(from requests&lt;3.0.0dev,&gt;=2.18.0-&gt;apache-bookkeeper-client-&gt;pulsar-client)
(2019.3.9)
Installing collected packages: pulsar-client
Successfully installed pulsar-client-2.3.1
[mmerli@pepito] ~ $ python3
Python 3.7.2 (default, Feb 12 2019, 08:15:36)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import pulsar
&gt;&gt;&gt;

```
----
2019-04-17 13:26:53 UTC - Laurent Chriqui: weird :
Looking in indexes: <https://pypi.org/simple>, <http://ec2-54-77-49-109.eu-west-1.compute.amazonaws.com:6543/pypi/>
Collecting pulsar-client
  Downloading <https://files.pythonhosted.org/packages/38/74/01281eedf5134fc26b9763611b364f90be36b2f3f6d601cf6e31ce988924/pulsar_client-2.3.1-cp37-cp37m-macosx_10_14_x86_64.whl>
(15.7MB)
    100% |████████████████████████████████|
15.7MB 25.8MB/s
Requirement already satisfied: apache-bookkeeper-client in ./venv/lib/python3.7/site-packages
(from pulsar-client) (4.9.1)
Requirement already satisfied: protobuf&gt;=3.6.1 in ./venv/lib/python3.7/site-packages
(from pulsar-client) (3.7.1)
Requirement already satisfied: ratelimit in ./venv/lib/python3.7/site-packages (from pulsar-client)
(2.2.1)
Requirement already satisfied: grpcio in ./venv/lib/python3.7/site-packages (from pulsar-client)
(1.20.0)
Requirement already satisfied: six in ./venv/lib/python3.7/site-packages (from pulsar-client)
(1.12.0)
Requirement already satisfied: prometheus-client in ./venv/lib/python3.7/site-packages (from
pulsar-client) (0.6.0)
Requirement already satisfied: fastavro in ./venv/lib/python3.7/site-packages (from pulsar-client)
(0.21.20)
Requirement already satisfied: mmh3&gt;=2.5.1 in ./venv/lib/python3.7/site-packages (from
apache-bookkeeper-client-&gt;pulsar-client) (2.5.1)
Requirement already satisfied: requests&lt;3.0.0dev,&gt;=2.18.0 in ./venv/lib/python3.7/site-packages
(from apache-bookkeeper-client-&gt;pulsar-client) (2.21.0)
Requirement already satisfied: pytz in ./venv/lib/python3.7/site-packages (from apache-bookkeeper-client-&gt;pulsar-client)
(2019.1)
Requirement already satisfied: setuptools&gt;=34.0.0 in ./venv/lib/python3.7/site-packages/setuptools-39.1.0-py3.7.egg
(from apache-bookkeeper-client-&gt;pulsar-client) (39.1.0)
Requirement already satisfied: chardet&lt;3.1.0,&gt;=3.0.2 in ./venv/lib/python3.7/site-packages
(from requests&lt;3.0.0dev,&gt;=2.18.0-&gt;apache-bookkeeper-client-&gt;pulsar-client)
(3.0.4)
Requirement already satisfied: certifi&gt;=2017.4.17 in ./venv/lib/python3.7/site-packages
(from requests&lt;3.0.0dev,&gt;=2.18.0-&gt;apache-bookkeeper-client-&gt;pulsar-client)
(2019.3.9)
Requirement already satisfied: idna&lt;2.9,&gt;=2.5 in ./venv/lib/python3.7/site-packages
(from requests&lt;3.0.0dev,&gt;=2.18.0-&gt;apache-bookkeeper-client-&gt;pulsar-client)
(2.8)
Requirement already satisfied: urllib3&lt;1.25,&gt;=1.21.1 in ./venv/lib/python3.7/site-packages
(from requests&lt;3.0.0dev,&gt;=2.18.0-&gt;apache-bookkeeper-client-&gt;pulsar-client)
(1.24.1)
Installing collected packages: pulsar-client
Successfully installed pulsar-client-2.3.1
You are using pip version 10.0.1, however version 19.0.3 is available.
You should consider upgrading via the ‘pip install --upgrade pip’ command.
(venv) MacBook-Pro-de-Laurent:python-instance laurentchriqui$ python
Python 3.7.2 (v3.7.2:9a3ffc0492, Dec 24 2018, 02:44:43)
[Clang 6.0 (clang-600.0.57)] on darwin
Type “help”, “copyright”, “credits” or “license” for more information.
&gt;&gt;&gt; import pulsar
----
2019-04-17 13:27:04 UTC - Laurent Chriqui: Segmentation fault: 11
----
2019-04-17 13:39:49 UTC - Matteo Merli: That looks weird :confused:
----
2019-04-17 13:41:04 UTC - Matteo Merli: Can you double check a couple of things.

```
ulimit -c unlimited 

python -v -c 'import pulsar'
```
----
2019-04-17 13:41:22 UTC - Matteo Merli: that should print from where it’s trying to load
it
----
2019-04-17 13:41:32 UTC - Matteo Merli: and leave a core file for the segfault
----
2019-04-17 13:42:57 UTC - Matteo Merli: then,
```
lldb `which python` -c /cores/core.XYZ
```

Should be able to load the core dump and `bt` should get a stack trace
----
2019-04-17 13:55:20 UTC - Alexandre DUVAL: @Matteo Merli who knows how to pass that? :stuck_out_tongue:
----
2019-04-17 14:33:12 UTC - Kai Levy: @Alexandre DUVAL not positive, but try passing it as `clientAuthenticationParameters:
"{\"tokenPublicKey\":\"&lt;path&gt;\"}"`
----
2019-04-17 14:35:18 UTC - Jianfeng Qiao: I start a broker by running 'bin/pulsar-daemon start
broker' on a bare-metal server and use the default configuration, I see the following messages
outputted on the screen
'starting broker, logging to /opt/broker/logs/pulsar-broker-dfv.log', but why I cannot find
the log file under logs?
----
2019-04-17 14:35:18 UTC - Alexandre DUVAL: @Kai Levy &lt;path&gt; should contains
file:// as prefix?
----
2019-04-17 14:36:27 UTC - Kai Levy: Most likely? Or you could also try pasting in the key
raw
----
2019-04-17 14:40:02 UTC - Alexandre DUVAL: @Kai Levy im sure to understand, but it's not clientauthparam,
it's the tokenpublickey for functions_worker
----
2019-04-17 14:45:50 UTC - Kai Levy: I believe that the functions worker just uses that one
parameter and passes that along: <https://github.com/apache/pulsar/blob/859c914a7a3f3ddb9f89c62eba3c048039bd852a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ThreadRuntimeFactory.java#L77>
----
2019-04-17 14:47:15 UTC - Laurent Chriqui: I get the following stack trace, I don’t understand
why it comes from a python2 file...
```Traceback (most recent call last):
  File "&lt;input&gt;", line 1, in &lt;module&gt;
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py",
line 52, in &lt;module&gt;
    import weakref
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/weakref.py",
line 14, in &lt;module&gt;
    from _weakref import (
ImportError: cannot import name _remove_dead_weakref````
----
2019-04-17 14:49:27 UTC - Alexandre DUVAL: Yes, but it's not the functions_worker as client
of pulsar problem, it's related to the auth system of the functions worker itself.
----
2019-04-17 14:49:45 UTC - Alexandre DUVAL: Something is missing on the configuration of it:
----
2019-04-17 14:49:48 UTC - Alexandre DUVAL: ```############################################
# security settings for worker service
############################################

# Enforce authentication
authenticationEnabled: true
# Enforce authorization on accessing functions api
authorizationEnabled: true
# Set of autentication provider name list, which is a list of class names
authenticationProviders:
  - org.apache.pulsar.broker.authentication.AuthenticationProviderToken
# Set of role names that are treated as "super-user", meaning they will be able to access
any admin-api
superUserRoles:
  - yolo```
----
2019-04-17 14:53:01 UTC - Kai Levy: That's functions_worker.yml?
----
2019-04-17 14:53:57 UTC - Kai Levy: Look at this: <https://github.com/apache/pulsar/blob/8b2929b3e5403fb44653976dc74be287666f7b96/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/AuthenticationConfig.java>
----
2019-04-17 14:57:24 UTC - Kai Levy: oh i think i understand what you're saying
----
2019-04-17 14:57:29 UTC - Kai Levy: nevermind
----
2019-04-17 14:58:11 UTC - Alexandre DUVAL: :slightly_smiling_face:
----
2019-04-17 15:33:57 UTC - Alexandre DUVAL: So anyone has an answer? :confused:
----
2019-04-17 15:36:42 UTC - Devin G. Bost: @Jerry Peng Any updates on this?
----
2019-04-17 15:39:52 UTC - Laurent Chriqui: I cleaned my python installations and it seems
to have fixed the problem, thanks for your help :slightly_smiling_face:
----
2019-04-17 15:54:36 UTC - David Kjerrumgaard: @Alexandre DUVAL The stacktrace indicates that
the issue is on the broker, `org.apache.pulsar.broker.PulsarServerException: Failed to load
an authentication provider.`.  Have you configured the brokers to authenticate clients?
----
2019-04-17 15:54:58 UTC - David Kjerrumgaard: by adding the following properties to the broker.conf?
----
2019-04-17 15:55:04 UTC - David Kjerrumgaard: 
----
2019-04-17 15:55:23 UTC - Alexandre DUVAL: yes, and i query the brokers with auth without
problems
----
2019-04-17 15:55:45 UTC - David Kjerrumgaard: which provider did you specify?
----
2019-04-17 15:56:10 UTC - Alexandre DUVAL: `authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken`
----
2019-04-17 15:56:39 UTC - David Kjerrumgaard: `Caused by: java.io.IOException: No secret key
was provided for token authentication
        at org.apache.pulsar.broker.authentication.AuthenticationProviderToken.getValidationKey(AuthenticationProviderToken.java:135)
~[org.apache.pulsar-pulsar-broker-common-2.3.1.jar:2.3.1]
        at `
----
2019-04-17 15:56:47 UTC - David Kjerrumgaard: did you specify the private key?
----
2019-04-17 15:57:19 UTC - David Kjerrumgaard: `tokenSecretKey=file:///path/to/secret.key`
----
2019-04-17 15:57:25 UTC - Alexandre DUVAL: no, the public
----
2019-04-17 15:57:37 UTC - David Kjerrumgaard: you need the above in your broker.conf
----
2019-04-17 15:57:47 UTC - Alexandre DUVAL: oh, ok
----
2019-04-17 16:02:09 UTC - Alexandre DUVAL: The private key and public key? Not the secret,
it's asym auth, no ?
----
2019-04-17 16:03:02 UTC - David Kjerrumgaard: <http://pulsar.apache.org/docs/en/security-token-admin/>
----
2019-04-17 16:04:11 UTC - David Kjerrumgaard: How did you create your key pair?
----
2019-04-17 17:46:11 UTC - Jerry Peng: sorry for the late reply
----
2019-04-17 17:46:48 UTC - Jerry Peng: You just have to specify in the jar field “builtin://&lt;name
of  connector&gt;”
----
2019-04-17 17:47:13 UTC - Jerry Peng: e.g. “<builtin://kafka>”
----
2019-04-17 18:03:03 UTC - Devin G. Bost: How do I specify the `sinkType` or `sourceType` (for
a kafka sink or source) when using the Java Admin API? I'm not finding any helpful unit tests
in the source code, and I don't see any applicable properties on the `SinkConfig` or `SourceConfig`
class objects that are expected by `adminClient.source().createSource(..)`, `adminClient.source().updateSource()`,
`adminClient.sink().createSink(..)`, and `adminClient.sink().updateSink(..)`.
I was able to locate the KafkaSourceConfig and KafkaSinkConfig classes, but they don't have
public properties like the SourceConfig and SinkConfig classes (such as for `tenant`, `namespace`,
etc.).
----
2019-04-17 18:03:53 UTC - Jerry Peng: @Devin G. Bost I replied in the thread we were discussing
on
----
2019-04-17 18:04:18 UTC - Devin G. Bost: Oh, sorry, I didn't see your response. Thanks for
getting back to me.
----
2019-04-17 18:08:16 UTC - Devin G. Bost: @Jerry Peng That's for the archive parameter?
----
2019-04-17 18:12:55 UTC - Devin G. Bost: It worked!!!
----
2019-04-17 18:53:46 UTC - Adam P.: is there a way to watch pulsar logs for functions being
created?  I'm trying to create a python function but keep getting this error, unless i use
the `localrun` mode
`Reason: javax.ws.rs.ProcessingException: Java heap space`
I'm not able to see anything go through the `logs/pulsar.log` or `logs/pulsar-standalone.log`
----
2019-04-17 18:54:39 UTC - Ali Ahmed: is that from the pulasr client ?
----
2019-04-17 18:54:52 UTC - Devin G. Bost: @Jerry Peng When I run update on the kafka sinks
and sources, I get this 500 error:

```
org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException: HTTP 500 Internal
Server Error
	at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:163)
	at org.apache.pulsar.client.admin.internal.SinkImpl.updateSink(SinkImpl.java:172)
	at deployment.manifest.Sink.upsert(Sink.java:88)
	at deployment.manifest.Topology.lambda$upsertComponents$2(Topology.java:73)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at deployment.manifest.Topology.upsertComponents(Topology.java:73)
	at deployment.test.DeploymentTest.topology_components_upsertComponents(DeploymentTest.java:118)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: org.apache.pulsar.shade.javax.ws.rs.InternalServerErrorException: HTTP 500 Internal
Server Error
	at org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1098)
	at org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:883)
	at org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:767)
	at org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:316)
	at org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:298)
	at org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:229)
	at org.apache.pulsar.shade.org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:414)
	at org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:765)
	at org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:456)
	at org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:340)
	at org.apache.pulsar.client.admin.internal.SinkImpl.updateSink(SinkImpl.java:170)
	... 27 more```
----
2019-04-17 18:55:26 UTC - Adam P.: it's the `bin/pulsar-admin functions create ...` command.
----
2019-04-17 18:55:47 UTC - Ali Ahmed: you need to increase the heap size for the pulsar-admin
----
2019-04-17 18:57:00 UTC - Adam P.: so, maybe i'm doing that wrong but when i run this from
docker-compose, i set these env vars to try to do that
```
- PULSAR_EXTRA_OPTS=-XX:+UnlockExperimentalVMOptions
- PULSAR_MEM=-Xms2g -Xmx4g -XX:MaxDirectMemorySize=6g```
am i missing something?  i don't think i should be running out of space with those numbers.
 my python package is 46M
----
2019-04-17 18:57:35 UTC - Thor Sigurjonsson: @Ali Ahmed Should that be in conf/pulsar_tools_env.sh
?
----
2019-04-17 18:57:42 UTC - Ali Ahmed: yes
----
2019-04-17 18:58:20 UTC - Thor Sigurjonsson: @Adam Williams Might have been doing that in
conf/pulsar_env.sh instead of the tools one.
----
2019-04-17 18:59:44 UTC - Thor Sigurjonsson: Quick question: Is there a way to change parallelism
on a function once it's deployed without doing an update with the `--jar` upload and all --
full redeploy.
----
2019-04-17 19:00:24 UTC - Matteo Merli: Yes, if you use the update command, you only specify
what to overwrite
----
2019-04-17 19:01:18 UTC - Thor Sigurjonsson: @Matteo Merli Very good, thanks!
----
2019-04-17 19:20:17 UTC - Jerry Peng: are there any errors in the broker logs?
----
2019-04-17 19:21:10 UTC - Devin G. Bost: Good question. I'll check.
----
2019-04-17 19:22:11 UTC - Devin G. Bost: If I'm running standalone, will the broker logs output
to the console?
----
2019-04-17 19:22:56 UTC - Jerry Peng: yes
----
2019-04-17 19:23:12 UTC - Devin G. Bost: I'll send you the output. There are definitely errors.
----
2019-04-17 19:24:13 UTC - Devin G. Bost: 
----
2019-04-17 19:25:56 UTC - Jerry Peng: taking a look
----
2019-04-17 19:26:08 UTC - Devin G. Bost: Thanks!
----
2019-04-17 19:34:16 UTC - Devin G. Bost: I'm wondering if it could be blowing up at the `.
. . ` in the `sasl.jaas.config` config.
----
2019-04-17 19:36:28 UTC - Kai Levy: Hey @Alexandre DUVAL I dug a little more and it looks
like the WorkerConfig never has it's properties set. So when the auth plugin calls `getProperty`
then it won't turn up anything.
----
2019-04-17 19:39:42 UTC - Devin G. Bost: That's not it. I tried passing in a valid `sasl.jaas.config`
string, and I still got the same error.
----
2019-04-17 19:39:58 UTC - Devin G. Bost: I only get the error with the kafka sinks and sources.
----
2019-04-17 19:41:39 UTC - Jerry Peng: ya it does seem there is some thing wrong when trying
to deserialize the config.  Let me investigate further
+1 : Devin G. Bost
----
2019-04-17 19:59:30 UTC - Devin G. Bost: I'm passing the configs as `Map&lt;String, Object&gt;`.
For the log file that I sent you, here's what the `sinkConfig` values look like.
----
2019-04-17 20:00:47 UTC - Devin G. Bost: I tried using this as the sasl.jaas.config value
instead, but I still got the same errors:

`"com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true useTicketCache=false
serviceName=\"kafka\" principal=\"pulsar_runtime@obfuscated.COM\" useKeyTab=true keyTab=\"/pulsar/conf/auth/pulsar_runtime_test.keytab\"
client=true;"`
----
2019-04-17 20:02:54 UTC - Jerry Peng: ok I can reproduce this error as well
----
2019-04-17 20:03:03 UTC - Jerry Peng: let me see what is wrong
----
2019-04-17 20:03:19 UTC - Jerry Peng: I think it has to do with nested jsons because of the
config
----
2019-04-17 20:03:30 UTC - Jerry Peng: but only happens on update
----
2019-04-17 20:04:08 UTC - Devin G. Bost: The fact that it only happens on update is the thing
that is most interesting to me.
----
2019-04-17 20:07:09 UTC - Jerry Peng: The error happens when trying to deserialize the config
json string that is stored internally via a protobuf object.
----
2019-04-17 20:07:56 UTC - Jerry Peng: It happens on update because we retrieve the old metadata
for the function and merge it with the new one the user has submitted
----
2019-04-17 20:08:21 UTC - Devin G. Bost: Oh, good to know.
----
2019-04-17 20:10:53 UTC - Adam Williams: I'm not sure I understand. Could you provide more
context?
----
2019-04-17 20:14:37 UTC - Jerry Peng: i know the issue.  I will have a fix up soon for this
issue
----
2019-04-17 20:20:54 UTC - Devin G. Bost: Thanks!!
----
2019-04-17 21:46:19 UTC - Jerry Peng: <https://github.com/apache/pulsar/pull/4067>
----
2019-04-17 21:46:21 UTC - Jerry Peng: is the fix
----
2019-04-17 22:21:28 UTC - Alexandre DUVAL: Okay
----
2019-04-17 23:09:25 UTC - Matteo Merli: I think the wrong handle was tagged :)
----
2019-04-18 08:30:04 UTC - Sébastien de Melo: Hello, we have a function that sometimes stops
working and cannot even be deleted. The other subcommands (get, status, stats) respond though.
 Do you know why it happens and what we can do to delete it?
----
2019-04-18 09:09:44 UTC - Alexandre DUVAL: Hi, if a Pulsar fonctions push message to inexistent
tenant/namespace/topic is it working? Or i have to create it before ?
----
2019-04-18 09:10:41 UTC - Ali Ahmed: topics can be created on demand tenant and namespace
have to exist before hand
----
Mime
View raw message