pulsar-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Pulsar Slack" <apache.pulsar.sl...@gmail.com>
Subject Slack digest for #general - 2018-12-20
Date Thu, 20 Dec 2018 09:11:02 GMT
2018-12-19 11:58:37 UTC - Samuel Sun: Hi, one question for schema , currently golang client
support it or not ? thanks
----
2018-12-19 11:59:34 UTC - Sijie Guo: it is not supported yet. we are targeting at supporting
it for 2.3.0 release.
----
2018-12-19 12:09:50 UTC - Ganga Lakshmanasamy: We would like to read data from a file and
stream it into ES. The file can be from any remote server with proper access being given.
Do we have any plugin already available to handle such condition?
----
2018-12-19 12:14:23 UTC - Samuel Sun: thanks
----
2018-12-19 12:17:56 UTC - Samuel Sun: another question, is it possible to get into the zk
while using standalone mode ? Thanks : )
----
2018-12-19 12:21:20 UTC - Samuel Sun: seems like the result is empty:
----
2018-12-19 12:21:27 UTC - Samuel Sun: ./pulsar-managed-ledger-admin print-managed-ledger --zkServer
127.0.0.1  --managedLedgerPath /
----
2018-12-19 12:21:45 UTC - Sijie Guo: bin/pulsar zookeeper-shell -server localhost:2181
----
2018-12-19 12:22:07 UTC - Sijie Guo: you can use `bin/pulsar zookeeper-shell`
----
2018-12-19 12:24:02 UTC - Samuel Sun: nice, but managed-ledger-admin not work ?
----
2018-12-19 12:40:19 UTC - Sijie Guo: managed-ledger-admin also works
----
2018-12-19 12:42:01 UTC - Samuel Sun: managedLedgerPath should be ?
----
2018-12-19 12:42:21 UTC - Samuel Sun: ./bin/pulsar-managed-ledger-admin print-managed-ledger
--zkServer localhost:2181 --managedLedgerPath /   no results
----
2018-12-19 12:50:05 UTC - Sijie Guo: managed ledger path should be `&lt;tenant&gt;/&lt;namespace&gt;/[persistent|non-persistent]/&lt;topic&gt;`
----
2018-12-19 12:50:16 UTC - Sijie Guo: e.g. ’public/default/persistent/test-topic`
----
2018-12-19 12:51:41 UTC - Samuel Sun: got it, it works, thanks.
----
2018-12-19 12:54:30 UTC - Sijie Guo: :+1:
----
2018-12-19 13:57:08 UTC - Chris Miller: I'm curious to know the reason why message TTL, retention
and backlog settings are applied at the namespace level. To me it seems more useful to have
them at the topic level (since some topics I might want to keep forever, some are transient,
some buffer for an hour etc). As it stands it seems I need to create a namespace for each
specific set of policies, and there's no obvious way to change the settings for a topic other
than move it to another namespace. Am I thinking about this the wrong way?
----
2018-12-19 13:58:14 UTC - Sijie Guo: @Chris Miller there is an outstanding task for adding
policies at topic-level
----
2018-12-19 13:58:31 UTC - Chris Miller: Ah ok thanks, I guess it's not just me then
----
2018-12-19 13:59:19 UTC - Chris Miller: Found it, thanks for the pointer: <https://github.com/apache/pulsar/issues/2688>
----
2018-12-19 14:00:16 UTC - Sijie Guo: you are welcome
----
2018-12-19 14:45:02 UTC - Ryan Samo: @Sijie Guo no problem at all. I am currently out on holiday
until next year but when I return to work I will try to put together a post. Thanks!
----
2018-12-19 16:30:04 UTC - David Kjerrumgaard: @Victor Li Try both of these NAR files
----
2018-12-19 16:30:45 UTC - David Kjerrumgaard: 
----
2018-12-19 16:31:19 UTC - David Kjerrumgaard: 
----
2018-12-19 16:49:06 UTC - Ganga Lakshmanasamy: @Ali Ahmed Is there any unique constraint checks
similar to sql unique available for pulsar topics? We basically want to avoid duplicate data
being put up into topics from 2 different threads. Is that configurable?
----
2018-12-19 17:28:32 UTC - Mike Card: Hey everyone- I am getting some unusual messages in my
log when I try to create partitioned topics in my code using the Pulsar admin client. My code
that creates the topics looks like this:

```
                try {admin.topics().createPartitionedTopic(FULL_MASTER_QUEUE_TOPIC_NAME, NUM_PARTITIONS);
} catch (Throwable t) { _log.error("DefaultDatabus.constructor: caught exception " + t.getClass().getName()
+ " creating " + FULL_MASTER_QUEUE_TOPIC_NAME + ", message = " + t.getMessage());}
                try {admin.topics().createPartitionedTopic(FULL_RESOLVER_QUEUE_TOPIC_NAME,
NUM_PARTITIONS); } catch (Throwable t) { _log.error("DefaultDatabus.constructor: caught exception
" + t.getClass().getName() + " creating " + FULL_RESOLVER_QUEUE_TOPIC_NAME + ", message =
" + t.getMessage());}
                try {admin.topics().createPartitionedTopic(FULL_RESOLVER_RETRY_QUEUE_TOPIC_NAME,
NUM_PARTITIONS); } catch (Throwable t) { _log.error("DefaultDatabus.constructor: caught exception
" + t.getClass().getName() + " creating " + FULL_RESOLVER_RETRY_QUEUE_TOPIC_NAME + ", message
= " + t.getMessage());}
```
----
2018-12-19 17:29:17 UTC - Mike Card: The error messages I am getting look like this:

```
WARN  [2018-12-19 16:21:41,643] org.apache.pulsar.client.admin.internal.BaseResource: [<http://emodb-pulsar-databus-pulsar-cluster-1:8080/admin/v2/persistent/public/default/master-queue/partitions>]
Failed to perform http put request: java.net.SocketException: Unexpected end of file from
server
ERROR [2018-12-19 16:21:41,643] com.bazaarvoice.emodb.databus.core.DefaultDatabus: DefaultDatabus.constructor:
caught exception org.apache.pulsar.client.admin.PulsarAdminException creating public/default/master-queue,
message = java.net.SocketException: Unexpected end of file from server
WARN  [2018-12-19 16:21:41,649] org.apache.pulsar.client.admin.internal.BaseResource: [<http://emodb-pulsar-databus-pulsar-cluster-1:8080/admin/v2/persistent/public/default/resolver-queue/partitions>]
Failed to perform http put request: java.net.SocketException: Unexpected end of file from
server
ERROR [2018-12-19 16:21:41,649] com.bazaarvoice.emodb.databus.core.DefaultDatabus: DefaultDatabus.constructor:
caught exception org.apache.pulsar.client.admin.PulsarAdminException creating public/default/resolver-queue,
message = java.net.SocketException: Unexpected end of file from server
WARN  [2018-12-19 16:21:41,654] org.apache.pulsar.client.admin.internal.BaseResource: [<http://emodb-pulsar-databus-pulsar-cluster-1:8080/admin/v2/persistent/public/default/resolver-retry-queue/partitions>]
Failed to perform http put request: java.net.SocketException: Unexpected end of file from
server
ERROR [2018-12-19 16:21:41,654] com.bazaarvoice.emodb.databus.core.DefaultDatabus: DefaultDatabus.constructor:
caught exception org.apache.pulsar.client.admin.PulsarAdminException creating public/default/resolver-retry-queue,
message = java.net.SocketException: Unexpected end of file from server
```
----
2018-12-19 17:29:34 UTC - Mike Card: Is there something I need to check in my config?
----
2018-12-19 17:44:14 UTC - Mike Card: @David Kjerrumgaard or @Matteo Merli have either of you
guys ever seen this error using the admin client? What should I be looking for to resolve
it?
----
2018-12-19 17:50:45 UTC - Sanjeev Kulkarni: @Ganga Lakshmanasamy Pulsar does not support sql
unique type feature. Because, at its heart, Pulsar is a pub-sub/log layer and not really a
database. Thus apart from scheme enforcement, very few additional checks are done during the
time of publishing. This results in writes being blazing fast, but sql unique type checks
aren’t done
----
2018-12-19 17:55:48 UTC - Sanjeev Kulkarni: @Mike Card This looks like something wrong with
the connectivity. Are you sure Pulsar is reachable from the machine where your client runs?
----
2018-12-19 17:56:40 UTC - Mike Card: Yes, all of the other pulsar activity conducted via the
Pulsar client API (like sending and receiving messages) works fine
----
2018-12-19 17:57:04 UTC - Mike Card: The only issues are these calls from the admin client
----
2018-12-19 17:57:27 UTC - Mike Card: I am wondering if my problem is that the admin client
doesn't work well going through a load balancer
----
2018-12-19 17:57:48 UTC - Mike Card: I have a listener for port 8080 but maybe that's not
the way to do the admin client
----
2018-12-19 18:01:32 UTC - Mike Card: The Pulsar client calls seem to work OK being sent through
an ELB, I have listeners for both 6650 and 8080
----
2018-12-19 18:29:53 UTC - Mike Card: @Sanjeev Kulkarni @David Kjerrumgaard @Matteo Merli Hey
guys I figured it out, it is that the Admin Java client does not like getting a URL that talks
to a ELB as opposed to directly to a broker node. Previously, the URL I had been using to
build the admin client was like this:

<http://emodb-pulsar-databus-pulsar-cluster-1:8080>

where emodb-pulsar-databus-pulsar-cluster-1 is a Route53 alias that points to my ELB, which
has a listener for port 8080. When I do it this way I get the unexpected end-of-file exception
on my calls to create partitioned topics per my earlier message.

However, if I change the URL to instead be the IP address of one of the broker nodes like
this:

<http://10.110.78.37:8080>

then it works fine and the topics get created as expected.

This strikes me as a bug, the regular pulsar client can talk via an ELB but the admin client
can't?
----
2018-12-19 19:08:10 UTC - Sanjeev Kulkarni: @Mike Card indeed. Could you please file an issue
on github?
----
2018-12-19 19:26:09 UTC - Mike Card: @Sanjeev Kulkarni OK opened #3220 <https://github.com/apache/pulsar/issues/3220>
+1 : David Kjerrumgaard, Ali Ahmed
----
2018-12-19 21:39:28 UTC - Irfan: Hi all. I'm trying to run a simple tiered storage example
on my local where I connect the standalone broker to a S3 bucket. However when I start up
the standalone broker I get the following exception: `java.io.IOException: No offloader found
for driver 'S3'. Please make sure you dropped the offloader nar packages under ${PULSAR_HOME}/offloaders.`
What do I need to do fix this?
----
2018-12-20 00:11:18 UTC - Sijie Guo: That would be awesome!
----
2018-12-20 06:00:48 UTC - Ganga Lakshmanasamy: ok Thanks
----
2018-12-20 06:01:38 UTC - Ganga Lakshmanasamy: @Sanjeev Kulkarni Do you have any inputs for
this query?
----
2018-12-20 06:57:19 UTC - Roger Yates: Hi all.
*What?* I'm trying to publish from a local pulsar client to a partitioned topic in the cloud,
but my client machine sits behind a corporate proxy.
*What's the symptom?* I get a failure to connect with `org.apache.pulsar.client.impl.ConnectionPool
- Failed to open connection to `... - the request isn't going via the corporate proxy, but
it needs to.
*What have you tried?* I've tried various methods of configuring a proxy (JVM ARGS, system
properties, for `socksProxyHost/Port` &amp; `http.ProxyHost/Port` &amp; tried creating
a `ProxySelector` and instantiating on the client, which initializes but never gets its select()
method called by the `Producer/PulsarClient`) , but none seem to have any effect.
I know the connectivity is there (firewalls have been burnt), as I have installed Proxifier
to intercept and route to the proxy, and the messages were sent successfully; however, I can't
install Proxifier on our servers so I need another way.
I haven't seen anything in the documentation to be able to specify a proxy on a  ClientBuilder
or Producer but I imagine this is a relatively common scenario in a corporate environment.
*Pulsar version:* 2.2.0
*So what's the question?*
Has anyone else been able to configure Pulsar to publish to an externally hosted topic from
behind a corporate proxy?
If so, what did you need to do? Can you point me the right direction?
Thanks in advance.
----
Mime
View raw message