pulsar-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Pulsar Slack" <apache.pulsar.sl...@gmail.com>
Subject Slack digest for #general - 2019-01-31
Date Thu, 31 Jan 2019 09:11:04 GMT
2019-01-30 16:01:29 UTC - naga: Any billing app to plugin on pulsar?
----
2019-01-30 16:12:59 UTC - David Kjerrumgaard: @r Are you referring to something that meters
the usage of Pulsar and creates a bill for an end user?
----
2019-01-30 16:16:08 UTC - naga: Yes @David Kjerrumgaard 
----
2019-01-30 16:48:45 UTC - David Kjerrumgaard: I am unaware of any such application at the
moment, but are always open to feature requests and/or contributions. :smiley:
----
2019-01-30 17:04:29 UTC - Joe Francis: @naga Pulsar provides usage metrics, but you will have
to aggregate them. How do you plan to bill?  Pulsar has  knobs for  provisioning. You  can
set   quotas for  storage and dispatch rates.
----
2019-01-30 17:26:22 UTC - naga: @David Kjerrumgaard and @Joe Francis thank you :grinning:

----
2019-01-30 18:27:04 UTC - Jon Bock: @naga Are you looking to offer Pulsar as a managed service,
or looking to build a tool to help assess cost of running Pulsar (e.g. for internal chargeback
or other)?
----
2019-01-30 19:56:45 UTC - Arun M. Krishnakumar: @Arun M. Krishnakumar has joined the channel
----
2019-01-30 21:11:20 UTC - Narciso Guillen de la Mora: @Narciso Guillen de la Mora has joined
the channel
----
2019-01-30 21:46:25 UTC - Ivan Serdyuk: @Ivan Serdyuk has joined the channel
----
2019-01-30 23:28:02 UTC - naga: @Jon Bock yes you are right. Trying to run as managed service
----
2019-01-30 23:59:27 UTC - Grant Wu: Hey, @Matteo Merli this happened again
----
2019-01-30 23:59:38 UTC - Grant Wu: Not 100% sure but I’m starting to think it might be
reproducible; this time it occurred when I deployed and shut down my company’s product in
a loop
----
2019-01-30 23:59:44 UTC - Grant Wu: Any thoughts on what could possibly be causing this?
----
2019-01-31 00:01:29 UTC - Matteo Merli: I never saw this :confused:
----
2019-01-31 00:01:43 UTC - Matteo Merli: any hint on how to reproduce?
----
2019-01-31 00:02:00 UTC - Grant Wu: uh….
----
2019-01-31 00:02:26 UTC - Grant Wu: Well, I can’t exactly send you our deploy scripts and
they wouldn’t be useful anyways
----
2019-01-31 00:03:12 UTC - Grant Wu: Let me see if I can reliably recreate this internally
first tomorrow
----
2019-01-31 00:03:16 UTC - Matteo Merli: Sure, can you describe when it happens? Or any config
that’s different from defaults, or any info that might help
----
2019-01-31 00:06:10 UTC - Grant Wu: I’m not an expert on how we deploy Pulsar
----
2019-01-31 00:07:16 UTC - Grant Wu: one thing that is strange is that we run three brokers
----
2019-01-31 00:07:19 UTC - Grant Wu: And that only 2 of them are dying
----
2019-01-31 00:07:58 UTC - Matteo Merli: It appears the issue is when recovering the topic
used for functions metadata and assignemntes
----
2019-01-31 00:08:09 UTC - Grant Wu: ah
----
2019-01-31 00:08:14 UTC - Grant Wu: okay, so it is related to Pulsar Functions… hrm
----
2019-01-31 00:08:22 UTC - Matteo Merli: `public/functions/persistent/coordinate`
----
2019-01-31 00:09:32 UTC - Grant Wu: well our deploys do have automation around creating and
deleting Pulsar functions
----
2019-01-31 00:09:52 UTC - Grant Wu: ```
put_pfs()
{
    find . -maxdepth 2 -name 'function-config.yaml' | while read -r i; do
        d=$(dirname "$i")
        pushd "$d" || exit 1
        sed -i "s/PULSAR_TENANT/${PULSAR_TENANT}/g" function-config.yaml
        sed -i "s/K8S_NAMESPACE/${K8S_NAMESPACE}/g" function-config.yaml

        pf_name=$(grep "^name:" function-config.yaml | awk -F": \"" '{print $2}' | sed 's/.$//')

        # Unfortunately, Pulsar uses the same exit code for both a missing function and a
connection error
        check_for_function=$($PULSAR_DIR/bin/pulsar-admin functions get --namespace "$K8S_NAMESPACE"
--tenant "$PULSAR_TENANT" --name "$pf_name" 2&gt;&amp;1)
        if printf "%s" "$check_for_function" | grep -q "HTTP 404 Not Found"; then
            echo "Creating pulsar function $pf_name"
            cmd="create"
        elif printf "%s" "$check_for_function" | grep -q "java.net.ConnectException"; then
            echo "Failed to connect to Pulsar"
            exit 2
        else
            echo "Updating pulsar function $pf_name"
            cmd="update"
        fi

        if ! $PULSAR_DIR/bin/pulsar-admin functions $cmd --functionConfigFile "$(realpath
function-config.yaml)"; then
            echo "Failed to put $pf_name"
            exit 3
        fi
        popd || exit 4
    done
}

delete_pfs()
{
    $PULSAR_DIR/bin/pulsar-admin functions list --namespace "$K8S_NAMESPACE" --tenant "$PULSAR_TENANT"
| while read -r pf_name; do
        echo "Deleting pulsar function $pf_name"
        $PULSAR_DIR/bin/pulsar-admin functions delete --namespace "$K8S_NAMESPACE" --tenant
"$PULSAR_TENANT" --name "$pf_name"
    done
}
```
----
2019-01-31 00:09:55 UTC - Grant Wu: this is what it looks like…
----
2019-01-31 00:10:25 UTC - Grant Wu: I did have a typo in this script that was breaking the
stopping part
----
2019-01-31 00:10:34 UTC - Matteo Merli: Actually, even though we don’t know how it got there,
we can add additional check to prevent failing to recover that topic
----
2019-01-31 00:10:50 UTC - Grant Wu: Is there any instrumentation we can add on our end to
help figure this out?
----
2019-01-31 00:13:52 UTC - Grant Wu: Just so you know <https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1547154563359500>
is from our company as well
----
2019-01-31 00:14:06 UTC - Grant Wu: We were using 2.1.1 then, but we’ve upgraded to 2.2.1
now
----
2019-01-31 00:29:41 UTC - Matteo Merli: The fix to avoid the bad failure is straighforward.
I’m trying to have unit test working to cover that .
----
2019-01-31 00:55:23 UTC - Matteo Merli: @Grant Wu <https://github.com/apache/pulsar/pull/3487>
----
2019-01-31 00:56:36 UTC - Ambud Sharma: are there other policies we can configure: ```--policy
producer_request_hold```  for `set-backlog-quota` ?
----
2019-01-31 00:57:19 UTC - Emma Pollum: @Ambud Sharma <http://pulsar.apache.org/docs/en/cookbooks-retention-expiry.html#backlog-quotas>
----
2019-01-31 00:57:22 UTC - Matteo Merli: <http://pulsar.apache.org/docs/en/admin-api-namespaces.html#set-backlog-quota-policies>
----
2019-01-31 00:57:23 UTC - Matteo Merli: :slightly_smiling_face:
----
2019-01-31 00:58:29 UTC - Ambud Sharma: thanks @Matteo Merli should we add a reference here:
<https://pulsar.apache.org/docs/en/cookbooks-retention-expiry/#pulsar-admin-2> ?
----
2019-01-31 01:00:23 UTC - Ambud Sharma: thanks @Emma Pollum
----
2019-01-31 03:08:03 UTC - bossbaby: my Bookie and my broker connect 2 zookeeper, I tried to
turn off any zookeeper and the result was a broker to stop. So the question is: Will zookeeper
never die to ensure the system works normally?
----
2019-01-31 04:21:35 UTC - David Kjerrumgaard: @bossbaby Zookeeper is just another java process,
so there is no way to guarantee that it never dies. However, if you configure a quorum of
3 zookeeper nodes, you can survive up to 2 ZK failures without any issue.  If you are properly
monitoring your environment, you should have sufficient time to restart or replace the failed
ZK nodes
+1 : bossbaby
----
2019-01-31 04:27:35 UTC - Ali Ahmed: @bossbaby there is no such nothing as a system not dying
you have increase the cluster size of zookeeper to have acceptable uptime for you application,
important production environments can ZK clusters to have 5 -9 nodes to satisfy quorum requirements
and to survive mutli rack / multi AZ failures.
+1 : bossbaby
----
2019-01-31 04:28:34 UTC - Matteo Merli: Do you have 2 ZK servers in your cluster?
----
2019-01-31 04:29:41 UTC - Matteo Merli: ZK is based on quorum, so you’d have to deploy an
odd number of servers
----
2019-01-31 04:29:56 UTC - Matteo Merli: 1, 3 or 5
----
2019-01-31 04:30:28 UTC - Matteo Merli: Deploying 2 servers gives you worse availability than
having 1 single node
----
2019-01-31 04:30:50 UTC - Matteo Merli: Any one node failure and the system is unavailable

----
2019-01-31 04:35:29 UTC - bossbaby: thanks you, i understand it
----
2019-01-31 04:37:39 UTC - bossbaby: so, if i have 3 zk, i can stop 1 zk or 5 zk and stop 2
zk
----
2019-01-31 04:37:42 UTC - bossbaby: it right?
----
2019-01-31 04:39:28 UTC - Matteo Merli: Correct
----
2019-01-31 04:41:18 UTC - bossbaby: If I have 5zk, can I stop 3 zk?
----
2019-01-31 04:41:36 UTC - Matteo Merli: Only 2 
----
2019-01-31 04:41:51 UTC - Matteo Merli: You need a majority of them available 
----
2019-01-31 04:42:18 UTC - Matteo Merli: You can deploy 7 and then you can afford to lose 3
----
2019-01-31 04:42:57 UTC - bossbaby: thanks you @Matteo Merli
----
2019-01-31 04:48:55 UTC - bossbaby: There was also a problem I encountered when setting up
geo-replication, when a new cluster involved in the clusters was available. How can consumers
in this new cluster read all messages sent previously from topic-global?
----
2019-01-31 04:49:50 UTC - Matteo Merli: The data is geo-replicated only from the point where
you enable it
----
2019-01-31 04:50:45 UTC - Matteo Merli: You can always rollback a replicator cursor, in the
same same way you can reset a subscription to an earlier point in time
----
2019-01-31 04:51:42 UTC - Matteo Merli: The name of the replicator “subscription” would
be `pulsar.repl.REGION`
----
2019-01-31 05:00:02 UTC - bossbaby: sorry @Matteo Merli, how to rollback a replicator cursor,
I did not find documentation about it
----
2019-01-31 05:06:03 UTC - Matteo Merli: eg: if you have 2 pulsar clusters (`west` and `east`),
if you want to replicate all the old data from `west` -&gt; `east`, you would have to
do:

In region `west`:
```
bin/pulsar-admin topics reset-cursor $MY_TOPIC -s pulsar.repl.east --time 24h
```

To roll it back by 24 hours
clock12 : Ali Ahmed
----
2019-01-31 05:10:47 UTC - bossbaby: Thanks you very much
+1 : Matteo Merli
----
2019-01-31 05:18:01 UTC - bossbaby: Why need a majority of them available.
My answer is that if the number of zookeeper is less than the number of dead, no one will
vote as the leader so the system will not work.
is it right?
----
2019-01-31 05:19:02 UTC - Matteo Merli: knowing whether a node is dead or just unreachable
(or slow) is actually an impossible problem to solve :slightly_smiling_face:
+1 : bossbaby
----
2019-01-31 06:19:05 UTC - bossbaby: clusters
```
["region_1","region_2","region_3","region_4"]
```
Error occur when run
```
bin/pulsar-admin topics reset-cursor <persistent://public/default/tx> -s pulsar.repl.region_4
--time 24h
```
```
Reason: Subscription not found
```
----
2019-01-31 06:29:58 UTC - Matteo Merli: Which region are you executing this command ?
----
2019-01-31 06:30:19 UTC - Matteo Merli: Also, which region is this topic being replicated
into?
----
2019-01-31 06:31:27 UTC - Matteo Merli: `pulsar-admin namespaces get-clusters public/default`
----
2019-01-31 06:34:52 UTC - bossbaby: topic being replicated in :
region_3
region_4
region_1
region_2
----
2019-01-31 06:35:12 UTC - bossbaby: i executing command in broker in region_1
----
2019-01-31 06:39:11 UTC - Vincent Ngan: I cannot find anywhere in the documentation describing
how Pulsar protects data loss. Supposing I have a multi-node cluster with 3 bookies, how many
bookie nodes can fail without causing loss of data?
----
2019-01-31 06:40:52 UTC - bossbaby: @Vincent Ngan <https://jack-vanlightly.com/blog/2018/10/21/how-to-not-lose-messages-on-an-apache-pulsar-cluster>
----
2019-01-31 06:45:27 UTC - Matteo Merli: it depends on the replication factor (eg: how many
guaranteed copy of the data you have)
----
2019-01-31 06:46:04 UTC - Vincent Ngan: Where is this configured?
----
2019-01-31 06:46:21 UTC - Matteo Merli: that’s configurable per each namespace. Defaults
per brokers are set in `broker.conf` :

```
# Number of bookies to use when creating a ledger
managedLedgerDefaultEnsembleSize=2

# Number of copies to store for each message
managedLedgerDefaultWriteQuorum=2

# Number of guaranteed copies (acks to wait before write is complete)
managedLedgerDefaultAckQuorum=2
```
----
2019-01-31 06:47:03 UTC - Matteo Merli: To be able to write, you need to have at least `EnsembleSize`
bookies available
----
2019-01-31 06:47:45 UTC - Matteo Merli: to not lose data, you can lose up to `AckQuorum -1`
bookies
----
2019-01-31 06:48:51 UTC - Matteo Merli: can you show internal stats for topic in region_1
?
----
2019-01-31 06:49:09 UTC - Matteo Merli: `pulsar-admin topics internal-stats  <persistent://public/default/tx>`
----
2019-01-31 06:57:46 UTC - bossbaby: intarnal-stats was wrong
i was run stats and result
```
{
  "msgRateIn" : 0.0,
  "msgThroughputIn" : 0.0,
  "msgRateOut" : 0.0,
  "msgThroughputOut" : 0.0,
  "averageMsgSize" : 0.0,
  "storageSize" : 1616,
  "publishers" : [ ],
  "subscriptions" : {
    "consumer-test-2" : {
      "msgRateOut" : 0.0,
      "msgThroughputOut" : 0.0,
      "msgRateRedeliver" : 0.0,
      "msgBacklog" : 31,
      "blockedSubscriptionOnUnackedMsgs" : false,
      "unackedMessages" : 0,
      "type" : "Exclusive",
      "msgRateExpired" : 0.0,
      "consumers" : [ ]
    },
    "consumer-test-1" : {
      "msgRateOut" : 0.0,
      "msgThroughputOut" : 0.0,
      "msgRateRedeliver" : 0.0,
      "msgBacklog" : 31,
      "blockedSubscriptionOnUnackedMsgs" : false,
      "unackedMessages" : 0,
      "msgRateExpired" : 0.0,
      "consumers" : [ ]
    }
  },
  "replication" : {
    "region_2" : {
      "msgRateIn" : 0.0,
      "msgThroughputIn" : 0.0,
      "msgRateOut" : 0.0,
      "msgThroughputOut" : 0.0,
      "msgRateExpired" : 0.0,
      "replicationBacklog" : 0,
      "connected" : true,
      "replicationDelayInSeconds" : 0,
      "inboundConnection" : "/192.168.1.133:54498",
      "inboundConnectedSince" : "2019-01-31T12:19:44.47+07:00",
      "outboundConnection" : "[id: 0x68d6d230, L:/192.168.1.133:54505 - R:pros-mbp/192.168.1.133:6652]",
      "outboundConnectedSince" : "2019-01-31T12:19:44.529+07:00"
    },
    "region_3" : {
      "msgRateIn" : 0.0,
      "msgThroughputIn" : 0.0,
      "msgRateOut" : 0.0,
      "msgThroughputOut" : 0.0,
      "msgRateExpired" : 0.0,
      "replicationBacklog" : 0,
      "connected" : true,
      "replicationDelayInSeconds" : 0,
      "outboundConnection" : "[id: 0x27d7afed, L:/192.168.1.133:54506 - R:pros-mbp/192.168.1.133:6654]",
      "outboundConnectedSince" : "2019-01-31T12:19:44.587+07:00"
    },
    "region_4" : {
      "msgRateIn" : 0.0,
      "msgThroughputIn" : 0.0,
      "msgRateOut" : 0.0,
      "msgThroughputOut" : 0.0,
      "msgRateExpired" : 0.0,
      "replicationBacklog" : 0,
      "connected" : true,
      "replicationDelayInSeconds" : 0,
      "inboundConnection" : "/192.168.1.133:54516",
      "inboundConnectedSince" : "2019-01-31T12:20:09.017+07:00",
      "outboundConnection" : "[id: 0x34765280, L:/192.168.1.133:54507 - R:pros-mbp/192.168.1.133:6656]",
      "outboundConnectedSince" : "2019-01-31T12:19:44.587+07:00"
    }
  },
  "deduplicationStatus" : "Disabled"
}
```
----
2019-01-31 07:00:48 UTC - Matteo Merli: sorry, the command was `stats-internal`
----
2019-01-31 07:01:29 UTC - Vincent Ngan: So, a 3-node cluster with the above settings can survive
a failure of 1 node. Is that right?
----
2019-01-31 07:03:12 UTC - Matteo Merli: Correct, and in this case availability (`3 - ensemble
= 1`) and durability `(ack - 1 = 1)` are the same
----
2019-01-31 07:03:27 UTC - bossbaby: result:
```
bin/pulsar initialize-cluster-metadata \
  --cluster region_4 \
  --zookeeper localhost:2184 \
  --configuration-store localhost:2190 \
  --web-service-url <http://localhost:8083> \
  --web-service-url-tls <https://localhost:8446> \
  --broker-service-url <pulsar://localhost:6656> \
  --broker-service-url-tls <pulsar+ssl://localhost:6657>


  {
  "entriesAddedCounter" : 1,
  "numberOfEntries" : 31,
  "totalSize" : 1616,
  "currentLedgerEntries" : 1,
  "currentLedgerSize" : 59,
  "lastLedgerCreatedTimestamp" : "2019-01-31T12:19:44.312+07:00",
  "waitingCursorsCount" : 7,
  "pendingAddEntriesCount" : 0,
  "lastConfirmedEntry" : "44:0",
  "state" : "LedgerOpened",
  "ledgers" : [ { ],
  "cursors" : {
    "consumer-test-1" : {
      "markDeletePosition" : "0:-1",
      "readPosition" : "0:0",
      "waitingReadOp" : false,
      "pendingReadOps" : 0,
      "messagesConsumedCounter" : -30,
      "cursorLedger" : 49,
      "cursorLedgerLastEntry" : 2,
      "individuallyDeletedMessages" : "[]",
      "lastLedgerSwitchTimestamp" : "2019-01-31T12:19:44.316+07:00",
      "state" : "Open",
      "numberOfEntriesSinceFirstNotAckedMessage" : 1,
      "totalNonContiguousDeletedMessagesRange" : 0,
      "properties" : { }
    },
    "consumer-test-2" : {
      "markDeletePosition" : "0:-1",
      "readPosition" : "0:0",
      "waitingReadOp" : false,
      "pendingReadOps" : 0,
      "messagesConsumedCounter" : -30,
      "cursorLedger" : 45,
      "cursorLedgerLastEntry" : 13,
      "individuallyDeletedMessages" : "[]",
      "lastLedgerSwitchTimestamp" : "2019-01-31T12:19:44.315+07:00",
      "state" : "Open",
      "numberOfEntriesSinceFirstNotAckedMessage" : 1,
      "totalNonContiguousDeletedMessagesRange" : 0,
      "properties" : { }
    },
    "pulsar.repl.region_2" : {
      "markDeletePosition" : "44:0",
      "readPosition" : "44:1",
      "waitingReadOp" : true,
      "pendingReadOps" : 0,
      "messagesConsumedCounter" : 1,
      "cursorLedger" : 46,
      "cursorLedgerLastEntry" : 1,
      "individuallyDeletedMessages" : "[]",
      "lastLedgerSwitchTimestamp" : "2019-01-31T12:19:44.317+07:00",
      "state" : "Open",
      "numberOfEntriesSinceFirstNotAckedMessage" : 1,
      "totalNonContiguousDeletedMessagesRange" : 0,
      "properties" : { }
    },
    "pulsar.repl.region_3" : {
      "markDeletePosition" : "44:0",
      "readPosition" : "44:1",
      "waitingReadOp" : true,
      "pendingReadOps" : 0,
      "messagesConsumedCounter" : 1,
      "cursorLedger" : 48,
      "cursorLedgerLastEntry" : 1,
      "individuallyDeletedMessages" : "[]",
      "lastLedgerSwitchTimestamp" : "2019-01-31T12:19:44.317+07:00",
      "state" : "Open",
      "numberOfEntriesSinceFirstNotAckedMessage" : 1,
      "totalNonContiguousDeletedMessagesRange" : 0,
      "properties" : { }
    },
    "pulsar.repl.region_4" : {
      "markDeletePosition" : "44:0",
      "readPosition" : "44:1",
      "waitingReadOp" : true,
      "pendingReadOps" : 0,
      "messagesConsumedCounter" : 1,
      "cursorLedger" : 47,
      "cursorLedgerLastEntry" : 1,
      "individuallyDeletedMessages" : "[]",
      "lastLedgerSwitchTimestamp" : "2019-01-31T12:19:44.317+07:00",
      "state" : "Open",
      "numberOfEntriesSinceFirstNotAckedMessage" : 1,
      "totalNonContiguousDeletedMessagesRange" : 0,
      "properties" : { }
    }
  }
}
```
----
2019-01-31 07:04:53 UTC - Matteo Merli: Uhm, then the reset-cursor command should have worked
----
2019-01-31 07:04:55 UTC - Matteo Merli: :confused:
----
2019-01-31 07:05:51 UTC - Matteo Merli: Maybe there’s a check in there that only allows
“regular” subscription, though it would be better for this to work too.
----
2019-01-31 07:06:02 UTC - Matteo Merli: Can you open an issue so that I don’t forget?
----
2019-01-31 07:10:44 UTC - bossbaby: if i use subscription: consumer-test-2, not error but
data no replicated
----
2019-01-31 07:11:04 UTC - bossbaby: ```
bin/pulsar-admin persistent reset-cursor <persistent://public/default/tx> -s consumer-test-2
--time 24h
```
----
2019-01-31 07:12:21 UTC - Matteo Merli: sure that’s just a local subscription
----
2019-01-31 07:13:35 UTC - bossbaby: i will open an issue  and hope it will work
----
2019-01-31 07:16:02 UTC - Matteo Merli: in the meantime,  any new message published should
be getting  replicated
----
2019-01-31 07:16:27 UTC - bossbaby: thanks you @Matteo Merli
----
2019-01-31 07:33:25 UTC - Vincent Ngan: Regarding message retention, what does `delete` means
in the following paragraph quoted from: <https://pulsar.apache.org/docs/en/concepts-messaging/#message-retention-and-expiry>
&gt; *Message retention and expiry*
&gt; By default, Pulsar message brokers:
&gt;
&gt; • immediately `delete` all messages that have been acknowledged by a consumer,
and
&gt; • persistently store all unacknowledged messages in a message backlog.
Does it mean physically deleted, or logically deleted with regard to the subscription the
consumer is subscribing?
----
2019-01-31 07:35:16 UTC - Vincent Ngan: Can I start another consumer using a different subscription
to receive the messages previously acknowledged?
----
2019-01-31 07:40:38 UTC - bossbaby: you can
----
2019-01-31 07:42:26 UTC - Vincent Ngan: My own tests show that acknowledgement of messages
by a consumer is only effective to the related subscription used by the consumer. I can start
another consumer with a different subscription to read the messages again.
----
2019-01-31 07:45:52 UTC - Vincent Ngan: This comes down to my question about when messages
are actually and physically deleted?
----
2019-01-31 07:48:26 UTC - Vincent Ngan: It looks to me that messages are perpetually persisted!
----
2019-01-31 08:02:35 UTC - jia zhai: It should be logically delete
----
2019-01-31 08:03:40 UTC - jia zhai: @Vincent Ngan yes, you are right.
----
2019-01-31 08:34:23 UTC - Vincent Ngan: If this is the case, how can we housekeep the physical
storage used by the bookies?
----
Mime
View raw message