geode-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Blum <jb...@pivotal.io>
Subject Re: Pulse is giving stale view of cluster -- lost updates
Date Thu, 21 Feb 2019 19:45:53 GMT
Dharam-

Before I delve into more details on Pulse, I want to cover 1 other
scenario, and that is, configuring and launcher a GemFire Server using
GemFire's API, such as with the comparable class (to the Spring class I
reference earlier), here (
https://github.com/jxblum/spring-session-data-gemfire-serialization-example/blob/master/native-gemfire-server/src/main/java/example/app/gemfire/server/NativeGemFireServerApplication.java
).

So, what do you think happens when you try to use *Gfsh* with this class, a
GemFire API based class, no less?  Let's see, shall we.


gfsh>connect

Connecting to Locator at [host=localhost, port=10334] ..

Connecting to Manager at [host=10.99.199.24, port=1099] ..

Successfully connected to: [host=10.99.199.24, port=1099]



gfsh>list members

                 Name                   | Id

--------------------------------------- |
--------------------------------------------------------------------------------------

NativeSessionGemFireSerializationServer |
10.99.199.24(NativeSessionGemFireSerializationServer:94332)<ec><v0>:1024
[Coordinator]



gfsh>describe member --name=NativeSessionGemFireSerializationServer

Name        : NativeSessionGemFireSerializationServer

Id          :
10.99.199.24(NativeSessionGemFireSerializationServer:94332)<ec><v0>:1024

Host        : 10.99.199.24

Regions     : Sessions

PID         : 94332

Groups      :

Used Heap   : 190M

Max Heap    : 3641M

Working Dir :
/Users/jblum/pivdev/spring-data-examples-workspace/spring-session-data-gemfire-serialization-example/native-gemfire-server/target

Log file    :
/Users/jblum/pivdev/spring-data-examples-workspace/spring-session-data-gemfire-serialization-example/native-gemfire-server/target

Locators    : localhost[10334]


Cache Server Information

Server Bind              : localhost

Server Port              : 52886

Running                  : true

Client Connections       : 0



gfsh>status server --name=NativeSessionGemFireSerializationServer

[error 2019/02/21 11:39:38.932 PST <Gfsh Launcher> tid=0x1f] Could not
execute "status server --name=NativeSessionGemFireSerializationServer".

java.lang.NullPointerException

at org.json.JSONTokener.nextCleanInternal(JSONTokener.java:128)

at org.json.JSONTokener.nextValue(JSONTokener.java:106)

at org.json.JSONObject.<init>(JSONObject.java:164)

at org.json.JSONObject.<init>(JSONObject.java:179)

at
org.apache.geode.management.internal.cli.json.GfJsonObject.<init>(GfJsonObject.java:82)

at
org.apache.geode.distributed.ServerLauncher$ServerState.fromJson(ServerLauncher.java:2572)

at
org.apache.geode.management.internal.cli.commands.lifecycle.StatusServerCommand.statusServer(StatusServerCommand.java:55)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at
org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:223)

at
org.apache.geode.management.internal.cli.remote.CommandExecutor.invokeCommand(CommandExecutor.java:97)

at
org.apache.geode.management.internal.cli.remote.CommandExecutor.execute(CommandExecutor.java:45)

at
org.apache.geode.management.internal.cli.remote.CommandExecutor.execute(CommandExecutor.java:39)

at
org.apache.geode.management.internal.cli.shell.GfshExecutionStrategy.execute(GfshExecutionStrategy.java:81)

at
org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)

at
org.apache.geode.management.internal.cli.shell.Gfsh.promptLoop(Gfsh.java:1011)

at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)

at java.lang.Thread.run(Thread.java:748)


Could not process command due to error. Error while processing command
<status server --name=NativeSessionGemFireSerializationServer> Reason : null



gfsh>status server --pid=94332

[error 2019/02/21 11:39:48.599 PST <Gfsh Launcher> tid=0x1f] Could not
execute "status server --pid=94332".

java.lang.NullPointerException

at
org.apache.geode.internal.process.MBeanProcessController.status(MBeanProcessController.java:140)

at
org.apache.geode.internal.process.MBeanProcessController.status(MBeanProcessController.java:84)

at
org.apache.geode.internal.process.MBeanOrFileProcessController.status(MBeanOrFileProcessController.java:37)

at
org.apache.geode.distributed.ServerLauncher.statusWithPid(ServerLauncher.java:1073)

at
org.apache.geode.distributed.ServerLauncher.status(ServerLauncher.java:1045)

at
org.apache.geode.management.internal.cli.commands.lifecycle.StatusServerCommand.statusServer(StatusServerCommand.java:74)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at
org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:223)

at
org.apache.geode.management.internal.cli.remote.CommandExecutor.invokeCommand(CommandExecutor.java:97)

at
org.apache.geode.management.internal.cli.remote.CommandExecutor.execute(CommandExecutor.java:45)

at
org.apache.geode.management.internal.cli.remote.CommandExecutor.execute(CommandExecutor.java:39)

at
org.apache.geode.management.internal.cli.shell.GfshExecutionStrategy.execute(GfshExecutionStrategy.java:81)

at
org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)

at
org.apache.geode.management.internal.cli.shell.Gfsh.promptLoop(Gfsh.java:1011)

at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)

at java.lang.Thread.run(Thread.java:748)


Could not process command due to error. Error while processing command
<status server --pid=94332> Reason : null


gfsh>




So, even when using the GemFire API, configuring and bootstrapping a
GemFire Server, JVM process using the Java launcher and a main class, no
good.

You would run into similar problems with *Pulse*.

However, you can still query your Region...


gfsh>list regions

List of regions

---------------

Sessions



gfsh>describe region --name=/Sessions

..................................................................

Name            : Sessions

Data Policy     : partition

Hosting Members : NativeSessionGemFireSerializationServer


Non-Default Attributes Shared By Hosting Members


 Type  |          Name           | Value

------ | ----------------------- | ---------

Region | data-policy             | PARTITION

       | entry-idle-time.timeout | 1800

       | size                    | 0

       | statistics-enabled      | true




And, after running my test class (again,
https://github.com/jxblum/spring-session-data-gemfire-serialization-example/blob/master/application-client/src/test/java/example/app/client/SessionSerializationWithDataSerializationDeltasAndJavaSerializationIntegrationTests.
<https://github.com/jxblum/spring-session-data-gemfire-serialization-example/blob/master/application-client/src/test/java/example/app/client/SessionSerializationWithDataSerializationDeltasAndJavaSerializationIntegrationTests.java>
java
<https://github.com/jxblum/spring-session-data-gemfire-serialization-example/blob/master/application-client/src/test/java/example/app/client/SessionSerializationWithDataSerializationDeltasAndJavaSerializationIntegrationTests.java>
)...


gfsh>describe region --name=/Sessions

..................................................................

Name            : Sessions

Data Policy     : partition

Hosting Members : NativeSessionGemFireSerializationServer


Non-Default Attributes Shared By Hosting Members


 Type  |          Name           | Value

------ | ----------------------- | ---------

Region | data-policy             | PARTITION

       | entry-idle-time.timeout | 1800

       | size                    | 1

       | statistics-enabled      | true




gfsh>query --query="SELECT session.id, attribute.key, attribute.value FROM
/Sessions session, session.attributes attribute ORDER BY session.id ASC"

Result : true

Limit  : 100

Rows   : 2


                 id                  |   key   | value

------------------------------------ | ------- |
-------------------------------

78b1c7d7-4721-48a3-8e4a-0fe26920a718 | janeDoe |
example.app.core.model.Customer

78b1c7d7-4721-48a3-8e4a-0fe26920a718 | jonDoe  |
example.app.core.model.Customer




Stay tuned...

-John


On Thu, Feb 21, 2019 at 10:01 AM John Blum <jblum@pivotal.io> wrote:

> Hi Dharam-
>
> You cannot use `status server` or `stop server` on a GemFire Server
> configured and bootstrapped with *Spring*. (i.e. *Spring Boot* with SDG,
> or alternatively and preferably SBDG).  Most other commands will work (e.g.
> `describe member`) as expected.
>
> For example, if you `describe member --name=S1`, I would expect this to
> work for you.
>
> The reason for this is that certain *Gfsh* commands (e.g. `status server`
> as well as `stop server`, depending on the options used, e.g. `--name`)
> are built with GemFire's MBeans.  Other commands use Functions while still
> other commands might be local and use the GemFire API.
>
> The MBean based commands related to the server (i.e. status/stop) assume
> the GemFire Server was started via *Gfsh* (i.e. `start server`) in the
> first place.  When you use the `start server` command, the
> o.a.g.distributed.ServerLauncher class is used (as the main class) to
> launch (i.e. start) the server, forked in a separate JVM process.  The
> MBean associated with `status/stop server` pulls information from the
> ServerLauncher class used to carry out the function of the commands.
> *Spring*, in all arrangements, does not and will not use the
> ServerLauncher class (ever) to bootstrap a GemFire Server.
>
> I am less familiar with the internals of *Pulse*, however, I do know it
> uses a fair amount of information derived from the MBeans.  This might
> explain some of the issues you are experiencing there.
>
> The missing Regions (in Pulse) as well as the querying issues in *Gfsh*,
> I have not experienced before.
>
> Recently, I have been helping a customer with integration of *Spring
> Session using Pivotal GemFire* (SSDG) in their legacy application.  For
> theirs and our purposes, I created an example repo, here (
> https://github.com/jxblum/spring-session-data-gemfire-serialization-example).
> The example contains a GemFire Server configured and bootstrapped with
> *Spring*, here (
> https://github.com/jxblum/spring-session-data-gemfire-serialization-example/blob/master/spring-gemfire-server/src/main/java/example/app/gemfire/server/SpringGemFireServerApplication.java
> ).
>
> When I start this server up, I can do the following in *Gfsh* without
> issue (NOTE: I am using Pivotal GemFire 9.5.2, which would be Apache Geode
> 1.6.0 since this correlates with what the customer is using):
>
> $ echo $GEMFIRE
>
> /Users/jblum/pivdev/pivotal-gemfire-9.5.2
>
> jblum-mbpro-2:lab jblum$
>
> jblum-mbpro-2:lab jblum$ gfsh
>
>     _________________________     __
>
>    / _____/ ______/ ______/ /____/ /
>
>   / /  __/ /___  /_____  / _____  /
>
>  / /__/ / ____/  _____/ / /    / /
>
> /______/_/      /______/_/    /_/    9.5.2
>
>
> Monitor and Manage Pivotal GemFire
>
> gfsh>
>
> Then...
>
> gfsh>connect
>
> Connecting to Locator at [host=localhost, port=10334] ..
>
> Connecting to Manager at [host=10.99.199.24, port=1099] ..
>
> Successfully connected to: [host=10.99.199.24, port=1099]
>
>
>
> gfsh>list members
>
>                    Name                     | Id
>
> ------------------------------------------- |
> ------------------------------------------------------------------------------------------
>
> SpringSessionDataGemFireSerializationServer |
> 10.99.199.24(SpringSessionDataGemFireSerializationServer:92975)<ec><v0>:1024
> [Coordinator]
>
>
>
> gfsh>describe member --name=SpringSessionDataGemFireSerializationServer
>
> Name        : SpringSessionDataGemFireSerializationServer
>
> Id          :
> 10.99.199.24(SpringSessionDataGemFireSerializationServer:92975)<ec><v0>:1024
>
> Host        : 10.99.199.24
>
> Regions     : Sessions
>
> PID         : 92975
>
> Groups      :
>
> Used Heap   : 60M
>
> Max Heap    : 3641M
>
> Working Dir :
> /Users/jblum/pivdev/spring-data-examples-workspace/spring-session-data-gemfire-serialization-example/spring-gemfire-server/target
>
> Log file    :
> /Users/jblum/pivdev/spring-data-examples-workspace/spring-session-data-gemfire-serialization-example/spring-gemfire-server/target
>
> Locators    : localhost[10334]
>
>
> Cache Server Information
>
> Server Bind              :
>
> Server Port              : 50606
>
> Running                  : true
>
> Client Connections       : 0
>
> However, if I execute `status server`, I see...
>
> gfsh>status server --name=SpringSessionDataGemFireSerializationServer
>
> [error 2019/02/21 09:17:51.226 PST <Gfsh Launcher> tid=0x1f] Could not
> execute "status server --name=SpringSessionDataGemFireSerializationServer".
>
> java.lang.NullPointerException
>
> at org.json.JSONTokener.nextCleanInternal(JSONTokener.java:128)
>
> at org.json.JSONTokener.nextValue(JSONTokener.java:106)
>
> at org.json.JSONObject.<init>(JSONObject.java:164)
>
> at org.json.JSONObject.<init>(JSONObject.java:179)
>
> at
> org.apache.geode.management.internal.cli.json.GfJsonObject.<init>(GfJsonObject.java:82)
>
> at
> org.apache.geode.distributed.ServerLauncher$ServerState.fromJson(ServerLauncher.java:2572)
>
> at
> org.apache.geode.management.internal.cli.commands.lifecycle.StatusServerCommand.statusServer(StatusServerCommand.java:55)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at
> org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)
>
> at
> org.apache.geode.management.internal.cli.remote.CommandExecutor.invokeCommand(CommandExecutor.java:97)
>
> at
> org.apache.geode.management.internal.cli.remote.CommandExecutor.execute(CommandExecutor.java:45)
>
> at
> org.apache.geode.management.internal.cli.remote.CommandExecutor.execute(CommandExecutor.java:39)
>
> at
> org.apache.geode.management.internal.cli.shell.GfshExecutionStrategy.execute(GfshExecutionStrategy.java:81)
>
> at
> org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)
>
> at
> org.apache.geode.management.internal.cli.shell.Gfsh.promptLoop(Gfsh.java:1011)
>
> at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
> Could not process command due to error. Error while processing command
> <status server --name=SpringSessionDataGemFireSerializationServer> Reason :
> null
>
>
>
> What if I try to execute `status server --pid=92975`...
>
> gfsh>status server --pid=92975
>
> [error 2019/02/21 09:18:40.040 PST <Gfsh Launcher> tid=0x1f] Could not
> execute "status server --pid=92975".
>
> org.apache.geode.lang.AttachAPINotFoundException: The Attach API classes
> could not be found on the classpath.  Please include JDK tools.jar on the
> classpath or add the JDK tools.jar to the jre/lib/ext directory.
>
> at
> org.apache.geode.internal.process.FileProcessController.checkPidSupport(FileProcessController.java:101)
>
> at
> org.apache.geode.distributed.ServerLauncher.statusWithPid(ServerLauncher.java:1072)
>
> at
> org.apache.geode.distributed.ServerLauncher.status(ServerLauncher.java:1045)
>
> at
> org.apache.geode.management.internal.cli.commands.lifecycle.StatusServerCommand.statusServer(StatusServerCommand.java:74)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at
> org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)
>
> at
> org.apache.geode.management.internal.cli.remote.CommandExecutor.invokeCommand(CommandExecutor.java:97)
>
> at
> org.apache.geode.management.internal.cli.remote.CommandExecutor.execute(CommandExecutor.java:45)
>
> at
> org.apache.geode.management.internal.cli.remote.CommandExecutor.execute(CommandExecutor.java:39)
>
> at
> org.apache.geode.management.internal.cli.shell.GfshExecutionStrategy.execute(GfshExecutionStrategy.java:81)
>
> at
> org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)
>
> at
> org.apache.geode.management.internal.cli.shell.Gfsh.promptLoop(Gfsh.java:1011)
>
> at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
> Could not process command due to error. Error while processing command
> <status server --pid=92975> Reason : The Attach API classes could not be
> found on the classpath.  Please include JDK tools.jar on the classpath or
> add the JDK tools.jar to the jre/lib/ext directory.
>
>
>
> OK!  That requires I have the JDK Attach API on the classpath of my
> server.  However, that will not longer work...
>
>
> https://github.com/apache/geode/blob/rel/v1.6.0/geode-core/src/main/java/org/apache/geode/internal/process/FileProcessController.java#L98-L102
>
> And, well...
>
>
> https://github.com/apache/geode/blob/rel/v1.8.0/geode-core/src/main/java/org/apache/geode/internal/process/FileProcessController.java#L98-L102
>
> Anyway, let's continue.
>
> Because, I have a Spring Session based example, I have a */Sessions*
> Region, here (
> https://github.com/jxblum/spring-session-data-gemfire-serialization-example/blob/master/spring-gemfire-server/src/main/java/example/app/gemfire/server/SpringGemFireServerApplication.java#L47
> ).
>
> I can...
>
> gfsh>list regions
>
> List of regions
>
> ---------------
>
> Sessions
>
>
>
> required --name: Name/Path of the region to be described.; no default value
>
> gfsh>describe region --name=/Sessions
>
>
> .................................................................................................................................................................
>
> Name            : Sessions
>
> Data Policy     : partition
>
> Hosting Members : SpringSessionDataGemFireSerializationServer
>
>
> Non-Default Attributes Shared By Hosting Members
>
>
>  Type  |          Name           | Value
>
> ------ | ----------------------- |
> ---------------------------------------------------------------------------------------------------------------------------
>
> Region | data-policy             | PARTITION
>
>        | entry-idle-time.timeout | 1800
>
>        | size                    | 0
>
>        | statistics-enabled      | true
>
>        | cache-listeners         |
> org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository.SessionEventHandlerCacheListenerAdapter
>
>
>
> Now, if I run my test class (
> https://github.com/jxblum/spring-session-data-gemfire-serialization-example/blob/master/application-client/src/test/java/example/app/client/SessionSerializationWithDataSerializationDeltasAndJavaSerializationIntegrationTests.java),
> after the test passes, it will create a Session and persist it to GemFire,
> where I can then see the size to the */Sessions* Region change and query
> the data...
>
> gfsh>describe region --name=/Sessions
>
>
> .................................................................................................................................................................
>
> Name            : Sessions
>
> Data Policy     : partition
>
> Hosting Members : SpringSessionDataGemFireSerializationServer
>
>
> Non-Default Attributes Shared By Hosting Members
>
>
>  Type  |          Name           | Value
>
> ------ | ----------------------- |
> ---------------------------------------------------------------------------------------------------------------------------
>
> Region | data-policy             | PARTITION
>
>        | entry-idle-time.timeout | 1800
>
>        | size                    | 1
>
>        | statistics-enabled      | true
>
>        | cache-listeners         |
> org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository.SessionEventHandlerCacheListenerAdapter
>
>
> gfsh>query --query="SELECT session.id, attribute.key, attribute.value
> FROM /Sessions session, session.attributes attribute ORDER BY session.id
> ASC"
>
> Result : true
>
> Limit  : 100
>
> Rows   : 2
>
>
>                  id                  |   key   | value
>
> ------------------------------------ | ------- |
> -------------------------------
>
> 736eb051-d1ae-4989-b7f3-4e7b91e5bb7a | janeDoe |
> example.app.core.model.Customer
>
> 736eb051-d1ae-4989-b7f3-4e7b91e5bb7a | jonDoe  |
> example.app.core.model.Customer
>
>
>
> So, (mostly) all works as expected.
>
> I will follow up in another email with details on *Pulse*.
>
> -John
>
>
>
> On Thu, Feb 21, 2019 at 8:54 AM Dharam Thacker <dharamthacker88@gmail.com>
> wrote:
>
>> One more thing about GFSH.
>>
>> gfsh>list regions
>> List of regions
>> ---------------
>> GroupDefinition
>> Sample1
>> Sample2
>>
>> gfsh>query --query="select * from /Sample1"
>> Result  : false
>> Message : Cannot find regions <[/Sample1]> in any of the members
>>
>> Thanks,
>> - Dharam Thacker
>>
>>
>> On Thu, Feb 21, 2019 at 10:15 PM Dharam Thacker <
>> dharamthacker88@gmail.com> wrote:
>>
>>> Hi Jens,
>>>
>>> I tried again as per your suggestions and traced logs as well again. You
>>> can see below that, both servers have joined but no regions created, no
>>> server counts changed and GFSH is behaving strangely as well.
>>>
>>> GFSH:
>>> gfsh>list members
>>>   Name   | Id
>>> -------- |
>>> ----------------------------------------------------------------
>>> locator1 | 192.168.31.62(locator1:2719:locator)<ec><v0>:41000
>>> [Coordinator]
>>> locator2 | 192.168.31.62(locator2:2854:locator)<ec><v1>:41001
>>> S2       | 192.168.31.62(S2:5021)<v10>:41002
>>> S1       | 192.168.31.62(S1:5107)<v11>:41003
>>>
>>> gfsh>status server --name=S1
>>> No Geode Cache Server with member name or ID S1 could be found.
>>>
>>>
>>> gfsh>status server --name=S2
>>> No Geode Cache Server with member name or ID S2 could be found.
>>>
>>>
>>> XHR logs: No regions as well as only locators as members even though
>>> servers are there.
>>>
>>> [image: image.png]
>>>
>>> [image: image.png]
>>>
>>> I see below logs as well on startup in case they help.
>>> [debug 2019/02/21 22:04:33.839 IST <main> tid=0x1] Notification Region
>>> created with Name : _notificationRegion_192.168.31.62<v10>41002
>>>
>>> [debug 2019/02/21 22:04:34.113 IST <unicast
>>> receiver,dharam-thakkar-34169> tid=0x1b] sending via JGroups:
>>> [HeartbeatMessage [requestId=1]] recipients:
>>> [192.168.31.62(locator2:2854:locator)<ec><v1>:41001]
>>>
>>> [warn 2019/02/21 22:04:34.590 IST <main> tid=0x1] (tid=1 msgId=0) *Could
>>> not load Command from*: class
>>> org.apache.geode.management.internal.cli.commands.DestroyIndexCommand due
>>> to org.apache.geode.management.internal.cli.commands.DestroyIndexCommand
>>> cannot be cast to org.springframework.shell.core.CommandMarker
>>>
>>> [warn 2019/02/21 22:04:34.597 IST <main> tid=0x1] (tid=1 msgId=1) *Could
>>> not load Command from*: class
>>> org.apache.geode.management.internal.cli.commands.BackupDiskStoreCommand
>>> due to
>>> org.apache.geode.management.internal.cli.commands.BackupDiskStoreCommand
>>> cannot be cast to org.springframework.shell.core.CommandMarker
>>>
>>> [warn 2019/02/21 22:04:34.600 IST <main> tid=0x1] (tid=1 msgId=2) *Could
>>> not load Command from:* class
>>> org.apache.geode.management.internal.cli.commands.PDXRenameCommand due to
>>> org.apache.geode.management.internal.cli.commands.PDXRenameCommand cannot
>>> be cast to org.springframework.shell.core.CommandMarker
>>>
>>> [debug 2019/02/21 22:04:34.632 IST <main> tid=0x1] *Closing Management
>>> Service*
>>>
>>> Thanks,
>>> Dharam
>>>
>>>
>>> On Wed, Feb 20, 2019 at 10:53 AM Jens Deppe <jensdeppe@apache.org>
>>> wrote:
>>>
>>>> Hi Dharam,
>>>>
>>>> I've tried to replicate this, but have not been successful - I've tried
>>>> restarting my Spring Boot app server at least 20 times, but it always shows
>>>> up in Pulse.
>>>>
>>>> What would be useful is to try and look at the data that Pulse is
>>>> retrieving in order to update it's display. If you're using Chrome, can you
>>>> open the developer console and select the 'Network' tab. From there, select
>>>> the 'XHR' filter tab - that should show you a 'pulseUpdate'
>>>> request/response every 5 seconds. I'd be interested to see the data (it's
a
>>>> JSON payload) that comes back when you have all the members in the view and
>>>> then the data that comes back when you are missing a member.
>>>>
>>>> Thanks
>>>> --Jens
>>>>
>>>> On Tue, Feb 19, 2019 at 2:47 PM Bruce Schuchardt <
>>>> bschuchardt@pivotal.io> wrote:
>>>>
>>>>> I can't comment on most of the content of your server1.log.  The
>>>>> java.net.SocketException doesn't seem to be causing any problems but
an
>>>>> internet search indicated that setting
>>>>>
>>>>> -Djava.net.preferIPv4Stack=true
>>>>>
>>>>> might fix that problem for the machine you're using for testing.  This
>>>>> exception is caught and logged but shouldn't cause any other problems.
>>>>> Indeed, I can see from the debug-level logging that UDP messaging was
>>>>> working okay in your run.
>>>>>
>>>>>
>>>>> On 2/16/19 6:42 AM, Dharam Thacker wrote:
>>>>>
>>>>> Hi Team,
>>>>>
>>>>> I am sure about this issue now and it's really critical and worth to
>>>>> look at. I would really appreciate to address it in upcoming release
as
>>>>> it's a BLOCKER for monitoring systems.
>>>>>
>>>>> I hope below one helps for your analysis. Please let me know if I can
>>>>> help with any more details for the same.
>>>>>
>>>>> Few quick glimpses
>>>>> On startup>>
>>>>> [debug 2019/02/16 19:41:45.642 IST <main> tid=0x1] Creating
>>>>> Management Region :
>>>>>
>>>>> [debug 2019/02/16 19:41:45.680 IST <main> tid=0x1] Management Service
>>>>> is not initialised hence returning from handleLockServiceCreation
>>>>>
>>>>> [warn 2019/02/16 19:41:46.500 IST <main> tid=0x1] Could not initialize
>>>>> class org.apache.logging.log4j.util.PropertiesUtil
>>>>> java.lang.NoClassDefFoundError: Could not initialize class
>>>>> org.apache.logging.log4j.util.PropertiesUtil
>>>>>
>>>>> ...
>>>>>
>>>>> *System Specification : *
>>>>> DISTRIB_ID=LinuxMint
>>>>> DISTRIB_RELEASE=18.3
>>>>> DISTRIB_CODENAME=sylvia
>>>>> DISTRIB_DESCRIPTION="Linux Mint 18.3 Sylvia"
>>>>>
>>>>> *Java : *
>>>>> openjdk version "1.8.0_191"
>>>>> OpenJDK Runtime Environment (build
>>>>> 1.8.0_191-8u191-b12-2ubuntu0.16.04.1-b12)
>>>>> OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
>>>>>
>>>>> *GEODE*: 1.8.0 *Spring-Data-Geode* : 2.1.4.RELEASE (Geode version
>>>>> overriden from 1.6.0 to 1.8.0)
>>>>>
>>>>> John,
>>>>> It's fully using spring-data-geode and worth looking at several issues
>>>>> related to that as well in server1.log
>>>>>
>>>>> The below link contains following artifacts for detailed analysis and
>>>>> re-generating issues,
>>>>>
>>>>> *Attachments:*
>>>>> https://drive.google.com/open?id=18AuPx05Aw-ezwNOKqdCfUJUwUycOqzTp
>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__drive.google.com_open-3Fid-3D18AuPx05Aw-2DezwNOKqdCfUJUwUycOqzTp&d=DwMFaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=JEKigqAv3f2lWHmA02pq9MDT5naXLkEStB4d4n0NQmk&m=KXGpMQ3vCHLW9I1372frvIz29jAVik7VeZ19pSYqNjU&s=Encm7VMywtgfrNZoO_gucw4q4RwpZmlQ3xpowDLpiNY&e=>
>>>>>
>>>>> 1. I have attached both locator (locator1,locator2) logs & properties
>>>>> file
>>>>> *Commands:*
>>>>> start locator --name=locator1 --port=10334
>>>>> --properties-file=/home/apps/work/geode/locator1/locator.properties
>>>>> --dir=/home/apps/work/geode/locator1/work
>>>>>
>>>>> start locator --name=locator2 --port=10335
>>>>> --properties-file=/home/apps/work/geode/locator2/locator.properties
>>>>> --dir=/home/apps/work/geode/locator2/work
>>>>>
>>>>> 2. I have attached server1.log with debug level & demo.tar to
>>>>> regenerate the same issue
>>>>> *Command* : java -jar demo-0.0.1-SNAPSHOT.jar --demo.name
>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__demo.name&d=DwMFaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=JEKigqAv3f2lWHmA02pq9MDT5naXLkEStB4d4n0NQmk&m=KXGpMQ3vCHLW9I1372frvIz29jAVik7VeZ19pSYqNjU&s=onawLLFvihcvBxkdkVHzB5jqnL6Cy1UmlVfSy1I7KMQ&e=>=S1
>>>>> --demo.port=40441 > server1.log &
>>>>>
>>>>> 3. Below is the pulse view where we can clearly say that, no more JMX
>>>>> notifications regarding region initialisation or cache server were recorded
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>> Thanks,
>>>>> Dharam
>>>>>
>>>>>
>>>>> I have
>>>>> - Dharam Thacker
>>>>>
>>>>>
>>>>> On Tue, Feb 5, 2019 at 6:28 PM Thacker, Dharam <
>>>>> dharam.thacker@jpmorgan.com> wrote:
>>>>>
>>>>>> Hi Team,
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have usually seen following sequence when new member joins the
>>>>>> cluster (member = cache-server)
>>>>>>
>>>>>>
>>>>>>
>>>>>> *JMX Notifications on pulse screen :*
>>>>>>
>>>>>>
>>>>>>
>>>>>> 1.       Member Joined <<SERVER_NAME>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2.       Region Created With Name /<<REGION_NAME>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 3.       Cache Server is Started in the VM
>>>>>>
>>>>>>
>>>>>>
>>>>>> I am using GEODE 1.8.0  + Spring data geode 2.1.4.RELEASE with
>>>>>> following properties and pulse in embedded mode.
>>>>>>
>>>>>>
>>>>>>
>>>>>> *locator1.properties*
>>>>>>
>>>>>> locators=dharam-thakkar[10440],dharam-thakkar[10440]
>>>>>>
>>>>>> mcast-port=0
>>>>>>
>>>>>> jmx-manager=true
>>>>>>
>>>>>> jmx-manager-start=true
>>>>>>
>>>>>> jmx-manager-port=1091
>>>>>>
>>>>>> jmx-manager-ssl-enabled=false
>>>>>>
>>>>>> jmx-manager-bind-address=dharam-thakkar
>>>>>>
>>>>>> enable-network-partition-detection=false
>>>>>>
>>>>>> http-service-port=9701
>>>>>>
>>>>>> http-service-bind-address=dharam-thakkar
>>>>>>
>>>>>> log-file=/local/var/tmp/demo-locator1/locator1.log
>>>>>>
>>>>>> log-file-size-limit=10
>>>>>>
>>>>>> log-level=config
>>>>>>
>>>>>> log-disk-space-limit=50
>>>>>>
>>>>>>
>>>>>>
>>>>>> I tried below sequence and I see that PULSE is missing “JMX
>>>>>> Notifications” and gives incorrect view of cluster.
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Steps to reproduce>>*
>>>>>>
>>>>>>
>>>>>>
>>>>>> 1.       gfsh start locator --name=demo-locator-1 --port=10440
>>>>>> --properties-file=locator1.properties --work-dir=/var/tmp/demo-locator1/work
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2.       java -jar demo-spring-boot-geode-server.jar
>>>>>> -DserverName=demo-server1 -DserverPort=40440
>>>>>>
>>>>>>
>>>>>>
>>>>>> 3.       java -jar demo-spring-boot-geode-server.jar
>>>>>> -DserverName=demo-server2 -DserverPort=40441
>>>>>>
>>>>>>
>>>>>>
>>>>>> 4.       Everything will look fine as of now and you will see all
>>>>>> notifications as explained in above sequence
>>>>>>
>>>>>>
>>>>>>
>>>>>> 5.       PID=`ps auxwww | fgrep 'java' | fgrep 'demo-server-1' |
awk
>>>>>> '{print $2}'` ; kill -INT $PID
>>>>>>
>>>>>>
>>>>>>
>>>>>> 6.       You should see *“Member Departed <<SERVER_NAME>>”*
message
>>>>>> on pulse
>>>>>>
>>>>>>
>>>>>>
>>>>>> 7.       Reboot the member -- java -jar
>>>>>> demo-spring-boot-geode-server.jar -DserverName=demo-server1
>>>>>> -DserverPort=40440
>>>>>>
>>>>>>
>>>>>>
>>>>>> 8.       Observe pulse notifications and member count
>>>>>>
>>>>>>
>>>>>>
>>>>>> 9.       You will only see *“Member Joined <<SERVER_NAME>>”
*
>>>>>> message on pulse and no update in member count
>>>>>>
>>>>>>
>>>>>>
>>>>>> 10.   If you don’t see situation as step-9, repeat steps-5 to
>>>>>> steps-7 few times and you will end up in this situation
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Note:* Please note that GFSH shows everything correctly but PULSE
>>>>>> has issues.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Dharam
>>>>>>
>>>>>> This message is confidential and subject to terms at: https://
>>>>>> www.jpmorgan.com/emaildisclaimer
>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.jpmorgan.com_emaildisclaimer&d=DwMFaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=JEKigqAv3f2lWHmA02pq9MDT5naXLkEStB4d4n0NQmk&m=KXGpMQ3vCHLW9I1372frvIz29jAVik7VeZ19pSYqNjU&s=wnMQ4KQ6EkepwerGG8L-HD4Bkb64Lv6lIQ77fjYolzs&e=>
>>>>>> including on confidentiality, legal privilege, viruses and monitoring
of
>>>>>> electronic messages. If you are not the intended recipient, please
delete
>>>>>> this message and notify the sender immediately. Any unauthorized
use is
>>>>>> strictly prohibited.
>>>>>>
>>>>>
>
> --
> -John
> john.blum10101 (skype)
>


-- 
-John
john.blum10101 (skype)

Mime
View raw message