ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stepan Pilschikov (Jira)" <j...@apache.org>
Subject [jira] [Created] (IGNITE-12658) [ML][Examples] TutorialStepByStepExample failed on cluster with more then 1 node
Date Tue, 11 Feb 2020 11:14:00 GMT
Stepan Pilschikov created IGNITE-12658:
------------------------------------------

             Summary: [ML][Examples] TutorialStepByStepExample failed on cluster with more
then 1 node
                 Key: IGNITE-12658
                 URL: https://issues.apache.org/jira/browse/IGNITE-12658
             Project: Ignite
          Issue Type: Bug
          Components: examples, ml
    Affects Versions: 2.8
         Environment: Ubuntu/Win
Java 8
            Reporter: Stepan Pilschikov


Steps to reproduce:
 1. Run Ignite node with org.apache.ignite.examples.ExampleNodeStartup (1 node will be enough)
 2. Run org.apache.ignite.examples.ml.tutorial.TutorialStepByStepExample

Actual:
 On Step_8_CV_with_Param_Grid_and_metrics starting to throw a lot of exceptions
{code:java}
Train with p: 2 and maxDeep: 1

>>> Trained model: if (x1 > 0.4368) then return 1.0000 else return 0.0000

>>> Accuracy 0.7679083094555874

>>> Test Error 0.2320916905444126
>>> Tutorial step 8 (cross-validation) example completed.
[13:25:40] Ignite node stopped OK [uptime=00:00:17.453]

>>> Tutorial step 8 (cross-validation with param grid) example started.
[13:25:40]    __________  ________________ 
[13:25:40]   /  _/ ___/ |/ /  _/_  __/ __/ 
[13:25:40]  _/ // (7 7    // /  / / / _/   
[13:25:40] /___/\___/_/|_/___/ /_/ /___/  
[13:25:40] 
[13:25:40] ver. 2.8.0#20200130-sha1:f478aa56
[13:25:40] 2020 Copyright(C) Apache Software Foundation
[13:25:40] 
[13:25:40] Ignite documentation: http://ignite.apache.org
[13:25:40] 
[13:25:40] Quiet mode.
[13:25:40]   ^-- Logging to file '/opt/buildagent/work/d501ae8146bd8253/i2test/var/suite-examples/app-ignite/work/log/ignite-e156b2f2.log'
[13:25:40]   ^-- Logging by 'Log4JLogger [quiet=true, config=null]'
[13:25:40]   ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat}
[13:25:40] 
[13:25:40] OS: Linux 4.15.0-65-generic amd64
[13:25:40] VM information: Java(TM) SE Runtime Environment 1.8.0_221-b11 Oracle Corporation
Java HotSpot(TM) 64-Bit Server VM 25.221-b11
[13:25:40] Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possible
problems in mixed environments.
[13:25:40] Configured plugins:
[13:25:40]   ^-- ml-inference-plugin 1.0.0
[13:25:40]   ^-- null
[13:25:40] 
[13:25:40] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]]
[13:25:40] Message queue limit is set to 0 which may lead to potential OOMEs when running
cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender
and receiver sides.
[13:25:40] Security status [authentication=off, tls/ssl=off]
[13:25:41] Performance suggestions for grid  (fix if possible)
[13:25:41] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[13:25:41]   ^-- Disable grid events (remove 'includeEventTypes' from configuration)
[13:25:41]   ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options)
[13:25:41]   ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]'
to JVM options)
[13:25:41]   ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC'
to JVM options)
[13:25:41] Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning
[13:25:41] 
[13:25:41] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[13:25:41] Data Regions Configured:
[13:25:41]   ^-- Default_Region [initSize=500.0 MiB, maxSize=18.9 GiB, persistence=false,
lazyMemoryAllocation=true]
[13:25:41] 
[13:25:41] Ignite node started OK (id=e156b2f2)
[13:25:41] Topology snapshot [ver=20, locNode=e156b2f2, servers=2, clients=0, state=ACTIVE,
CPUs=5, offheap=38.0GB, heap=3.0GB]
[13:25:41]   ^-- Baseline [id=0, size=2, online=2, offline=0]
[2020-02-11 13:25:42,428][ERROR][sys-#593][GridTaskWorker] Failed to obtain remote job result
policy for result from ComputeTask.result(..) method (will fail the whole task): GridJobResultImpl
[job=C2 [c=o.a.i.ml.dataset.impl.cache.util.ComputeUtils$DeployableCallable@30e27659], sib=GridJobSiblingImpl
[sesId=f9aced33071-e156b2f2-d116-4389-bd43-8536dc598888, jobId=1aaced33071-e156b2f2-d116-4389-bd43-8536dc598888,
nodeId=f1135598-73c8-4324-92af-45c110a6a637, isJobDone=false], jobCtx=GridJobContextImpl [jobId=1aaced33071-e156b2f2-d116-4389-bd43-8536dc598888,
timeoutObj=null, attrs=HashMap {}], node=TcpDiscoveryNode [id=f1135598-73c8-4324-92af-45c110a6a637,
consistentId=127.0.0.1,192.168.112.2:47500, addrs=ArrayList [127.0.0.1, 192.168.112.2], sockAddrs=HashSet
[/127.0.0.1:47500, 87bf29244900/192.168.112.2:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1581416741501, loc=false, ver=2.8.0#20200130-sha1:f478aa56, isClient=false],
ex=class o.a.i.IgniteException: Failed to deserialize object [typeName=o.a.i.i.processors.closure.GridClosureProcessor$C2],
hasRes=true, isCancelled=false, isOccupied=true]
class org.apache.ignite.IgniteException: Remote job threw user exception (override or implement
ComputeTask.result(..) method if you would like to have automatic failover for this exception):
Failed to deserialize object [typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2]
	at org.apache.ignite.compute.ComputeTaskAdapter.result(ComputeTaskAdapter.java:102)
	at org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1062)
	at org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1055)
	at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7037)
	at org.apache.ignite.internal.processors.task.GridTaskWorker.result(GridTaskWorker.java:1055)
	at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:862)
	at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1146)
	at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1370)
	at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1843)
	at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1468)
	at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
	at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1365)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2]
	at org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:467)
	at org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1265)
	at org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2089)
	... 7 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2]
	at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10310)
	at org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:448)
	... 9 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object
[typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2]
	at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:913)
	at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
	at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
	at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:307)
	at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:101)
	at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:81)
	at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10304)
	... 10 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object
[typeName=org.apache.ignite.ml.dataset.impl.cache.util.ComputeUtils$DeployableCallable]
	at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:913)
	at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
	at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
	at org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1800)
	at org.apache.ignite.internal.binary.BinaryReaderExImpl.readObject(BinaryReaderExImpl.java:1331)
	at org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2.readBinary(GridClosureProcessor.java:1872)
	at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:864)
	... 16 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to read field [name=fun]
	at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
	at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:874)
	... 22 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object
[typeName=java.lang.invoke.SerializedLambda]
	at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:913)
	at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
	at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
	at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
	at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
	at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
	... 23 more
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message