cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joshua Schaeffer <jschaef...@harmonywave.com>
Subject Network error when uploading local ISO
Date Fri, 02 Jul 2021 15:34:33 GMT
I posted a couple weeks back about some issues getting ISO's and templates uploaded in ACS
and received some excellent help. I've changed a few things around based on that previous
issue and am finally circling back to it but still running into a problem uploading an ISO.
When I try to upload an ISO through primate I now get two messages [1]:

Network Error: Unable to reach the management server or a browser extension may be blocking
the network request.

Upload Failed: Failed to upload ISO - Error: Network Error

After I click "OK" on the "Upload ISO from Local" dialog box I get these two messages/errors
and nothing else happens, the dialog box doesn't go away and I can click "OK" again as many
times as I want (the messages always repeat). Also, If I refresh the ISO page then I see that
my ISO that I tried to upload is now listed [2], but is not actually ready [3].

To give some background, I am running ACS 4.15.0.0 with two load balanced management servers
and a KVM compute host. I've tried the following:

1. Disabled one of the management servers.
2. Connected directly to a management server.
3. Used different browsers.

Nothing listed above has changed the outcome and the same message is always thrown, so I don't
think it is a load balancer or browser issue. As this is related to my previous issue in getting
the ssvm started I wanted to show that it is now up and running [4], I can connect to it,
and the /usr/local/cloud/systemvm/ssvm-check.sh script doesn't throw any errors:

root@s-41-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
================================================
First DNS server is  10.2.81.1
PING 10.2.81.1 (10.2.81.1): 56 data bytes
64 bytes from 10.2.81.1: icmp_seq=0 ttl=63 time=0.357 ms
64 bytes from 10.2.81.1: icmp_seq=1 ttl=63 time=0.399 ms
--- 10.2.81.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.357/0.378/0.399/0.000 ms
Good: Can ping DNS server
================================================
Good: DNS resolves cloudstack.apache.org
================================================
nfs is currently mounted
Mount point is /mnt/SecStorage/df2ca46d-aee0-302e-9ad0-2e94252341e4
Good: Can write to mount point
================================================
Management server is bllcloudlb01.harmonywave.cloud. Checking connectivity.
Good: Can connect to management server bllcloudlb01.harmonywave.cloud port 8250
================================================
Good: Java process is running
================================================
Tests Complete. Look for ERROR or WARNING above.

The cloud service inside the system VM is also running:

root@s-41-VM:~# service cloud status
● cloud.service - CloudStack Agent service
   Loaded: loaded (/etc/systemd/system/cloud.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2021-06-26 02:58:24 UTC; 6 days ago
 Main PID: 3011 (bash)
    Tasks: 48 (limit: 543)
   Memory: 157.0M
   CGroup: /system.slice/cloud.service
           ├─3011 bash /usr/local/cloud/systemvm/_run.sh
           └─3241 java -Djavax.net.ssl.trustStore=./certs/realhostip.keystore
-Djdk.tls.ephemeralDHKeySize=2048 -Djsse.enableSNIExtension=false -Dlog.hom

Jul 02 15:16:41 s-41-VM _run.sh[3011]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Jul 02 15:16:41 s-41-VM _run.sh[3011]:         at java.base/java.lang.Thread.run(Thread.java:834)
Jul 02 15:16:41 s-41-VM _run.sh[3011]: 15:16:41,284  INFO Agent:835 - Processing agent ready
command, agent id = 16
Jul 02 15:16:41 s-41-VM _run.sh[3011]: 15:16:41,284  INFO Agent:375 - Set agent id 16
Jul 02 15:16:41 s-41-VM _run.sh[3011]: 15:16:41,285  INFO Agent:842 - Ready command is processed
for agent id = 16
Jul 02 15:16:41 s-41-VM _run.sh[3011]: 15:16:41,459  INFO Agent:835 - Processing agent ready
command, agent id = 16
Jul 02 15:16:41 s-41-VM _run.sh[3011]: 15:16:41,459  INFO Agent:375 - Set agent id 16
Jul 02 15:16:41 s-41-VM _run.sh[3011]: 15:16:41,469  INFO Agent:791 - Processed new management
server list: bllcloudlb01.harmonywave.cloud@static
Jul 02 15:16:41 s-41-VM _run.sh[3011]: 15:16:41,469  INFO Agent:842 - Ready command is processed
for agent id = 16
Jul 02 15:16:45 s-41-VM _run.sh[3011]: 15:16:45,134  INFO Agent:549 - Connected to the host:
bllcloudlb01.harmonywave.cloud

Finally I took a network capture from the browser when I hit "OK" and I see the following:
[5] https://drive.google.com/file/d/1w-gnARJ1XegK5D24c91T3qseU8GDNAPd/view?usp=sharing

I'm not quite sure what to make of that and can provide more details if needed. I have been
able to verify that port 443 is up on the SSVM:

root@s-41-VM:~# ss -tupanl | grep 443
tcp     LISTEN   0        511        192.41.41.161:443          
0.0.0.0:*       users:(("apache2",pid=28140,fd=4),("apache2",pid=28139,fd=4),("apache2",pid=28136,fd=4))

The logs don't seem to provide me with any hint of the issue:

###################
# MANAGEMENT LOGS #
###################
2021-07-02 14:37:39,767 DEBUG [c.c.a.ApiServlet] (qtp1644231115-26571:ctx-3657db16) (logid:2a9a3fec)
===START===  172.16.44.18 -- GET  name=ubuntu-20.04.2-live-server-amd64.iso&displaytext=Ubuntu+20.04+(&zoneid=9f6f5b49-0e12-4af4-a13b-2ace6c47de43&bootable=true&ostypeid=aa18f3ad-cd73-11eb-b1da-5254008f72d5&isextractable=false&ispublic=true&isfeatured=false&format=ISO&command=getUploadParamsForIso&response=json
2021-07-02 14:37:39,787 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-11:null) (logid:)
SeqA 15-71382: Processing Seq 15-71382:  { Cmd , MgmtId: -1, via: 15, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"40","_loadInfo":"{
  "connections": []
}","wait":"0"}}] }
2021-07-02 14:37:39,805 DEBUG [c.c.a.ApiServer] (qtp1644231115-26571:ctx-3657db16 ctx-d1285d49)
(logid:2a9a3fec) CIDRs from which account 'Acct[f8d6949d-cd74-11eb-b1da-5254008f72d5-admin]'
is allowed to perform API calls: 0.0.0.0/0,::/0
2021-07-02 14:37:39,826 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-11:null) (logid:)
SeqA 15-71382: Sending Seq 15-71382:  { Ans: , MgmtId: 90520733511963, via: 15, Ver: v1,
Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0"}}] }
2021-07-02 14:37:40,000 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (qtp1644231115-26571:ctx-3657db16
ctx-d1285d49) (logid:2a9a3fec) template 217 is not in store:2, type:Image
2021-07-02 14:37:40,032 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (qtp1644231115-26571:ctx-3657db16
ctx-d1285d49) (logid:2a9a3fec) template 217 is already in store:2, type:Image
2021-07-02 14:37:40,058 DEBUG [c.c.r.ResourceLimitManagerImpl] (qtp1644231115-26571:ctx-3657db16
ctx-d1285d49) (logid:2a9a3fec) Updating resource Type = template count for Account = 2 Operation
= increasing Amount = 1
2021-07-02 14:37:40,289 DEBUG [c.c.a.ApiServlet] (qtp1644231115-26571:ctx-3657db16 ctx-d1285d49)
(logid:2a9a3fec) ===END===  172.16.44.18 -- GET  name=ubuntu-20.04.2-live-server-amd64.iso&displaytext=Ubuntu+20.04+(&zoneid=9f6f5b49-0e12-4af4-a13b-2ace6c47de43&bootable=true&ostypeid=aa18f3ad-cd73-11eb-b1da-5254008f72d5&isextractable=false&ispublic=true&isfeatured=false&format=ISO&command=getUploadParamsForIso&response=json
2021-07-02 14:37:40,496 DEBUG [c.c.a.ApiServlet] (qtp1644231115-10794:ctx-8910f7db) (logid:debd3cae)
===START===  172.16.44.18 -- GET  listall=true&isofilter=all&showunique=true&page=1&pagesize=10&command=listIsos&response=json

#################
# LOAD BALANCER #
#################
Jul  2 14:37:40 bllcloudlb01 haproxy[216322]: 172.16.44.18:54094 [02/Jul/2021:14:37:39.761]
ft_cloudstack_gui~ bk_cloudstack_gui/cloudstack_gui_1 1/0/0/530/531 200 1131 - - --VN 4/1/0/0/0
0/0 "GET /client/api/?name=ubuntu-20.04.2-live-server-amd64.iso&displaytext=Ubuntu+20.04+(&zoneid=9f6f5b49-0e12-4af4-a13b-2ace6c47de43&bootable=true&ostypeid=aa18f3ad-cd73-11eb-b1da-5254008f72d5&isextractable=false&ispublic=true&isfeatured=false&format=ISO&command=getUploadParamsForIso&response=json
HTTP/1.1"
Jul  2 14:37:40 bllcloudlb01 haproxy[216322]: 172.16.44.18:54094 [02/Jul/2021:14:37:40.491]
ft_cloudstack_gui~ bk_cloudstack_gui/cloudstack_gui_1 0/0/0/164/164 200 994 - - --VN 4/1/0/0/0
0/0 "GET /client/api/?listall=true&isofilter=all&showunique=true&page=1&pagesize=10&command=listIsos&response=json
HTTP/1.1"

Any help would be appreciated.

[1] (https://drive.google.com/file/d/1cIZtMXfpy83etjq8NGdo-x8UVcPDiZZw/view?usp=sharing)
[2] (https://drive.google.com/file/d/1IHDmteKXFjKli0u7rzFQVznIjSw8chD7/view?usp=sharing)
[3] (https://drive.google.com/file/d/1FWyR4Dzo20UNDO887V6tpyIej5o0FRUd/view?usp=sharing)
[4] (https://drive.google.com/file/d/1N1dTIYvxIjTSgSaf56SCmoQlQNou7vNW/view?usp=sharing)
[5] (https://drive.google.com/file/d/1w-gnARJ1XegK5D24c91T3qseU8GDNAPd/view?usp=sharing)

-- 
Thanks,
Joshua Schaeffer


Mime
View raw message