libcloud-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "charles walker (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LIBCLOUD-991) Can not retrieve more than 500VM on Google cloud
Date Tue, 10 Apr 2018 17:06:00 GMT

    [ https://issues.apache.org/jira/browse/LIBCLOUD-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16432591#comment-16432591
] 

charles walker commented on LIBCLOUD-991:
-----------------------------------------

Hi,

 

Thx for the feedbacks. I m not sure about "no need to call page()". I made a quick test with
the following code: [https://github.com/charly37/testlibcloud/blob/master/test.py]

and get the following output:

[XXXXXX@XXXXXX test]# python3.6 test.py
Starting
Test1: Not calling page
GCEList init
Page Test 1
nbvmTest1: 500
Test2: calling page
GCEList init
Page Test2
Page Test2
nbvmTest2: 632
Starting Test 3
GCEList init
Page test 3
Page test 3
nbvmTest3: 632

 

Either I miss something or made an error on my code but I think it is require to call page()
when you have more than 500 results. I also come accross this PR which tend to confirms that:
[https://github.com/apache/libcloud/pull/1095/files]

Nevertheless I m able to get my 635 VMs and thus confirms that the 500 limitation can be handle
by using the existing code (with the help of GCEList class) but still wonder if it would be
better to have this hiden in libcloud (especialy since you said it yourself "were about to
add it in a third place!". It is strange that all users need to add this logic on their side
(I would understand that if i was using a specific google lib but libcloud should abstract
this). What do you think ?

Thx again for your feedback and support in resolving/understanding this issue.

Charles.

> Can not retrieve more than 500VM on Google cloud
> ------------------------------------------------
>
>                 Key: LIBCLOUD-991
>                 URL: https://issues.apache.org/jira/browse/LIBCLOUD-991
>             Project: Libcloud
>          Issue Type: Bug
>          Components: Compute
>         Environment: Google cloud project with more than 500VMs (python 3 and last libcloud
version AVL on pip)
>            Reporter: charles walker
>            Priority: Minor
>
> Today I notice that some of our VMs were not retrieve by our internal toolings. After
some investigation it seems that libcloud "list_nodes()" method on google can only retrieve
max 500VMs:
>  
> [root@stackviewer-5cbcd8695-x964t /]# grep -o Vm response.log | wc -l
> 500
>  
> when we have in reality around 750 VMs. So i had a quick look and my first guess is that
it comes from the call made by liblcoud using this REST API:
> [https://cloud.google.com/compute/docs/reference/rest/v1/instances/aggregatedList]
> with an optional argument "maxResults" that has a default value of 500 (which seems also
to be the max value BTW). I think i m on the good lead due to this 500 limit mentioned in
the doc and also observe on my side.
> I will have a deeper look but just quickly open this ticket in case other people already
saw this limit
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message