cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas Vazquez <Nicolas.Vazq...@shapeblue.com>
Subject Re: listUsageRecords Update frequency
Date Tue, 03 Nov 2020 14:03:17 GMT
Hi Hean,

Looks like no data is being processed at all, the usage server takes the raw data from usage_event
table and parses it into helper tables (such as usage_network). The data from the usage_event
at the cloud_usage database is copied from the usage_event table on the cloud database. Can
you check the records from those tables and see if there is a discrepancy? Also, can you verify
if the usage db.properties is configured properly?


Regards,

Nicolas Vazquez

________________________________
From: Hean Seng <heanseng@gmail.com>
Sent: Saturday, October 31, 2020 5:41 AM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: listUsageRecords Update frequency

I think selec the wrong id, but still is return empty set:


mysql> select * from usage_network where account_id=59;

Empty set (0.00 sec)

root@mgmt01 usage]# more usage.log | grep  "accountId: 59"

2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0

2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
0; curABR: 0; curABW: 0; ubr: 0; ubw: 0

2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0

2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
0; curABR: 0; curABW: 0; ubr: 0; ubw: 0

2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0

2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
0; curABR: 0; curABW: 0; ubr: 0; ubw: 0

2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0

2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
0; curABR: 0; curABW: 0; ubr: 0; ubw: 0

2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0

2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
0; curABR: 0; curABW: 0; ubr: 0; ubw: 0

2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0

2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
0; curABR: 0; curABW: 0; ubr: 0; ubw: 0

2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0

2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
0; curABR: 0; curABW: 0; ubr: 0; ubw: 0

2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0

2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
0; curABR: 0; curABW: 0; ubr: 0; ubw: 0

2020-10-31 02:00:00,854 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0

2020-10-31 02




Nicolas.Vazquez@shapeblue.comĀ 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
  
 

On Sat, Oct 31, 2020 at 2:57 AM Hean Seng <heanseng@gmail.com> wrote:

> Hi Nicolas
>
> The cloudstacl_usage is running .   confirm from log
>
> 20-10-30 18:45:11,016 INFO  [cloudstack.quota.QuotaStatementImpl]
> (Usage-Job-1:null) (logid:) For 5 it is already more than 29 days, will
> send statement in next cycle
>
> 2020-10-30 18:45:11,016 INFO  [cloudstack.quota.QuotaStatementImpl]
> (Usage-Job-1:null) (logid:) For 10 it is already more than 29 days, will
> send statement in next cycle
>
> 2020-10-30 18:45:11,016 INFO  [cloudstack.quota.QuotaStatementImpl]
> (Usage-Job-1:null) (logid:) For 11 it is already more than 59 days, will
> send statement in next cycle
>
> 2020-10-30 18:45:11,016 INFO  [cloudstack.quota.QuotaStatementImpl]
> (Usage-Job-1:null) (logid:) For 12 it is already more than 29 days, will
> send statement in next cycle
>
> 2020-10-30 18:45:11,016 INFO  [cloudstack.quota.QuotaStatementImpl]
> (Usage-Job-1:null) (logid:) For 13 it is already more than 58 days, will
> send statement in next cycle
>
> 2020-10-30 18:45:11,016 INFO  [cloudstack.quota.QuotaStatementImpl]
> (Usage-Job-1:null) (logid:) For 14 it is already more than 29 days, will
> send statement in next cycle
>
> 2020-10-30 18:45:11,017 INFO  [cloudstack.quota.QuotaStatementImpl]
> (Usage-Job-1:null) (logid:) For 23 it is already more than 29 days, will
> send statement in next cycle
>
> 2020-10-30 18:45:11,017 INFO  [cloudstack.quota.QuotaStatementImpl]
> (Usage-Job-1:null) (logid:) For 24 it is already more than 29 days, will
> send statement in next cycle
>
> 2020-10-30 18:45:11,107 INFO  [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) usage job complete
>
> ^C
>
> It looks like it did not write to usage table as you say:
>
> mysql> select * from usage_network where
> account_id="a40e9892-3051-41b6-baf2-fabe3af06c92" ;
>
> Empty set, 1 warning (0.00 sec)
>
>
>
>
> Where is process to update the usage helper?
>
>
>
>
> On Sat, Oct 31, 2020 at 1:36 AM Nicolas Vazquez <
> Nicolas.Vazquez@shapeblue.com> wrote:
>
>> Hi Hean,
>>
>> The listVirtualMachineMetrics show the "current" usage of the virtual
>> machine, and the usage records display the aggregated information (hourly,
>> daily) per account. If the records are not updating can you please check
>> the usage server logs if there is any error when the usage jobs run? Can
>> you also check if data is being added on the helper table for network usage
>> (usage_network table)? The information from this table is summarized into
>> the resulting usage records when the usage jobs run.
>>
>>
>> Regards,
>>
>> Nicolas Vazquez
>>
>> ________________________________
>> From: Hean Seng <heanseng@gmail.com>
>> Sent: Friday, October 30, 2020 4:18 AM
>> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
>> Subject: Re: listUsageRecords Update frequency
>>
>> Hi Nicolas
>>
>> Thanks for answering.
>>
>> Some questions puzzle me, what is different between  listUsageRecords and
>> listVirtualMachineMetrics , in terms of Network Read and Write data.
>> (other
>> then one is per account basis, and another one is per VM basis) ,  the
>> data
>> in listVirtualMachineMetrics almost instant update .
>>
>> Is listUsageReocrds (type 4 and type 5) only keep metric with Public IP ,
>> and not calculated the SharedNetwork ?
>>
>> I facing issue of AdvanceNetwork withSG, and listVirtualMachineMetrics
>> network sent and receive update almost instant, and listUsageRecords never
>> update , I do not know where to check if there is error when updating this
>> Account.  Or if the account have setting/options to choose if process
>> usage
>> by Cloudstack-Usage ?
>>
>>
>>
>>
>>
>>
>> On Fri, Oct 30, 2020 at 5:55 AM Nicolas Vazquez <
>> Nicolas.Vazquez@shapeblue.com> wrote:
>>
>> > Yes, the usage jobs information is stored in the cloud_usage database on
>> > the usage_job table.
>> >
>> > The start_date and end_date columns indicate which usage range was
>> parsed
>> > and the start_millis and end_millis the jobs execution start and end
>> time
>> > in milliseconds, which can be converted to date.
>> >
>> >
>> > Regards,
>> >
>> > Nicolas Vazquez
>> >
>> > ________________________________
>> > From: Hean Seng <heanseng@gmail.com>
>> > Sent: Thursday, October 29, 2020 4:58 PM
>> > To: users@cloudstack.apache.org <users@cloudstack.apache.org>
>> > Subject: Re: listUsageRecords Update frequency
>> >
>> > Hi NIC
>> >
>> > Do know where can confirm the last run for usage server ?
>> >
>> > It seems my counter is not running , my setup is AdvanceZone Security
>> > Group.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Oct 29, 2020 at 4:34 AM Nicolas Vazquez <
>> > Nicolas.Vazquez@shapeblue.com> wrote:
>> >
>> > > Hi Hean,
>> > >
>> > > You are looking at the right configuration, if you set
>> > > 'usage.stats.job.aggregation.range' = 60 then you only need to restart
>> > the
>> > > usage server to propagate the changes.
>> > >
>> > > Also please check the 'usage.stats.job.exec.time' configuration, you
>> can
>> > > set that exec time to a previous time in the day so after the usage
>> > server
>> > > restarts, the usage jobs start running, and then the interval remains
>> > > hourly (by the previous configuration)
>> > >
>> > >
>> > > Regards,
>> > >
>> > > Nicolas Vazquez
>> > >
>> > > ________________________________
>> > > From: Hean Seng <heanseng@gmail.com>
>> > > Sent: Wednesday, October 28, 2020 3:10 PM
>> > > To: users@cloudstack.apache.org <users@cloudstack.apache.org>
>> > > Subject: listUsageRecords Update frequency
>> > >
>> > > Hi
>> > >
>> > > I notice the usage record update  once in a day .
>> > >
>> > > Is that possible to update the Usage data more frequently? May be
>> hourly
>> > > etc.
>> > >
>> > > I try to update usage.stats.job.aggregation.range  but seem that is
>> not I
>> > > am looking for, it  only affect startdate and enddate of the record,
>> but
>> > > not update usage record interval.
>> > >
>> > >
>> > > --
>> > > Regards,
>> > > Hean Seng
>> > >
>> > > Nicolas.Vazquez@shapeblue.com
>> > > www.shapeblue.com<http://www.shapeblue.com>
>> > > 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
>> > > @shapeblue
>> > >
>> > >
>> > >
>> > >
>> >
>> > --
>> > Regards,
>> > Hean Seng
>> >
>> > Nicolas.Vazquez@shapeblue.com
>> > www.shapeblue.com<http://www.shapeblue.com>
>> > 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
>> > @shapeblue
>> >
>> >
>> >
>> >
>>
>> --
>> Regards,
>> Hean Seng
>>
>> Nicolas.Vazquez@shapeblue.com
>> www.shapeblue.com<http://www.shapeblue.com>
>> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
>> @shapeblue
>>
>>
>>
>>
>
> --
> Regards,
> Hean Seng
>


--
Regards,
Hean Seng
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message