cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hean Seng <heans...@gmail.com>
Subject Re: listUsageRecords Update frequency
Date Fri, 13 Nov 2020 18:22:01 GMT
Hi  Nicolas,

i checked both, the data is tally , it has following type  in usage_event:

| NET.IPASSIGN            |

| VOLUME.CREATE           |

| VM.CREATE               |

| NETWORK.OFFERING.ASSIGN |

| VM.START                |

| SG.ASSIGN               |

| VMSNAPSHOT.CREATE       |

| VMSNAPSHOT.ON_PRIMARY



I check the ListUsageRecord ,  it  has usage type of  3, 6, 13 m  , just do
not have type of  type4 and 5 which for network bytes



On Tue, Nov 3, 2020 at 10:03 PM Nicolas Vazquez <
Nicolas.Vazquez@shapeblue.com> wrote:

> Hi Hean,
>
> Looks like no data is being processed at all, the usage server takes the
> raw data from usage_event table and parses it into helper tables (such as
> usage_network). The data from the usage_event at the cloud_usage database
> is copied from the usage_event table on the cloud database. Can you check
> the records from those tables and see if there is a discrepancy? Also, can
> you verify if the usage db.properties is configured properly?
>
>
> Regards,
>
> Nicolas Vazquez
>
> ________________________________
> From: Hean Seng <heanseng@gmail.com>
> Sent: Saturday, October 31, 2020 5:41 AM
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Re: listUsageRecords Update frequency
>
> I think selec the wrong id, but still is return empty set:
>
>
> mysql> select * from usage_network where account_id=59;
>
> Empty set (0.00 sec)
>
> root@mgmt01 usage]# more usage.log | grep  "accountId: 59"
>
> 2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>
> 2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>
> 2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>
> 2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>
> 2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>
> 2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>
> 2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>
> 2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>
> 2020-10-31 02:00:00,854 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 02
>
>
>
>
> Nicolas.Vazquez@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
> @shapeblue
>
>
>
> On Sat, Oct 31, 2020 at 2:57 AM Hean Seng <heanseng@gmail.com> wrote:
>
> > Hi Nicolas
> >
> > The cloudstacl_usage is running .   confirm from log
> >
> > 20-10-30 18:45:11,016 INFO  [cloudstack.quota.QuotaStatementImpl]
> > (Usage-Job-1:null) (logid:) For 5 it is already more than 29 days, will
> > send statement in next cycle
> >
> > 2020-10-30 18:45:11,016 INFO  [cloudstack.quota.QuotaStatementImpl]
> > (Usage-Job-1:null) (logid:) For 10 it is already more than 29 days, will
> > send statement in next cycle
> >
> > 2020-10-30 18:45:11,016 INFO  [cloudstack.quota.QuotaStatementImpl]
> > (Usage-Job-1:null) (logid:) For 11 it is already more than 59 days, will
> > send statement in next cycle
> >
> > 2020-10-30 18:45:11,016 INFO  [cloudstack.quota.QuotaStatementImpl]
> > (Usage-Job-1:null) (logid:) For 12 it is already more than 29 days, will
> > send statement in next cycle
> >
> > 2020-10-30 18:45:11,016 INFO  [cloudstack.quota.QuotaStatementImpl]
> > (Usage-Job-1:null) (logid:) For 13 it is already more than 58 days, will
> > send statement in next cycle
> >
> > 2020-10-30 18:45:11,016 INFO  [cloudstack.quota.QuotaStatementImpl]
> > (Usage-Job-1:null) (logid:) For 14 it is already more than 29 days, will
> > send statement in next cycle
> >
> > 2020-10-30 18:45:11,017 INFO  [cloudstack.quota.QuotaStatementImpl]
> > (Usage-Job-1:null) (logid:) For 23 it is already more than 29 days, will
> > send statement in next cycle
> >
> > 2020-10-30 18:45:11,017 INFO  [cloudstack.quota.QuotaStatementImpl]
> > (Usage-Job-1:null) (logid:) For 24 it is already more than 29 days, will
> > send statement in next cycle
> >
> > 2020-10-30 18:45:11,107 INFO  [cloud.usage.UsageManagerImpl]
> > (Usage-Job-1:null) (logid:) usage job complete
> >
> > ^C
> >
> > It looks like it did not write to usage table as you say:
> >
> > mysql> select * from usage_network where
> > account_id="a40e9892-3051-41b6-baf2-fabe3af06c92" ;
> >
> > Empty set, 1 warning (0.00 sec)
> >
> >
> >
> >
> > Where is process to update the usage helper?
> >
> >
> >
> >
> > On Sat, Oct 31, 2020 at 1:36 AM Nicolas Vazquez <
> > Nicolas.Vazquez@shapeblue.com> wrote:
> >
> >> Hi Hean,
> >>
> >> The listVirtualMachineMetrics show the "current" usage of the virtual
> >> machine, and the usage records display the aggregated information
> (hourly,
> >> daily) per account. If the records are not updating can you please check
> >> the usage server logs if there is any error when the usage jobs run? Can
> >> you also check if data is being added on the helper table for network
> usage
> >> (usage_network table)? The information from this table is summarized
> into
> >> the resulting usage records when the usage jobs run.
> >>
> >>
> >> Regards,
> >>
> >> Nicolas Vazquez
> >>
> >> ________________________________
> >> From: Hean Seng <heanseng@gmail.com>
> >> Sent: Friday, October 30, 2020 4:18 AM
> >> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> >> Subject: Re: listUsageRecords Update frequency
> >>
> >> Hi Nicolas
> >>
> >> Thanks for answering.
> >>
> >> Some questions puzzle me, what is different between  listUsageRecords
> and
> >> listVirtualMachineMetrics , in terms of Network Read and Write data.
> >> (other
> >> then one is per account basis, and another one is per VM basis) ,  the
> >> data
> >> in listVirtualMachineMetrics almost instant update .
> >>
> >> Is listUsageReocrds (type 4 and type 5) only keep metric with Public IP
> ,
> >> and not calculated the SharedNetwork ?
> >>
> >> I facing issue of AdvanceNetwork withSG, and listVirtualMachineMetrics
> >> network sent and receive update almost instant, and listUsageRecords
> never
> >> update , I do not know where to check if there is error when updating
> this
> >> Account.  Or if the account have setting/options to choose if process
> >> usage
> >> by Cloudstack-Usage ?
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Fri, Oct 30, 2020 at 5:55 AM Nicolas Vazquez <
> >> Nicolas.Vazquez@shapeblue.com> wrote:
> >>
> >> > Yes, the usage jobs information is stored in the cloud_usage database
> on
> >> > the usage_job table.
> >> >
> >> > The start_date and end_date columns indicate which usage range was
> >> parsed
> >> > and the start_millis and end_millis the jobs execution start and end
> >> time
> >> > in milliseconds, which can be converted to date.
> >> >
> >> >
> >> > Regards,
> >> >
> >> > Nicolas Vazquez
> >> >
> >> > ________________________________
> >> > From: Hean Seng <heanseng@gmail.com>
> >> > Sent: Thursday, October 29, 2020 4:58 PM
> >> > To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> >> > Subject: Re: listUsageRecords Update frequency
> >> >
> >> > Hi NIC
> >> >
> >> > Do know where can confirm the last run for usage server ?
> >> >
> >> > It seems my counter is not running , my setup is AdvanceZone Security
> >> > Group.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Oct 29, 2020 at 4:34 AM Nicolas Vazquez <
> >> > Nicolas.Vazquez@shapeblue.com> wrote:
> >> >
> >> > > Hi Hean,
> >> > >
> >> > > You are looking at the right configuration, if you set
> >> > > 'usage.stats.job.aggregation.range' = 60 then you only need to
> restart
> >> > the
> >> > > usage server to propagate the changes.
> >> > >
> >> > > Also please check the 'usage.stats.job.exec.time' configuration, you
> >> can
> >> > > set that exec time to a previous time in the day so after the usage
> >> > server
> >> > > restarts, the usage jobs start running, and then the interval
> remains
> >> > > hourly (by the previous configuration)
> >> > >
> >> > >
> >> > > Regards,
> >> > >
> >> > > Nicolas Vazquez
> >> > >
> >> > > ________________________________
> >> > > From: Hean Seng <heanseng@gmail.com>
> >> > > Sent: Wednesday, October 28, 2020 3:10 PM
> >> > > To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> >> > > Subject: listUsageRecords Update frequency
> >> > >
> >> > > Hi
> >> > >
> >> > > I notice the usage record update  once in a day .
> >> > >
> >> > > Is that possible to update the Usage data more frequently? May be
> >> hourly
> >> > > etc.
> >> > >
> >> > > I try to update usage.stats.job.aggregation.range  but seem that is
> >> not I
> >> > > am looking for, it  only affect startdate and enddate of the record,
> >> but
> >> > > not update usage record interval.
> >> > >
> >> > >
> >> > > --
> >> > > Regards,
> >> > > Hean Seng
> >> > >
> >> > > Nicolas.Vazquez@shapeblue.com
> >> > > www.shapeblue.com<http://www.shapeblue.com>
> >> > > 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
> >> > > @shapeblue
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >> > --
> >> > Regards,
> >> > Hean Seng
> >> >
> >> > Nicolas.Vazquez@shapeblue.com
> >> > www.shapeblue.com<http://www.shapeblue.com>
> >> > 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
> >> > @shapeblue
> >> >
> >> >
> >> >
> >> >
> >>
> >> --
> >> Regards,
> >> Hean Seng
> >>
> >> Nicolas.Vazquez@shapeblue.com
> >> www.shapeblue.com<http://www.shapeblue.com>
> >> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
> >> @shapeblue
> >>
> >>
> >>
> >>
> >
> > --
> > Regards,
> > Hean Seng
> >
>
>
> --
> Regards,
> Hean Seng
>


-- 
Regards,
Hean Seng

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message