cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From motty cruz <motty.c...@gmail.com>
Subject Re: Status: update from 4.4.0 -> 4.4.1 still broken ?
Date Thu, 30 Oct 2014 16:29:34 GMT
Hi Vadim,
I finally was able to upgrade successfully Cloudstack 4.4.0 to Cloudstack
4.4.1. I used this guider:
http://cloudstack-release-notes.readthedocs.org/en/latest/upgrade/upgrade-4.4.html

Download System VMs from here: http://cloudstack.apt-get.eu/systemvm/4.4/
I specifically used this:
http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.1-xen.vhd.bz2

instead of the one suggested in the guide.

Also, I executed the following two command:
yum upgrade cloudstack-management
once that finished successfully
yum update (updates java)

This also failed for me:

nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a >
sysvm.log 2>&1 &

but I run that command any way:

Also I log in to my database server:

ALTER TABLE `snapshot_policy` ADD  `display` TINYINT( 1 ) NOT > NULL
DEFAULT  '1'

restarted management server and all worked.

Hope this help you!
ThanksMotty



On Thu, Oct 30, 2014 at 12:08 AM, Vadim Kimlaychuk <
Vadim.Kimlaychuk@elion.ee> wrote:

> I have tried to do upgrade 2nd time and failed again. The sequence of
> actions is described here:
> http://cloudstack-release-notes.readthedocs.org/en/latest/upgrade/upgrade-4.4.html
>
> Followed instructions until the end. VM upgrade has taken more than 5
> hrs.  The result of update is:
> -----------------
> Stopping and starting 1 secondary storage vm(s)...
> ERROR: Failed to stop secondary storage vm with id 25
>
> Done stopping and starting secondary storage vm(s)
>
> Stopping and starting 1 console proxy vm(s)...
> ERROR: Failed to stop console proxy vm with id 1
>
> Done stopping and starting console proxy vm(s) .
>
> Stopping and starting 1 running routing vm(s)...
> 2
> Done restarting router(s).
> -------------------
>
> After restarting managment server workers are not querying status and
> resources anymore. The last lines at management server log are:
>
> -------------------
>
> 2014-10-30 08:48:10,207 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin UcsManagerImpl
> 2014-10-30 08:48:10,208 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin VpcVirtualRouterElement
> 2014-10-30 08:48:10,208 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin BaremetalPxeManagerImpl
> 2014-10-30 08:48:10,209 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin BaremetalManagerImpl
> 2014-10-30 08:48:10,209 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin ApiDiscoveryServiceImpl
> 2014-10-30 08:48:10,209 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin ManagementServerImpl
> 2014-10-30 08:48:10,211 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin DedicatedResourceManagerImpl
> 2014-10-30 08:48:10,211 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin InternalLoadBalancerElement
> 2014-10-30 08:48:10,211 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin LdapManagerImpl
> 2014-10-30 08:48:10,211 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin ApiRateLimitServiceImpl
> 2014-10-30 08:48:10,211 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin BaremetalDhcpManagerImpl
> 2014-10-30 08:48:10,212 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin ContrailVpcElementImpl
> 2014-10-30 08:48:10,215 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin ContrailElementImpl
> 2014-10-30 08:48:10,215 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin MidoNetElement
> 2014-10-30 08:48:10,215 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin NetscalerElement
> 2014-10-30 08:48:10,216 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin NiciraNvpElement
> 2014-10-30 08:48:10,216 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin OpenDaylightControllerResourceManagerImpl
> 2014-10-30 08:48:10,216 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin OvsElement
> 2014-10-30 08:48:10,216 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin PaloAltoExternalFirewallElement
> 2014-10-30 08:48:10,217 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin VirtualRouterElement
> 2014-10-30 08:48:10,217 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin SspElement
> 2014-10-30 08:48:10,217 DEBUG [c.c.a.ApiServer] (main:null) Discovered
> plugin BigSwitchVnsElement
> 2014-10-30 08:48:10,250 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (NetworkStatsUpdater-1:ctx-ca0c3513) Successfully updated aggregate network
> stats
> 2014-10-30 08:48:10,379 INFO  [c.c.a.ApiServer] (Thread-11:null) ApiServer
> listening on port 8096
> ------------------------------------------
>
> Full log is here: http://pastebin.com/Wpe7wsAV
>
> Log is not growing and changing. Cloudstack console is missing with 404
> error.  I thing VM update should not influence this. Any System VM in fact
> can be just re-created by "killing" from management GUI.
>
> Vadim.
>
>
> -----Original Message-----
> From: Andrija Panic [mailto:andrija.panic@gmail.com]
> Sent: Wednesday, October 29, 2014 10:21 AM
> To: users@cloudstack.apache.org
> Subject: Re: Status: update from 4.4.0 -> 4.4.1 still broken ?
>
> If it makes it any easier - it failed for me, but with no error simply got
> "transaction revoked..." error, no description, nothing...
>
> On 28 October 2014 23:19, motty cruz <motty.cruz@gmail.com> wrote:
>
> > Thanks Andrija,
> > but that did not fixed for me:
> > mysql> describe snapshot_policy;
> >
> +-----------+---------------------+------+-----+---------+----------------+
> > | Field     | Type                | Null | Key | Default | Extra
>   |
> >
> +-----------+---------------------+------+-----+---------+----------------+
> > | id        | bigint(20) unsigned | NO   | PRI | NULL    |
> auto_increment |
> > | uuid      | varchar(40)         | YES  | UNI | NULL    |
>   |
> > | volume_id | bigint(20) unsigned | NO   | MUL | NULL    |
>   |
> > | schedule  | varchar(100)        | NO   |     | NULL    |
>   |
> > | timezone  | varchar(100)        | NO   |     | NULL    |
>   |
> > | interval  | int(4)              | NO   |     | 4       |
>   |
> > | max_snaps | int(8)              | NO   |     | 0       |
>   |
> > | active    | tinyint(1) unsigned | NO   |     | NULL    |
>   |
> > | display   | tinyint(1)          | NO   |     | 1       |
>   |
> >
> +-----------+---------------------+------+-----+---------+----------------+
> > 9 rows in set (0.00 sec)
> >
> > I changed it as suggest:
> > ALTER TABLE  `snapshot_policy` ADD  `display` TINYINT( 1 ) NOT NULL
> > DEFAULT  '1'
> >
> > am I doing something wrong? this is my 3rd attempt to upgrade.failed
> > with the same result.
> >
> > - Please help!
> >
> >
> >
> >
> > On Tue, Oct 28, 2014 at 2:56 PM, Andrija Panic
> > <andrija.panic@gmail.com>
> > wrote:
> >
> > > Motty, here is copy/paste from someone's email here on mailing list:
> > > Maybe check if this is your issue...
> > > I did not manage to upgrade, and this solution came to late for
> > > me... did clean install, since this was test install in my case...
> > >
> > > <thomas.schneider@euskill.com> wrote:
> > > > Hi,
> > > >
> > > > I solved the problem.
> > > >
> > > > The column display from the snapshot _policy was missing after
> update.
> > > >
> > > > Just do: ALTER TABLE  `snapshot_policy` ADD  `display` TINYINT( 1
> > > > ) NOT NULL DEFAULT  '1'
> > > >
> > > > Regards,
> > >
> > > On 28 October 2014 22:13, motty cruz <motty.cruz@gmail.com> wrote:
> > >
> > > > forgot to mentioned about this logs in /etc/cloudstack/sysvm.log
> > > > nohup: ignoring input
> > > >
> > > > Stopping and starting 1 secondary storage vm(s)...
> > > > curl: (7) couldn't connect to host
> > > > ERROR: Failed to stop secondary storage vm with id 15
> > > >
> > > > Done stopping and starting secondary storage vm(s)
> > > >
> > > > Stopping and starting 1 console proxy vm(s)...
> > > > curl: (7) couldn't connect to host
> > > > ERROR: Failed to stop console proxy vm with id 16
> > > >
> > > > Done stopping and starting console proxy vm(s) .
> > > >
> > > > Stopping and starting 1 running routing vm(s)...
> > > > curl: (7) couldn't connect to host
> > > > 2
> > > > Done restarting router(s).
> > > >
> > > >
> > > > Also /etc/cloudstack/cloud.log:
> > > > [2014.10.28-13.48.43] Stopping and starting 1 secondary storage
> > vm(s)...
> > > > [2014.10.28-13.48.43] INFO: Stopping secondary storage vm with id
> > > > 15 [2014.10.28-13.48.43] ERROR: Failed to stop secondary storage
> > > > vm with
> > id
> > > 15
> > > > [2014.10.28-13.48.43] Done stopping and starting secondary storage
> > vm(s).
> > > > [2014.10.28-13.48.43] Stopping and starting 1 console proxy vm(s)...
> > > > [2014.10.28-13.48.43] INFO: Stopping console proxy with id 16
> > > > [2014.10.28-13.48.43] ERROR: Failed to stop console proxy vm with
> > > > id 16 [2014.10.28-13.48.43] Done stopping and starting console proxy
> vm(s) .
> > > > [2014.10.28-13.48.43] Stopping and starting 1 running routing
> vm(s)...
> > > > [2014.10.28-13.48.43] INFO: Restarting router with id 17
> > > > [2014.10.28-13.48.43] ERROR: Failed to restart domainRouter with
> > > > id 17; unable to submit the job [2014.10.28-13.48.53] Done
> > > > restarting router(s).
> > > >
> > > > Thanks
> > > > -Motty
> > > >
> > > > On Tue, Oct 28, 2014 at 2:08 PM, motty cruz <motty.cruz@gmail.com>
> > > wrote:
> > > >
> > > > > Thanks Vadim,
> > > > > I was able to restore back to ACS 4.4.0. upgraded again with the
> > > > > same
> > > bad
> > > > > result. has anybody successsfully upgrade from ACS 4.4.0 to
> > ACS-4.4.1?
> > > if
> > > > > yes do you mind sharing notes?
> > > > > I had used this URL:
> > > > >
> > > >
> > >
> > http://cloudstack-release-notes.readthedocs.org/en/latest/upgrade/upgr
> > ade-4.4.html
> > > > >
> > > > > here is my logs:
> > > > > upgradeACS4.4.1-http-status-404
> > > > > *http://pastebin.com/raw.php?i=eeYQsVyD
> > > > > <http://pastebin.com/raw.php?i=eeYQsVyD>*
> > > > >
> > > > > *I really apreciate if someone can help me! * *in /etc/sudoers*
> > > > >
> > > > > *cloud ALL=NOPASSWD : ALL*
> > > > >
> > > > > *my suspecion, the issue have to do with database but am not
> > > > > certain
> > > and
> > > > > won't know how to fix it. *
> > > > >
> > > > > *Thanks, *
> > > > > *-Motty*
> > > > >
> > > > > On Tue, Oct 28, 2014 at 12:30 AM, Vadim Kimlaychuk <
> > > > > Vadim.Kimlaychuk@elion.ee> wrote:
> > > > >
> > > > >> Motty,
> > > > >>
> > > > >>         I got 404 after downgrading as well, but this was
> > > > >> database
> > > > issue.
> > > > >> You need to restore your DB first and then restart your
> > > > >> cloudstack management server.  Better way (from my point of
> > > > >> view) - having DB
> > and
> > > > >> Management servers as VMs and restore them from snapshots. It
> > > > >> is
> > much
> > > > >> faster :)
> > > > >>
> > > > >>         I will try to upgrade once again as Rohit Yadav suggested.
> > > > >> Didn't think about old VM-s running at hypervisors. They were
> > > > >> not
> > > > upgraded
> > > > >> that is true.  Will keep in mind issue 7781 pointed by Nux!
> > > > >>         Thank you all. Be back after update.
> > > > >>
> > > > >> Vadim.
> > > > >>
> > > > >> -----Original Message-----
> > > > >> From: motty cruz [mailto:motty.cruz@gmail.com]
> > > > >> Sent: Tuesday, October 28, 2014 12:39 AM
> > > > >> To: users@cloudstack.apache.org
> > > > >> Subject: Re: Status: update from 4.4.0 -> 4.4.1 still broken
?
> > > > >>
> > > > >> Nux,
> > > > >> I just finished downgrading back to 4.4.0. but the error I get
is:
> > > > >> HTTP STATUS 404
> > > > >> type status report
> > > > >>
> > > > >> I will try again to upgrade as soon as is fully restore back
to
> > > 4.4.0. I
> > > > >> am running XenServer 6.2 hypervison with Centos 6.5 as manager.
> > > > >> has
> > > > anybody
> > > > >> run into this issue?
> > > > >>
> > > > >> Vadim, I don't meant to hijack your post.
> > > > >>
> > > > >> Thanks,
> > > > >> Motty
> > > > >>
> > > > >> On Mon, Oct 27, 2014 at 3:09 PM, Nux! <nux@li.nux.ro> wrote:
> > > > >>
> > > > >> > Motty,
> > > > >> >
> > > > >> > Please provide DEBUG logs from management server and
> hypervisors.
> > > > >> > It's possible you may be hitting this (as I remember similar
> > > symptoms)
> > > > >> > https://issues.apache.org/jira/browse/CLOUDSTACK-7781
> > > > >> >
> > > > >> > Lucian
> > > > >> >
> > > > >> > --
> > > > >> > Sent from the Delta quadrant using Borg technology!
> > > > >> >
> > > > >> > Nux!
> > > > >> > www.nux.ro
> > > > >> >
> > > > >> > ----- Original Message -----
> > > > >> > > From: "motty cruz" <motty.cruz@gmail.com>
> > > > >> > > To: users@cloudstack.apache.org
> > > > >> > > Sent: Monday, 27 October, 2014 20:39:54
> > > > >> > > Subject: Re: Status: update from 4.4.0 -> 4.4.1
still broken ?
> > > > >> >
> > > > >> > > Hello Vadim,
> > > > >> > > did you find the solution?
> > > > >> > >
> > > > >> > > I am having similar issue. I am running CentOS 6.5
> > > > >> > > "nohup: ignoring input
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > Stopping and starting 1 secondary storage vm(s)...
> > > > >> > >
> > > > >> > > curl: (56) Failure when receiving data from the peer
> > > > >> > >
> > > > >> > > ERROR: Failed to stop secondary storage vm with id
14
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > Done stopping and starting secondary storage vm(s)
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > Stopping and starting 1 console proxy vm(s)...
> > > > >> > >
> > > > >> > > curl: (7) couldn't connect to host
> > > > >> > >
> > > > >> > > ERROR: Failed to stop console proxy vm with id 13
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > Done stopping and starting console proxy vm(s) .
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > Stopping and starting 1 running routing vm(s)...
> > > > >> > >
> > > > >> > > curl: (7) couldn't connect to host
> > > > >> > >
> > > > >> > > 2
> > > > >> > >
> > > > >> > > Done restarting router(s)."
> > > > >> > >
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > >
> > > > >> > > Motty
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Mon, Oct 27, 2014 at 8:52 AM, Rohit Yadav
> > > > >> > > <rohit.yadav@shapeblue.com>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > >> Hi Vadim,
> > > > >> > >>
> > > > >> > >> See my reply in-line;
> > > > >> > >>
> > > > >> > >> > On 27-Oct-2014, at 8:46 pm, Vadim Kimlaychuk
> > > > >> > <Vadim.Kimlaychuk@Elion.ee>
> > > > >> > >> wrote:
> > > > >> > >> >
> > > > >> > >> > Yes, I did follow this guide.  APT does not
have
> > > > >> > >> > pre-compiled packages
> > > > >> > >> so I have compiled them by myself. This is the
only
> difference.
> > > > >> > >> >
> > > > >> > >> > cloudstack-sysvmadm  script that is mentioned
in the
> > > > >> > >> > last
> > step
> > > > >> > >> > has the
> > > > >> > >> first line as follows:
> > > > >> > >> >
> > > > >> > >> > . /etc/rc.d/init.d/functions
> > > > >> > >> >
> > > > >> > >> > But there is no /etc/rc.d/  folder at my Ubuntu
server.
> > Script
> > > > >> > >> > failed
> > > > >> > at
> > > > >> > >> the very beginning so I haven't waited until VMs
are going
> > > > >> > >> to
> > > > >> migrate.
> > > > >> > >>
> > > > >> > >> You’re correct. This is supposed to be CentOS/RHEL/Fedora
> > > specific.
> > > > >> > >> I think your upgrade went fine it’s just this
script that
> > > > >> > >> may
> > > need
> > > > >> > >> to be fixed. Can you create a ticket on CloudStack’s
JIRA?
> > > > >> > >> As a workaround you may try to remove this line
in the
> > > > >> > >> script, make
> > > sure
> > > > >> > >> “curl" is
> > > > >> > installed on
> > > > >> > >> Ubuntu and try again.
> > > > >> > >>
> > > > >> > >> > I have downgraded back to 4.4.0 until DEB
packages will
> > > > >> > >> > be
> > > > >> released.
> > > > >> > >> Problem may be there though I have no errors building
CS
> > > > >> > >> from
> > the
> > > > >> > source.
> > > > >> > >>
> > > > >> > >> I think you should be able to upgrade; until sysvmadm
> > > > >> > >> script is fixed so that it works on Debian based
distro,
> > > > >> > >> after the
> > upgrade
> > > > >> > >> you may reboot
> > > > >> > or
> > > > >> > >> kill the existing/old 4.4.0 based systemvms using
> > > > >> > >> something
> > like
> > > a
> > > > >> > custom
> > > > >> > >> bash/python script or a tool like CloudMonkey or
manually
> > > > >> > >> if
> > the
> > > > >> > >> deploy size is small.
> > > > >> > >>
> > > > >> > >> Regards.
> > > > >> > >>
> > > > >> > >> >
> > > > >> > >> > Vadim.
> > > > >> > >> >
> > > > >> > >> > -----Original Message-----
> > > > >> > >> > From: Rohit Yadav [mailto:rohit.yadav@shapeblue.com]
> > > > >> > >> > Sent: Monday, October 27, 2014 5:00 PM
> > > > >> > >> > To: users@cloudstack.apache.org
> > > > >> > >> > Subject: Re: Status: update from 4.4.0 ->
4.4.1 still
> > > > >> > >> > broken
> > ?
> > > > >> > >> >
> > > > >> > >> > Hi Vadim,
> > > > >> > >> >
> > > > >> > >> > Did you follow the upgrade instructions (especially
the
> > > systemvm
> > > > >> > >> upgrading):
> > > > >> > >> >
> > > > >> > >>
> > > > >> >
> > > > http://cloudstack-release-notes.readthedocs.org/en/4.4.1/upgrade/u
> > > > pgra
> > > > >> > de-4.4.html
> > > > >> > >> >
> > > > >> > >> > If consoles are not coming up, please upgrade
systemvms
> > > > >> > >> > to
> > > 4.4.1
> > > > >> > >> > ones
> > > > >> > >> and delete existing console proxy VM(s) and retry.
> > > > >> > >> >
> > > > >> > >> > Any error log would help as well.
> > > > >> > >>
> > > > >> > >> Regards,
> > > > >> > >> Rohit Yadav
> > > > >> > >> Software Architect, ShapeBlue M. +91 88 262 30892
|
> > > > >> > >> rohit.yadav@shapeblue.com
> > > > >> > >> Blog: bhaisaab.org | Twitter: @_bhaisaab
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> Find out more about ShapeBlue and our range of
CloudStack
> > related
> > > > >> > services
> > > > >> > >>
> > > > >> > >> IaaS Cloud Design & Build<
> > > > >> > >> http://shapeblue.com/iaas-cloud-design-and-build//>
> > > > >> > >> CSForge – rapid IaaS deployment
> > > > >> > >> framework<http://shapeblue.com/csforge/
> > > > >> > >
> > > > >> > >> CloudStack Consulting<
> > > http://shapeblue.com/cloudstack-consultancy/
> > > > >
> > > > >> > >> CloudStack Infrastructure Support<
> > > > >> > >> http://shapeblue.com/cloudstack-infrastructure-support/>
> > > > >> > >> CloudStack Bootcamp Training Courses<
> > > > >> > >> http://shapeblue.com/cloudstack-training/>
> > > > >> > >>
> > > > >> > >> This email and any attachments to it may be confidential
> > > > >> > >> and
> > are
> > > > >> > intended
> > > > >> > >> solely for the use of the individual to whom it
is addressed.
> > Any
> > > > >> > >> views
> > > > >> > or
> > > > >> > >> opinions expressed are solely those of the author
and do
> > > > >> > >> not necessarily represent those of Shape Blue Ltd
or
> > > > >> > >> related
> > > companies.
> > > > >> > >> If you are not
> > > > >> > the
> > > > >> > >> intended recipient of this email, you must neither
take
> > > > >> > >> any
> > > action
> > > > >> > >> based upon its contents, nor copy or show it to
anyone.
> > > > >> > >> Please contact the
> > > > >> > sender
> > > > >> > >> if you believe you have received this email in
error.
> > > > >> > >> Shape
> > Blue
> > > > >> > >> Ltd is
> > > > >> > a
> > > > >> > >> company incorporated in England & Wales. ShapeBlue
> > > > >> > >> Services
> > India
> > > > >> > >> LLP
> > > > >> > is a
> > > > >> > >> company incorporated in India and is operated under
> > > > >> > >> license
> > from
> > > > >> > >> Shape
> > > > >> > Blue
> > > > >> > >> Ltd. Shape Blue Brasil Consultoria Ltda is a company
> > incorporated
> > > > >> > >> in
> > > > >> > Brasil
> > > > >> > >> and is operated under license from Shape Blue Ltd.
> > > > >> > >> ShapeBlue SA
> > > Pty
> > > > >> > >> Ltd
> > > > >> > is
> > > > >> > >> a company registered by The Republic of South Africa
and
> > > > >> > >> is
> > > traded
> > > > >> > >> under license from Shape Blue Ltd. ShapeBlue is
a
> > > > >> > >> registered
> > > > >> trademark.
> > > > >> > >>
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > --
> > > > >> > > Thanks for your support,
> > > > >> > > Motty
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Thanks for your support,
> > > > >> Motty
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks for your support,
> > > > > Motty
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks for your support,
> > > > Motty
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > > --------------------------------------
> > >   http://admintweets.com
> > > --------------------------------------
> > >
> >
> >
> >
> > --
> > Thanks for your support,
> > Motty
> >
>
>
>
> --
>
> Andrija Panić
> --------------------------------------
>   http://admintweets.com
> --------------------------------------
>



-- 
Thanks for your support,
Motty

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message