The listener can be removed, but nothing ever does. Right now it can never know if it should be removed. And also, all that level checking is cached in StatusLogger. If all you do is change the status level of the listener it has no effect on the cached value in StatusLogger. It may end up having no effect.
This is some of the stuff I was trying to clean up with my fix that I have been delinquent with.
I will try to simplify this on the branch and see if it makes sense in the next hour or two.