Napoleon bar end mirror problems

I had to remove my bar end mirror (Napoleon brand) to change the grips on my bike. I should note that the mirror has always been on very tight, and not moved around at all. However, upon reinstalling the mirror, I am now having trouble with it. When I crank down the bolt to get it tight, the rubber block starts pushing out of the end of the bar and the mirror works itself out of the bar after a couple minutes. Just wondering if anyone else has had this problem.... and what they used to make it stay put. I was thinking about putting some threadlocker on the rubber block and letting that set up inside the handlebar for a while before trying to tighten it... and ideas?

...Well, for anyone who might stumble across this, what I ended up doing was replacing the expanding rubber part of the mirror with a very thick piece of hose that was closer to the same I.D. of the handlerbars. I put sandpaper on a dowel and scuffed up the inside of the bar first too. I think that the rubber on the mirror was just kinda worn out, if that makes any sense...