Okay, so I gave a bit more elbow grease getting the pulley off. Then I cleaned it up a bit more, though the real issue is the bore on the bracket. What I did do is have the pulley near the end of the shaft and used red (medium) Loctite to keep it on/from spinning on the shaft.