Metal supply houses typically offer a shim stock assortment pack that while not cheap can prove to be a bargain in the long run. I think K&S, the hobbyist's supplier for telescoping brass, aluminum and copper tubing, also offering various flat and angle stock, does offer an assortment of brass pieces of various thicknesses, though not as comprehensive nor inclusive as a regular machinist's set of shim stock, yet much cheaper. Think I'd look at what K&S has to offer first.
On another note, it may be possible to "clamp" the shim stock into the bearing by having part of it "pinched" into the joint so that it cannot move with the shaft's rotation.
https://ksmetals.com/https://ksmetals.com/products/brshim