Even without a vise, you can still just mount your backing punch in a block of wood by drilling an appropriate sized hole in the block.
On the other hand, it is only YOU that needs to be happy with your end results, so it really is best to do it exactly the way that you prefer!