Hi Tim,
I have used ballast on the portable layouts I have made just glued to the boards but have relied on the mineral finish on the roofing felt for appearance on the garden railways made. Ballast on a garden railway which looks great can cause maintenance problems, freezing can cause it to break up and if it gets loose rain can cause it to jump up higher than the track. It is also more difficult to clean however a lot of garden railways use it both glued and loose and it does look well. If you intend to run steam locomotives and you can arrange the track maybe only a short length to be high as possible it will help in working on the locomotive and putting bogie stock on the tracks.
I hope this might help.
Tony.