My mini outdoor table layout was elevated (track ~30" high), but I was still surprised how little dust there was from the track ballast or environment. The ballast was held in place using diluted (~1:1) Tite-Bond waterproof glue. When I dismantled the layout, the ballast was well-adhered to the layout table. Rock-solid. The track actually broke/bent before the glue gave way. I wonder if you glued your ballast matrix between the sleepers as well as an inch or two (or more) from the outer edges of the track, if that would reduce the dust in the immediate vicinity of the track. It would be easy enough to test in a limited area. Tite-bond does change the color of the ballast. In my case, the gray granite acquired a red tint.
There are also some bicycle lubricants that might be worth looking into. Some are called 'dry.' I used one brand (I forget which) on my former commuter bike and the chain stayed cleaner longer. I think it had an alcohol carrier that evaporated shortly after application leaving the lubricant behind.