Fast Exact Bayesian Inference for High-Dimensional Models