Improving numerical reproducibility and stability in large-scale numerical simulations on GPUs