Wow! Just had to share this for all of you struggling with the RGBDToolkit calibration process.
I just remounted my depth sensor to my camera, and re-ran the correspondence calibration. It failed completely - a total fail. But then I hit Regenerate RGB/Depth correspondence
again, without changing anything - and I got this really solid-looking calibration!

It's weird - same calibration button, same data. James had told me a while back there was a little bit of randomness in the algorithm, and I thought that meant it could go from not-so-good to good-ish. But clearly it can go from fail to great! Some hope for those struggling.

Here is a picture so you get some sense of the light condition...