I'm afraid I haven't had an opportunity to test with Mac OS X 10.8.5.
How are you connected to the NXT? Right through the USB cable? Do you get an error message, and if not, what happens?
Do you happen to have any other software that will connect to the NXT that you might try? (If you don't have LEGO's NXT-G software, how about trying out NeXT Tools