Actually, I have figured out the problem. This is probably not a serious bug, BUT, what was causing the crash was the “Version” response that the mysql-interface was returning to SQLYOG. (The piece that may say something like 5.0.32-debian_blahblah). I had my interface returning “test” with no numbers. I would guess that SQLYOG uses this part of the packet to calculate something, and since it did not return any numbers to start, it crashed. Other clients I have tried as mentioned above don't seem to depend on this part of the server greeting (As well as the test-connection piece).
After I have my interface returning 5.0.32-test, it connects without error.