How to Setup Visual Studio (TFS) Test Agents in the Cloud

We ran into some issues trying to get the Visual Studio Test Agents to register and communicate with the Visual Studio Test Controller when the controller is inside our network, and the agents are in the cloud. There are some strange quirks that need to be addressed before this will work. However, following these steps you should be able to run test agents on VMs in Amazon AWS EC2, Microsoft Windows Azure, etc.