We were busy setting up a TeamCity server and an agent in order to get some continuous integration going. The setup we wanted was:
- Have the TeamCity server setup in the cloud. We are not all working in the same office, so we need to have general access to it.
- Run the agents locally in one or more of our offices. Fast hardware is not that expensive, and machines in the cloud cost money too. And after setting up these local machines, there is basically no need to attend them further.
A colleague had already set up TeamCity on our server, and I was going to set up the first agent locally. I thought that it was going to be simple to do this securely, but unfortunately I hit a few bumps.
The first bump was a minor one. During the installation, I was asked which account the agent was going to run in. I could either specify “system account” (the default), or some specific account. I did not want to use system account, because that would give the agent administrative rights. Specifying another account however, gave an authorization error.
The way to solve this in Windows is rather simple, but you have to know how to do it. Instead of specifying a specific account, you will have to tell the installer to use the system account and let it proceed. Then, after it is finished, you can go to the services control panel and change the logon account for the TeamCity Build Agent service. If you pass the right credentials, you will see a popup that mentions that the account has been given the right to run as a service.
When I fired up the new agent, it did not get a connection to the server. I looked in the logs. It said that the connection was lost. I remembered that the installer asked for an agent port. I looked up the documentation:
It says: “Server should be able to open HTTP connections to the agent”.
I could not find an option to turn this off. I can understand that it is sometimes useful to give the server a direct link to the agent. But in this case, I would have to open up a port in our office to a machine in our own local network. I expected that this was just an option, and that I could turn the connection to the agent off. After all, other web based app’s can also do their thing without opening connections to their clients. The agent would just have to poll for work. But alas, no option.
And then I read: "Please note that by default, these connections are not secured and thus are exposing possibly sensitive data to any third party that can listen to the traffic between the server and the agents. Moreover, since the agent and server can send "commands" to each other an attacker that can send HTTP requests and capture responses can in theory trick agent into executing arbitrary command and perform other actions with a security impact."
The connection to the server can be configured to use https, but for the connection to the agent this option does not exist. All the settings that are configured in the TeamCity web GUI are passed over this link. This includes configured authentication information for the version control system.
In short: in order to do this securely, I would have to set up a VPN between the server and the agent.This however would require changes at our remote server, and if not done properly it might disrupt it. And possibly I might have to configure our local router too, which could cause trouble on our local network. That was a bit too much for an initial setup.
To be able to deal with this, I set up the agent firewall so that it only accepts connections for a single port from our TeamCity server. I also limited outgoing connections to one or two ports on our server. Our internal network is now invisible to the agent machine, and if it gets compromised, there is nowhere to go (assuming an attacker cannot intercept responses to our server).
Also, we avoided the standard TeamCity mechanism for checking out sources. Its server-side checkout might send sources to the agent out in the open, while for its agent-side checkout it might tell the world about some credentials used for our version control system.
Instead, we let the agent check out the sources over https via TortoiseSVN, using locally stored credentials. Sure, if the machine gets compromised over the agent link it may be possible to find them, but given the firewall rules it will be hard to get them out.
And finally, we made sure that the team city agent does not have write access to our version control system. A matter of damage control.
I know that this setup is not ideal, and it may well be that we go for a VPN solution after all. But a VPN raises questions too. Our TeamCity server is basically a “public” machine. If it gets compromised we will not be happy, but it will just be the server. If it were part of a VPN on the other hand, the situation might be worse: via the compromised server it may be possible to easily reach other machines within the VPN and compromise these too.
In any case, what amazes me the most is that a well-known system like TeamCity requires a connection to the agent. Without it, we could just have used https to the server and be done with it. No VPN setup, no special router configuration, no firewall restrictions (or at least less severe ones), and no restrictions on using TeamCity’s check-out mechanism.
It surely would have made things a lot simpler.