This blog is part 6 of a series
We decided to install Octopus on an internal server this meant that all developers could access Octopus Deploy. But once it was installed on the internal server we ran into a problem. When I had been testing Octopus Deploy on my machine the deployment of packages from the internal Octopus nuget repository to the tentacle had been really fast, just a few seconds. But now the connection speed from the internal server to the tentacle was very slow. When it did work it was taking over 20 minutes to deploy a 13 Mb package, and often the connection would simply time out with the following exception:
Pipefish.PipefishException: The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided. ---> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
I contacted our network administrator who assured me that the problem was with Octopus Deploy. Then I contacted Octopus Deploy support. I had a great experience with Octopus Deploy support, they were really interested in trying to diagnose and fix the problem. I tried many approaches in concert with them including creating a self-signed SSL certificate when we were not sure if the problem was with the certificate being trusted, then we realised this was a “goose hunt” and that it was definitely a connection timeout issue. Then they asked me to try the Octopus ping pong tool: https://github.com/OctopusDeploy/TentaclePing which logs the pings from the Octopus server to the tentacle, and the “pongs” from the tentacle side, listening on a specific port. Unfortunately the results of these ping logs did not shed any light on the problem either.
At this point I decided to try to approach the problem from a different angle. Octopus Deploy is very flexible, there is usually more than one way to approach a problem, which is one of its strengths. In this case I discovered that as well as using the built in nuget repository on the Octopus Deploy server, it is possible to set up your own repository, push the nuget packages there from my development machine, and then configure the tentacle to download the nuget packages straight from this repository. Thereby bypassing the need for the Octopus Deploy server to upload lots of data and avoiding the bottleneck.
Private NuGet Server
Creating our own nuget server was incredibly simple, I simply created a new solution and product named NugetServer, downloaded the NuGet.Server package from NuGet and then published the project. I copied this to the same server that the Octopus Tentacle was installed on. This meant that the tentacle would be able to fetch the nuget package very quickly. Then I Created a new website in IIS with a physical path pointing to this new project. I added a private ApiKey to the web.config of the nuget server, which I would have to pass when pushing packages to the server. Then in my build script (see part 2) I simply had to pass in these properties to the MSBuild task:
Then when I published my solution using the build script the nuget package was pushed to the new nuget repository instead of the Octopus repository.
To link the Octopus Deploy server to this private nuget repository was also easy. I went to the Library => External feeds tab, clicked on the add feed button and entered in the repository url. Then I could select this feed from the NuGet Feed drop down for each step in the deploy process. I also made sure to check the “Each Tentacle will download the package directly from the NuGet server” radio button, otherwise the Octopus Server would download and reupload the package, completely negating the whole exercise.
This way of deploying the NuGet packages solved the slow connection problem which we had, there was no longer an issue with performance. Now the only time I notice the slow connection is when there is an update to Octopus Deploy and it has to upload the update to the tentacle. Also, using an external NuGet repository means that it is not possible to schedule a lifecycle to deploy automatically as soon as a new NuGet package is pushed, we have to do this manually.
Next: less & CSS
Now that the performance issue was solved there was one more hurdle in the way before we could start using Octopus Deploy every day. We use less to compile our css files and have variables which must be changed depending on which website we want to deploy to. Unfortunately the css compilation happens before it is packaged into a nuget package. In the next blog I detail how I wrote a powershell script to compile the css for each website and integrated this into our build script.