Hey there, it's Baker Searles the Technical Director at Veridian Forge.
For my kick-off post I wanted to talk a bit about the technology we're using as a basis for our company. Like our Creative Director Brad did with his kickoff post, I'll try to include some helpful links.
Goals and Budget
There was a lot to figure out when getting started, but one thing I wanted to focus on was making sure we had a realistic plan for getting the game out the door. Our technology stack needed to allow us to create and share, reliably and rapidly, across time zones.
We spent a lot of time, and still do, thinking about how the software we use fits into our budget. We decided early on that anything that would help us move as quickly as possible would be worth investing in. We evaluate all our software on its “Bang for the Buck.” Sometimes that means learning some new software package that is free, other times it’s parting with some cash to get what’s best.
We have a shared budgeting spreadsheet I put together that auto-updates a lot of information based on current spending patterns, so we always know how much "runway" we have based on several projected scenarios.
Engine
Selecting a game engine was the first technology choice we made. We evaluated multiple engines for the type and style of game we were planning to make, and we settled on the Unreal Engine. We've both had experience with it before, and it seemed to be particularly suited for our genre and the tools were well suited for creating content in the art style we are working towards.
Their royalty system has a very generous setup as well, and absolutely free to immediately get started on.
Version Control
A Version Control system allows multiple developers to work on the same project simultaneously, without stepping on each other's toes.
Selection Process
Our choice of source control, usually a contentious issue, was made a lot simpler by selecting Unreal. They have a very tight integration with Perforce “Helix Core” built into their editor. It handles large binary asset files just fine.
Pricing
It is free to use for up to 5 developers, which is nice for a startup. One thing I’m not crazy about is that there is no clear pricing on their website for once we need more licenses than that. It’s been a while since I’ve been at a studio using Perforce, so I know what it used to cost, but I usually take opaque pricing like this as a red flag.
Private Networking
Not an area of expertise of mine, I was pointed toward ZeroTier as a possible solution by a friend. It is a virtual private network (VPN) that allows you to create a private network for your team, and is very good at just handling tricky network topologies. It's a nice way to just setup on each machine and it just looks like a LAN connection to the machine. They have applications for most platforms, including phones. One nitpick here is that at this time there's no built-in DNS solution, so we connect directly by IP address. I setup an MDNS server on our VM, but it seemed to cause slower connections and occasional timeouts. Future versions may fix this.
But this has worked out extremely well, and was pretty easy to setup.
Server Hosting
By hosting our Perforce server on a cloud-based virtual machine (VM), we can take advantage of scalable and reliable cloud infrastructure, without having to worry about maintaining our own hardware.
Setup
There are lots of great guides on setting up your own server on a Linux machine out there. I have some experience developing software on, and basic management of Linux machines. If you do not, it may make sense to look into the turnkey solutions out there, that come with management services.
Optimizing for our Team
Since our team is spread across time zones, we needed to find out what datacenter was going to work best for all of us. Our initial setup was located in NYC, which was fine for me but pretty slow for Brad.
Testing Datacenters
I created a test matrix across four different cloud providers, in different locations across the U.S., and involving our client computers across 3 cities. I’d spin up a VM in the test datacenter, install an Iperf3 server and ZeroTier, and we’d measure speed across the external IP directly and also across the ZeroTier network. Of note is that at typical internet speeds the ZeroTier encryption overhead was negligible.
Our Selection
We landed on using a Google Cloud Platform (GCP) VM in a warehouse that provided the best combined speed to our home offices. From the perspective of a software engineer with pretty basic IT skills, I thought that the creation and maintenance interfaces I liked the most were GCP and DigitalOcean.
Implementation Details
GCP VM
On GCP we’re using a very basic shared-core machine (e2-micro) with only 1GB of RAM. I created a dual disk setup, where the second drive is setup as a data drive for Perforce depot files. We can expand resources on demand, but this is working quite nicely so far, and is very low cost.
Backups
GCP has a nice built-in feature where you can enable disk snapshots at some interval. I setup snapshots to occur weekly, and it keeps the last 10 disk images around.
It’s important to test your backups, so make sure you do a “dry run” restoring a set of data and make sure it works. The nice part about the whole disk snapshots is you just attach them to a VM and start it up, it’s already configured other than having to join it to the ZeroTier network.
Communications
We're using Google Workspace to provide email services, shared virtual drives, and collaborative Google Docs. We used the Starter package at first, but we bumped up to the Standard set to get access to the configurable shared Drives and much expanded storage. We use Google Drive for Desktop so we each have a G: drive that contains all of our shared drives. The integration with Windows Explorer is pretty great.
We tried several voice/chat packages, and ended up settling on Discord for our everyday online presence and communication. It gets the job done, and has a great phone app. The streaming quality with Discord Nitro is pretty great, although I sure would like the ‘draw on the screen’ functionality of Slack.
We're potentially interested in starting a public Discord server for Veridian Forge, so it's been good to dig in and setup roles and permissions for a bunch of channels and groups for our company server. We setup a playtest section that we used for all of our external prototype playtesting, and it worked great.
Having just used Discord for simple gaming chat prior, I was pleasantly surprised at how much depth there is here.
Planning and Internal Documentation
We use Atlassian's cloud-hosted Jira and Confluence to plan the project and quickly create internal documentation. Both of these tools are offered by Atlassian and are commonly used in the software development industry.
Jira is a project management tool that allows you to track and manage your team's tasks and issues. At our small size it’s perhaps a bit overkill, but their kanban board works quite nicely.
Confluence is a collaboration platform that enables your team to share and collaborate on documents. I really like how easy it is to just throw down documentation quickly, with the formatting being pretty intuitive and rich. The main knock I’ve found on Confluence is the search functionality can be lacking as your content grows, so we’ll need to keep an eye on this.
If you use the build in heading depths, just do a “/table of contents” at the top for a nice index into the page. It’s easy to organize pages hierarchically, and you can add a ‘Children Display’ widget on a page to list out all of the sub-pages at a configurable depth. I’m a fan, although much like Jira we’re not using any deep features.
Build Environment
We're using Visual Studio and JetBrains Rider for development. I've been a Visual Studio die-hard for many years, but I was easily convinced of the utility of JetBrains. It has lightning fast search utilities, and even has an Unreal plugin that allows you to see when properties in your classes are used in Blueprints, which is a nice touch.
The JetBrains toolsets are not the lowest cost options out there, but they really pack a punch. It’s worth noting that they have a hefty discount program for startups. You do have to send them proof of your organization.
Automation
Finally, we have an older machine of mine running Jenkins to automatically build, package, and post the game to a shared Google Drive location. We used Jenkins at a previous workplace and it seemed to get the job done.
Whenever changes are committed to the relevant areas of Perforce, a post-commit hook sends a notification to Jenkins to start a build. When the build is done it will place a zip file of the build onto a shared drive and send a success/failure notification to a designated channel in our discord.
Before we switched to GCP, which has easy VM status notifications, I also had an hourly script that would run on our VM to check how full our disks were and post status to Jenkins. So the percentages would show up on our Build Monitor Panel, and trigger a failure over a threshold.
It is nothing fancy just yet, but it’s an easy way to verify changes on an independent machine. I’d love to add some automated testing to this in the future.
Setup Documentation
I strongly recommend creating documentation of setting up all of the software, step-by-step, with diagrams. This pays for itself pretty quick when others need to get contributing, and even helps when you’re setting up secondary laptops and such.
In particular, having all of the commands used to setup the VM from scratch was very useful and I’ve had to reference it multiple times.
Conclusion
That describes a basic outline of our setup. Feel free to drop me a message if you found any of this useful, or if you think there’s a better choice somewhere along our stack!
Comments