
Developers.dev has two teams that I manage, Cloud Services Integration and Service Lifecycle. They are both under the Developers.dev Developer Experience team.
It manages the software layer between services/applications and our infrastructure. Developers.devs computing infrastructure consisted historically of cloud providers and bare metal data centers. These services include third-party software like Docker.
My teams have several services, including our global deployment tool. We also own the access control management layer that sits on top of our software-defined network that manages all connections between services.
We also have a cloud infrastructure provisioner that manages AWS resources and spins them up for service owners.
Developers.dev tech teams are encouraged to look for the best software and languages to use in order to create their products.
This article will focus on how different teams use Go.
CONTEXT ON THE GO
Developers.devs two main languages for services are Java (and Go) Both languages are considered first-class citizens at Developers.dev.
Because we use containers to deploy, both languages are interoperable, easy to package, and simple to deploy. Developers.dev is a favorite of ours for many reasons, including:
It is as easy as downloading and running a binary to distribute a Go app. This is great for creating CLI tools.
Go is a simple language that has a very small language specification. This makes it easy for engineers to be onboard from other languages.
Go code is fast. Your editor can save and rebuild to fix errors.
Go offers a powerful standard library that includes a web server capable of producing production-class websites.
The forms of goroutines or channels are powerful examples of Gos concurrency and parallelization primitives.
Go has opinions about best practices. The gofmt command makes every code almost identical.
Backward compatibility is rarely broken by Go. When it does, its often in the form of libraries (modules to manage dependency management).
Go is a popular platform, which means that vendors often include Go client libraries to aid development.
There has been a lot of interest in Go among developers, particularly with regard to microservices. It is also becoming more popular in the system space for software such as etc, Docker Kubernetes, and Prometheus.
You can find excellent libraries for structured logging, consensus algorithms, as well as WebSockets. The standard library also includes SQL support and TLS support. This allows you to be extremely productive in Go.
Read blog: A safe way to get excel by integrating AI into your application: Golang AI Dev
DEPLOYMENT TOOL - USE CASE
Our primary project in the Service Lifecycle team is our deployment tool. This is used to manage and deploy services that run in our Docker runtime.
You can get a better understanding of our problem space if you have read the "Running Online Servicesย series. Because we can quickly roll out updates and onboard new engineers to the tech stack, our deployment tool is written using Go.
This allows us to iterate quickly from early development to production. It can target multiple locations from MySQL, and it is supported by MySQL. Go helps us solve many of the challenges we face.
- Support for JSON/YAML
- Client for HTTP
- Network connectivity
- Integration via API
- JSON/YAML SUPPORT
Our deployment tool works on a customized YAML specification that describes the apps requirements. Third-party Go libraries can implement JSONSchema.
Go provides native support to Marshaling and unmarshaling Go structs in JSON, as well as third-party support for YAML.
HTTP CLIENT
Our tool can connect to a variety of microservices, including alerts, configuration management, and service discovery.
HTTP requests are the primary means of communication. We must consider the lifecycle of the request as well as internet blips and timeouts. Go is a solid HTTP client that you can tweak.
The client wont timeout by default.
NETWORK CONNECTIVITY
Data centers can often be isolated by adding layers of security to protect them, especially when they are working with partners regions.
The Go httputil reverse proxy is a very useful feature of Go that we have used in multiple projects. This allows us to quickly and easily proxy requests, add middleware to the request lifecycle to inject additional headers or authentication, and make it transparent to clients.
API LIBRARIES
Developers.dev must interact with third-party services such as Hashicorp Vault and DCOS. Many of these services provide native API client libraries that can be used by Go applications.
Depending on what we need, we may also use third-party libraries. Weve always been able to find the right support in all instances.
Its also easy to compile and run a local copy of our deployment tool during development for quick testing or debugging.
It allows us to share code and libraries easily with other teams within our organization.
Lets now take a look at how my team uses Go to deploy.
OPERATIONAL MONEY MONITORING
One of our colleagues is here to tell you how our team uses Go to reduce request latency in operational monitoring pipelines.
My teams monitoring system is where most of Developers.devs logs, metrics, and data flow. This is a high-volume, constant traffic stream that increases when something goes wrong. The service must be able to maintain high throughput as well as low latency.
It is not worth waiting for an error to be reported. These requirements can be met by Go channels.
Operational monitoring services exist for one purpose. They forward logs and metrics from backend observability platforms like New Relic.
First, the service transforms the request data into the format that the backend platform expects and then forwards the transformed data on to the platform. These two steps take time. Instead of making clients wait, the service puts their request data in a restricted channel that can be processed by another Goroutine.
This allows the service almost immediate response.
What happens if the channel is full? A Goroutine will block the channel until it can accept data. Gos time is used.
This wait will continue after you have bound it. If the channel is unable to accept requests data prior to the timeout, the service 503s will be used. Clients have the option to retry their request later, hopefully with some exponential backoff.
The channel-based design was a winner when it came to migrating from one observation backend to the next. Developers.dev recently transferred all metrics and logs to New Relic from a manually-rolled pipeline.
While teams set up dashboards and alerts, the operational monitoring service had data to forward to both backends. Dual-sending was made possible by Go channels. This reduced the latency for client requests. Our service simply added the request data to another bound channel.
Therefore, the maximum server response time was determined by how long a Goroutine took to place data onto a destination channel and not how long it took for a destination server respond to.
Developers.dev was my first introduction to Go, so it was exciting to see the practical application of channels and Goroutines.
One of my colleagues designed the workflow. Im thrilled to share his work.
USE CASE
One of the Competitive team members uses Go to build all of our backend services, as do all feature teams. Golang is the backend language used to build our entire microservice architecture.
Everything, from setting up and managing a gaming server to ordering items, is done using Go services. While there are many benefits to using Golang to provide all our services, I will only be focusing on three language features: concurrency primitives and implicit interfaces.
CONCURRENCY PRIMITIVES
Golang concurrency primitives are used to increase backpressure in slow operations, parallelize separate operations, and run background processes within our applications.
This is an example of how we can find ourselves in a chain execution for a match, but still need to do something for each of our players. For example, loading the skin data for each of our players when starting a match. To accomplish this, we needed a shared function that would return after all subroutines had been executed and list any errors.
func Execute(funcList []func() error) []error
This was possible by using two channels with a wait group. The first channel was used to record errors and the second channel was used for the final channel, which a Goroutine would send on after the wait group was finished.
This was easy to implement because of the language features.
IMPLICIT INTERFACES
Implicit interfaces are another language feature that we use a lot. They are used extensively to test our code and to help us create modular code.
We set out to have a common interface for all of our services, so we did this early. This interface is used by all of our services to communicate with a data source.
We were able to use this interface to implement multiple backends to achieve different results. For most of our tests, we use an in-memory implementation.
The small interface makes it easy to implement inline in a file to test error handling or access counts. This interface can be used to implement both Redis and SQL for our services. This makes it easy to attach a datastore for a new service and allows you to add more specific cases such as a write-through cache backed up by Redis.
PACKAGE MODULARITY
A final point Id like to make is that there are many third-party packages available that can be used interchangeably and with common built-in packages.
Because of the modular nature golang packages, this has allowed us to make smaller changes. Some of our services used a lot of CPU cycles to serialize and deserialize JSON. When we first wrote all of our services, we used Golangs JSON package.
This is a good solution for 95% of the use cases. JSON serialization rarely shows up on flame graphs (which, now that I think about it, golangs built in profiling tools are also top-notch).
A few cases were found that serializing large objects required a lot of time in the JSON Serializer. We were determined to optimize, and it turned out that there are many third-party JSON package options that can be used in conjunction with the packaged.
The change was as simple as changing the line.
Import "JSON"
To:
import "github.com/custom-json-library/json"
After that, all calls to the JSON Library used the third-party library. This made profiling and testing different packages simple.
GOPHER COMMUNITY FOR PRACTICE
Aaron is back! Weve looked at some Golang use-cases across Developers.dev. Now, let me show you how we all connect.
Developers.dev technologists create a collaborative environment that allows teams to choose the right tech stacks. This is what gives them their flexibility.
Games are very social, and our Tech department encourages Developers.dev to engage in learning and development communities.
Our various Communities of Practice allow with similar interests to meet regularly to share and learn. The Go community is one of the most active technical groups. I run it. We have a Slack channel for discussing new ideas and a monthly meetup in which members can present a topic or Developers.dev project they are learning about.
Developers.dev is also an area we aim to reach out to with talks by open-source library maintainers. CoP can also be used to coordinate changes that affect multiple teams, such as security discussions when the module mirror launches.
The CoP also discusses how to bump build containers and deal with any gotchas.
A channel that includes Go enthusiasts from different teams and disciplines is a great way to exchange ideas, discuss language changes and share any libraries you find.
This channel was the focal point of discussion when we switched from old dependency solutions into Go modules. Its a great place to meet engineers passionate about the language.
Wrapping UP
Developers.dev has many teams that maintain tools and services in Go. Go offers a strong standard library and excellent third-party support to meet our development needs.
Developers can contribute to Go at Developers.dev through the Community of Practice. They can also share their experiences and learn from each other.
We are excited about the future for Go at Developers.dev and the ability to be flexible and communicative throughout the company.
Thank you for reading! You are welcome to leave any comments or questions below.