I remember one of my first major production outages. We had a high-throughput microservice written in Go that was processing critical financial transactions. Every time we deployed a new version, the deployment script would simply kill the existing process to make room for the new one. Because we hadn't implemented a way to listen for termination signals, the service would die mid-flight. We'd end up with orphaned database records, half-written files, and a whole lot of manual reconciliation work on Monday morning.

That was my hard lesson in graceful shutdown. In a distributed system, you can't just pull the plug. You have to tell your service, "Hey, we're shutting down. Finish what you're doing, close your connections, and then exit gracefully."

The Problem: The Brutal SIGKILL

When you deploy code via Kubernetes, Docker, or even just a systemd service, the orchestration layer eventually sends a signal to your process. Usually, it starts with SIGTERM (a polite request to stop) and, if your process doesn't exit within a certain grace period, it follows up with SIGKILL (an immediate, unblockable termination).

If your code doesn't catch that SIGTERM, it's game over. Any goroutines currently processing requests, any open TCP connections, and any un-flushed buffers are lost instantly. This leads to data corruption and a poor user experience.

The Solution: Listening for Signals

Go makes this incredibly straightforward using the os/signal package. The pattern involves creating a buffered channel to receive OS signals and then blocking your main function until a signal arrives. Once the signal is received, you trigger your cleanup logic.

Here is a production-ready pattern I use for most of my web services. It handles both the HTTP server shutdown and a simulated background worker.

package main

import (
	"context"
	"errors"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	// 1. Create a channel to listen for OS signals.
	// It must be buffered as per the os/signal documentation.
	sigChan := make(chan os.Signal, 1)

	// 2. Register the signals we want to intercept.
	// SIGINT is Ctrl+C, SIGTERM is the standard termination signal.
	signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)

	// 3. Initialize our server/components
	server := &http.Server{
		Addr: ":8080",
		Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			time.Sleep(2 * time.Second) // Simulate heavy work
			w.Write([]byte("Transaction complete"))
		}),
	}

	// Start the server in a separate goroutine so it doesn't block main.
	go func() {
		log.Println("Starting server on :8080")
		if err := server.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
			log.Fatalf("ListenAndServe error: %v", err)
		}
	}()

	// 4. Block here until we receive a signal
	sig := <-sigChan
	log.Printf("Received signal: %v. Starting graceful shutdown...\n", sig)

	// 5. Create a context with a timeout for the shutdown process.
	// We don't want to wait forever if a connection is hung.
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()

	// 6. Shutdown the HTTP server.
	// This will stop accepting new requests and wait for active ones to finish.
	if err := server.Shutdown(ctx); err != nil {
		log.Printf("Server forced to shutdown: %v", err)
	} else {
		log.Println("Server exited gracefully")
	}

	// 7. Perform any other cleanup (DB connections, file handles, etc.)
	cleanupResources()

	log.Println("Shutdown complete.")
}

func cleanupResources() {
	log.Println("Closing database connections...")
	// Simulate cleanup work
	time.Sleep(500 * time.Millisecond)
}

Breaking Down the Pattern

There are a few critical details in the code above that separate a "toy" implementation from a production one:

  • The Buffered Channel: Always use a buffered channel of size 1 (make(chan os.Signal, 1)). The signal.Notify function does not block when sending signals; if your channel isn't buffered and the receiver isn't ready at the exact millisecond the signal arrives, the signal could be lost.
  • http.ErrServerClosed: When you call server.Shutdown(), the ListenAndServe() method immediately returns an error. This error is http.ErrServerClosed. You must check for this specifically, otherwise, your logs will be flooded with "errors" that are actually part of the normal shutdown flow.
  • The Shutdown Timeout: This is the most important part. You must use a context.WithTimeout. If a client has a connection open that hangs indefinitely, your service will hang forever during deployment, eventually getting SIGKILL'ed by Kubernetes anyway. The timeout ensures that you give your tasks a fair chance, but you ultimately draw the line.
Pro Tip: If you have complex background workers (like a Kafka consumer or a worker pool), don't just shut down the HTTP server. Use a sync.WaitGroup to track your background goroutines and wait for them to finish inside your shutdown logic.

Why This Matters for Scale

When you're running a single instance, it might not seem critical. But in a Kubernetes environment with 50 replicas, rolling updates happen constantly. Every time you push code, 10-20% of your fleet is cycling through these shutdown phases.

If your shutdown isn't graceful, you'll see spikes in 5xx errors during every deployment. By implementing this pattern, your load balancer stops sending new traffic to the terminating pod, the pod finishes its current work, and the transition becomes invisible to the end user. That's the difference between a junior implementation and a senior-level architecture.

Summary Checklist

  1. Use os/signal to catch SIGTERM and SIGINT.
  2. Run your blocking listeners (like ListenAndServe) in a separate goroutine.
  3. Implement a context.WithTimeout to prevent the shutdown from hanging forever.
  4. Handle http.ErrServerClosed to keep your logs clean.
  5. Clean up non-HTTP resources (DB, Redis, Files) after the server has stopped accepting new requests.