How to Implement Zero-Downtime Deployments

Zero-downtime deployments keep applications available to users throughout the entire deployment process without service interruptions or error spikes. The challenge isn't avoiding instance restarts—it's ensuring that at every moment during deployment, some instances are healthy and accepting traffic, database migrations don't break running code, connection draining completes before instances terminate, and health checks accurately reflect readiness to serve requests. Teams attempting zero-downtime deployments often discover that brief error spikes or timeout clusters count as downtime even when most requests succeed.

This guide covers zero-downtime deployment implementation including readiness and liveness probe configuration, rolling update strategies, connection draining and graceful shutdown, backward-compatible database migrations, load balancer health check tuning, and validation procedures that confirm truly zero downtime occurred. The approach addresses real-world complications like startup time variability, dependency initialization order, and concurrent request handling during shutdown.

The structure progresses from foundational concepts through specific technical implementations to operational practices that maintain zero-downtime guarantees as systems evolve.

Zero-Downtime Deployment Fundamentals

Zero-downtime deployments require that capacity exceeds load throughout the deployment process. If you run 10 instances handling 8000 requests per minute (800 RPM each) and deploy by replacing instances one at a time, you temporarily run 9 instances handling 889 RPM each. If instances max out at 900 RPM, you're fine. If they max at 850 RPM, the deployment causes latency spikes or errors—technical downtime even though instances are running.

The deployment process must overlap old and new versions running simultaneously. Start new instances, wait for them to become healthy, add them to the load balancer, remove old instances from the load balancer, drain their connections, then terminate them. This sequence ensures traffic always has healthy instances to route to. If you terminate old instances before new ones are healthy, a gap exists where insufficient capacity causes errors.

Common Causes of Deployment Downtime

Insufficient capacity during deployments happens when rolling updates replace instances too quickly. Replacing 3 of 10 instances simultaneously drops capacity 30%, which might exceed headroom. Poor health check configuration causes load balancers to route traffic to instances that aren't ready—endpoints return 200 OK but databases aren't connected yet, causing errors.

Database migrations that break backward compatibility cause errors when old code hits new schemas. Abrupt instance termination kills in-flight requests—HTTP connections close mid-request, causing client errors even though new instances are healthy. Configuration mismatches where new instances load different settings create version skew that manifests as inconsistent behavior users perceive as failures.

Prerequisites for Zero-Downtime Success

Applications must handle graceful shutdown signals (SIGTERM) by stopping new request acceptance, completing in-flight requests, then exiting. Load balancers need connection draining configured with timeouts exceeding longest expected request duration. Health checks must distinguish between "process running" and "ready to serve traffic"—startup initialization might take 30 seconds before the app is truly ready.

Monitoring must have sufficient granularity to detect brief error spikes. Checking error rates every 5 minutes misses 30-second spikes during deployments. Query metrics at 10-second intervals around deployments to validate truly zero errors occurred. Deployments require spare capacity to absorb load during instance replacement—running at 90% capacity leaves insufficient headroom for zero-downtime deployments.

Health Check Configuration

Readiness versus Liveness Probes

Kubernetes separates readiness from liveness with different probe types and consequences. Readiness probes determine if a pod should receive traffic—failing readiness removes the pod from Service endpoints but doesn't restart it. Use readiness probes to signal "I'm starting up" or "my database connection is down temporarily." Liveness probes detect if a pod is deadlocked and needs restarting—failing liveness kills and recreates the pod.

Configure readiness probes to check actual application readiness: database connectivity, required cache warmup, dependency health. A simple HTTP GET to /health that returns 200 isn't sufficient if the handler doesn't verify backend connections. Liveness probes should be simpler—checking if the process responds at all, not if it's fully functional. A liveness probe that checks database connectivity will kill pods during temporary database issues rather than waiting for recovery.

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    image: myapp:v2.0
    ports:
    - containerPort: 8080
    readinessProbe:
      httpGet:
        path: /health/ready
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
      failureThreshold: 3
      successThreshold: 1
    livenessProbe:
      httpGet:
        path: /health/alive
        port: 8080
      initialDelaySeconds: 30
      periodSeconds: 10
      failureThreshold: 3

Set initialDelaySeconds based on application startup time. If startup typically takes 20 seconds, set readiness initialDelaySeconds to 15-20 seconds to avoid premature health checks that fail unnecessarily. Set liveness initialDelaySeconds higher (30-60 seconds) because liveness probe failures kill pods—give applications time to complete slow startup before checking liveness.

Health Check Endpoint Implementation

Implement dedicated health check endpoints that verify actual readiness. A /health/ready endpoint should test database connectivity with a simple query, check that cache connections are established, verify message queue subscriptions are active, and confirm required configuration loaded successfully. Return 200 OK only when all dependencies are ready.

Avoid expensive operations in health checks. Don't run complex queries or call slow external APIs—these slow down health checks and create cascading failures if dependencies are slow. A database connectivity check should be a trivial SELECT 1 query, not complex application logic. Health checks run frequently (every 5-10 seconds), so efficiency matters.

// Example Node.js readiness endpoint
app.get('/health/ready', async (req, res) => {
  try {
    // Check database
    await db.query('SELECT 1');

    // Check Redis
    await redis.ping();

    // Check critical config loaded
    if (!config.apiKey) {
      throw new Error('API key not configured');
    }

    res.status(200).json({ status: 'ready' });
  } catch (error) {
    res.status(503).json({
      status: 'not ready',
      error: error.message
    });
  }
});

Load Balancer Health Check Tuning

Configure load balancer health check intervals and thresholds to detect failures quickly without false positives. AWS ALB default is 30-second intervals with 2 consecutive failures triggering unhealthy status. This means 60 seconds minimum to detect a failure—too slow for zero-downtime deployments. Reduce interval to 5-10 seconds and require 2 consecutive failures, detecting issues within 10-20 seconds.

Set unhealthy threshold (failures required to mark unhealthy) lower than healthy threshold (successes required to mark healthy). Requiring 2 failures to go unhealthy but 5 successes to go healthy prevents flapping—instances that toggle between healthy and unhealthy cause traffic routing instability. Conservative thresholds (2 to mark unhealthy, 3 to mark healthy) work for most applications.

Rolling Update Strategies

Kubernetes RollingUpdate Configuration

Kubernetes Deployment strategy RollingUpdate controls how pods are replaced. The maxUnavailable setting limits how many pods can be unavailable during updates—set to 0 for true zero downtime, ensuring at least the desired number of pods always exist. The maxSurge setting controls how many extra pods can exist during updates—set to 25% or 1 to create new pods before terminating old ones.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:v2.0
      terminationGracePeriodSeconds: 60

With maxUnavailable: 0 and maxSurge: 1, Kubernetes creates 1 new pod, waits for it to be ready, adds it to the Service, then terminates 1 old pod. This progresses until all 10 pods are replaced. With maxSurge: 2, Kubernetes creates 2 new pods at a time, speeding up deployment at the cost of running up to 12 pods (10 + 2 surge) during the update.

Progressive Rollout Velocity

Control deployment speed by adjusting maxSurge and deployment progression pauses. Deploying 100 replicas with maxSurge: 10 replaces 10 pods at a time, completing in roughly 10 waves. Deployments with maxSurge: 1 take longer (100 waves) but have smaller blast radius if issues exist—catching problems after 5 pods deploy is better than after 50.

Combine rolling updates with pauses for manual verification. Deploy 10% of pods, pause for 15 minutes to monitor metrics, then continue. Use kubectl rollout pause and kubectl rollout resume to control progression. Automated canary deployments with tools like Argo Rollouts codify this pattern—progressive rollout with metric-based gates between stages.

Auto Scaling During Deployments

Horizontal Pod Autoscaler (HPA) can interfere with deployments if not configured carefully. During deployments, new pods consume CPU for initialization, potentially triggering HPA to scale up. Then when initialization completes, CPU drops and HPA scales down, possibly removing newly deployed pods. This creates deployment thrashing.

Configure HPA with appropriate cooldown periods (scaleDownStabilizationWindowSeconds) to prevent rapid scale-down after scale-up. Set to 5-10 minutes so HPA doesn't react to temporary CPU spikes during pod initialization. Alternatively, pause HPA during deployments by scaling the HPA minReplicas to current replica count, deploy, then restore HPA settings.

Graceful Shutdown and Connection Draining

Implementing Graceful Shutdown

Applications must handle SIGTERM signals by initiating graceful shutdown: stop accepting new requests, complete in-flight requests within timeout period, close database connections cleanly, flush logs and metrics, then exit. Kubernetes sends SIGTERM when terminating pods, waits for terminationGracePeriodSeconds (default 30), then sends SIGKILL to force termination.

// Node.js graceful shutdown example
let isShuttingDown = false;

process.on('SIGTERM', async () => {
  console.log('SIGTERM received, starting graceful shutdown');
  isShuttingDown = true;

  // Stop accepting new requests
  server.close(async () => {
    console.log('HTTP server closed');

    // Close database connections
    await db.close();

    // Close Redis connections
    await redis.quit();

    console.log('Graceful shutdown complete');
    process.exit(0);
  });

  // Force shutdown after 50 seconds (before 60s grace period)
  setTimeout(() => {
    console.error('Forced shutdown after timeout');
    process.exit(1);
  }, 50000);
});

// Reject new requests during shutdown
app.use((req, res, next) => {
  if (isShuttingDown) {
    res.status(503).json({ error: 'Server shutting down' });
  } else {
    next();
  }
});

Set terminationGracePeriodSeconds based on longest expected request duration plus cleanup time. If requests typically complete in 5 seconds with max 30 seconds, set grace period to 60 seconds providing buffer for cleanup. Applications that fail to exit within grace period get SIGKILL, abruptly terminating in-flight requests.

Load Balancer Connection Draining

Connection draining (deregistration delay in AWS) keeps connections open to instances being removed from the load balancer, allowing in-flight requests to complete. When an instance is deregistered, the load balancer stops sending new requests but maintains existing connections for the draining period. Configure draining timeout based on request duration—30 seconds for APIs, 300 seconds for file uploads or streaming.

In Kubernetes, connection draining happens automatically when pods are removed from Service endpoints. The combination of pod graceful shutdown and endpoint removal provides draining—new requests stop routing to the pod (endpoint removal) while existing requests complete (graceful shutdown). Ensure readiness probe fails quickly during shutdown so endpoint removal happens promptly.

PreStop Hooks for Deregistration Delays

Kubernetes removes pods from Service endpoints and sends SIGTERM simultaneously, creating a race condition. If SIGTERM arrives before endpoint removal propagates to all kube-proxy instances, new requests might route to the shutting-down pod after it stops accepting connections. PreStop hooks delay SIGTERM to let endpoint removal propagate.

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    image: myapp:v2.0
    lifecycle:
      preStop:
        exec:
          command:
          - /bin/sh
          - -c
          - sleep 15
  terminationGracePeriodSeconds: 60

The preStop hook sleeps for 15 seconds, delaying SIGTERM while endpoint removal propagates across the cluster. This ensures kube-proxy instances update their routing tables before the application stops accepting requests. The sleep duration depends on cluster size—larger clusters need longer propagation times. 10-20 seconds suffices for most clusters under 100 nodes.

Database Migration Strategies

Expand-Migrate-Contract Pattern

The expand-migrate-contract pattern enables zero-downtime schema changes through three phases. Expand: add new schema elements (columns, tables, indexes) without removing old ones. Migrate: deploy application code that writes to both old and new schema elements. Contract: after all instances run new code, remove old schema elements no longer needed.

Example: renaming a column from "name" to "full_name." Expand phase adds full_name column. Migration phase deploys code that writes to both name and full_name and reads from either. After all instances deploy, contract phase removes the name column. This allows old and new code to run simultaneously during deployment without errors.

Online Schema Change Tools

Large table schema changes can lock tables for minutes or hours, blocking writes and causing downtime. Online schema change tools like gh-ost or pt-online-schema-change (for MySQL) or pg_repack (for PostgreSQL) modify tables without blocking. They create shadow tables, copy data in chunks, apply ongoing changes through triggers or binlog replication, then swap tables atomically.

These tools avoid locking by working incrementally. Instead of ALTER TABLE that locks the entire table, they copy rows in small batches (1000-10000 rows at a time), allowing normal operations to proceed between batches. The cutover swap is brief (milliseconds) and happens when no active writes hold locks. For multi-TB tables, online schema changes take hours but never block production traffic.

Database Migration Timing

Run additive migrations (adding columns, tables, indexes) before deploying application code that uses them. This ensures new code finds the schema it expects. Run destructive migrations (dropping columns, tables) after deploying application code that stopped using them and verifying no old code remains. This prevents old code from breaking when schema elements disappear.

Use migration tools like Flyway, Liquibase, or language-specific ORMs (Sequelize, Alembic) that track applied migrations and prevent duplicate execution. These tools ensure migrations run exactly once even if multiple application instances attempt migration simultaneously during deployment. They also provide rollback scripts for reversing migrations if needed.

Request Routing and Session Handling

Session Persistence Strategies

Applications storing session state in-memory (like Express sessions in Node.js process memory) lose sessions when instances restart during deployments. Users get logged out or lose shopping carts mid-deployment. Fix this by externalizing sessions to Redis, Memcached, or databases that persist across application deployments.

With external sessions, multiple application instances share session storage. When instance A handles login and instance B handles the next request, both access the same session data. During deployments, sessions survive instance restarts because they live in shared storage. Configure session stores with appropriate TTLs (30 minutes for short-lived sessions, 7 days for long-lived) to balance memory usage and user experience.

// Example Redis session configuration
const session = require('express-session');
const RedisStore = require('connect-redis')(session);

app.use(session({
  store: new RedisStore({
    client: redisClient,
    ttl: 86400  // 24 hours
  }),
  secret: process.env.SESSION_SECRET,
  resave: false,
  saveUninitialized: false,
  cookie: {
    secure: true,
    httpOnly: true,
    maxAge: 86400000  // 24 hours
  }
}));

Sticky Session Considerations

Sticky sessions (session affinity) route requests from the same client to the same backend instance. This enables in-memory caching and reduces session storage load but complicates zero-downtime deployments. When an instance terminates during deployment, its sticky sessions must migrate to other instances or clients must re-establish sessions.

If you must use sticky sessions, configure connection draining generously (5-10 minutes) to let long-running sessions complete. Implement session migration by persisting sessions to shared storage even with stickiness, so if an instance terminates, the next instance can load the session. Prefer stateless designs where session stickiness isn't required—this simplifies deployments significantly.

Request Timeout and Retry Handling

Set client request timeouts higher than typical request duration but low enough to detect failures quickly. A 30-second timeout for requests that typically take 2 seconds gives 15x buffer for variability while failing fast if instances are unresponsive. Implement exponential backoff retry logic in clients to handle transient errors during deployments.

Distinguish retryable errors (503 Service Unavailable, connection timeouts) from non-retryable errors (400 Bad Request, 401 Unauthorized). Retry only transient failures likely to succeed on retry. Implement circuit breakers that stop retrying an instance after repeated failures, preventing cascading failures where retry storms overwhelm struggling instances during deployments.

Monitoring Zero-Downtime Deployments

Metrics to Track During Deployment

Monitor error rate at fine granularity (10-30 second intervals) during deployments to detect brief spikes. A 20-second spike in 500 errors counts as downtime even though 5-minute aggregates might look acceptable. Track request latency percentiles (p95, p99) to catch performance degradation. New versions that are slower but don't error still degrade user experience.

Track active instance count throughout deployment to verify capacity remains adequate. Plot desired instances, actual healthy instances, and instances receiving traffic over time. Drops in any metric indicate problems—too few healthy instances means health checks are failing, too few receiving traffic means endpoint propagation delays.

Synthetic Monitoring Through Deployments

Run synthetic monitors continuously executing critical transactions throughout deployments. These detect issues immediately rather than waiting for user reports. A synthetic monitor might log in, search for a product, add to cart, and checkout every 60 seconds. If any step fails during deployment, you know users are experiencing errors.

Use tools like Pingdom, Datadog Synthetic Monitoring, or custom scripts that hit production endpoints and assert correct responses. Configure aggressive alerting during deployment windows—any synthetic failure triggers immediate investigation. Outside deployments, relax thresholds to reduce noise from transient network issues.

Deployment Verification Checklist

After deployment completes, verify zero downtime occurred by reviewing metrics for the deployment window. Check error rate never spiked above baseline, request latency didn't increase significantly, all pods became ready within expected time, and no pod restarts occurred post-deployment. Query logs for connection errors, timeouts, or 500 responses.

Compare deployment to baseline periods (same time previous day, same traffic patterns before deployment). If error rate was 0.1% baseline and 0.15% during deployment, investigate the increase even though absolute numbers are low. Zero downtime means no measurable degradation, not just "didn't break completely."

Platform-Specific Implementations

AWS ECS Rolling Deployments

ECS services support rolling deployments with minimum healthy percent and maximum percent settings. Set minimum healthy percent to 100% to ensure full capacity always exists. Set maximum percent to 150-200% to allow creating new tasks before terminating old ones. This temporarily runs 1.5-2x desired tasks during deployment, ensuring zero downtime.

{
  "serviceName": "my-service",
  "desiredCount": 10,
  "deploymentConfiguration": {
    "minimumHealthyPercent": 100,
    "maximumPercent": 200
  },
  "healthCheckGracePeriodSeconds": 60,
  "loadBalancers": [{
    "targetGroupArn": "arn:aws:elasticloadbalancing:...",
    "containerName": "app",
    "containerPort": 8080
  }]
}

Configure ALB health checks with appropriate thresholds. Set deregistration delay (connection draining) to match task shutdown time. ECS sends SIGTERM to containers, waits for stopTimeout (default 30 seconds), then sends SIGKILL. Ensure deregistration delay exceeds stopTimeout so tasks finish serving requests before forceful termination.

AWS Lambda Alias Versioning

Lambda aliases enable zero-downtime deployments by shifting traffic between versions. Publish a new function version, update the alias to point to it with weighted routing (90% to old version, 10% to new version), monitor error metrics, then gradually shift to 100% new version. If errors spike, instantly shift back to 100% old version.

Use Lambda deployment preferences in SAM or CloudFormation to automate canary deployments. Configure CodeDeploy to manage traffic shifting with automatic rollback based on CloudWatch alarms. This provides zero-downtime deployments with built-in safety through progressive traffic shifting and metric-based rollback.

Google Cloud Run Revisions

Cloud Run creates immutable revisions for each deployment and routes traffic using splits between revisions. Deploy a new revision, configure traffic split to send 10% to the new revision and 90% to old, monitor metrics, then adjust splits to 100% new revision. Cloud Run manages container lifecycle, health checking, and scaling automatically.

Cloud Run's concurrency setting controls how many concurrent requests each container handles. Set this based on application characteristics—CPU-bound apps might handle 1-10 concurrent requests efficiently while I/O-bound apps might handle 100-1000. Appropriate concurrency settings prevent overload during deployments when some containers are starting or stopping.

Advanced Zero-Downtime Techniques

Versioned API Deployments

Run multiple API versions simultaneously to support clients on old versions during zero-downtime backend upgrades. Clients specify desired API version in request headers (X-API-Version: v2) or URL paths (/v2/users). Backend routes requests to appropriate version handlers. This enables deprecating old API versions gradually rather than forcing all clients to upgrade immediately.

Implement version sunset schedules where new versions run alongside old versions for 6-12 months before removal. Announce deprecation timelines, provide migration guides, and track client version usage to identify clients needing upgrade assistance. This pattern is common for public APIs where you can't control client upgrade timing.

Feature Flags for Risk Isolation

Deploy code with new features disabled by default, enabled through feature flags. This separates deployment risk from feature risk. Deploy code changes with zero downtime, then gradually enable features for increasing user percentages. If features have issues, disable flags without redeploying. This reduces deployment risk since code deploys in inactive state.

Use feature flag platforms like LaunchDarkly, Split.io, or Unleash that provide real-time flag updates, user targeting, and rollout automation. These platforms change flag states instantly across all instances without deployments, enabling immediate feature disablement if issues arise.

Idempotent Operations for Retry Safety

Design operations to be safely retryable so clients can retry failed requests during deployments without causing duplicate actions. If a request to charge a credit card times out mid-deployment, the client should be able to retry without charging the card twice. Implement idempotency keys or database constraints preventing duplicate processing.

For payment processing, generate an idempotency key (UUID) on the client, send it with the request, and store it with the charge record in the database. If the request times out and client retries with the same idempotency key, the server returns the original charge result instead of creating a duplicate charge. This makes operations safe to retry during deployment disruptions.

Frequently Asked Questions

How do we test zero-downtime deployments before production?

Deploy to staging environments with production-like traffic using load testing tools. Run deployments while load tests execute, monitoring for error spikes or latency increases. Staging won't match production traffic patterns exactly, but it validates deployment mechanics work. Also deploy during low-traffic production windows initially to limit blast radius if issues occur.

What if our application requires 5 minutes to start up?

Long startup times complicate zero-downtime deployments because health checks must wait for initialization. Set readiness probe initialDelaySeconds to expected startup time plus buffer. Consider optimizing startup by lazy-loading non-critical components or using snapshots to reduce initialization time. Alternatively, pre-warm instances before adding to production traffic.

How do we handle deployments during traffic spikes?

Avoid deployments during predictable high-traffic periods (Black Friday sales, major product launches) when capacity margins are thin. If emergency deployments are necessary during traffic spikes, increase maxSurge to create more extra capacity during deployment. Consider pausing auto-scaling during deployment to prevent scaling interference.

Can we achieve zero downtime with stateful applications?

Stateful applications require more sophisticated strategies. Use StatefulSets in Kubernetes with partition-based rolling updates to control which replicas update. Implement data replication between old and new versions during deployment windows. For databases, use blue-green or canary deployments at the database level with replication keeping versions synchronized.

What causes brief connection errors even with proper health checks?

Timing issues between service endpoint updates and health check propagation cause brief errors. When pods start, they might receive traffic before fully initialized if endpoint updates propagate faster than readiness probe passes. Use initialDelaySeconds and preStop hooks to coordinate timing. Also implement retry logic in clients to handle transient connection errors gracefully.

How do we verify that zero downtime actually occurred?

Query metrics at fine granularity (10-second intervals) for the deployment window and compare to baseline periods. Check that error rate, latency, and request rate remained stable. Review application logs for connection errors or exceptions. Run synthetic monitors throughout deployment and verify all completed successfully. Any spike indicates downtime occurred even if brief.

Should we use blue-green or rolling updates for zero downtime?

Both can achieve zero downtime. Blue-green requires double infrastructure but provides instant rollback. Rolling updates are more resource-efficient but slower to rollback. Choose based on infrastructure cost tolerance and rollback speed requirements. For critical services where instant rollback justifies cost, use blue-green. For most services, rolling updates suffice.

How do we handle zero-downtime deployments in development environments?

Development environments often don't need true zero downtime since user counts are low and tolerating brief interruptions is acceptable. However, practicing zero-downtime procedures in development validates they work before production use. Run simplified versions—maybe without canary stages but still with health checks and graceful shutdown.

What if zero-downtime deployments take too long?

Balance deployment speed with safety. Increase maxSurge to replace more instances simultaneously, shortening deployment time at the cost of higher peak resource usage. For very large deployments (100+ instances), consider batched rolling updates where you deploy to subsets of instances (25 at a time) rather than all instances sequentially.

How do we coordinate zero-downtime deployments across microservices?

Deploy microservices independently with backward-compatible API contracts. Each service deploys with zero downtime on its own schedule without coordinating with others. Ensure API versioning and compatibility testing so Service A on version 2 works with Service B on version 1 during overlapping deployment windows. Avoid hard dependencies on simultaneous multi-service deployments.

Conclusion

Zero-downtime deployments require careful coordination of health checks, graceful shutdown, connection draining, backward-compatible database migrations, and adequate capacity during instance replacement. Success comes from addressing every potential interruption point—instance startup readiness, request routing timing, in-flight request handling, and database compatibility. Teams achieving consistent zero downtime deploy with confidence multiple times per day because automation and validation remove human error from the deployment critical path.

Start by implementing health checks that accurately reflect readiness, graceful shutdown that completes in-flight requests, and rolling updates with maxUnavailable: 0. Validate with synthetic monitoring running throughout deployments. Progressively add sophistication like preStop hooks, connection draining tuning, and automated metric verification. The investment pays off through faster feature delivery without the anxiety of user-facing errors during deployments.