CI/CD Pipelines: Automating Build, Test, and Deploy

Continuous Integration and Continuous Delivery (CI/CD) pipelines automate the journey from code commit to production deployment. This guide covers GitHub Actions and GitLab CI with practical examples for caching, secrets, artifacts, and matrix builds.

GitHub Actions

GitHub Actions workflows live in .github/workflows/. A workflow triggers on events and runs jobs on runners.

Basic Workflow

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run tests
        run: pytest --junitxml=results.xml

      - name: Upload test results
        uses: actions/upload-artifact@v4
        with:
          name: test-results
          path: results.xml

Caching Dependencies

Speed up builds by caching package managers:

      - name: Cache pip packages
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-

Secrets and Variables

Store sensitive values in repository or organization secrets:

      - name: Deploy to server
        env:
          DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }}
          SERVER_HOST: ${{ vars.PRODUCTION_HOST }}
        run: |
          echo "$DEPLOY_KEY" > /tmp/deploy_key
          chmod 600 /tmp/deploy_key
          rsync -avz -e "ssh -i /tmp/deploy_key" ./dist/ user@$SERVER_HOST:/var/www/

Never print secrets in logs. GitHub automatically masks known secret values.

Matrix Builds

Test across multiple versions or operating systems:

  test:
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest]
        python-version: ["3.10", "3.11", "3.12"]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - run: pip install -r requirements.txt && pytest

Deploy Job

Gate deployments behind successful tests:

  deploy:
    needs: test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    environment: production
    steps:
      - uses: actions/checkout@v4
      - name: Deploy
        run: ./scripts/deploy.sh

GitLab CI

GitLab CI configuration lives in .gitlab-ci.yml at the repository root.

# .gitlab-ci.yml
stages:
  - build
  - test
  - deploy

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

cache:
  paths:
    - .cache/pip
    - venv/

build:
  stage: build
  image: python:3.12-slim
  script:
    - python -m venv venv
    - source venv/bin/activate
    - pip install -r requirements.txt
  artifacts:
    paths:
      - venv/
    expire_in: 1 hour

test:
  stage: test
  image: python:3.12-slim
  script:
    - source venv/bin/activate
    - pytest --junitxml=report.xml
  artifacts:
    reports:
      junit: report.xml

deploy_prod:
  stage: deploy
  image: alpine:latest
  only:
    - main
  script:
    - apk add --no-cache openssh rsync
    - rsync -avz ./dist/ deployer@$PROD_HOST:/var/www/
  environment:
    name: production
    url: https://example.com

Key GitLab CI Features

• Stages define the execution order. Jobs in the same stage run in parallel.
• Artifacts pass files between stages. Use expire_in to auto-clean.
• Cache persists across pipelines for dependency directories.
• Variables are set in the CI/CD settings UI for secrets, or inline for non-sensitive values.

Pipeline Best Practices

1. Fail fast -- put linting and unit tests in the earliest stage.
2. Cache aggressively -- dependency installs are often the slowest step.
3. Use artifacts to pass build outputs between stages instead of rebuilding.
4. Pin action/image versions to SHA or tag to avoid surprises.
5. Protect deploy branches -- require approvals before production deployments.
6. Keep secrets out of code -- use the platform's secret store.

---

Return to the DevOps hub or continue to Git for Sysadmins and Docker Guide.