⚙️DevOps – Terraform, State, GitLab CI/CD & Gestion d'incidents (Part 3)

Procédure standard

Arbre de décision rapide

Job failed ou pending
                            │
                            ▼
                            Job pending ?
                            │
                            ├── Oui
                            │     ├── runner actif ?
                            │     ├── tags corrects ?
                            │     ├── runner protégé ?
                            │     └── quota/concurrency ?
                            │
                            └── Non
                            │
                            ▼
                            Erreur avant script ?
                            │
                            ├── image introuvable
                            ├── pull denied
                            └── runner/executor
                            │
                            ▼
                            Erreur dans script ?
                            │
                            ├── path
                            ├── variable
                            ├── terraform init
                            ├── terraform plan
                            ├── artifact
                            └── cloud/IAM

Commandes de debug sûres

echo "Branch: $CI_COMMIT_BRANCH"
                            echo "Pipeline source: $CI_PIPELINE_SOURCE"
                            echo "Environment: $CI_ENVIRONMENT_NAME"
                            echo "Terraform root: $TF_ROOT"
                            pwd
                            ls -la
                            terraform --version

                            # Vérifier présence sans exposer
                            if [ -z "$AWS_ACCESS_KEY_ID" ]; then
                            echo "AWS_ACCESS_KEY_ID is missing"
                            else
                            echo "AWS_ACCESS_KEY_ID is present"
                            fi

Symptômes runner

Commandes runner

# Vérifier runners
                            gitlab-runner verify
                            gitlab-runner list

                            # Service runner
                            systemctl status gitlab-runner --no-pager
                            journalctl -u gitlab-runner -n 100 --no-pager
                            journalctl -u gitlab-runner -f

                            # Ressources machine runner
                            uptime
                            free -h
                            df -h
                            top

Tags runner : exemple

terraform_plan_prod:
                            stage: plan
                            tags:
                            - protected
                            - terraform
                            - prod
                            script:
                            - terraform plan -out=tfplan

Matrice runner

Diagramme runner

Job GitLab
                            │
                            ├── tags demandés
                            ├── branche protégée ?
                            ├── environnement ?
                            └── secrets nécessaires ?
                            │
                            ▼
                            Runner compatible
                            ├── online
                            ├── protected si prod
                            ├── executor OK
                            ├── réseau OK
                            └── ressources suffisantes

Causes fréquentes

Debug sécurisé

check_var() {
                            local name="$1"
                            local value="${!name:-}"

                            if [ -z "$value" ]; then
                            echo "$name is missing"
                            return 1
                            fi

                            echo "$name is present"
                            }

                            check_var "AWS_ACCESS_KEY_ID"
                            check_var "AWS_SECRET_ACCESS_KEY"
                            check_var "TF_VAR_environment"

Scoping recommandé

Variables utiles à afficher

echo "CI_COMMIT_BRANCH=$CI_COMMIT_BRANCH"
                            echo "CI_PIPELINE_SOURCE=$CI_PIPELINE_SOURCE"
                            echo "CI_ENVIRONMENT_NAME=$CI_ENVIRONMENT_NAME"
                            echo "TF_ROOT=$TF_ROOT"
                            echo "TF_WORKSPACE=${TF_WORKSPACE:-not-set}"

À ne pas faire

# Bad examples
                            env
                            printenv
                            set -x
                            echo "$AWS_SECRET_ACCESS_KEY"
                            echo "$TF_VAR_db_password"

Symptômes courants

Debug chemins

echo "TF_ROOT=$TF_ROOT"
                            pwd
                            ls -la
                            find . -maxdepth 3 -type f | sort | head -100

                            cd "$TF_ROOT"
                            pwd
                            ls -la

Artifacts Terraform propres

terraform_plan:
                            stage: plan
                            script:
                            - cd "$TF_ROOT"
                            - terraform plan -out=tfplan
                            - terraform show -no-color tfplan > plan.txt
                            artifacts:
                            paths:
                            - "$TF_ROOT/tfplan"
                            - "$TF_ROOT/plan.txt"
                            expire_in: 3 days

                            terraform_apply:
                            stage: apply
                            dependencies:
                            - terraform_plan
                            script:
                            - cd "$TF_ROOT"
                            - terraform apply tfplan
                            when: manual

Plan vs apply

plan job
                            │
                            ├── cd $TF_ROOT
                            ├── terraform plan -out=tfplan
                            └── artifact: $TF_ROOT/tfplan
                            │
                            ▼
                            apply job
                            │
                            ├── récupère artifact
                            ├── cd même $TF_ROOT
                            └── terraform apply tfplan

Diagnostic par commande

Bloc CI robuste

before_script:
                            - echo "TF_ROOT=$TF_ROOT"
                            - cd "$TF_ROOT"
                            - terraform --version
                            - terraform init -input=false

                            terraform_validate:
                            stage: validate
                            script:
                            - terraform fmt -check -recursive
                            - terraform validate

                            terraform_plan:
                            stage: plan
                            script:
                            - terraform plan -input=false -out=tfplan
                            - terraform show -no-color tfplan > plan.txt

Erreurs Terraform fréquentes

Comparaison local / CI

# À comparer entre local et CI
                            terraform --version
                            terraform providers
                            terraform workspace show
                            echo "$TF_ROOT"
                            echo "$TF_VAR_environment"
                            terraform plan -var-file="staging.tfvars"

Symptômes state / lock

Commandes state utiles

terraform state list
                            terraform state show RESOURCE
                            terraform plan -refresh-only
                            terraform state pull > state-backup.json

                            # Dangereux : seulement après preuve que le lock est orphelin
                            terraform force-unlock LOCK_ID

Procédure lock Terraform

Lock détecté
                            │
                            ▼
                            Identifier lock info
                            ├── ID
                            ├── user
                            ├── operation
                            ├── created time
                            └── state path
                            │
                            ▼
                            Chercher job actif
                            ├── GitLab pipeline running ?
                            ├── apply en cours ?
                            ├── runner encore actif ?
                            └── logs récents ?
                            │
                            ▼
                            Décision
                            ├── job actif : attendre
                            ├── job échoué : vérifier état
                            └── lock orphelin prouvé : force-unlock
                            │
                            ▼
                            Relancer terraform plan

Prévention en CI

Exemple GitLab resource group

terraform_apply_prod:
                            stage: apply
                            resource_group: terraform-prod
                            script:
                            - terraform apply tfplan
                            when: manual

Logs utiles à ajouter

echo "Job: $CI_JOB_NAME"
                            echo "Stage: $CI_JOB_STAGE"
                            echo "Branch: $CI_COMMIT_BRANCH"
                            echo "Pipeline source: $CI_PIPELINE_SOURCE"
                            echo "Environment: ${CI_ENVIRONMENT_NAME:-not-set}"
                            echo "TF_ROOT: $TF_ROOT"
                            echo "Working directory:"
                            pwd
                            echo "Terraform version:"
                            terraform --version

Debug contrôlé des fichiers

echo "Repository files:"
                            find . -maxdepth 2 -type f | sort

                            echo "Terraform root files:"
                            cd "$TF_ROOT"
                            ls -la
                            find . -maxdepth 2 -type f | sort

À ne pas logger

Template de log de diagnostic

debug_context:
                            script:
                            - echo "Job=$CI_JOB_NAME"
                            - echo "Branch=$CI_COMMIT_BRANCH"
                            - echo "Source=$CI_PIPELINE_SOURCE"
                            - echo "Env=${CI_ENVIRONMENT_NAME:-not-set}"
                            - echo "TF_ROOT=$TF_ROOT"
                            - pwd
                            - test -d "$TF_ROOT" || exit 1
                            - cd "$TF_ROOT"
                            - terraform --version
                            - terraform validate

Procédure production

Cas production typiques

Diagramme production

Pipeline prod failed
                            │
                            ▼
                            Plan ou apply ?
                            │
                            ├── Plan failed
                            │     ├── variables
                            │     ├── backend
                            │     ├── provider
                            │     └── IAM read
                            │
                            └── Apply failed
                            ├── vérifier état partiel
                            ├── relancer plan
                            ├── vérifier lock
                            ├── surveiller prod
                            └── décider rollback/mitigation

Checklist diagnostic rapide

Checklist sécurité

Checklist correction

Mini-cheat-sheet

# Runner
                            gitlab-runner verify
                            gitlab-runner list
                            systemctl status gitlab-runner --no-pager
                            journalctl -u gitlab-runner -n 100 --no-pager

                            # Safe CI debug
                            echo "Branch=$CI_COMMIT_BRANCH"
                            echo "Source=$CI_PIPELINE_SOURCE"
                            echo "Env=${CI_ENVIRONMENT_NAME:-not-set}"
                            echo "TF_ROOT=$TF_ROOT"
                            pwd
                            ls -la
                            terraform --version

                            # Terraform
                            terraform init
                            terraform validate
                            terraform plan -out=tfplan
                            terraform show -no-color tfplan
                            terraform state list
                            terraform plan -refresh-only

                            # Avoid
                            env
                            printenv
                            set -x
                            echo "$SECRET"
                            terraform force-unlock LOCK_ID

Principe

Version N active
                            │
                            ├── instance 1 : vN
                            ├── instance 2 : vN
                            ├── instance 3 : vN
                            └── instance 4 : vN
                            │
                            ▼
                            Rolling update
                            │
                            ├── instance 1 → vN+1
                            ├── healthcheck OK
                            ├── instance 2 → vN+1
                            ├── healthcheck OK
                            └── continuer
                            │
                            ▼
                            Version N+1 active partout

Conditions nécessaires

Exemple Kubernetes

apiVersion: apps/v1
                            kind: Deployment
                            metadata:
                            name: api
                            spec:
                            replicas: 4
                            strategy:
                            type: RollingUpdate
                            rollingUpdate:
                            maxUnavailable: 1
                            maxSurge: 1
                            template:
                            spec:
                            containers:
                            - name: api
                            image: registry.example.com/api:1.8.2
                            readinessProbe:
                            httpGet:
                            path: /health/
                            port: 8000

Commandes utiles

kubectl rollout status deployment/api
                            kubectl get pods -l app=api
                            kubectl describe deployment/api

                            # Rollback Kubernetes
                            kubectl rollout undo deployment/api

                            # Vérifier version image
                            kubectl get deployment api -o wide

Risques rolling

Principe

BLUE actif
                            ├── version N
                            └── reçoit 100% trafic

                            GREEN préparé
                            ├── version N+1
                            ├── smoke tests
                            ├── health checks
                            └── validation métier
                            │
                            ▼
                            Bascule trafic
                            ├── load balancer
                            ├── DNS pondéré
                            └── service mesh
                            │
                            ▼
                            GREEN actif
                            BLUE conservé temporairement pour rollback

Avantages

Exemple de bascule load balancer

Load Balancer
                            │
                            ├── Target Group BLUE
                            │     └── app version N
                            │
                            └── Target Group GREEN
                            └── app version N+1

                            Avant :
                            100% → BLUE

                            Après validation :
                            100% → GREEN

                            Rollback :
                            100% → BLUE

Risques blue/green

Progression canary

Version N : 100%
                            Version N+1 : 0%
                            │
                            ▼
                            Canary 5%
                            ├── surveiller 5xx
                            ├── latence
                            ├── erreurs métier
                            └── logs
                            │
                            ▼
                            Canary 25%
                            │
                            ▼
                            Canary 50%
                            │
                            ▼
                            Canary 100%
                            │
                            ▼
                            Ancienne version retirée

Critères de promotion

Exemple routing canary

Load Balancer / Service Mesh
                            │
                            ├── 95% → version stable
                            └── 5%  → version canary

                            Si OK :
                            75% stable / 25% canary
                            50% stable / 50% canary
                            0% stable / 100% canary

                            Si KO :
                            100% stable / 0% canary

Exemple Kubernetes conceptuel

# Stable deployment
                            apiVersion: apps/v1
                            kind: Deployment
                            metadata:
                            name: api-stable
                            spec:
                            replicas: 9

                            ---
                            # Canary deployment
                            apiVersion: apps/v1
                            kind: Deployment
                            metadata:
                            name: api-canary
                            spec:
                            replicas: 1

Quand éviter canary ?

Quand l’utiliser ?

Runbook maintenance

Before:
                            - annoncer fenêtre
                            - vérifier backup
                            - vérifier rollback
                            - préparer page maintenance
                            - arrêter jobs asynchrones
                            - ouvrir dashboards

                            During:
                            - activer maintenance
                            - vérifier que trafic utilisateur est bloqué proprement
                            - exécuter migration / déploiement
                            - exécuter smoke tests internes

                            After:
                            - désactiver maintenance
                            - vérifier health
                            - vérifier logs
                            - vérifier métriques
                            - confirmer retour utilisateur

Exemple Nginx maintenance simple

server {
                            listen 443 ssl;
                            server_name example.com;

                            if (-f /var/www/maintenance/ON) {
                            return 503;
                            }

                            error_page 503 @maintenance;

                            location @maintenance {
                            root /var/www/maintenance;
                            rewrite ^(.*)$ /maintenance.html break;
                            }

                            location / {
                            proxy_pass http://app_backend;
                            }
                            }

Points de vigilance

Types de rollback

Bloc rollback dans une MR

Rollback plan:
                            - Previous version: api:1.8.1
                            - New version: api:1.8.2
                            - Rollback command:
                            kubectl rollout undo deployment/api

                            - Conditions to rollback:
                            - 5xx > 2% for 5 minutes
                            - p95 latency > 800ms for 10 minutes
                            - critical business flow failing

                            - Validation after rollback:
                            - health endpoint OK
                            - 5xx back to baseline
                            - no new critical logs
                            - business smoke test OK

Commandes rollback courantes

# Kubernetes
                            kubectl rollout status deployment/api
                            kubectl rollout undo deployment/api
                            kubectl rollout history deployment/api

                            # Docker compose
                            docker compose pull api
                            docker compose up -d api

                            # systemd app release symlink
                            ln -sfn /opt/app/releases/2026-04-previous /opt/app/current
                            sudo systemctl restart app.service

                            # Vérification
                            curl -fsS https://example.com/health/
                            journalctl -u app.service -n 100 --no-pager

Diagramme rollback

Signal KO
                            │
                            ├── 5xx
                            ├── latence
                            ├── erreurs métier
                            └── saturation
                            │
                            ▼
                            Décision rollback
                            │
                            ├── stop rollout
                            ├── restore previous version
                            ├── reduce canary to 0%
                            └── switch blue/green
                            │
                            ▼
                            Post-checks
                            ├── health
                            ├── logs
                            ├── metrics
                            └── business test

Pattern expand / migrate / contract

1. Expand
                            ├── ajouter nouvelle colonne nullable
                            ├── ajouter nouvelle table
                            ├── ajouter index sans casser ancien code
                            └── garder compatibilité

                            2. Migrate
                            ├── backfill progressif
                            ├── double-write éventuel
                            ├── lecture nouvelle donnée
                            └── validation

                            3. Contract
                            ├── supprimer ancien champ
                            ├── retirer ancien code
                            └── nettoyer dette après stabilisation

Règles DB pour rolling/canary

Checklist migration DB

Déploiement incompatible

Si migration non compatible
                            │
                            ├── rolling impossible
                            ├── canary dangereux
                            ├── blue/green limité par DB commune
                            └── maintenance window souvent nécessaire
                            │
                            ▼
                            Actions
                            ├── backup
                            ├── stop writes
                            ├── migration
                            ├── deploy app
                            ├── smoke tests
                            └── reopen traffic

Pipeline type

Commit / MR
                            │
                            ├── tests
                            ├── build image
                            ├── security scan
                            └── package
                            │
                            ▼
                            Staging
                            ├── deploy
                            ├── migrations
                            ├── smoke tests
                            └── validation
                            │
                            ▼
                            Production approval
                            ├── choose strategy
                            ├── rolling / canary / blue-green
                            ├── monitor
                            └── promote or rollback

Exemple GitLab CI simplifié

stages:
                            - test
                            - build
                            - deploy_staging
                            - smoke
                            - deploy_prod
                            - postcheck

                            deploy_staging:
                            stage: deploy_staging
                            script:
                            - ./scripts/deploy.sh staging rolling

                            smoke_staging:
                            stage: smoke
                            script:
                            - curl -fsS https://staging.example.com/health/

                            deploy_prod_canary:
                            stage: deploy_prod
                            when: manual
                            environment:
                            name: production
                            script:
                            - ./scripts/deploy.sh production canary

                            postcheck_prod:
                            stage: postcheck
                            script:
                            - ./scripts/postcheck.sh production

Variables utiles

Garde-fous CI

Checklist avant déploiement

Checklist pendant déploiement

Checklist après déploiement

Mini-cheat-sheet

Strategies:
                            - rolling: progressive instance replacement
                            - blue/green: switch traffic between two environments
                            - canary: expose small traffic percentage
                            - maintenance: controlled downtime
                            - feature flag: enable/disable behavior

                            Signals:
                            - health endpoint
                            - HTTP 5xx
                            - latency p95/p99
                            - application errors
                            - DB locks / slow queries
                            - queue backlog
                            - business smoke test

                            Rollback:
                            - previous image
                            - previous config
                            - switch back blue/green
                            - canary to 0%
                            - disable feature flag
                            - DB mitigation if needed

Golden signals production

Signaux faibles avant incident

• La latence p95 monte alors que le taux d’erreur reste bas.
• Les queues accumulent du retard, mais les workers ne crashent pas encore.
• La DB approche de la limite de connexions.
• Le disque est à 80%, mais les logs grossissent très vite.
• Le CPU est correct, mais l’I/O wait augmente.
• Les timeouts externes augmentent après un déploiement.

Dashboard capacité minimal

Production Capacity Dashboard
                            │
                            ├── HTTP RPS
                            ├── HTTP 5xx rate
                            ├── Latency p50 / p95 / p99
                            ├── CPU / load average
                            ├── RAM / swap / OOM
                            ├── Disk usage / I/O wait
                            ├── DB connections / locks / slow queries
                            ├── Redis / cache hit ratio
                            ├── Queue backlog / oldest message age
                            ├── Worker count / failed jobs
                            └── Last deployment marker

Corrélation utile

Commandes Linux essentielles

# Charge système
                            uptime

                            # Process CPU/RAM
                            top
                            ps aux --sort=-%cpu | head
                            ps aux --sort=-%mem | head

                            # Mémoire
                            free -h
                            vmstat 1 5

                            # OOM killer
                            dmesg | grep -i "killed process"
                            journalctl -k | grep -i "out of memory"

                            # CPU détaillé si outils installés
                            mpstat 1 5
                            pidstat 1 5

CPU : ce qu’il faut comprendre

RAM : signaux dangereux

Diagnostic CPU/RAM

Latence élevée
                            │
                            ▼
                            CPU haut ?
                            ├── Oui
                            │   ├── code CPU-bound
                            │   ├── trop de requêtes
                            │   ├── compression/chiffrement
                            │   └── batch concurrent
                            │
                            └── Non
                            ▼
                            RAM ou I/O ?
                            ├── swap
                            ├── OOM
                            ├── I/O wait
                            └── blocage DB/API externe

Seuils indicatifs

Commandes disque

# Espace disque
                            df -h

                            # Inodes
                            df -i

                            # Gros dossiers
                            du -sh /* 2>/dev/null | sort -h
                            du -sh /var/log/* 2>/dev/null | sort -h
                            du -sh /tmp/* 2>/dev/null | sort -h

                            # Journald
                            journalctl --disk-usage
                            sudo journalctl --vacuum-time=7d

                            # Fichiers supprimés encore ouverts
                            lsof +L1

                            # I/O si outils installés
                            iostat -xz 1 5
                            iotop

Diagnostic disque

Seuils disque recommandés

Signaux DB à surveiller

PostgreSQL : commandes utiles

-- Connexions actives
                            select state, count(*)
                            from pg_stat_activity
                            group by state;

                            -- Requêtes longues
                            select pid, now() - query_start as duration, state, query
                            from pg_stat_activity
                            where state <> 'idle'
                            order by duration desc
                            limit 10;

                            -- Locks
                            select * from pg_locks where not granted;

                            -- Taille des tables
                            select relname, pg_size_pretty(pg_total_relation_size(relid))
                            from pg_catalog.pg_statio_user_tables
                            order by pg_total_relation_size(relid) desc
                            limit 10;

MySQL / MariaDB : commandes utiles

-- Processlist
                            show full processlist;

                            -- Variables importantes
                            show variables like 'max_connections';
                            show status like 'Threads_connected';
                            show status like 'Threads_running';

                            -- Requêtes lentes
                            show variables like 'slow_query_log';
                            show variables like 'long_query_time';

                            -- InnoDB status
                            show engine innodb status\G

Diagnostic DB

Application lente
                            │
                            ▼
                            DB impliquée ?
                            │
                            ├── connexions hautes
                            ├── locks
                            ├── slow queries
                            ├── CPU DB haut
                            ├── I/O DB haut
                            └── replication lag
                            │
                            ▼
                            Actions
                            ├── index
                            ├── query optimization
                            ├── pool sizing
                            ├── réduire transactions
                            ├── read replicas
                            └── scaling DB prudent

Pool de connexions

Métriques queues essentielles

Commandes Celery utiles

# Inspecter workers
                            celery -A project inspect active
                            celery -A project inspect reserved
                            celery -A project inspect scheduled
                            celery -A project inspect stats

                            # Redémarrage contrôlé via systemd
                            systemctl status celery.service --no-pager
                            journalctl -u celery.service -n 100 --no-pager
                            sudo systemctl restart celery.service

Diagnostic backlog

Backlog monte
                            │
                            ▼
                            Workers actifs ?
                            ├── Non
                            │   ├── service down
                            │   ├── crash
                            │   └── broker inaccessible
                            │
                            └── Oui
                            ▼
                            Throughput suffisant ?
                            ├── Non
                            │   ├── jobs lents
                            │   ├── DB lente
                            │   ├── API externe lente
                            │   └── workers insuffisants
                            │
                            └── Oui
                            └── pic temporaire à surveiller

Actions possibles

Seuils indicatifs de départ

Structure d’une bonne alerte

Alert:
                            - Name: API latency p95 high
                            - Condition: p95 > 800ms for 10 minutes
                            - Severity: warning / critical
                            - Environment: production
                            - Impact: users may experience slow responses
                            - Dashboard: production-api-overview
                            - Runbook: runbooks/api-latency.md
                            - Owner: platform-team
                            - First checks:
                            - last deployment
                            - 5xx rate
                            - DB slow queries
                            - CPU/RAM
                            - queue backlog

Alerte utile vs bruit

Types de scaling

Décision scaling

Saturation détectée
                            │
                            ▼
                            Quel goulot ?
                            ├── CPU app → horizontal/vertical
                            ├── RAM app → leak ou vertical
                            ├── DB → query/index/pool/scale DB
                            ├── queue → workers/concurrency
                            ├── disque → cleanup/resize/I/O
                            └── réseau → LB/cache/architecture
                            │
                            ▼
                            Tester en staging si possible
                            │
                            ▼
                            Apply contrôlé
                            │
                            ▼
                            Surveiller post-change

Autoscaling : métriques possibles

Garde-fous autoscaling

• Définir un minimum et un maximum réalistes.
• Ajouter cooldown pour éviter l’oscillation.
• Surveiller coût et limites cloud.
• Protéger la DB contre trop de workers/app instances.
• Tester le scale down, pas seulement le scale up.
• Éviter une métrique unique si le système est complexe.

Procédure en 10 étapes

1. Confirmer l'impact utilisateur.
                            2. Identifier le service et l'environnement.
                            3. Vérifier dernier déploiement ou changement infra.
                            4. Ouvrir dashboard capacité.
                            5. Vérifier 5xx, latence p95/p99 et trafic.
                            6. Identifier la ressource saturée :
                            CPU, RAM, disque, DB, queue, réseau.
                            7. Vérifier logs applicatifs et erreurs récentes.
                            8. Mitiger :
                            rollback, scale, rate limit, désactivation feature, cleanup.
                            9. Surveiller retour à la normale.
                            10. Documenter cause, action et prévention.

Commandes terrain

# Linux
                            uptime
                            top
                            free -h
                            df -h
                            df -i
                            ss -tulpn
                            journalctl -p err -n 100 --no-pager

                            # Service
                            systemctl status app.service --no-pager
                            journalctl -u app.service -n 100 --no-pager

                            # HTTP
                            curl -fsS https://example.com/health/
                            curl -I https://example.com/

                            # Logs proxy
                            tail -n 100 /var/log/nginx/error.log
                            grep " 500 " /var/log/nginx/access.log | tail

Mitigations rapides

Timeline incident capacité

Alerte capacité
                            │
                            ▼
                            Impact confirmé ?
                            │
                            ├── Non : surveillance renforcée
                            └── Oui
                            │
                            ▼
                            Identifier goulot
                            │
                            ▼
                            Mitigation courte
                            │
                            ▼
                            Vérifier retour nominal
                            │
                            ▼
                            RCA
                            ├── pourquoi alerte ?
                            ├── pourquoi saturation ?
                            ├── pourquoi non anticipée ?
                            └── action préventive

Checklist dashboard production

Checklist alertes

Checklist incident capacité

Mini-cheat-sheet

# Linux
                            uptime
                            top
                            free -h
                            df -h
                            df -i
                            vmstat 1 5
                            ss -tulpn
                            journalctl -p err -n 100 --no-pager

                            # Disk
                            du -sh /var/log/* | sort -h
                            journalctl --disk-usage
                            lsof +L1

                            # Service
                            systemctl status app.service --no-pager
                            journalctl -u app.service -n 100 --no-pager

                            # HTTP
                            curl -fsS https://example.com/health/

                            # Key signals
                            CPU, RAM, disk, I/O wait
                            5xx, latency p95/p99
                            DB connections, locks, slow queries
                            Queue backlog, oldest message age

Questions de triage

Commandes de première lecture

# Côté Linux / service
                            systemctl status postgresql --no-pager
                            systemctl status mariadb --no-pager
                            journalctl -u postgresql -n 100 --no-pager
                            journalctl -u mariadb -n 100 --no-pager

                            # Système
                            uptime
                            free -h
                            df -h
                            df -i
                            top

                            # Réseau local
                            ss -tulpn

Arbre de triage

Incident DB
                            │
                            ▼
                            Application down ou lente ?
                            │
                            ├── Down
                            │   ├── DB service up ?
                            │   ├── disque plein ?
                            │   ├── max connections ?
                            │   └── credentials/réseau ?
                            │
                            └── Lente
                            ├── slow queries ?
                            ├── locks ?
                            ├── CPU DB haut ?
                            ├── I/O wait ?
                            ├── replication lag ?
                            └── pool saturé ?

Triage par symptôme

PostgreSQL : connexions

-- Connexions par état
                            select state, count(*)
                            from pg_stat_activity
                            group by state
                            order by count(*) desc;

                            -- Connexions par application / utilisateur
                            select usename, application_name, client_addr, state, count(*)
                            from pg_stat_activity
                            group by usename, application_name, client_addr, state
                            order by count(*) desc;

                            -- Requêtes actives longues
                            select pid, now() - query_start as duration, state, wait_event_type, wait_event, query
                            from pg_stat_activity
                            where state <> 'idle'
                            order by duration desc
                            limit 20;

                            -- Limite configurée
                            show max_connections;

PostgreSQL : terminer une requête

-- Annuler une requête sans tuer la connexion
                            select pg_cancel_backend(PID);

                            -- Terminer une connexion
                            -- À utiliser seulement si compris et validé
                            select pg_terminate_backend(PID);

MySQL / MariaDB : connexions

-- Connexions et threads
                            show status like 'Threads_connected';
                            show status like 'Threads_running';
                            show variables like 'max_connections';

                            -- Processlist détaillée
                            show full processlist;

                            -- Tuer une requête ou connexion
                            -- Attention en production
                            kill QUERY_ID;

Causes fréquentes

Calcul simple de pool

Connexions potentielles =
                            instances applicatives
                            × workers par instance
                            × pool size

                            Exemple :
                            6 instances
                            × pool 20
                            = 120 connexions possibles

                            Si max_connections = 150,
                            il reste peu de marge pour admin, migrations, workers, cron.

PostgreSQL : locks et bloqueurs

-- Sessions bloquées et bloqueurs
                            select
                            blocked.pid as blocked_pid,
                            blocked.query as blocked_query,
                            blocking.pid as blocking_pid,
                            blocking.query as blocking_query,
                            now() - blocked.query_start as blocked_duration
                            from pg_stat_activity blocked
                            join pg_locks blocked_locks
                            on blocked_locks.pid = blocked.pid
                            join pg_locks blocking_locks
                            on blocking_locks.locktype = blocked_locks.locktype
                            and blocking_locks.database is not distinct from blocked_locks.database
                            and blocking_locks.relation is not distinct from blocked_locks.relation
                            and blocking_locks.page is not distinct from blocked_locks.page
                            and blocking_locks.tuple is not distinct from blocked_locks.tuple
                            and blocking_locks.virtualxid is not distinct from blocked_locks.virtualxid
                            and blocking_locks.transactionid is not distinct from blocked_locks.transactionid
                            and blocking_locks.classid is not distinct from blocked_locks.classid
                            and blocking_locks.objid is not distinct from blocked_locks.objid
                            and blocking_locks.objsubid is not distinct from blocked_locks.objsubid
                            and blocking_locks.pid != blocked_locks.pid
                            join pg_stat_activity blocking
                            on blocking.pid = blocking_locks.pid
                            where not blocked_locks.granted
                            and blocking_locks.granted;

PostgreSQL : transactions longues

select pid, usename, application_name,
                            now() - xact_start as transaction_duration,
                            state, wait_event_type, wait_event, query
                            from pg_stat_activity
                            where xact_start is not null
                            order by transaction_duration desc
                            limit 20;

MySQL / MariaDB : locks

-- Vue générale
                            show full processlist;

                            -- InnoDB status
                            show engine innodb status\G

                            -- Selon version, informations locks
                            select * from information_schema.innodb_trx\G
                            select * from information_schema.innodb_locks\G
                            select * from information_schema.innodb_lock_waits\G

Décision sur un lock

Diagramme locks

Requête bloquée
                            │
                            ▼
                            Identifier bloqueur
                            │
                            ├── migration ?
                            ├── batch ?
                            ├── transaction app ?
                            ├── admin query ?
                            └── maintenance ?
                            │
                            ▼
                            Évaluer impact
                            ├── lectures seulement ?
                            ├── écritures bloquées ?
                            ├── service critique ?
                            └── durée ?
                            │
                            ▼
                            Décision
                            ├── attendre
                            ├── cancel query
                            ├── terminate session
                            └── rollback / mitigation

PostgreSQL : requêtes lentes

-- Requêtes actives les plus longues
                            select pid, now() - query_start as duration, state, wait_event_type, wait_event, query
                            from pg_stat_activity
                            where state <> 'idle'
                            order by duration desc
                            limit 20;

                            -- Si pg_stat_statements est activé
                            select calls,
                            total_exec_time,
                            mean_exec_time,
                            rows,
                            query
                            from pg_stat_statements
                            order by total_exec_time desc
                            limit 20;

                            -- Plan d'exécution
                            explain analyze
                            select * from my_table where column_name = 'value';

MySQL / MariaDB : requêtes lentes

-- Processlist
                            show full processlist;

                            -- Slow query log
                            show variables like 'slow_query_log';
                            show variables like 'long_query_time';

                            -- Explain
                            explain select * from my_table where column_name = 'value';

                            -- Index existants
                            show index from my_table;

Causes classiques de lenteur

Optimisation prudente

Requête lente
                            │
                            ▼
                            Mesurer
                            ├── fréquence
                            ├── durée moyenne
                            ├── durée p95/p99
                            ├── lignes lues
                            └── impact métier
                            │
                            ▼
                            Expliquer
                            ├── plan SQL
                            ├── indexes
                            ├── locks
                            ├── stats
                            └── volume
                            │
                            ▼
                            Corriger
                            ├── index
                            ├── rewrite SQL
                            ├── cache
                            ├── pagination
                            └── réduction appels

Diagnostic système

# Espace et inodes
                            df -h
                            df -i

                            # Gros répertoires
                            du -sh /var/lib/* 2>/dev/null | sort -h
                            du -sh /var/log/* 2>/dev/null | sort -h
                            du -sh /backup/* 2>/dev/null | sort -h

                            # Fichiers supprimés encore ouverts
                            lsof +L1

                            # Journaux systemd
                            journalctl --disk-usage
                            sudo journalctl --vacuum-time=7d

À ne jamais faire

# Dangerous examples:
                            rm -rf /var/lib/postgresql/*
                            rm -rf /var/lib/mysql/*
                            rm -f ibdata1
                            rm -f *.ibd
                            rm -f pg_wal/*
                            rm -f mysql-bin.* without purge procedure

Causes fréquentes

Mitigation disque plein

Disque DB critique
                            │
                            ▼
                            Identifier ce qui grossit
                            ├── logs OS/app
                            ├── backups
                            ├── WAL/binlogs
                            ├── data files
                            └── temp files
                            │
                            ▼
                            Libérer sans risque
                            ├── logs non DB
                            ├── backups hors rétention validée
                            ├── extension volume
                            └── purge DB via commande officielle
                            │
                            ▼
                            Vérifier reprise écritures
                            │
                            ▼
                            Plan préventif

Concepts essentiels

Commandes backup exemples

# PostgreSQL dump logique
                            pg_dump -Fc -f backup.dump my_database

                            # PostgreSQL restore logique
                            pg_restore -d restored_database backup.dump

                            # MySQL / MariaDB dump logique
                            mysqldump --single-transaction --routines --triggers my_database > backup.sql

                            # Restore MySQL / MariaDB
                            mysql restored_database < backup.sql

Checklist backup sérieux

Architecture backup

Production DB
                            │
                            ├── full backup
                            ├── incremental / WAL / binlogs
                            ├── snapshots
                            └── metadata restore
                            │
                            ▼
                            Backup storage
                            ├── chiffré
                            ├── rétention
                            ├── accès limité
                            └── monitoring
                            │
                            ▼
                            Restore test
                            ├── sandbox
                            ├── vérification intégrité
                            ├── test applicatif
                            └── mesure RTO

Types de restauration

Runbook restore décisionnel

Before restore:
                            - Confirm incident type: corruption, deletion, bad migration, outage.
                            - Freeze risky writes if needed.
                            - Identify last known good time.
                            - Confirm available backups / WAL / binlogs.
                            - Estimate data loss window.
                            - Validate with business owner.
                            - Prefer restore to parallel environment first.
                            - Document decision and approvers.

                            During restore:
                            - Follow tested procedure.
                            - Preserve original data if possible.
                            - Monitor logs, disk, replication.

                            After restore:
                            - Verify application health.
                            - Verify data consistency.
                            - Run business checks.
                            - Document RPO/RTO observed.

Restore parallèle recommandé

Incident data
                            │
                            ▼
                            Ne pas écraser immédiatement prod
                            │
                            ▼
                            Restaurer backup en environnement isolé
                            │
                            ├── vérifier intégrité
                            ├── comparer données
                            ├── extraire lignes/tables
                            └── préparer correction
                            │
                            ▼
                            Décision
                            ├── correction ciblée
                            ├── restore complet
                            ├── PITR
                            └── abandon restore

Questions avant restore complet

Dashboard DB minimal

Seuils d’alerte indicatifs

Alerte DB bien écrite

Alert:
                            - Name: PostgreSQL connections near limit
                            - Condition: active connections > 90% for 5 minutes
                            - Severity: critical
                            - Impact: application may fail to acquire DB connections
                            - Dashboard: db-production-overview
                            - Runbook: runbooks/db-connections.md
                            - First checks:
                            - pg_stat_activity by application_name
                            - last deployment
                            - worker count
                            - pool configuration
                            - long transactions
                            - First mitigations:
                            - reduce worker concurrency
                            - restart leaking service if confirmed
                            - block non-critical batch
                            - scale DB only if validated

Signaux faibles DB

• Connexions idle augmentent après un déploiement.
• Durée moyenne des requêtes monte lentement.
• WAL/binlogs grossissent plus vite que d’habitude.
• Le nombre de deadlocks augmente mais reste faible.
• Les backups prennent de plus en plus longtemps.
• Le replica lag apparaît uniquement pendant les batchs.

Checklist triage immédiat

Checklist action prudente

Checklist post-incident

Mini-cheat-sheet

# System
                            df -h
                            df -i
                            free -h
                            top
                            journalctl -u postgresql -n 100 --no-pager
                            journalctl -u mariadb -n 100 --no-pager

                            # PostgreSQL
                            select state, count(*) from pg_stat_activity group by state;
                            select pid, now() - query_start as duration, state, query
                            from pg_stat_activity
                            where state <> 'idle'
                            order by duration desc
                            limit 20;
                            select pg_cancel_backend(PID);
                            select pg_terminate_backend(PID);

                            # MySQL / MariaDB
                            show full processlist;
                            show status like 'Threads_connected';
                            show status like 'Threads_running';
                            show variables like 'max_connections';
                            show engine innodb status\G

                            # Backup examples
                            pg_dump -Fc -f backup.dump my_database
                            mysqldump --single-transaction --routines --triggers my_database > backup.sql

Carte type d’un flux web

Browser / Client API
                            │
                            ▼
                            DNS resolver
                            │
                            ▼
                            CDN / WAF éventuel
                            │
                            ▼
                            Load Balancer
                            │
                            ▼
                            Firewall / Security Group
                            │
                            ▼
                            Nginx / Reverse Proxy
                            │
                            ▼
                            Application server
                            │
                            ├── database
                            ├── cache
                            ├── queue
                            └── external APIs

Questions à poser

Interpréter les symptômes

Tests croisés

# Depuis son poste
                            curl -I https://example.com/
                            dig example.com

                            # Depuis le serveur proxy
                            curl -I http://127.0.0.1:8000/health/
                            curl -I http://app_backend:8000/health/

                            # Depuis une autre région / machine
                            curl -I https://example.com/

                            # Forcer résolution vers une IP connue
                            curl -vk --resolve example.com:443:203.0.113.10 https://example.com/

Commandes DNS utiles

# Résolution simple
                            dig example.com
                            dig www.example.com

                            # Type de record
                            dig A example.com
                            dig AAAA example.com
                            dig CNAME www.example.com
                            dig MX example.com
                            dig TXT example.com

                            # Trace complète
                            dig +trace example.com

                            # Interroger un resolver précis
                            dig @1.1.1.1 example.com
                            dig @8.8.8.8 example.com

                            # Voir TTL
                            dig example.com +noall +answer

Points à vérifier

Scénarios DNS fréquents

Runbook changement DNS

Avant changement
                            ├── réduire TTL si possible
                            ├── vérifier zone autoritative
                            ├── préparer rollback record
                            └── noter valeur actuelle

                            Pendant
                            ├── modifier record
                            ├── tester plusieurs resolvers
                            ├── tester curl vers domaine
                            └── surveiller erreurs

                            Après
                            ├── confirmer propagation
                            ├── restaurer TTL normal
                            └── documenter

Commandes TLS

# Tester HTTPS avec détails
                            curl -vk https://example.com/

                            # Voir headers uniquement
                            curl -I https://example.com/

                            # Inspecter certificat
                            openssl s_client -connect example.com:443 -servername example.com

                            # Extraire dates du certificat
                            echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
                            | openssl x509 -noout -dates -subject -issuer

                            # Certbot
                            certbot certificates
                            systemctl status certbot.timer --no-pager
                            journalctl -u certbot -n 100 --no-pager

À vérifier

Symptômes TLS

Nginx TLS typique

server {
                            listen 443 ssl http2;
                            server_name example.com www.example.com;

                            ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
                            ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

                            location / {
                            proxy_pass http://app_backend;
                            proxy_set_header Host $host;
                            proxy_set_header X-Forwarded-Proto https;
                            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                            }
                            }

Commandes Nginx

# Tester configuration
                            nginx -t

                            # Statut service
                            systemctl status nginx --no-pager

                            # Recharger sans couper brutalement
                            sudo systemctl reload nginx

                            # Logs
                            tail -n 100 /var/log/nginx/error.log
                            tail -n 100 /var/log/nginx/access.log
                            journalctl -u nginx -n 100 --no-pager

                            # Voir config chargée
                            nginx -T | less

Tester upstream local

# Depuis le serveur Nginx
                            curl -I http://127.0.0.1:8000/health/
                            curl -I http://localhost:8000/health/
                            curl -I http://app_backend:8000/health/

                            # Ports écoutés
                            ss -tulpn | grep -E ':80|:443|:8000|:8080'

Codes d’erreur proxy

Exemple proxy robuste

location / {
                            proxy_pass http://127.0.0.1:8000;
                            proxy_http_version 1.1;

                            proxy_set_header Host $host;
                            proxy_set_header X-Real-IP $remote_addr;
                            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                            proxy_set_header X-Forwarded-Proto $scheme;

                            proxy_connect_timeout 5s;
                            proxy_send_timeout 60s;
                            proxy_read_timeout 60s;
                            }

Commandes ports et firewall

# Ports ouverts localement
                            ss -tulpn
                            ss -ltnp

                            # Tester un port depuis une autre machine
                            nc -vz example.com 443
                            nc -vz 10.0.1.20 8000

                            # UFW
                            ufw status verbose

                            # iptables
                            iptables -L -n -v
                            iptables -S

                            # nftables
                            nft list ruleset

                            # Firewalld
                            firewall-cmd --list-all

Interprétation

Checklist firewall cloud

Flux port à valider

Client / LB
                            │
                            ▼
                            Security group inbound
                            │
                            ▼
                            OS firewall
                            │
                            ▼
                            Nginx port 80/443
                            │
                            ▼
                            Upstream app port 8000
                            │
                            ▼
                            Application listening
                            │
                            ▼
                            DB/cache/queue ports internes

Points à vérifier

Healthcheck robuste

# Exemple endpoint recommandé
                            GET /health/

                            Expected:
                            - HTTP 200
                            - réponse rapide
                            - pas de dépendance lourde
                            - pas d'appel externe lent
                            - vérification minimale app ready

Diagnostic target unhealthy

Target unhealthy
                            │
                            ▼
                            Healthcheck répond localement ?
                            │
                            ├── Non
                            │   ├── app down
                            │   ├── route /health cassée
                            │   ├── port incorrect
                            │   └── startup trop long
                            │
                            └── Oui
                            ▼
                            LB peut joindre target ?
                            ├── security group
                            ├── subnet / route
                            ├── NACL
                            ├── mauvais port
                            └── mauvais protocol HTTP/HTTPS

Erreurs fréquentes LB

Zones de contrôle

Tests depuis machine

# Route locale
                            ip route
                            ip addr

                            # Résolution interne
                            dig internal.service.local

                            # Test port interne
                            nc -vz 10.0.2.15 5432
                            nc -vz redis.internal 6379

                            # Traceroute si disponible
                            traceroute example.com
                            tracepath example.com

                            # Connexion HTTP interne
                            curl -v http://internal-service:8080/health/

Diagnostic routes

Timeout entre deux services
                            │
                            ▼
                            Résolution DNS OK ?
                            │
                            ├── Non → zone DNS / resolver
                            └── Oui
                            ▼
                            Route existe ?
                            ├── Non → route table
                            └── Oui
                            ▼
                            Firewall autorise ?
                            ├── Non → SG / NACL / OS firewall
                            └── Oui
                            ▼
                            Service écoute ?
                            ├── Non → app/service down
                            └── Oui → analyser protocole/timeouts

Pièges cloud fréquents

Procédure en 10 étapes

1. Confirmer l’impact : qui est touché, depuis où, depuis quand.
                            2. Noter le dernier changement : DNS, TLS, Nginx, firewall, LB, deploy.
                            3. Tester DNS : dig, resolvers publics, TTL, A/CNAME/AAAA.
                            4. Tester TLS : curl -vk, openssl s_client, expiration, SNI.
                            5. Tester HTTP : curl -I, codes 301/302/4xx/5xx.
                            6. Tester Nginx : nginx -t, logs, upstream local.
                            7. Tester ports : ss, nc, firewall OS, security groups.
                            8. Tester LB : targets healthy, listener, rules, healthcheck.
                            9. Mitiger : rollback DNS/config, reload proxy, retirer règle fautive.
                            10. Documenter timeline, cause, correctif et prévention.

Commandes incident rapide

dig example.com
                            dig +trace example.com
                            curl -I https://example.com/
                            curl -vk https://example.com/health/
                            openssl s_client -connect example.com:443 -servername example.com
                            nginx -t
                            systemctl status nginx --no-pager
                            tail -n 100 /var/log/nginx/error.log
                            ss -tulpn
                            nc -vz example.com 443
                            ip route
                            df -h

Mitigations fréquentes

Timeline incident

T0 alerte utilisateur
                            │
                            ├── test DNS
                            ├── test TLS
                            ├── test HTTP
                            ├── test proxy
                            ├── test port interne
                            ├── mitigation
                            ├── validation retour nominal
                            └── RCA réseau

Checklist diagnostic immédiat

Checklist avant correction

Checklist post-incident

Mini-cheat-sheet

# DNS
                            dig example.com
                            dig +trace example.com
                            dig @1.1.1.1 example.com
                            dig A example.com
                            dig AAAA example.com

                            # HTTP / TLS
                            curl -I https://example.com/
                            curl -vk https://example.com/health/
                            openssl s_client -connect example.com:443 -servername example.com

                            # Nginx
                            nginx -t
                            nginx -T | less
                            systemctl status nginx --no-pager
                            journalctl -u nginx -n 100 --no-pager
                            tail -n 100 /var/log/nginx/error.log

                            # Ports / firewall
                            ss -tulpn
                            nc -vz example.com 443
                            ufw status verbose
                            iptables -L -n -v
                            nft list ruleset

                            # Network
                            ip route
                            ip addr
                            traceroute example.com
                            tracepath example.com

Glossaire operationnel

Decision IAM

Une action est demandee
                            │
                            ▼
                            Identite connue ?
                            │
                            ▼
                            Role assumable ?
                            │
                            ▼
                            Policy autorise action ?
                            │
                            ▼
                            Resource autorisee ?
                            │
                            ▼
                            Conditions respectees ?
                            │
                            ▼
                            Aucun deny explicite ?
                            │
                            ▼
                            Action autorisee

Questions a poser sur chaque droit

Exemple de policy trop large

{
                            "Effect": "Allow",
                            "Action": "*",
                            "Resource": "*"
                            }

Exemple plus cible

{
                            "Effect": "Allow",
                            "Action": [
                            "s3:GetObject",
                            "s3:PutObject"
                            ],
                            "Resource": [
                            "arn:aws:s3:::my-app-prod-artifacts/*"
                            ]
                            }

Matrice de roles recommandee

Naming clair

Pattern:
                            <environment>-<system>-<purpose>-role

                            Examples:
                            dev-api-readonly-role
                            staging-terraform-plan-role
                            prod-terraform-apply-role
                            prod-security-audit-role
                            prod-breakglass-admin-role

Separation par usage

Terraform pipeline
                            │
                            ├── plan role
                            │   ├── read cloud resources
                            │   ├── read remote state
                            │   └── no destructive change
                            │
                            └── apply role
                            ├── create/update/delete allowed scope
                            ├── protected environment only
                            ├── manual approval
                            └── audit mandatory

Role design

Structure d’une policy

{
                            "Version": "2012-10-17",
                            "Statement": [
                            {
                            "Sid": "ReadSpecificBucket",
                            "Effect": "Allow",
                            "Action": [
                            "s3:GetObject",
                            "s3:ListBucket"
                            ],
                            "Resource": [
                            "arn:aws:s3:::my-app-prod-artifacts",
                            "arn:aws:s3:::my-app-prod-artifacts/*"
                            ]
                            }
                            ]
                            }

Controle des ressources

Permissions dangereuses a surveiller

Exemple condition MFA

{
                            "Effect": "Allow",
                            "Action": "sts:AssumeRole",
                            "Resource": "arn:aws:iam::123456789012:role/prod-breakglass-admin-role",
                            "Condition": {
                            "Bool": {
                            "aws:MultiFactorAuthPresent": "true"
                            }
                            }
                            }

Flux assume-role

Identite source
                            ├── humain via SSO
                            ├── GitLab OIDC
                            ├── service account
                            └── role intermediaire
                            │
                            ▼
                            sts:AssumeRole
                            │
                            ▼
                            Trust policy du role cible
                            ├── principal autorise ?
                            ├── conditions respectees ?
                            ├── MFA / OIDC / branch ?
                            └── session duration OK ?
                            │
                            ▼
                            Credentials temporaires
                            │
                            ▼
                            Actions cloud selon policy

Commande AWS CLI

aws sts assume-role \
                            --role-arn arn:aws:iam::123456789012:role/prod-readonly-role \
                            --role-session-name incident-readonly-check

Verifier son identite

aws sts get-caller-identity

Trust policy conceptuelle

{
                            "Effect": "Allow",
                            "Principal": {
                            "AWS": "arn:aws:iam::123456789012:role/gitlab-ci-base-role"
                            },
                            "Action": "sts:AssumeRole",
                            "Condition": {
                            "StringEquals": {
                            "aws:PrincipalTag/environment": "production"
                            }
                            }
                            }

Risques assume-role

Modele GitLab CI recommande

Exemple GitLab CI

terraform_plan_prod:
                            stage: plan
                            environment:
                            name: production
                            script:
                            - aws sts get-caller-identity
                            - terraform init
                            - terraform plan -out=tfplan
                            rules:
                            - if: '$CI_COMMIT_BRANCH == "main"'

                            terraform_apply_prod:
                            stage: apply
                            environment:
                            name: production
                            script:
                            - aws sts get-caller-identity
                            - terraform apply tfplan
                            when: manual
                            rules:
                            - if: '$CI_COMMIT_BRANCH == "main"'

OIDC : principe

GitLab job
                            │
                            ▼
                            OIDC token emis par GitLab
                            │
                            ▼
                            Cloud IAM trust policy
                            ├── project id
                            ├── ref / branch
                            ├── environment
                            ├── audience
                            └── protected context
                            │
                            ▼
                            Temporary credentials
                            │
                            ▼
                            Terraform plan/apply

Controles obligatoires

Matrice des droits

Separation plan / apply

Terraform plan
                            ├── read resources
                            ├── read state
                            ├── generate plan
                            └── no modification

                            Terraform apply
                            ├── modify resources
                            ├── requires protected env
                            ├── manual trigger
                            ├── approval
                            └── audit trail

Regles de separation

Workflow securise

Auteur MR
                            │
                            ├── propose changement
                            └── explique le besoin
                            │
                            ▼
                            Reviewer
                            │
                            ├── relit plan
                            ├── verifie IAM impact
                            └── approuve risque
                            │
                            ▼
                            Authorized deployer
                            │
                            ├── lance apply
                            ├── surveille prod
                            └── documente resultat

Caracteristiques d’un bon break-glass

Procedure break-glass

Before:
                            - Confirm incident severity.
                            - Open incident ticket.
                            - Identify why normal path is unavailable.
                            - Get approval from incident commander or senior owner.
                            - Use MFA.
                            - Start audit note.

                            During:
                            - Assume break-glass role.
                            - Execute minimal required actions.
                            - Avoid unrelated changes.
                            - Capture commands and results.

                            After:
                            - Exit session.
                            - Revoke temporary access if needed.
                            - Review audit logs.
                            - Document RCA.
                            - Fix normal path.

Flux break-glass

Incident critique
                            │
                            ▼
                            Chemin normal indisponible ?
                            │
                            ├── Non
                            │   └── utiliser procedure standard
                            │
                            └── Oui
                            ▼
                            Approval urgence
                            │
                            ▼
                            MFA + assume break-glass role
                            │
                            ▼
                            Action minimale
                            │
                            ▼
                            Audit logs + timeline
                            │
                            ▼
                            Revocation / review / RCA

Ce qui doit declencher une revue

Elements a auditer

Questions d’audit

- Who assumed the role?
                            - Which role was assumed?
                            - From which system or pipeline?
                            - Was MFA or OIDC used?
                            - What actions were executed?
                            - Which resources were modified?
                            - Was this linked to a MR, ticket or incident?
                            - Was the access still needed?
                            - Did any permission exceed least privilege?

Revue periodique des acces

Audit trail ideal

MR IAM
                            │
                            ├── diff policy
                            ├── risk analysis
                            ├── security approval
                            └── plan artifact
                            │
                            ▼
                            Pipeline
                            ├── OIDC identity
                            ├── assume-role
                            ├── job logs
                            └── manual trigger
                            │
                            ▼
                            Cloud audit
                            ├── API calls
                            ├── resources changed
                            ├── session name
                            └── timestamp
                            │
                            ▼
                            Review
                            ├── expected actions?
                            ├── excessive privileges?
                            └── cleanup needed?

Checklist design IAM

Checklist CI/CD IAM

Checklist audit & break-glass

Mini-cheat-sheet

# Identity
                            aws sts get-caller-identity

                            # Assume role
                            aws sts assume-role \
                            --role-arn arn:aws:iam::123456789012:role/prod-readonly-role \
                            --role-session-name ticket-1234-check

                            # Terraform IAM safety
                            terraform fmt -recursive
                            terraform validate
                            terraform plan -out=tfplan
                            terraform show -no-color tfplan

                            # Review points
                            - Who can assume the role?
                            - What actions are allowed?
                            - Which resources are in scope?
                            - Are there wildcards?
                            - Is MFA/OIDC required?
                            - Is production separated?
                            - Is break-glass audited?
                            - When was last access review?

Scénarios typiques

Chaîne d’attaque possible

Dépendance ou image compromise
                            │
                            ▼
                            Pipeline CI exécute le code
                            │
                            ├── accès variables
                            ├── accès artifacts
                            ├── accès registry
                            └── accès cloud role
                            │
                            ▼
                            Exfiltration ou injection
                            ├── secret volé
                            ├── artifact modifié
                            ├── image pushée
                            └── deploy déclenché
                            │
                            ▼
                            Production impactée
                            ├── code malveillant
                            ├── données exposées
                            ├── infra modifiée
                            └── audit difficile

Contrôles par niveau

Bonnes pratiques Docker

Dockerfile plus sûr

FROM python:3.12.3-slim

                            ENV PYTHONDONTWRITEBYTECODE=1
                            ENV PYTHONUNBUFFERED=1

                            RUN addgroup --system app && adduser --system --group app

                            WORKDIR /app

                            COPY requirements.txt .
                            RUN pip install --no-cache-dir -r requirements.txt

                            COPY . .

                            USER app

                            CMD ["gunicorn", "config.wsgi:application", "--bind", "0.0.0.0:8000"]

Scan image en CI

container_scan:
                            stage: security
                            image:
                            name: aquasec/trivy:latest
                            entrypoint: [""]
                            script:
                            - trivy image --exit-code 1 --severity HIGH,CRITICAL "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
                            rules:
                            - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

Image promotion model

Build image
                            │
                            ├── tag commit SHA
                            ├── scan
                            ├── SBOM
                            └── signature
                            │
                            ▼
                            Push registry
                            │
                            ├── immutable digest
                            ├── restricted push
                            └── retention policy
                            │
                            ▼
                            Deploy staging
                            │
                            ▼
                            Promote same digest to prod
                            │
                            ▼
                            No rebuild between staging and prod

Risques à surveiller

Contrôles dépendances

Exemple Python

# requirements.txt
                            Django==4.2.11
                            gunicorn==21.2.0
                            psycopg[binary]==3.1.18

                            # Installation reproductible
                            pip install --require-hashes -r requirements.txt

Exemple Node.js

# Installer depuis lockfile en CI
                            npm ci

                            # Audit
                            npm audit --audit-level=high

Pipeline dépendances

Dependency change
                            │
                            ▼
                            MR dédiée
                            │
                            ├── lockfile updated
                            ├── changelog reviewed
                            ├── SCA scan
                            ├── tests
                            └── security approval if critical
                            │
                            ▼
                            Build image
                            │
                            ▼
                            Scan image
                            │
                            ▼
                            Deploy staging
                            │
                            ▼
                            Promote production

Signaux d’alerte

Modèle de runners

Contrôles runner

# Vérifier runners
                            gitlab-runner verify
                            gitlab-runner list

                            # Service
                            systemctl status gitlab-runner --no-pager
                            journalctl -u gitlab-runner -n 100 --no-pager

                            # Machine
                            df -h
                            free -h
                            docker ps
                            docker images
                            docker system df

Runner sécurisé : principes

Job non fiable
                            │
                            └── runner sans secrets
                            │
                            ▼
                            Job build image
                            │
                            └── runner build isolé
                            │
                            ▼
                            Job production
                            │
                            ├── protected runner
                            ├── protected branch
                            ├── protected environment
                            ├── variables scoped
                            └── audit logs

Risques runner

Types d’outputs CI

Artifacts Terraform propres

terraform_plan:
                            stage: plan
                            script:
                            - cd "$TF_ROOT"
                            - terraform plan -out=tfplan
                            - terraform show -no-color tfplan > plan.txt
                            - terraform show -json tfplan > plan.json
                            artifacts:
                            paths:
                            - "$TF_ROOT/tfplan"
                            - "$TF_ROOT/plan.txt"
                            - "$TF_ROOT/plan.json"
                            expire_in: 3 days
                            when: always

Règles artifacts/cache

À ne pas faire

# Dangerous
                            artifacts:
                            paths:
                            - .

                            # Dangerous debug
                            script:
                            - env
                            - printenv
                            - cat .env
                            - cat terraform.tfvars

Debug sûr

echo "Job=$CI_JOB_NAME"
                            echo "Branch=$CI_COMMIT_BRANCH"
                            echo "TF_ROOT=$TF_ROOT"
                            pwd
                            ls -la

                            if [ -z "$AWS_ACCESS_KEY_ID" ]; then
                            echo "AWS_ACCESS_KEY_ID is missing"
                            else
                            echo "AWS_ACCESS_KEY_ID is present"
                            fi

Éléments de provenance

Metadata dans une image

docker build \
                            --label org.opencontainers.image.revision="$CI_COMMIT_SHA" \
                            --label org.opencontainers.image.source="$CI_PROJECT_URL" \
                            --label org.opencontainers.image.created="$CI_JOB_STARTED_AT" \
                            -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .

Provenance flow

Commit SHA
                            │
                            ▼
                            Pipeline ID
                            │
                            ▼
                            Build job
                            │
                            ├── dependencies locked
                            ├── image scanned
                            ├── SBOM generated
                            └── image signed
                            │
                            ▼
                            Image digest
                            │
                            ▼
                            Deploy staging
                            │
                            ▼
                            Promote same digest
                            │
                            ▼
                            Production release
                            │
                            ▼
                            Audit trail complete

Questions d’audit

Pipeline type

MR pipeline
                            │
                            ├── lint
                            ├── unit tests
                            ├── dependency scan
                            ├── secret scan
                            ├── build image
                            ├── image scan
                            └── SBOM
                            │
                            ▼
                            Merge main
                            │
                            ▼
                            Build release image
                            ├── tag commit SHA
                            ├── digest
                            ├── scan
                            └── signature
                            │
                            ▼
                            Deploy staging
                            │
                            ▼
                            Approval production
                            │
                            ▼
                            Deploy same digest

Exemple GitLab CI simplifié

stages:
                            - test
                            - security
                            - build
                            - publish
                            - deploy

                            dependency_scan:
                            stage: security
                            script:
                            - echo "Run dependency scan"

                            secret_scan:
                            stage: security
                            script:
                            - echo "Run secret detection"

                            build_image:
                            stage: build
                            script:
                            - docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .
                            - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

                            image_scan:
                            stage: security
                            script:
                            - trivy image --exit-code 1 --severity HIGH,CRITICAL "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

                            deploy_prod:
                            stage: deploy
                            when: manual
                            environment:
                            name: production
                            script:
                            - ./deploy.sh "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

Gates recommandés

Protected production

deploy_prod:
                            stage: deploy
                            environment:
                            name: production
                            when: manual
                            rules:
                            - if: '$CI_COMMIT_BRANCH == "main"'
                            tags:
                            - protected
                            - prod
                            script:
                            - echo "Deploying image digest: $IMAGE_DIGEST"
                            - ./deploy.sh "$IMAGE_DIGEST"

Runbook suspicion compromission

1. Stopper promotions et déploiements automatiques.
                            2. Identifier le maillon suspect :
                            - dépendance
                            - image base
                            - runner
                            - template CI
                            - registry
                            - artifact
                            3. Lister les pipelines exécutés depuis le changement suspect.
                            4. Identifier les secrets potentiellement exposés.
                            5. Révoquer / renouveler tokens et credentials.
                            6. Bloquer ou supprimer images/artifacts suspects.
                            7. Rebuilder depuis commit propre et environnement sain.
                            8. Redéployer une image vérifiée.
                            9. Auditer logs cloud, registry, GitLab et runners.
                            10. Rédiger RCA et actions préventives.

Premières actions sûres

Analyse d’impact

Maillon suspect
                            │
                            ▼
                            Quels pipelines touchés ?
                            │
                            ├── MR only
                            ├── main
                            ├── release
                            └── production deploy
                            │
                            ▼
                            Quels secrets accessibles ?
                            ├── none
                            ├── staging
                            ├── production
                            └── cloud admin
                            │
                            ▼
                            Quels outputs produits ?
                            ├── artifacts
                            ├── images
                            ├── packages
                            └── terraform plans
                            │
                            ▼
                            Actions
                            ├── revoke
                            ├── rebuild
                            ├── redeploy
                            ├── audit
                            └── RCA

Questions incident

Checklist projet

Checklist avant production

Checklist audit / incident

Mini-cheat-sheet

# Docker
                            docker image inspect IMAGE
                            docker history IMAGE
                            docker scan IMAGE
                            trivy image IMAGE

                            # GitLab runner
                            gitlab-runner verify
                            gitlab-runner list
                            systemctl status gitlab-runner --no-pager

                            # Registry / image
                            docker pull registry/app:tag
                            docker inspect --format='index .RepoDigests 0' registry/app:tag

                            # CI safety
                            - no latest
                            - no secrets in logs
                            - no secrets in artifacts
                            - pin templates by tag
                            - protected runner for prod
                            - deploy digest, not mutable tag
                            - same image from staging to prod
                            - revoke secrets if exposed

Glossaire opérationnel

Commandes de lecture

kubectl config current-context
                            kubectl get namespaces
                            kubectl get all -n my-namespace
                            kubectl get pods -n my-namespace -o wide
                            kubectl describe pod POD_NAME -n my-namespace
                            kubectl logs POD_NAME -n my-namespace
                            kubectl get events -n my-namespace --sort-by=.lastTimestamp

Relations entre objets

Deployment
                            │
                            ▼
                            ReplicaSet
                            │
                            ▼
                            Pods
                            │
                            ├── labels: app=api
                            └── containers
                            │
                            ▼
                            Service
                            │
                            ├── selector: app=api
                            └── stable DNS name
                            │
                            ▼
                            Ingress
                            │
                            ├── host: api.example.com
                            └── path: /

Labels et selectors

Exemple labels

metadata:
                            labels:
                            app: api
                            environment: production

                            spec:
                            selector:
                            matchLabels:
                            app: api

Deployment minimal

apiVersion: apps/v1
                            kind: Deployment
                            metadata:
                            name: api
                            namespace: production
                            spec:
                            replicas: 3
                            selector:
                            matchLabels:
                            app: api
                            strategy:
                            type: RollingUpdate
                            rollingUpdate:
                            maxUnavailable: 1
                            maxSurge: 1
                            template:
                            metadata:
                            labels:
                            app: api
                            spec:
                            containers:
                            - name: api
                            image: registry.example.com/api:1.8.2
                            ports:
                            - containerPort: 8000
                            readinessProbe:
                            httpGet:
                            path: /health/
                            port: 8000
                            livenessProbe:
                            httpGet:
                            path: /health/
                            port: 8000

Commandes rollout

kubectl get deploy -n production
                            kubectl rollout status deployment/api -n production
                            kubectl rollout history deployment/api -n production
                            kubectl describe deployment/api -n production

                            # Rollback
                            kubectl rollout undo deployment/api -n production

                            # Voir les pods associés
                            kubectl get pods -n production -l app=api -o wide

États fréquents des pods

Service ClusterIP

apiVersion: v1
                            kind: Service
                            metadata:
                            name: api
                            namespace: production
                            spec:
                            type: ClusterIP
                            selector:
                            app: api
                            ports:
                            - name: http
                            port: 80
                            targetPort: 8000

Ingress HTTP

apiVersion: networking.k8s.io/v1
                            kind: Ingress
                            metadata:
                            name: api
                            namespace: production
                            spec:
                            rules:
                            - host: api.example.com
                            http:
                            paths:
                            - path: /
                            pathType: Prefix
                            backend:
                            service:
                            name: api
                            port:
                            number: 80

Commandes réseau

kubectl get svc -n production
                            kubectl get endpoints -n production
                            kubectl get ingress -n production
                            kubectl describe svc api -n production
                            kubectl describe ingress api -n production

Chemin réseau Kubernetes

Client externe
                            │
                            ▼
                            DNS public
                            │
                            ▼
                            Load Balancer cloud
                            │
                            ▼
                            Ingress Controller
                            │
                            ▼
                            Ingress rule
                            │
                            ▼
                            Service
                            │
                            ▼
                            Endpoints
                            │
                            ▼
                            Pods ready

Pannes réseau fréquentes

Debug depuis un pod temporaire

kubectl run debug-shell -n production --rm -it \
                            --image=curlimages/curl -- sh

                            # Dans le pod
                            curl -v http://api.production.svc.cluster.local/health/
                            nslookup api.production.svc.cluster.local

ConfigMap

apiVersion: v1
                            kind: ConfigMap
                            metadata:
                            name: api-config
                            namespace: production
                            data:
                            DJANGO_SETTINGS_MODULE: "config.settings.production"
                            LOG_LEVEL: "INFO"
                            FEATURE_X_ENABLED: "false"

Secret

apiVersion: v1
                            kind: Secret
                            metadata:
                            name: api-secret
                            namespace: production
                            type: Opaque
                            stringData:
                            DATABASE_URL: "postgres://user:password@db:5432/app"
                            SECRET_KEY: "change-me"

Injection dans un Deployment

envFrom:
                            - configMapRef:
                            name: api-config
                            - secretRef:
                            name: api-secret

Différences ConfigMap / Secret

Commandes de vérification

kubectl get configmap -n production
                            kubectl get secret -n production
                            kubectl describe pod POD_NAME -n production

                            # Ne pas afficher les secrets en clair pendant un debug partagé
                            kubectl get secret api-secret -n production -o yaml

Helm : structure typique

chart-api/
                            ├── Chart.yaml
                            ├── values.yaml
                            ├── values-staging.yaml
                            ├── values-production.yaml
                            └── templates/
                            ├── deployment.yaml
                            ├── service.yaml
                            ├── ingress.yaml
                            ├── configmap.yaml
                            └── hpa.yaml

Commandes Helm

helm lint ./chart-api

                            helm template api ./chart-api \
                            -f values-production.yaml

                            helm upgrade --install api ./chart-api \
                            -n production \
                            -f values-production.yaml \
                            --atomic \
                            --timeout 5m

                            helm list -n production
                            helm history api -n production
                            helm rollback api 3 -n production

Kustomize : overlays

k8s/
                            ├── base/
                            │   ├── deployment.yaml
                            │   ├── service.yaml
                            │   └── kustomization.yaml
                            │
                            └── overlays/
                            ├── staging/
                            │   └── kustomization.yaml
                            └── production/
                            └── kustomization.yaml

Helm vs Kustomize

Bonnes pratiques values

Pipeline type

Commit / MR
                            │
                            ├── tests
                            ├── build image
                            ├── image scan
                            └── helm lint/template
                            │
                            ▼
                            Merge main
                            │
                            ├── build release image
                            ├── push registry
                            └── deploy staging
                            │
                            ▼
                            Smoke tests
                            │
                            ▼
                            Production approval
                            │
                            ▼
                            helm upgrade --install
                            │
                            ▼
                            rollout status + post-checks

Exemple GitLab CI simplifié

stages:
                            - test
                            - build
                            - scan
                            - deploy_staging
                            - deploy_prod

                            build_image:
                            stage: build
                            script:
                            - docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .
                            - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

                            helm_lint:
                            stage: test
                            script:
                            - helm lint ./chart

                            deploy_staging:
                            stage: deploy_staging
                            environment:
                            name: staging
                            script:
                            - helm upgrade --install api ./chart -n staging
                            --set image.tag="$CI_COMMIT_SHA"

                            deploy_prod:
                            stage: deploy_prod
                            environment:
                            name: production
                            when: manual
                            script:
                            - helm upgrade --install api ./chart -n production
                            --set image.tag="$CI_COMMIT_SHA"
                            - kubectl rollout status deployment/api -n production

Garde-fous CI/CD

Variables utiles

KUBE_NAMESPACE=production
                            HELM_RELEASE=api
                            HELM_CHART_PATH=./chart
                            IMAGE_TAG=$CI_COMMIT_SHA
                            KUBE_CONTEXT=production-cluster

Runbook debug rapide

# 1. Vérifier contexte
                            kubectl config current-context

                            # 2. Voir les objets
                            kubectl get all -n production

                            # 3. Voir les pods
                            kubectl get pods -n production -o wide

                            # 4. Décrire le pod
                            kubectl describe pod POD_NAME -n production

                            # 5. Lire les logs
                            kubectl logs POD_NAME -n production
                            kubectl logs POD_NAME -n production --previous

                            # 6. Lire les events
                            kubectl get events -n production --sort-by=.lastTimestamp

                            # 7. Vérifier rollout
                            kubectl rollout status deployment/api -n production

                            # 8. Vérifier service/endpoints
                            kubectl get svc,endpoints -n production

Debug logs

# Logs d'un container spécifique
                            kubectl logs POD_NAME -c CONTAINER_NAME -n production

                            # Follow logs
                            kubectl logs -f deployment/api -n production

                            # Logs précédents si crash
                            kubectl logs POD_NAME -n production --previous

Diagnostic par symptôme

Arbre de décision

Application KO
                            │
                            ▼
                            Pods running ?
                            ├── Non
                            │   ├── Pending
                            │   ├── ImagePullBackOff
                            │   └── CrashLoopBackOff
                            │
                            └── Oui
                            ▼
                            Pods ready ?
                            ├── Non → probes / dependencies
                            └── Oui
                            ▼
                            Service endpoints ?
                            ├── Non → labels/selectors
                            └── Oui
                            ▼
                            Ingress / LB / DNS

Responsabilités recommandées

Architecture propre

Terraform
                            ├── cloud network
                            ├── IAM / service accounts
                            ├── Kubernetes cluster
                            ├── node groups
                            ├── DNS / registry
                            └── platform add-ons optional

                            Helm / Kustomize
                            ├── application deployment
                            ├── service
                            ├── ingress
                            ├── configmap
                            ├── hpa
                            └── environment values

                            GitLab CI
                            ├── build image
                            ├── scan image
                            ├── deploy staging
                            ├── smoke tests
                            └── deploy production

Pièges Terraform + Kubernetes

Règle de propriété

Resource ownership:
                            - VPC: Terraform
                            - EKS/GKE/AKS cluster: Terraform
                            - Node pools: Terraform
                            - Ingress controller: Platform chart
                            - Application Deployment: Helm/Kustomize
                            - Application image tag: CI/CD release
                            - Runtime secrets: Secret manager integration

Checklist avant déploiement

Checklist debug

Checklist production

Mini-cheat-sheet

# Context
                            kubectl config current-context
                            kubectl get ns

                            # Workloads
                            kubectl get all -n production
                            kubectl get pods -n production -o wide
                            kubectl describe pod POD -n production
                            kubectl logs POD -n production
                            kubectl logs POD -n production --previous

                            # Rollout
                            kubectl rollout status deployment/api -n production
                            kubectl rollout history deployment/api -n production
                            kubectl rollout undo deployment/api -n production

                            # Network
                            kubectl get svc,endpoints -n production
                            kubectl get ingress -n production
                            kubectl describe ingress api -n production

                            # Helm
                            helm list -n production
                            helm history api -n production
                            helm rollback api REVISION -n production

Modele de comptes recommande

Commandes utilisateurs

# Creer un utilisateur nominatif
                            sudo adduser alice
                            sudo usermod -aG sudo alice

                            # Voir groupes
                            id alice
                            groups alice

                            # Voir sudo autorise
                            sudo -l

                            # Verifier comptes avec shell
                            awk -F: '$7 !~ /(nologin|false)/ {print $1, $7}' /etc/passwd

                            # Verifier derniers logins
                            last
                            lastlog

Sudoers propre

# Toujours utiliser visudo
                            sudo visudo

                            # Exemple fichier dedie
                            sudo visudo -f /etc/sudoers.d/deploy

                            # Exemple limite
                            deploy ALL=(root) NOPASSWD: /bin/systemctl restart myapp.service, /bin/systemctl status myapp.service

Principes sudo

Audit rapide des comptes

Audit users
                            │
                            ├── comptes humains actifs
                            ├── comptes techniques
                            ├── membres sudo
                            ├── shells ouverts
                            ├── cles SSH autorisees
                            └── comptes inactifs a supprimer

Configuration SSH recommandee

# /etc/ssh/sshd_config.d/hardening.conf

                            PermitRootLogin no
                            PasswordAuthentication no
                            PubkeyAuthentication yes
                            PermitEmptyPasswords no
                            ChallengeResponseAuthentication no
                            X11Forwarding no
                            AllowTcpForwarding no
                            ClientAliveInterval 300
                            ClientAliveCountMax 2
                            MaxAuthTries 3

                            # Exemple : limiter aux users autorises
                            AllowUsers guillaume deploy

Validation SSH

# Tester syntaxe
                            sudo sshd -t

                            # Recharger sans couper les sessions existantes
                            sudo systemctl reload ssh
                            sudo systemctl status ssh --no-pager

                            # Garder une session ouverte avant test nouvelle connexion
                            ssh -v user@server

Audit SSH

# Logs auth Debian/Ubuntu
                            sudo tail -n 100 /var/log/auth.log
                            sudo grep "Failed password" /var/log/auth.log | tail
                            sudo grep "Accepted publickey" /var/log/auth.log | tail

                            # Voir cles autorisees
                            sudo find /home -name authorized_keys -type f -print

                            # Permissions importantes
                            ls -ld ~/.ssh
                            ls -l ~/.ssh/authorized_keys

Permissions SSH

Firewall SSH

# UFW : exemple restrictif
                            sudo ufw allow from 203.0.113.10 to any port 22 proto tcp
                            sudo ufw deny 22/tcp
                            sudo ufw status verbose

Commandes ports

# Ports en ecoute
                            sudo ss -tulpn
                            sudo ss -ltnp

                            # Process associe a un port
                            sudo lsof -iTCP -sTCP:LISTEN -P -n

                            # Test externe depuis une autre machine
                            nc -vz example.com 443
                            nc -vz example.com 22

UFW exemple

sudo ufw default deny incoming
                            sudo ufw default allow outgoing

                            # Web public
                            sudo ufw allow 80/tcp
                            sudo ufw allow 443/tcp

                            # SSH limite a une IP admin
                            sudo ufw allow from 203.0.113.10 to any port 22 proto tcp

                            sudo ufw enable
                            sudo ufw status verbose

nftables / iptables

# nftables
                            sudo nft list ruleset

                            # iptables
                            sudo iptables -L -n -v
                            sudo iptables -S

Matrice d'exposition

Diagnostic firewall

Service inaccessible
                            │
                            ▼
                            Service ecoute ?
                            ├── Non → app/service down
                            └── Oui
                            ▼
                            Port ouvert OS firewall ?
                            ├── Non → UFW/nft/iptables
                            └── Oui
                            ▼
                            Security Group / cloud firewall ?
                            ├── Non → ouvrir source minimale
                            └── Oui → verifier route / LB / DNS

Exemple service systemd

# /etc/systemd/system/myapp.service
                            [Unit]
                            Description=My Django Application
                            After=network.target
                            Wants=network-online.target

                            [Service]
                            User=app
                            Group=app
                            WorkingDirectory=/opt/myapp/current
                            EnvironmentFile=/etc/myapp/myapp.env
                            ExecStart=/opt/myapp/current/venv/bin/gunicorn config.wsgi:application --bind 127.0.0.1:8000
                            Restart=on-failure
                            RestartSec=5
                            TimeoutStopSec=30
                            KillSignal=SIGTERM

                            NoNewPrivileges=true
                            PrivateTmp=true
                            ProtectSystem=full
                            ProtectHome=true

                            [Install]
                            WantedBy=multi-user.target

Commandes systemd

sudo systemctl daemon-reload
                            sudo systemctl enable myapp
                            sudo systemctl start myapp
                            sudo systemctl status myapp --no-pager
                            sudo systemctl restart myapp
                            sudo journalctl -u myapp -n 100 --no-pager
                            sudo journalctl -u myapp -f

Options importantes

Diagnostic systemd

Service KO
                            │
                            ▼
                            systemctl status
                            │
                            ├── inactive
                            ├── failed
                            ├── restart loop
                            └── active but unhealthy
                            │
                            ▼
                            journalctl -u service
                            │
                            ├── erreur config
                            ├── permission denied
                            ├── port already in use
                            ├── env missing
                            └── dependency failed

Service hardening check

systemd-analyze security myapp.service

Commandes journald

# Logs d'un service
                            journalctl -u nginx -n 100 --no-pager
                            journalctl -u myapp -f

                            # Logs depuis une date
                            journalctl -u myapp --since "2026-04-30 10:00:00"

                            # Erreurs systeme recentes
                            journalctl -p err -n 100 --no-pager

                            # Kernel logs
                            journalctl -k -n 100 --no-pager

                            # Taille journald
                            journalctl --disk-usage

Rotation journald

# Nettoyage ponctuel
                            sudo journalctl --vacuum-time=14d
                            sudo journalctl --vacuum-size=1G

                            # /etc/systemd/journald.conf
                            SystemMaxUse=1G
                            MaxRetentionSec=14day

Logs classiques

Audit acces

# Connexions SSH reussies
                            grep "Accepted" /var/log/auth.log | tail

                            # Echecs SSH
                            grep "Failed password" /var/log/auth.log | tail

                            # Sudo
                            grep "sudo:" /var/log/auth.log | tail

                            # Derniers logins
                            last
                            who

Ce qu'il ne faut pas logger

Commandes permissions

# Voir permissions
                            ls -la /opt/myapp
                            namei -l /opt/myapp/current/.env

                            # Changer owner
                            sudo chown -R app:app /opt/myapp/current

                            # Permissions dossiers/fichiers
                            sudo find /opt/myapp/current -type d -exec chmod 750 {} \;
                            sudo find /opt/myapp/current -type f -exec chmod 640 {} \;

                            # Executable script
                            sudo chmod 750 /opt/myapp/current/scripts/deploy.sh

Secrets applicatifs

# Exemple
                            sudo mkdir -p /etc/myapp
                            sudo chown root:app /etc/myapp
                            sudo chmod 750 /etc/myapp

                            sudo chown root:app /etc/myapp/myapp.env
                            sudo chmod 640 /etc/myapp/myapp.env

Permissions recommandees

Recherche permissions dangereuses

# World writable
                            sudo find / -xdev -type d -perm -0002 -print 2>/dev/null

                            # Fichiers avec permission 777
                            sudo find / -xdev -perm -0777 -print 2>/dev/null

                            # SUID files
                            sudo find / -xdev -perm -4000 -type f -print 2>/dev/null

Commandes Debian / Ubuntu

sudo apt update
                            apt list --upgradable

                            # Simuler upgrade
                            sudo apt -s upgrade

                            # Appliquer
                            sudo apt upgrade

                            # Security updates si unattended-upgrades configure
                            sudo unattended-upgrade --dry-run --debug

                            # Reboot requis ?
                            test -f /var/run/reboot-required && cat /var/run/reboot-required

Verifier versions

uname -a
                            cat /etc/os-release
                            nginx -v
                            python3 --version
                            openssl version
                            systemctl --version

Process patch production

Avant patch
                            ├── verifier backups / snapshot
                            ├── lire packages upgradable
                            ├── identifier services critiques
                            ├── prevoir fenetre
                            └── preparer rollback

                            Pendant
                            ├── appliquer updates
                            ├── surveiller logs
                            ├── redemarrer services si besoin
                            └── reboot si necessaire

                            Apres
                            ├── health checks
                            ├── logs propres
                            ├── ports OK
                            ├── metrics stables
                            └── note de changement

Politique recommandee

Procedure audit rapide serveur

1. Identifier serveur, role et environnement.
                            2. Verifier uptime, disque, RAM, CPU.
                            3. Lister ports ecoutes.
                            4. Verifier firewall OS et cloud.
                            5. Verifier SSH : root login, password auth, users.
                            6. Verifier services systemd critiques.
                            7. Lire logs auth et logs services.
                            8. Verifier permissions des secrets.
                            9. Verifier updates disponibles et reboot requis.
                            10. Documenter risques et actions.

Commandes audit rapide

hostnamectl
                            uptime
                            df -h
                            free -h
                            ss -tulpn
                            systemctl --failed
                            systemctl list-units --type=service --state=running
                            journalctl -p err -n 100 --no-pager
                            sudo ufw status verbose
                            sudo grep -E "PermitRootLogin|PasswordAuthentication" /etc/ssh/sshd_config /etc/ssh/sshd_config.d/*

Mitigations courantes

Timeline action sensible

Avant action
                            ├── contexte confirme
                            ├── rollback connu
                            ├── session SSH de secours
                            └── commande relue

                            Action
                            ├── modification minimale
                            ├── test syntaxe
                            ├── reload plutot que restart si possible
                            └── logs surveilles

                            Apres action
                            ├── service OK
                            ├── ports OK
                            ├── healthcheck OK
                            └── note de changement

Checklist securite serveur

Checklist avant changement sensible

Checklist diagnostic production

Mini-cheat-sheet

# System
                            hostnamectl
                            uptime
                            df -h
                            df -i
                            free -h
                            ss -tulpn

                            # SSH
                            sudo sshd -t
                            sudo systemctl reload ssh
                            grep "Failed password" /var/log/auth.log | tail
                            grep "Accepted publickey" /var/log/auth.log | tail

                            # Firewall
                            sudo ufw status verbose
                            sudo nft list ruleset
                            sudo iptables -L -n -v

                            # Systemd
                            systemctl --failed
                            systemctl status myapp --no-pager
                            journalctl -u myapp -n 100 --no-pager
                            systemd-analyze security myapp.service

                            # Updates
                            sudo apt update
                            apt list --upgradable
                            test -f /var/run/reboot-required && cat /var/run/reboot-required

Niveaux de progression

Competences par palier

Escalier d'autonomie

Niveau 0
                            Observer
                            │
                            ▼
                            Niveau 1
                            Expliquer
                            │
                            ▼
                            Niveau 2
                            Executer en sandbox
                            │
                            ▼
                            Niveau 3
                            Changer staging
                            │
                            ▼
                            Niveau 4
                            Production accompagnee
                            │
                            ▼
                            Niveau 5
                            Production standard autonome
                            │
                            ▼
                            Niveau 6
                            Incident lead junior supervise

Critere pour passer au niveau suivant

Catalogue d'exercices

Exercice : lire un plan Terraform

Mission:
                            - Ouvrir plan.txt
                            - Compter create / update / delete
                            - Identifier ressources critiques
                            - Expliquer pourquoi le changement est attendu
                            - Identifier rollback ou mitigation

                            Questions:
                            - Y a-t-il un destroy ?
                            - Le state semble-t-il correspondre au bon environnement ?
                            - Les tags/naming sont-ils corrects ?
                            - Le changement touche-t-il production ?
                            - Qui doit approuver ?

Exercice : pipeline GitLab casse

Scenario:
                            - terraform_plan echoue
                            - Message: TF_ROOT directory not found
                            - Objectif: diagnostiquer sans modifier production

                            Steps:
                            1. Lire le log depuis le debut.
                            2. Afficher CI_COMMIT_BRANCH, CI_PIPELINE_SOURCE, TF_ROOT.
                            3. Verifier pwd et ls.
                            4. Corriger la variable dans la MR.
                            5. Relancer uniquement le job utile.
                            6. Documenter la cause.

Exercice : service systemd

Scenario:
                            - myapp.service est failed
                            - Le endpoint /health ne repond plus

                            Commands:
                            systemctl status myapp --no-pager
                            journalctl -u myapp -n 100 --no-pager
                            ss -tulpn | grep 8000
                            cat /etc/myapp/myapp.env
                            ls -la /opt/myapp/current

                            Expected:
                            - Identifier cause
                            - Proposer mitigation
                            - Relancer proprement si justifie
                            - Verifier health endpoint

Bibliotheque d'erreurs pedagogiques

Format d'un exercice d'erreur

Exercise:
                            - Title: Broken GitLab artifact
                            - Environment: sandbox only
                            - Injected error: wrong artifact path
                            - Expected symptom: apply job cannot find tfplan
                            - Allowed actions: read logs, inspect job config, MR fix
                            - Forbidden actions: bypass apply, recreate state
                            - Success: trainee explains cause and fix

Cycle d'apprentissage par erreur

Erreur volontaire
                            │
                            ▼
                            Junior observe
                            ├── logs
                            ├── metrics
                            ├── pipeline
                            └── config
                            │
                            ▼
                            Hypothese
                            ├── cause probable
                            ├── risque
                            └── test de validation
                            │
                            ▼
                            Correction
                            ├── MR
                            ├── commande sandbox
                            └── verification
                            │
                            ▼
                            Debrief
                            ├── ce qui a marche
                            ├── ce qui a manque
                            ├── runbook update
                            └── point de vigilance

Regles de securite

Checklist de revue MR DevOps

Exemple commentaire MR

Review comment:
                            - The change is clear and limited to staging.
                            - Terraform plan shows 2 creates and 0 destroy.
                            - Please add the rollback note:
                            "Revert this MR and re-apply previous module version."
                            - Please include post-check:
                            curl -fsS https://staging.example.com/health/
                            - Approved after rollback note is added.

Revue de plan Terraform

Terraform plan review
                            │
                            ├── resources added
                            ├── resources changed
                            ├── resources destroyed
                            ├── IAM changes
                            ├── network exposure
                            ├── database impact
                            ├── cost impact
                            └── rollback path

Feedback senior utile

Rituels recommandés

Agenda type : revue de plan

Duration: 30 minutes

                            1. Context of the change
                            2. Read Terraform plan summary
                            3. Identify risky resources
                            4. Discuss IAM/network/database impact
                            5. Define rollback
                            6. Define post-checks
                            7. Capture learning points

Cycle de rituel d'equipe

Chaque semaine
                            │
                            ├── 1 plan Terraform commente
                            ├── 1 pipeline explique
                            ├── 1 log incident lu
                            └── 1 runbook ameliore
                            │
                            ▼
                            Chaque mois
                            ├── game day staging
                            ├── access review simple
                            ├── backup/restore drill
                            └── documentation cleanup
                            │
                            ▼
                            Chaque incident
                            ├── timeline
                            ├── RCA
                            ├── action preventive
                            └── formation associee

Bonnes pratiques de rituel

Reflexes production

Phrase que le junior doit savoir formuler

I am changing:
                            - what: the exact component
                            - where: the exact environment
                            - why: the business or technical reason
                            - risk: what can go wrong
                            - rollback: how to revert
                            - validation: how to prove it works
                            - owner: who is accountable

Decision production

Changement demande
                            │
                            ▼
                            Risque faible ?
                            ├── Oui
                            │   ├── MR
                            │   ├── plan/review
                            │   └── apply standard
                            │
                            └── Non
                            ▼
                            Risque moyen/fort
                            ├── fenetre de changement
                            ├── senior approver
                            ├── rollback teste
                            ├── dashboards ouverts
                            ├── communication
                            └── post-checks

Ce qu'il faut enseigner explicitement

Grille d'evaluation

Exercice d'evaluation

Scenario:
                            - A staging deployment failed.
                            - The app is returning 502.
                            - GitLab pipeline was green.
                            - Nginx is running.
                            - The app service is failed.

                            Expected trainee actions:
                            1. Confirm environment.
                            2. Read Nginx error logs.
                            3. Check systemd service status.
                            4. Read journalctl.
                            5. Identify missing EnvironmentFile.
                            6. Propose fix.
                            7. Validate health endpoint.
                            8. Write short RCA.

Notation qualitative

Signaux positifs

• Le junior lit les logs avant de relancer.
• Il annonce l'environnement cible avant une commande.
• Il detecte les secrets ou outputs sensibles.
• Il propose un rollback avant un changement.
• Il documente clairement ce qu'il a appris.
• Il escalade quand l'incertitude devient dangereuse.

Plan 30 jours

Plan 60 jours

Plan 90 jours

Roadmap visuelle

Jours 1-30
                            Lire, comprendre, observer
                            │
                            ▼
                            Jours 31-60
                            Pratiquer, corriger staging, documenter
                            │
                            ▼
                            Jours 61-90
                            Contribuer production accompagnee
                            │
                            ▼
                            Apres 90 jours
                            Autonomie standard + apprentissage continu

Checklist onboarding

Checklist exercice

Checklist autonomie production

Mini-cheat-sheet formation

Training essentials:
                            - Start with reading and observation
                            - Use sandbox before staging
                            - Use staging before production
                            - Teach rollback before apply
                            - Teach logs before restart
                            - Teach plan review before merge
                            - Teach secret hygiene before CI debug
                            - Use game days for incident reflexes
                            - Use MR reviews for real feedback
                            - Document every repeated lesson

Termes production essentiels

Difference SLA / SLO / SLI

SLI = ce que l'on mesure
                            └── ex: taux de 5xx

                            SLO = objectif interne
                            └── ex: 5xx < 1% sur 30 jours

                            SLA = engagement externe
                            └── ex: credit client si disponibilite non respectee

Termes lies au changement

Phrase type

Before production change:
                            - SLI to watch: HTTP 5xx and p95 latency
                            - SLO risk: latency may exceed target during rollout
                            - Rollback: deploy previous image digest
                            - RTO target: service restored within 15 minutes
                            - RPO impact: no expected data loss

Glossaire GitLab CI/CD

Flux CI/CD

Commit
                            │
                            ▼
                            Pipeline
                            ├── lint
                            ├── test
                            ├── build
                            ├── security scan
                            ├── terraform plan
                            └── deploy
                            │
                            ▼
                            Artifacts
                            ├── image
                            ├── plan
                            ├── reports
                            └── release notes

Termes souvent confondus

Phrase de debug

The pipeline failed in the plan stage.
                            The job ran on the protected runner.
                            The artifact tfplan was not produced.
                            The apply job cannot continue because it depends on that artifact.

Glossaire Terraform

Cycle Terraform

Code Terraform
                            │
                            ▼
                            terraform fmt
                            │
                            ▼
                            terraform validate
                            │
                            ▼
                            terraform plan
                            │
                            ├── create
                            ├── update
                            ├── replace
                            └── destroy
                            │
                            ▼
                            Review humaine
                            │
                            ▼
                            terraform apply
                            │
                            ▼
                            State mis a jour

Commandes associees

terraform fmt -recursive
                            terraform validate
                            terraform init
                            terraform plan -out=tfplan
                            terraform show -no-color tfplan
                            terraform show -json tfplan
                            terraform apply tfplan
                            terraform state list
                            terraform state show RESOURCE
                            terraform import RESOURCE ID

Phrase de revue

The plan creates 2 resources, updates 1 security group,
                            and destroys 0 resources. The backend points to staging,
                            not production. No unexpected replacement is visible.

Glossaire cloud

Chemin reseau type

User
                            │
                            ▼
                            DNS
                            │
                            ▼
                            Load Balancer
                            │
                            ▼
                            Security Group
                            │
                            ▼
                            Subnet / Route Table
                            │
                            ▼
                            Instance / Pod / Service
                            │
                            ▼
                            Database / Cache / Queue

Confusions frequentes

Commandes terrain

dig example.com
                            curl -vk https://example.com/health/
                            nc -vz example.com 443
                            ip route
                            ss -tulpn
                            traceroute example.com
                            tracepath example.com

Glossaire securite DevOps

Exemples de phrases securite

This role violates least privilege because it allows Action="*".

                            The GitLab job should use OIDC instead of long-lived cloud keys.

                            The secret was printed in logs, so it must be rotated.

                            This break-glass access must be linked to an incident ticket.

                            The blast radius is limited to staging because the role cannot assume prod.

Termes a associer

Glossaire observabilite

Golden signals

Golden signals
                            │
                            ├── Latency
                            │   └── how long requests take
                            │
                            ├── Traffic
                            │   └── how much demand exists
                            │
                            ├── Errors
                            │   └── how often requests fail
                            │
                            └── Saturation
                            └── how full the system is

Questions d'observabilite

Phrase incident

The alert fired because the p95 latency exceeded the SLO threshold.
                            Traffic is normal, but saturation increased on the database.
                            The logs show lock waits starting after the last deployment.

Glossaire incident

Cycle incident

Detection
                            │
                            ▼
                            Triage
                            ├── impact
                            ├── severity
                            ├── owner
                            └── timeline
                            │
                            ▼
                            Mitigation
                            ├── rollback
                            ├── scale
                            ├── disable feature
                            └── traffic control
                            │
                            ▼
                            Recovery
                            ├── service stable
                            ├── metrics normal
                            └── users OK
                            │
                            ▼
                            Post-mortem
                            ├── RCA
                            ├── action items
                            ├── runbook update
                            └── prevention

Phrases utiles en crise

We are still in triage.
                            The current mitigation is rollback to the previous image.
                            The customer impact is limited to login latency.
                            The service has recovered, but RCA is not complete.
                            The action item is to add an alert on DB lock waits.

En revue Terraform

The Terraform plan is clean: no unexpected destroy.

                            This change increases the blast radius because it opens the security group.

                            The state backend points to production, so the apply must be protected.

                            The drift should be investigated before applying more changes.

                            The module version should be pinned to avoid uncontrolled updates.

En debug CI/CD

The job failed before producing the artifact.

                            The runner does not have access to the protected variable.

                            The pipeline is running from a merge request, so production secrets are not available.

                            The apply job should depend on the exact plan artifact.

                            The include should be pinned to a versioned template tag.

En incident production

The impact is confirmed on production traffic.

                            The first mitigation is to rollback the last deployment.

                            The current hypothesis is database connection saturation.

                            We need to check logs before restarting the service.

                            The service is stable again, but the RCA is still pending.

En securite

The role is too broad and violates least privilege.

                            The secret was exposed in logs and must be rotated.

                            The production runner should be protected and isolated.

                            Break-glass access must be temporary, audited and linked to a ticket.

                            OIDC is preferable to long-lived cloud credentials in CI.

Checklist vocabulaire pour junior

Checklist communication incident

Mini-glossaire prioritaire

Mini-cheat-sheet vocabulaire

Use precise words:
                            - impact, not "it is broken"
                            - rollback, not "we will fix somehow"
                            - hypothesis, not "root cause" too early
                            - artifact, not cache
                            - SLI/SLO/SLA with correct meaning
                            - state, not Terraform code
                            - mitigation, not permanent fix
                            - post-check, not "looks fine"