adeelarshad414/devops-interns-proj: devSecOps teaching target, so a few checks are designed to report

DevOps by Breaking It on Purpose" width="880" />

A deliberately breakable three-tier platform for teaching DevOps. (aka the DevOps Interns Project — Cloud · DevOps · SRE · DevSecOps · Platform Engineering)

Built as the training substrate for a DevOps intern rotation.

The core stack now runs, and CI proves it. Every push builds all four images and stands the full stack up — postgres, redis, orders, kitchen, dispatch, web — then places a real order end-to-end (Integration - full stack, a required merge gate). What is not yet executed — a live terraform apply, the OpenBao end-to-end flow, a real cluster — is tracked honestly in VERIFICATION.md. "I ran it" and "it should work" are different claims, and this repo does not blur them — which is itself the first thing it teaches.

⚠️ Reading the CI signal (some red is the point)

This repo is a DevSecOps teaching target, so a few checks are designed to report findings:

• SAST / dependency / IaC-policy jobs flag the deliberately-planted vulnerabilities (services/orders/src/insecure.js, the TLS-less ALB, the inline k8s Secret…). A scanner going red here means it's working — that is the lesson (see SECURITY.md).
• Core CI (static checks, unit tests, image build) is expected green — that's the badge above and the required merge gate on main.
• Application dependencies are at 0 known vulnerabilities; the security jobs fire on the planted application code, not on outdated packages.

📚 Start here — the learning path

1. TRAINING.md — one-stop handbook: every area's concepts, cheatsheets and guidelines (Linux → Docker → k8s → IaC → CI/CD → DevSecOps → Vault → SRE → chaos), each grounded in a real file here.
2. ASSIGNMENT.md — the capstone project: goal, seven phased tasks in sequence, required evidence, and an acceptance checklist.
3. Worked answers — step-by-step solutions and checkpoint answers. Instructor-held in the private devops-interns-proj-solutions repo so this public repo stays answer-free. Request access from the maintainer.

🎤 Running a kickoff? docs/daig-intern-bootcamp.pptx is the instructor-led deck — 18 slides (foundations + the hands-on track) with speaker notes on every slide. PDF for quick viewing.

🃏 Print for every intern: the one-page cheat-card — the loop, the commands, the diagnostic pivot, the 7-phase checklist, and the golden rules.

🖼️ See it in diagrams: a domains & concepts coverage map plus flow diagrams for CI/CD, Kubernetes, IaC, observability, Docker and traffic — in TRAINING.md.

🤖 AIOps POC: an AI SRE Incident Copilot reads the live observability signals during an incident, proposes ranked root-cause hypotheses, and ships an eval harness that grades how often it's right (make copilot / make copilot-eval).

🥊 Break/fix arena: challenges inject a fault into the running stack; you fix it; the auto-grader probes the live system and scores your fix, per check, with a hint on every miss (make challenges / make grade C=...).

🛡️ LLMOps + AI-security lab: a support agent built twice — vulnerable vs hardened — against the OWASP-LLM Top 10, with a red-team eval gate and a token-cost SLO wired into CI (make redteam).

🗺️ New to the field? roadmap.sh/devops is the canonical what to learn map — daig is the do it companion. TRAINING.md maps each area to its roadmap.sh track (kubernetes, terraform, devsecops, …).

What this is

Daig is a food-delivery platform. Customers order, restaurants accept, riders deliver. Three services, one database, one payment integration.

It is not good software. It is good teaching software. Every tier is real, every layer is instrumented, and several things are wrong on purpose so there is something true to find.

The name: a daig is the pot you cook in when you are feeding everybody at once. That is the entire engineering problem in one word — and it is why the system's defining failure is the forty minutes before iftar, when an entire country orders dinner simultaneously against a deadline set by the sun.

Design principles

How it works

flowchart LR
    subgraph learn["The learning loop"]
        direction LR
        A["<b>1. Watch it work</b><br/>make up · make obs<br/>place an order"]
        B["<b>2. Break it</b><br/>chaos/dayN-*.sh<br/>one symptom, hidden cause"]
        C["<b>3. Diagnose</b><br/>logs · metrics<br/>traces · profiles"]
        D["<b>4. Fix it</b><br/>at the right layer"]
        E["<b>5. Gate it</b><br/>a test, a rule, a policy<br/>so it cannot regress"]
        A --> B --> C --> D --> E
        E -.->|next day, next layer| A
    end

Every day of the rotation is one pass around that loop at a different layer — Linux, then infrastructure, then containers, then pipelines, then security, then orchestration. Step 5 is the one that gets skipped and the one that matters: a fix without a gate regresses within two sprints.

Quick start

git clone https://github.com/adeelarshad414/devops-interns-proj && cd devops-interns-proj
cp .env.example .env
cp .githooks/pre-commit .git/hooks/pre-commit && chmod +x .git/hooks/pre-commit

make up          # the three tiers
make seed        # restaurants and menu items
make smoke       # verify every tier answers
make obs         # add observability
make load        # simulate the iftar spike

The overlays

Each is a separate compose file so a laptop is not asked to run everything at once.

make obs          # observability: Prometheus, Loki, Tempo, Pyroscope, Grafana
make vault-up     # OpenBao: initialise, unseal, provision
make vault-app    # run Daig with credentials from OpenBao
make sonar        # SonarQube (wants ~2GB RAM on its own)
make sec-up       # ZAP, Falco

make down stops everything. make nuke also deletes volumes — including the database.

Architecture

Three tiers

flowchart TB
    client(["Browser"])

    subgraph t1["TIER 1 · PRESENTATION"]
        web["<b>web</b><br/>nginx :8080<br/>static SPA, no build step"]
    end

    subgraph t2["TIER 2 · APPLICATION"]
        orders["<b>orders</b> :3001<br/>takes the order"]
        kitchen["<b>kitchen</b> :3002<br/>accept / reject · surge"]
        dispatch["<b>dispatch</b> :3003<br/>finds a rider"]
    end

    subgraph t3["TIER 3 · DATA"]
        pg[("<b>postgres</b> :5432<br/>orders · events · riders")]
        redis[("<b>redis</b> :6379<br/>cache")]
    end

    subgraph plat["PLATFORM"]
        bao["<b>OpenBao</b> :8200<br/>credentials"]
        otel["<b>OTel Collector</b><br/>:4317 / :4318"]
    end

    client --> web
    web -->|proxy| orders
    orders -->|HTTP| kitchen
    kitchen -->|HTTP| dispatch

    orders --> pg
    kitchen --> pg
    dispatch --> pg
    orders --> redis

    orders -.->|AppRole| bao
    kitchen -.->|AppRole| bao
    dispatch -.->|AppRole| bao

    orders -.->|OTLP| otel
    kitchen -.->|OTLP| otel
    dispatch -.->|OTLP| otel

Why the tier boundaries are enforced rather than documented. The data network is internal: true in Swarm; the AWS security groups only permit 5432 from the application tier's group; the Kubernetes NetworkPolicy restricts egress. The three-tier model is not a diagram someone has to remember — it is what the network will actually allow.

Observability

flowchart LR
    subgraph svc["Services"]
        s["orders · kitchen · dispatch"]
    end

    s -->|OTLP traces| col["OTel<br/>Collector"]
    s -->|/metrics scrape| prom["Prometheus"]
    s -->|stdout JSON| pt["Promtail"]
    s -->|push| pyro["Pyroscope"]

    col --> tempo["Tempo<br/>traces"]
    col -->|spanmetrics| prom
    pt --> loki["Loki<br/>logs"]

    prom --> graf["<b>Grafana</b>"]
    loki --> graf
    tempo --> graf
    pyro --> graf

    graf -.->|"metrics → trace"| tempo
    graf -.->|"trace → logs"| loki
    graf -.->|"trace → profile"| pyro

The dotted arrows are the point. They are wired in observability/grafana/provisioning/datasources/datasources.yaml so that one slow request can be followed from "something is wrong" to "this line of code" in four clicks:

Four pillars exist because four different questions need answering. Day 4 proves it: one deliberate defect is findable only in traces, another only in profiles, and neither in metrics alone.

Request flow

One order, end to end. This sequence is what produces a single trace crossing three services and two data stores.

sequenceDiagram
    autonumber
    participant C as Browser
    participant W as web
    participant O as orders
    participant K as kitchen
    participant D as dispatch
    participant P as postgres

    C->>W: POST /api/orders
    W->>O: proxy + traceparent
    Note over W,O: nginx forwards the trace header,<br/>which is why the trace joins up

    O->>P: BEGIN
    O->>P: INSERT orders (PLACED)
    loop each line item
        O->>P: SELECT menu_item (available?)
        O->>P: INSERT order_item
    end
    O->>P: UPDATE total · INSERT order_event
    O->>P: COMMIT
    Note over O,P: committed BEFORE calling downstream.<br/>The order survives a kitchen failure.

    O->>K: POST /api/kitchen/accept
    K->>P: SELECT open orders
    Note over K: computeSurgeScore()<br/>⚠ O(n²) when CHAOS_HOT_SURGE_LOOP=true
    K->>P: UPDATE state · INSERT order_event

    K->>D: POST /api/dispatch/assign
    D->>P: pickRider()
    Note over D,P: ⚠ N+1 when CHAOS_SLOW_DISPATCH=true<br/>one query per rider
    D->>P: INSERT assignment · UPDATE state
    D-->>K: rider assigned
    K-->>O: accepted + surge
    O-->>W: 201 order_id
    W-->>C: order placed

Order state machine

stateDiagram-v2
    [*] --> PLACED: orders
    PLACED --> ACCEPTED: kitchen, restaurant open
    PLACED --> REJECTED: kitchen, restaurant closed
    ACCEPTED --> COOKING
    COOKING --> READY
    READY --> ASSIGNED: dispatch found a rider
    ASSIGNED --> DELIVERED
    PLACED --> CANCELLED
    ACCEPTED --> CANCELLED
    REJECTED --> [*]
    DELIVERED --> [*]
    CANCELLED --> [*]

Enforced by a CHECK constraint in db/init/001_schema.sql. Every transition appends a row to order_events — nothing is ever overwritten, so the history of an order is reconstructable. Temporal rows, additive migrations only.

Credential flow

sequenceDiagram
    autonumber
    participant S as orders
    participant B as OpenBao
    participant P as postgres

    Note over S: startup — index.js
    S->>S: start telemetry
    S->>B: POST /v1/auth/approle/login<br/>role_id + secret_id
    B-->>S: client_token, TTL 1h

    S->>B: GET /v1/daig/data/database
    B->>B: check policy daig-orders
    B-->>S: url, host, username

    opt BAO_DYNAMIC_DB=true
        S->>B: GET /v1/daig-db/creds/orders
        B->>P: CREATE USER v-orders-x PASSWORD ...
        B-->>S: username, password, lease, TTL
        Note over S,P: a user that expires by itself
    end

    S->>S: server.js — open the port
    S->>P: connect

    Note over S,B: no vault reachable and NODE_ENV=production<br/>→ exit 78. It does NOT fall back silently.

That last note is the design decision worth defending. A service that quietly degrades from "secrets from the vault" to "secrets from the environment" is how a misconfiguration becomes a credential leak nobody notices for six months. services/_shared/secrets.js refuses.

Full detail, including AppRole rationale and the production delta table: vault/README.md.

Deployment

Environments

flowchart LR
    dev["<b>Local</b><br/>docker compose<br/>env credentials"]
    stg["<b>Staging</b><br/>one cloud<br/>OpenBao / cloud secrets"]
    prod["<b>Production</b><br/>K8s or managed containers<br/>signed images only"]

    dev -->|"CI: 7 gates"| stg
    stg -->|"CD: canary 10% → 100%"| prod

The pipeline

flowchart TB
    push(["git push"]) --> g1

    subgraph gates["Gates, ordered by cost of feedback"]
        direction TB
        g1["<b>1 · Secrets</b> ~20s<br/>gitleaks"]
        g2["<b>2 · SAST</b> ~2m<br/>Semgrep · CodeQL"]
        g3["<b>3 · SCA</b> ~1m<br/>npm audit · OSV"]
        g4["<b>4 · IaC + policy</b> ~1m<br/>Trivy · Checkov · Conftest"]
        g5["<b>5 · Image</b> ~3m<br/>Trivy · syft SBOM · cosign"]
        g6["<b>6 · DAST</b> ~5m<br/>OWASP ZAP"]
        g1 --> g2 --> g3 --> g4 --> g5 --> g6
    end

    g6 --> guard{"deploy window?<br/>16:00–21:00 PKT<br/>= blocked"}
    guard -->|blocked| stop(["refused"])
    guard -->|allowed| canary["canary 10%"]
    canary --> watch{"SLO healthy<br/>5 min?"}
    watch -->|no| rb(["rollback"])
    watch -->|yes| full["100%"]
    full --> annotate["annotate Grafana<br/>with the deploy marker"]

Why that ordering. Cheapest and fastest first. A leaked key must never merge and takes 20 seconds to detect. DAST needs the whole stack running. If a developer waits twelve minutes to learn about a typo they stop trusting the pipeline — and a pipeline people work around protects nobody.

The deploy-window guard in .github/workflows/cd.yml physically refuses to deploy during the iftar peak. That is the platform engineer's answer: do not ask people to remember, make the wrong thing impossible.

Targets

The three cloud stacks are deliberately parallel so Day 2 can make the point that matters: the concepts are identical and only the nouns change. Apply one properly, read the other two.

Full procedures: docs/DEPLOYMENT.md.

The documentation, and how to use it

Fifteen markdown files, each with one job. Read them in this order.

Start here

Running the rotation

Topic guides

Reference

The convention these files follow

• Every claim is falsifiable. No "should work" presented as "works".
• VERIFICATION.md is the authority on execution status. Not the README, not a commit message.
• Instructor material is separate and kept private. Answers live in the devops-interns-proj-solutions repo; nothing in this public repo does.
• Deliberate defects are labelled at the point of definition, not only in documentation, because that is where someone reading the code will be.

Directory structure

daig/
│
├── README.md                     ← you are here
├── VERIFICATION.md               what has actually been executed
├── PROGRESS.md                   living source of truth
├── CHANGELOG.md · CONTRIBUTING.md · SECURITY.md · LICENSE
├── DUMMY-VALUES.md               every registered placeholder credential
├── Makefile                      every workflow, self-documenting (make help)
├── .env.example                  copy to .env
│
├── docker-compose.yml            base: the three tiers
├── docker-compose.obs.yml        overlay: observability
├── docker-compose.vault.yml      overlay: OpenBao
├── docker-compose.sonar.yml      overlay: SonarQube
├── docker-compose.security.yml   overlay: ZAP, Falco
│
├── services/                     ── APPLICATION ──────────────────────
│   ├── _shared/                  cross-service modules
│   │   ├── secrets.js            OpenBao → env → exit 78
│   │   ├── telemetry.js          OpenTelemetry bootstrap
│   │   ├── metrics.js            four Prometheus series that matter
│   │   ├── logger.js             JSON logs with trace_id injected
│   │   ├── db.js                 pool with per-query timing
│   │   └── guard.js              exit-78 config guard
│   ├── web/                      TIER 1 · nginx + SPA, no build step
│   ├── orders/                   TIER 2 · entry point
│   │   ├── src/index.js          bootstrap: telemetry, then credentials
│   │   ├── src/server.js         the application
│   │   ├── src/insecure.js       ⚠ six deliberate vulnerabilities
│   │   ├── test/                 unit tests
│   │   └── Dockerfile            multi-stage, incl. the :broken target
│   ├── kitchen/                  TIER 2 · accept/reject · ⚠ O(n²) surge loop
│   └── dispatch/                 TIER 2 · rider assignment · ⚠ N+1 query
│
├── db/init/                      ── DATA ─────────────────────────────
│   ├── 001_schema.sql            additive only · ⚠ two indexes commented out
│   └── 002_seed_reference.sql    restaurants and riders
│
├── vault/                        ── CREDENTIALS ──────────────────────
│   ├── config/config.hcl         server config, annotated
│   ├── policies/*.hcl            four least-privilege policies
│   ├── agent/                    sidecar alternative to in-app integration
│   ├── bootstrap.sh              init → unseal → provision → verify
│   ├── demo.sh                   guided walkthrough
│   └── README.md
│
├── observability/                ── TELEMETRY ────────────────────────
│   ├── otel-collector.yaml       the seam between code and vendors
│   ├── prometheus.yml · loki.yaml · tempo.yaml · promtail.yaml
│   ├── rules/slo.yml             SLO recording rules + four alerts
│   └── grafana/                  provisioned datasources and dashboard
│
├── security/                     ── DEVSECOPS ────────────────────────
│   ├── semgrep/daig.yml          custom SAST rules (one is an exercise)
│   ├── policy/*.rego             OPA: Dockerfile, Kubernetes, Terraform
│   ├── kyverno/policies.yaml     admission control + image signature
│   ├── zap/rules.tsv             DAST, with triage decisions recorded
│   ├── falco/rules.yaml          runtime detection
│   ├── scan-all.sh               the whole toolchain, locally
│   ├── sign-and-verify.sh        SBOM, provenance, cosign
│   └── vault-demo.sh
│
├── infra/                        ── INFRASTRUCTURE AS CODE ───────────
│   ├── aws/                      ECS Fargate · RDS · ALB · ECR
│   ├��─ gcp/                      Cloud Run · Cloud SQL · Artifact Registry
│   └── azure/                    Container Apps · Flexible Server · ACR
│
├── ansible/                      ── CONFIG MANAGEMENT ────────────────
│   ├── site.yml · ansible.cfg
│   ├── inventories/dev/
│   └── roles/{common,docker,daig_app,postgres}/
│
├── k8s/                          ── ORCHESTRATION ────────────────────
│   ├── base/                     kustomize: deployments, services, PDBs, HPA
│   └── chaos/                    Demo Day scenario manifests
├── swarm/daig-stack.yml          the ladder to Kubernetes
│
├── chaos/                        ── EXERCISES ────────────────────────
│   ├── day1-network.sh           three planted causes, one symptom
│   ├── day2-drift.sh             Terraform state vs reality
│   ├── day3-crashloop.sh         config missing / wrong / environment wrong
│   ├── day4-latency.sh           the N+1 and the hot loop
│   ├── day5-demoday.sh           nine scenarios, assigned at random
│   └── day6-security.sh          find and fix six vulnerabilities
│
├── ai/                           ── AI / AIOps ───────────────────────
│   ├── incident-copilot/         AI SRE copilot: telemetry → ranked RCA, eval-graded
│   └── support-agent/            LLMOps + AI-security lab (OWASP-LLM, red-teamed)
├── grader/                       ── THE ARENA ────────────────────────
│   └── grader.py · checks.py     pluggable check engine (file/shell/http/prometheus)
├── challenges/                   break/fix challenges graded against live state
│
├── load/iftar-spike.js           zero-dependency load generator
├── scripts/                      smoke · seed · integration · init-repo
│   └── cost-model.py             deterministic cost model, all three clouds
├── .githooks/pre-commit          CI checks, locally, in two seconds
│
├── .github/
│   ├── workflows/                ci · cd · security · quality · devsecops
│   │                             · challenges · ai-security · infracost · drift
│   ├── ISSUE_TEMPLATE/
│   ├── CODEOWNERS · dependabot.yml · pull_request_template.md
│
└── docs/                         ── DOCUMENTATION ────────────────────
    ├── ARCHITECTURE.md · REQUIREMENTS.md · DEPLOYMENT.md
    ├── CHARTER.md                objective, questions, milestones, targets, tasks
    ├── COVERAGE.md               the scope decision
    ├── COST.md                   what it costs on each cloud
    ├── CHEATSHEET.md             hand out Day 1. commands only.
    │                             (worked answers → private solutions repo)
    ├── DAY1–DAY5.md
    └── GIT.md · DOCKER-NETWORKS-VOLUMES.md · SWARM.md
        · SONARQUBE.md · DEVSECOPS.md

⚠ marks a deliberate defect.

Requirements

Summarised. Full list with acceptance criteria and IDs: docs/REQUIREMENTS.md.

Functional

Non-functional

NFR-12 is a real constraint, not a slogan. It is why there are three services rather than twelve, and why the frontend has no build step.

Deliberate defects

Nothing here is an accident.

The most important one is the vulnerability tools cannot find. The IDOR in insecure.js has parameterised SQL, no unsafe call, and nothing any scanner can point at. What is missing is a business rule. That is why code review exists, and it is the single most valuable thing in this repository.

Instructors: the private devops-interns-proj-solutions repo has every answer (docs/INSTRUCTOR.md). Request access; interns don't get it.