Start: 09:00

Morning 09:00-10:45

• Intro, logistics, and labs.arista environment overview and user assignment
• Evolution of the Network Labs and why better labs matter
• Dev Containers, Codespaces and deeplink hands-on
• Pre-built containers, GitHub Actions and GHCR
• A survival guide to SELinux, the kernel and permissions
• The story of two platforms: ARMing your lab

Break 10:45-11:15

Afternoon 11:15-13:00

• GitHub Actions: reusable workflows and matrix builds
• Add Containerlab, code-server and inventory
• Using ENTRYPOINT for image import and more
• Build a simple deeplink API to trigger lab deployment
• Code-server - the perfect UI
  • Use tasks and other VSCode features to customize the lab

• It's a container
  • When it runs, it's not much different to "normal" container
  • However it's empowered by Dev Container specification
• Development Container Specification @ containers.dev
  • Originally a proprietary standard for VSCode remote development
  • Open-sourced by Microsoft in 2022
  • Defines metadata to build, open and customize the dev environment
  • Works best with a supporting tool
    • The most well known is VS Code with Dev Containers extension
  • Keeps the environment definition close to the repo

// .devcontainer/devcontainer.json
{
  // Start from a known base image
  "image": "mcr.microsoft.com/devcontainers/base:ubuntu",

  // Add standard tools during create
  "features": {
    "ghcr.io/devcontainers/features/python:1": { "version": "3.11" },
    "ghcr.io/devcontainers/features/docker-in-docker:2": {}
  },

  // Make the repo ready on first open
  "postCreateCommand": "pip install -r requirements.txt",

  // Useful for docs preview or a small lab UI
  "forwardPorts": [8000]
}

• Init a new empty repo on Github
  • Call it ac5-workshop
• Clone the repo to the Sandbox Lab environment
  • Do not forget to authenticate!
• Create following files:
  • .devcontainer/devcontainer.json
  • requirements.txt
  • init.sh
• Make init script executable:
  • chmod +x init.sh
• Commit and push
• Start the Codespace
• Check
• Stop and delete the Codespace

// .devcontainer/devcontainer.json
{
  "image": "ghcr.io/aristanetworks/avd/universal:python3.12-avd-v6.1.0",
  "remoteUser": "avd",
  "onCreateCommand": "${containerWorkspaceFolder}/init.sh"
}

# requirements.txt
rich==13.9.4

#!/usr/bin/env bash

# init.sh

set +e
pip install -r requirements.txt
wget https://raw.githubusercontent.com/textualize/rich/refs/heads/master/examples/tree.py
echo "alias rich='python3 tree.py $(pwd)'" >> /home/avd/.zshrc
cp -r /home/avd/.ansible/collections/ansible_collections/arista/avd/examples/single-dc-l3ls/* .

Fun fact: AVD Container Images might be the first ever pre-build dev environment for network engineers

• containers/my_container/.devcontainer/devcontainer.json:

  {
    "image": "ghcr.io/aristanetworks/avd/universal:python3.12-avd-v6.1.0",
    "remoteUser": "avd",
    "onCreateCommand": "${containerWorkspaceFolder}/init.sh"
  }
  

• containers/my_container/init.sh (do not forget chmod +x):

  #!/usr/bin/env bash
  
  set +e
  pip install -r requirements.txt
  wget https://raw.githubusercontent.com/textualize/rich/refs/heads/master/examples/tree.py
  echo "alias rich='python3 tree.py $(pwd)'" >> /home/avd/.zshrc
  cp -r /home/avd/.ansible/collections/ansible_collections/arista/avd/examples/single-dc-l3ls/* .
  

• containers/my_container/requirements.txt

  rich==13.9.4
  

• same files as before, but the workflow requires some comments

You can now test ghcr.io/<your-github-account>/ac5-workshop/my_container:latest

---
# .github/workflows/build_image.yml
name: Reusable build container workflow

env:
  # BUILDX_NO_DEFAULT_ATTESTATIONS must be set to build only arm64 and amd64 images.
  # The devcontainers/ci@v0.3 build will fail if this env variable is not set.
  BUILDX_NO_DEFAULT_ATTESTATIONS: 1

on:
  push:
    branches: ['**']

permissions:
  packages: write

jobs:
  build_image:
    runs-on: ubuntu-22.04
    steps:
      - name: Checkout code 
        uses: actions/checkout@v6

      - name: Convert Github repository name to lowercase 
        id: gh_repo
        run: echo "name_lowcase=${GITHUB_REPOSITORY,,}" >> $GITHUB_OUTPUT

      - name: Setup QEMU for multi-arch builds 
        uses: docker/setup-qemu-action@v4
        with:
          platforms: ${{ inputs.platform }}

      - name: Setup Docker buildX for multi-arch builds 
        uses: docker/setup-buildx-action@v4

      - name: Login to the container registry 
        uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Pre-build dev container image 
        uses: devcontainers/ci@v0.3
        env:
          FROM_IMAGE: ${{ inputs.from_image }}
          FROM_VARIANT: ${{ inputs.from_variant }}
          USERNAME: ${{ inputs.username }}
          UID: ${{ inputs.user_id }}
          GID: ${{ inputs.group_id }}
        with:
          subFolder: containers/my_container
          imageName: ghcr.io/${{ steps.gh_repo.outputs.name_lowcase }}/my_container
          imageTag: latest
          platform: linux/arm64/v8,linux/amd64
          push: always

• env
  • disables default attestations for ARM images. If missing: the workflow can fail
• on.push
  • run the workflow on every branch, filter to specific branch in prod
• permissions.packages: write
  • allow pushes to ghcr.io. Check your GitHub Actions configuration as well
• actions/checkout
  • clone repo files onto the runner first
• gh_repo
  • lowercase the repo name for GHCR image tags
  • if skipped: mixed-case names can break image naming.
• setup-qemu
  • QEMU is required on an amd64 runner to build arm64 (cross-platform builds)
  • you can simply use ARM-runner instead, which is likely a better way
• setup-buildx
  • enable the multi-arch builder
• docker/login-action
  • sign in to ghcr.io with ${{ secrets.GITHUB_TOKEN }}

---
# .github/workflows/build_image.yml
name: Reusable build container workflow

env:
  # BUILDX_NO_DEFAULT_ATTESTATIONS must be set to build only arm64 and amd64 images.
  # The devcontainers/ci@v0.3 build will fail if this env variable is not set.
  BUILDX_NO_DEFAULT_ATTESTATIONS: 1

on:
  push:
    branches: ['**']

permissions:
  packages: write

jobs:
  build_image:
    runs-on: ubuntu-22.04
    steps:
      - name: Checkout code 
        uses: actions/checkout@v6

      - name: Convert Github repository name to lowercase 
        id: gh_repo
        run: echo "name_lowcase=${GITHUB_REPOSITORY,,}" >> $GITHUB_OUTPUT

      - name: Setup QEMU for multi-arch builds 
        uses: docker/setup-qemu-action@v4
        with:
          platforms: ${{ inputs.platform }}

      - name: Setup Docker buildX for multi-arch builds 
        uses: docker/setup-buildx-action@v4

      - name: Login to the container registry 
        uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

• The devcontainer/ci action allows pre-building a Dev Container image inside a CI pipeline
• It's optional, but makes build easier as we can refer to a Dev Container definition in our repository
• This is the same Dev Container we would use locally or in a Codespace
• There are many other ways to build

      - name: Pre-build dev container image 
        # Reuse the same Dev Container definition we open locally.
        uses: devcontainers/ci@v0.3
        env:
          # Feed build-time values into the Dockerfile / devcontainer template.
          FROM_IMAGE: ${{ inputs.from_image }}
          FROM_VARIANT: ${{ inputs.from_variant }}
          USERNAME: ${{ inputs.username }}
          UID: ${{ inputs.user_id }}
          GID: ${{ inputs.group_id }}
        with:
          # Build from one container folder in this repository.
          subFolder: containers/my_container
          # Publish to a predictable path in GitHub Container Registry.
          imageName: ghcr.io/${{ steps.gh_repo.outputs.name_lowcase }}/my_container
          imageTag: latest
          # One manifest, two CPU families.
          platform: linux/arm64/v8,linux/amd64
          # The runner is ephemeral, so publish before it disappears.
          push: always

• Go to your sandbox terminal and use

  • docker run --rm -it -v $(pwd):/home/avd/workspace -w /home/avd/workspace ghcr.io/<username>/ac5-workshop/my_container:latest zsh

• Note following errors:

  $ rich
  zsh: command not found: rich
  $ touch test.tmp
  touch: cannot touch 'test.tmp': Permission denied
  

1. Failing rich command
   • We must specify init.sh. Better way - add ENTRYPOINT
   • ENTRYPOINT - is the command/script that is executed when a container starts
2. Permission denied
   • UIDs are incredibly important when dealing with non-root users on most operating systems and must match
   • Root is not good!
   • Tools like VSCode can re-build images to fix UID, but we'll simply build another container image

• Never deploy your labs on "random" machines!
• Containers do NOT provide full isolation and host machine is part of your environment
• Understand UIDs, permissions and the impact of the root user account
• Avoid SELinux and other great security features, when not strictly required
  • Focus on lab isolation and low lifetime instead
  • When you must - build a good level of understanding first! Deploying random staff on SELinux hurts!
• Pick a friendly Linux distribution - Debian / Ubuntu

• Watch your kernel!
  • Also during upgrades!
• Kernel challenge examples:
  • 6.10 Linuxkit with CONFIG_TCP_MD5SIG disabled. cEOS-lab BGP password authentication breaks
  • legacy ip_tables are not loaded by default. This was breaking VSCode D-in-D helper script

# list loaded kernel modules
lsmod
# check kernel version
uname -a
# check kernel configuration
cat /lib/modules/$(uname -r)/config

• Matrices is a GitHub actions feature that allows permutations of various parameters
• We'll use a parent workflow with matrix to trigger the child workflow making the build
  • Reusable workflows feature
• Create .github/workflows/build_parent_matrix.yml

---
name: build container images

on:
  push:
    paths:
      - .github/workflows/build_child.yml
      - .github/workflows/build_parent_matrix.yml
      - containers/lab/**

permissions:
  packages: write

jobs:
  build-lab-containers:
    uses: ./.github/workflows/build_child.yml
    strategy:
      matrix:
        from_image: ["ghcr.io/aristanetworks/avd/universal"]
        from_variant: ["python3.12-avd-v6.1.0"]
        user_id: ["1000", "1009"]
        include:
          - user_id: "1000"
            group_id: "1000"
          - user_id: "1009"
            group_id: "1009"
    with:
      from_image: ${{ matrix.from_image }}
      from_variant: ${{ matrix.from_variant }}
      user_id: ${{ matrix.user_id }}
      group_id: ${{ matrix.group_id }}

• containers/lab/.devcontainer/Dockerfile

  ARG FROM_IMAGE
  ARG FROM_VARIANT
  
  FROM ${FROM_IMAGE}:${FROM_VARIANT}
  
  ARG USERNAME
  ARG UID
  ARG GID
  
  USER root
  
  # This magic sed edits are very important
  # They allow to build a container with different UID/GID ot of a base
  ENV OLD_GID="1000"
  RUN sed -i -e "s/\(${USERNAME}:[^:]*:\)[^:]*:[^:]*/\1${UID}:${GID}/" /etc/passwd; \
      sed -i -e "s/\([^:]*:[^:]*:\)${OLD_GID}:/\1${GID}:/" /etc/group; \
      chown -R $UID:$GID /home/$USERNAME
  
  # copy code-server entrypoint
  COPY ./entrypoint.sh /bin/entrypoint
  RUN chmod +x /bin/entrypoint
  ENTRYPOINT [ "/bin/entrypoint" ]
  
  USER ${USERNAME}
  

• Do NOT forget chmod +x for the entrypoint.sh

• Add containers/lab/.devcontainer/devcontainer.json

  {
      "build": {
          "dockerfile": "Dockerfile",
          "args": {
              "FROM_IMAGE": "${localEnv:FROM_IMAGE}",
              "FROM_VARIANT": "${localEnv:FROM_VARIANT}",
              "USERNAME": "${localEnv:USERNAME}",
              "UID": "${localEnv:UID}",
              "GID": "${localEnv:GID}"
          }
      }
  }
  

• Add basic ENTRYPOINT containers/lab/.devcontainer/entrypoint.sh

  #!/usr/bin/env bash
  
  set +e
  
  # on Codespaces this will not work correctly
  if ! ${CODESPACES:-false}; then
      # Execute command from docker cli if any.
      if [ ${@+True} ]; then
          exec "$@"
      # Otherwise just enter sh or zsh.
      else
          if [ -f "/bin/zsh" ]; then
              exec zsh
          else
              exec sh
          fi
      fi
  fi
  

• Rename build_image.yml to build_child.yml

• Change on: section:

  on:
    workflow_call:
      inputs:
        from_image:
          required: false
          type: string
          default: ghcr.io/aristanetworks/avd/universal
        from_variant:
          required: false
          type: string
          default: latest
        username:
          required: false
          type: string
          default: avd
        user_id:
          required: false
          type: string
          default: 1000
        group_id:
          required: false
          type: string
          default: 1000
  

• Edit Pre-build dev container image in jobs: ci section:

        - name: Pre-build dev container image 
          uses: devcontainers/ci@v0.3
          env:
            FROM_IMAGE: ${{ inputs.from_image }}
            FROM_VARIANT: ${{ inputs.from_variant }}
            USERNAME: ${{ inputs.username }}
            UID: ${{ inputs.user_id }}
            GID: ${{ inputs.group_id }}
          with:
            subFolder: containers/lab
            imageName: ghcr.io/${{ steps.gh_repo.outputs.name_lowcase }}/lab
            imageTag: uid-${{ inputs.user_id }}
            platform: linux/arm64/v8,linux/amd64
            push: always
  

• Test with:

  • docker run --rm -it -v $(pwd):/home/avd/workspace -w /home/avd/workspace ghcr.io/<username>/ac5-workshop/lab:uid-1009
  • touch test.tmp

Tip: Make sure to exit the container!

• Let's use a small AI generated script to emulated deeplink-like API
• Yes! We use AI in this workshop!
• https://127.0.0.1/<lab-name> will start specific lab
• Install Python requirements
  • pip install fastapi uvicorn
• Add --detach to your Make shortcuts for every lab
• Tunnel ports 5000 and 5001
• python3 lab.py (keep the tab open to check logs)
• (Optional): Add https://<tunnel-url>/<lab-name> to your README
• Go and click that link!

import subprocess
from fastapi import FastAPI, HTTPException
import uvicorn

app = FastAPI()

@app.get("/{lab_name}")
def run_lab(lab_name: str):
    if any(char in lab_name for char in [";", "&", "|", "..", "/"]):
        raise HTTPException(status_code=400, detail="Invalid lab name")

    try:
        result = subprocess.run(
          ["make", lab_name], capture_output=True, text=True, check=True)
        return {"status": "success", "output": result.stdout}
    except subprocess.CalledProcessError as e:
        raise HTTPException(status_code=500, detail=f"Make failed:\n{e.stderr}")

if __name__ == "__main__":
    uvicorn.run(app, host="127.0.0.1", port=5001)

• VSCode / Code Server is the best base you can imagine for your lab
• We can further tweak and customize our labs with very low effort
• Let's add following to our setup:
  • Containerlab extension
  • A random Python script that will greet us on lab start

// .vscode/tasks.json
{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "init_lab",
            "command": "python3 greet.py",
            "type": "shell",
            "presentation": {
                "reveal": "always"
            },
            "runOptions": {
                "runOn": "folderOpen"
            }
        }
    ]
}