spasic.net
spasic@net: ~
➜ ssh spasic@edge-01
Welcome to Ubuntu 24.04 LTS (GNU/Linux 6.8.0 x86_64)
➜ nvidia-smi
+----------------------------+
| GPU NAME UTIL MEM |
| 0 H100 80GB ▓▓▓▓ 94% |
| 1 H100 80GB ▓▓▓▓ 88% |
+----------------------------+
gpu-0 ██████████░░ 94% 74G/80G
gpu-1 █████████░░░ 88% 71G/80G
➜ vllm serve Llama-3.1-70B --tp 2
INFO vLLM engine v0.6.4 · tensor-parallel = 2 GPUs
INFO model loaded in 38.2s · KV cache 92% util
INFO Uvicorn running on http://0.0.0.0:8000
✓ engine ready · 12,847 tok/s · p99 41ms
➜ kubectl get pods -n ai
NAME READY STATUS AGE
api-7f9c4 1/1 Running 3d
inference-0 1/1 Running 6h
vector-db-2 1/1 Running 3d
➜ curl -s :8000/v1/chat/completions -d @req.json
{"model":"llama-3.1-70b","latency_ms":41}
{"content":"All systems nominal — 3 pods healthy."}
➜ docker build -t spasic.net:latest .
[+] Building 12.4s (18/18) FINISHED
→ pushed to registry ✓
➜ git push origin main
Everything up-to-date
→ live at https://spasic.net ✓
ai · terminal · devops