Day 1 OTel-Shop: Setting Up k3d, Jaeger & Postgres

Why I Started This Series

So here's the thing. I really wanted to genuinely understand OpenTelemetry and distributed tracing. Not just skim docs and go "yeah, got it," but actually build something end to end. That's how this challenge, OTel-Shop, was born: a small microservices lab in Go where all traffic gets visualized in Jaeger. Fun, right?

Today (Day 1) wasn't about features, it was about foundations. My rule: if the base is solid, everything after is just stacking bricks. So most of the day went into setting up the lab infrastructure.

What I Got Done Today

Quick recap so I don't ramble:

• Initialized the repo from scratch with a .gitignore and ran a gitleaks scan so no secrets sneak in.
• Set up a Go workspace with 4 modules: order, inventory, payment, and telemetry.
• Wrote Kubernetes manifests for PostgreSQL, OpenTelemetry Collector, and Jaeger.
• Brought up a local k3d cluster. All three core components came up: database, collector, and Jaeger UI.
• Sent a tiny fake trace and watched it show up in Jaeger. The pipeline was proven end to end from day one, not just on paper.

The most satisfying part wasn't things running smoothly — it was sending that tiny little trace and seeing it land in Jaeger. That's the "oh, so this is observability" moment.

The Drama Along the Way

Don't imagine everything was smooth — that's where the real value was.

First, k3d (the tool that creates local Kubernetes clusters) refuses to use Podman. It wants Docker, period. My machine was all-in on Podman. Brief drama, then Docker Desktop got switched on. Lesson: check what container runtime your tools actually support before you get excited about deploying.

Second, image downloads were painfully slow. One took nine minutes. My deploy script even timed out on it. Exhausting? Yes. But it reinforced why you pin image versions — so you're not silently pulling whatever "latest" is and wondering why behavior changed.

Third, a small classic error: go build ./... from the repo root. Turns out with a multi-module workspace that command doesn't work at the root. Fix: build per module. Small thing, but it made me understand the workspace structure better.

Lessons Learned

Quick takeaway, four things:

• Check your toolchain before you start. If I'd checked that k3d needs Docker first, I wouldn't have spent time fighting Podman.
• Pin every image version. "Latest" is a silent commitment that makes your infra non-reproducible.
• Get one tiny "proof" early. Seeing a trace land in Jaeger on Day 1 is far more motivating than waiting for every feature to finish.
• Neat infra is your future self's treat. Tomorrow it's just deploying services, no cluster debugging first.

Conclusion

Day 1 wrapped in a good place: cluster up, database seeded, collector running, and Jaeger already receiving its first trace. Tired, but relieved — the fiddliest foundation is done.

Tomorrow, the real services: the checkout flow, stock check, and payment. Hopefully less drama than today.

Repo is here: https://github.com/stayrelevantid/otel-shop