Hacker News

"It is simple to use, easy to organize and manage, and very robust."

This is why nobody uses it. Cloud stuff has to be as baroque as possible.

I love it so much I have created a thin statless orchestrator layer upon it:

https://github.com/daitangio/misterio

It works very well!

> Extremely debatable. They still have never fully implemented health checks and auto healing.

Agree.

Plus there's the monitoring of the host that is always overlooked in articles. I've ended up chucking Monit on there to monitor disk usage et al, and also used it for monitoring compose too and restarting containers.

And then there's Healthchecks.io, and external uptime monitoring... the list goes on. Properly monitoring systems, even single server systems, is not simple.

I'm totally onboard with k3s/k8s being better in a lot of cases.

But docker compose can actually be very sufficient for what many projects actually need.

Granted I am a guy pushing for compose based localdevs and such but going further you often just cannot beat the simplicity of doing update QA or other CI/CD workloads in compose based projects. I have had dozens of projects where we replaced flaky slow and maintenance heavy pipelines with just docker compose up --build --wait in the past years. How come you say health checks are still broken?

Why use a docker volume? Seems like a good way to accidentally lose your files. Just volume-mount from the external filesystem and back it up the same way you would any linux application's files - maybe stop it, maybe not, depends on how it uses files.

The "easiest" way is to use bind mounts to a local directory (or multiple directories) instead of volumes. Then you can just use normal backup tooling.

Docker volumes (and bind mounts) however have the minor problem of being hard to get a consistent copy to without stopping the service. You can work around this by, e. G., having ZFS or btrfs as the underlying FS and making a snapshot there. Otherwise, your software (like PostgreSQL) might also have other online backup tooling.

For databases, I usually host them on a separate server. This could either be through Docker Compose or a managed DB server. If a managed DB is affordable enough I'd reach for it.

It's because I like keeping my servers stateless when possible. It makes it easier to upgrade them in a zero downtime way later.

If your web server has your DB too, then you can't do zero downtime system upgrades. For example I would never upgrade Debian 12 to 13 on a live server. Instead, I'd make a new server with 13, get it all ready to go and tested and then when I'm ready flip over DNS or a floating IP address to the new server. This pattern works because both the old and new server can be writing to a database on a different server.

With all that said, if you were ok with 1 server, then yeah I'd for sure run it in Docker Compose.

At this stage the volume/persistence configuration for all of the major DBs is arguably extremely well understood and has been for years. The only real risk in running the DB as a container for most people is not configuring volumes for persistence correctly.

For most DBs it's one or two paths in the container, and virtually all DBs vendors have a reference Docker Compose example somewhere showing volume config. I can't remember the last time I ever "natively" installed a DB personally!

Not OP, but I think it depends heavily on your use case and where you are deploying... I've used containerized DBs as well as leaned into hosted DBs in a given cloud environment. I've tended to favor PostgreSQL for container dev simply because it is well supported in pretty much every first and second tier cloud provider out there.

It really comes down to YMMV... Sometimes for a singular app surface, it's easier to just use a compose file that includes the database. mailu/mailcow is a good example... you don't necessarily want to comingle email on the same server as other services.

That said, if you need to share a single DB or set of DBs across an applicaiton with several instances/deployments, then it makes much more sense for a central deployment. I almost never do my own host level install, instead relying on cloud hosting and mgt. The only real exception is MS-SQL on internal servers... MS-SQL in Docker is barely acceptable for dev, and missing a few key features you may actually want/need.

What effect does docker compose have on database resilience?

Thanks a lot, I really appreciate it.

It makes things amazingly simple and portable, dealing with things otherwise seems so horrible now.

Then you either need to try it or research it. Other people can't tell you if you like turkey or not.

The analogy you are replying to is pointing out that it works just fine for production, so if you should use it or not is simply a matter of taste.

More to the point, there is no objectively right answer of what stack you should use. There are plenty of objectively wrong answers, but compose isn't one of them.

Yes. I clearly believe we should not motivate choices in technology.

I also would not eat a turkey sandwich for lunch on a Tuesday.

*shudder*

I wouldn't use Docker in production.

I note the absence of a motivation.

Well, the fashion changes. At one time, sugar too was advertised as beneficial for tooth health.

GP is talking about podman with generated systemd unit files (a.k.a. podman quadlet[0]), not the docker-compose-compatible podman-compose ...and I'd agree, systemd can manage services on a system just fine, and even better than any compose workload ever could.

journald will help with logs, and the pull policy[1] helps with mutable tags. What help do you need with "orphan containers"?

[0]: https://docs.podman.io/en/latest/markdown/podman-quadlet.1.h...

[1]: https://docs.podman.io/en/latest/markdown/podman-image.unit....

I'm not OP, but the whole podman compose topic gets quite confusing, as initially Podman didn't seem to know what they were trying to do. I've given some more context around it in previous comments.

You shouldn't be using podman compose. It's flimsy and doesn't work very well (at least it was last time I used it prior to Podman v3), and I'm pretty sure it doesn't have Red Hat's direct support.

Instead, activate Podman's Docker API compatibility socket, and simply set your `DOCKER_HOST` env var to that socket, and from there you can use your general docker client commands such as `docker`, `docker compose` and anything else that uses the Docker API. There are very few things that don't work with this, and the few things that don't are advanced setups.

For what it's worth, podman has also a thin wrapper (podman compose) which executes `docker-compose` or the old `podman-compose`. The docs should explain which it picks.

Note:

- `podman-compose` is an early attempt at remaking `docker-compose` v1 but for Podman. This used parsed the compose config and converts them to podman commands, and executes it.

- Later Podman wrote a Docker compatible socket instead, which can work with most docker clis that accept a `DOCKER_HOST` argument, including `docker` and `docker-compose` (both v1 and v2)

- `podman compose` is a thin wrapper that automatically selects `docker-compose` or `podman-compose` depending on which is installed.

Generally all you need is podman, docker-compose (the v2 binary), and that's it. From there you can use `podman` and/or `podman compose`.

Docker (Compose) has some quirks compared to Podman (Compose), e.g. when using gvisor or a lot of internal networks. Depending on what you do your milage will vary, though.

Podman in general is much higher quality than Docker. For the last 5 years, Docker was throwing all kinds of stuff against the wall to see what sticks, neglecting foundational issues.

Moreover, podman-compose integrates really easily with systemd. You can create a service by just running "podman-compose systemd" and following the prompt! No quadlet nonsense required.

Then you learn podman can't even list containers for all users properly and it kind of starts smelling like the whole ip4 vs ip6 debacle: bunch of vocal proponents wanting you to subject yourself to endless torture for no discernible reason.

I find the podman man pages quite readable and thorough if you've had experience configuring systemd services. Good examples as well.

https://docs.podman.io/en/latest/markdown/podman-systemd.uni...

Can't comment on Raspbian, but Ubuntu LTS (has/had) a seriously outdated podman version. This is the kind of nuisance the Debian derivatives have been running into for more than 20 years: they are extremely conservative, and if that is all you need, then that is great, but if not, you'll have to either run the latest Ubuntu (not LTS), or you upgrade to something like fedora.

> Having your whole application with its containers, volumes, and networks all defined together in one easy-to-read YAML file is a way better experience. Deployment is two steps: 1. `git clone foo` 2. `docker compose up -d`. You can see the state of the application containers with `docker compose ps`. You can run multiple compose applications on the same host and manage them separately by putting them in different directories.

I always felt it the other way around: docker compose files are weird blobs of YAML that I have to hunt down the location of or parse their under-speced labels to find the location of. I can't make them depend on any non-container services[0], the break my firewall rules[1], and I have to use a whole mess of bespoke tooling just to do normal start/stop/restart operations with them instead of using the same commands I use for literally any other service.

> With quadlets, you delegate everything to systemd. You have to break the configuration up into a bunch of tiny unit files and then separately copy them to /etc or a dedicated user's dotfiles.

The nice thing about quadlets is exactly that, they integrate with systemd and by extension the rest of the system. I don't have to think about `webapp.container` as a "Docker container" I can think of it as just `webapp.service`, like any other piece of software I would install and run. All the related files are in one of the well-speced file locations that follow the same hierarchy as anything else on the system (user -> etc -> /usr), optionally grouped in folders[2].

> Good luck SSH'ing into an unfamiliar system and understanding at a glance what it's doing.

Just use the same tools you'd use on any other systemd system: `systemctl list-units`, `systemctl status`, etc. Versus having to hunt down compose files either manually or by parsing the under-specified labels on the containers.

> (Even moreso if you created dedicated users for each application, which I understand is the recommended solution.)

TBH I've rarely seen this advice. Most people I know just run it as root (which is what I do) or as a `podman` user. But even in this situation it should be pretty easy to figure out whats' running, as you know it's all running as one user and is hard-namespaced to only rely on resources available in that account.

> If I'm just holding it wrong and there exists some better tooling to manage podman in prod that I don't know about, I'm happy to hear about it.

Quadlets are just files that created systemd services, so basically any configuration management or deployment tool will manage them fine. Ansible has a dedicated Quadlet role that works pretty well, or just git clones+`systemctl start`. This would probably be the recommended way if you're not using k8s/etc.

Alternatively, you can just `git clone /etc/containers/systemd/`, `systemctl start container` like with docker compose. If you're running multiple containers, either refer to them with `Wants=`/etc in the Quadlet files, create a `.target` file that references them all, or put them all in a `.pod` and start the pod. I think this is the part were most people stumble though: when you're used to treating containerized software as a separate kind of "thing" it's a little weird to go back to treating it like normal services.

I've been writing something to help with deploying quadlets GitOps-style[3] that will hopefully fill the "more than one server but less than kubernetes" deployment gap.

[0] Unless I wrap the compose steps in a systemd unit, at which point now I have two problems.

[1] Caveat, this has probably gotten better overall but I still run into compose-related firewall issues about once or twice a year

[2] The newer versions of Podman also support `.quadlets` files, that merge all the quadlets into one file.

[3] https://github.com/stryan/materia . There's also https://github.com/orches-team/orches and https://github.com/ubiquitous-factory/quadit

Honestly with the hell I'm going through just trying to get Podman to run properly on macos, I can't imagine trusting the Podman people with a production deployment. I was not particularly impressed with the Docker tooling, but Podman has been even worse. This is a not-remotely exhaustive list of things I've run into in the last 24 hours:

* Podman fails to build a 16GB container image (after 30 minutes of downloading dependencies) despite having 90GB free out of a 200GB podman virtual machine

* Podman machine will, for reasons I don't understand, create a filesystem in a block device with wildly different sizes, and it seems like it's just random

* Pushing podman images to a container image registry via the Podman Desktop UI gives no indication that it's doing anything or even recognized the "push image" click, a success or error notification _might_ appear several or tens of minutes later or possibly not at all

* Starting a podman machine might work, but it fails ~75% of the time with not-particularly-exotic options (a bunch of ram and disk) and very cryptic error messages, frequently telling me to file bug tickets (I have)

* Podman Desktop won't let me create a podman machine with more than 44GB of disk, but the podman machine CLI won't let me create a machine with fewer than 100GB (IIRC--it's some number larger than 44, in any case)

Apart from the container image being absurdly large (Python developers love massive packages, I guess), I'm not doing anything exotic.

Yep, I've got a couple of single node k3s clusters and it just works. It can be a useful way to transition from applications running on their own dedicated servers to running on a true k8s cluster.

What it gets you is a more powerful Docker Compose running on server that you can interact with via kubectl. No SSH, no custom scripts etc. Just kubectl and YAML.

If you go cloud managed (EKS etc), there isn't really much to take care of. I do dev work and even keep a bare metal cluster going as well as a cloud managed one.

Probably needs a real generalist though which maybe doesn't include your average dev.

https://www.macchaffee.com/blog/2024/you-have-built-a-kubern...

Like, if that works for you, more power to you. But that is a lot of moving parts in exchange for using a tool whose value prop is that it doesn't have many.

This is the way, ended up using identical setup.

What would the config look like if I have my docker containers split up over multiple VMs?

Could you elaborate on your setup? Is the docker host also your web server on which you run caddy?

A really nifty thing is that you can also of course bind this to the device's tailscale ip!

Also you don't even need the loopback address if the traffic is between one container and another, just a bridge network is fine.

This is surely the easiest and I would guess the safest way, and has the added benefit that your proxy (nginx in my case) can handle SSL for you, making certificate deployment a breeze.

It makes sense but that's more overhead and the spirit of the post seems to be "can we just docker compose and be done with it?"

Well, as an example we usually set incoming rules to filter SSH only from administrator IP addresses, TCP 10050 only from zabbix monitoring server and leave few icmp types required and rest is dropped and logged.

For forward chain we set docker network ranges to route between themselves and only services actually used in containers. Allow container outgoing connections to our DNS servers, centralized HTTP proxy server and monitoring - nothing else containers are allowed to route to.

And for output is similar, only allow our DNS servers, NTP, HTTP proxy, centralized rsyslog where everything goes and zabbix monitoring server and a few icmp types - nothing else gets out and is logged.

With the advent of these supply chain attacks we read about often here it's just a matter of time some container is compromised and this seems like only viable way to at least somehow limit impact when such an event occurs.

To expand, you can use privileged containers, host network, capabilities, etc if the software really needs it. In that case, Docker basically becomes an init system/service manager but you get a singular daemon managing everything

It is yeah – I was thinking along the lines of the decision being between docker compose and something like podman quadlets

Same, I've tried three or four times to make it work, including one attempt that just translated compose.yaml into k8s yaml, and every time I came away thinking, "just use k8s". K8s yaml looks complex, and can start to feel very boilerplate, but attempts to hide the complexity often just lead to something not-flexible-enough because it encodes convention over configuration, and inevitably some project runs into limitations and pretty soon you've just built an abstraction layer that leaks or is equally complex/verbose and now you have to learn something new.

Just use k8s and follow similar patterns is the conclusion I've arrived at personally.

I've recently been dipping my toes into k8s / kustomize / helm, and I recently had a situation where having a base deployment yml template that I wanted to reuse across various deployments. I had a look at Helm and I was frankly shocked how bad the templating was with Go templates, it was close to unreadable and felt very brittle!

That's fair; good solution.

It's not quite what you suggested, but you can use podman with systemd.

https://docs.podman.io/en/latest/markdown/podman-systemd.uni...

Or more simply just wrap some orchestrator in a single service file. Could just be a bash script used for ExecStart.

i don't have dozens of services. My complex SaaS has total 6 services and each one is running on its own server.

Service file lives in the mono repo where all 6 services live.

Makes it trivial to make changes and redeploy.

I mean no disrespect. This is more of a rant at how things are today. It is telling that over-complicated solutions have become so common that, for the current generation of devs, Kubernetes is the obvious way of doing stuff and a simple systemd service is the obscure one. I am sure there are good reasons for this, but it still feels like a loss when simplicity is no longer obvious.

Here's a CLI plugin that automates this: https://github.com/wowu/docker-rollout

Blue Green Deployment. There must be a docker container to handle this or at least a bash script.

edit: thanks to next comment for referencing one

Why do you say it's an AI company? It seems like their business is "Distribute your application to self-managed customers" not especially AI focused.

I think it just means "This is how Distr works", which seems clear enough.

That's the problem. If there are many heavy apps, you don't want to start all of them at once. There's no control about that, that I'm aware of. Also, if you have dependson, this is a compose feature and it's ignored when docker daemon is restarted.

> IP ratelimited for Docker Hub

How? What I'm describing is using Docker less.

> The build process can exhaust resources on the host

Maybe, but I've yet to have a host where that's the case for usual CRUD fare.

> The advantage is simplicity, and it's often worth the risk.

That's basically what I'm evaluating for here.

For bog standard LAMP or similar stack applications, I've not understood the advantage of going through the build-image-then-pull-on-host rigmarole. There's more layers involved there than something like provisioning with Ansible and just having a deploy script to run the usual suspects.

But I have seen that done fairly often, hence was wondering what the point was.

> you end up having to copy all the build dependencies (source code) to the host

> disk, i/o exhaustion

This is why I mentioned specifically for ecosystems like PHP, which are interpreted. I'm specifically asking for that use case.

I'm not building binaries, my "build" steps are actually deployment steps (npm build, composer install, etc) that I'd be running in exactly the same way on the host. The image I'm deploying by definition also contains my source code because I'm not deploying anything compiled.

I guess what I'm asking for is what the point is of a "baked" image for interpreted language ecosystems. Already using multi stage builds.

"Builds" are the same as deploys, so when working with server(s) instead of larger scale deployments, I'm not seeing the benefit of the whole "build image, pull on server" pipeline when I can just ditch the registry and added layers by doing those steps on the server as I would normally in other kinds of scenarios.

But I have seen this in action, which is why I'm wondering if I'm missing something.

The clearer benefit to me seems to be in this scenario to use it as a fast environment provisioning tool.