Here's rough list of things that I had to do, because I'm doing right now exactly that: trying to deploy kubernetes.
1. Prepare server by installing and configuring containerd. Few simple steps.
2. Load one kernel module. Configure ip forward sysctl.
3. Install kubernetes binaries from apt repository.
4. Run kubeadm init
5. Install helm (this part is optional, but makes things simpler): download binary or install it from snap.
6. Install network plugin like flannel or calico.
7. Install nginx ingress plugin.
8. Install storage provider plugin. Also you should have some storage service like NFS server, so Kubernetes can ask for storage.
If you want single-node Kubernetes, I think that should be enough. May be #6 is not even necessary. If you want real cluster, you would need to tinker with load balancer for which I don't have clear picture right now. I'm using external load balancer.
If you're using docker, my understanding is that containerd is already installed.
I actually spent few weeks trying to understand those parts and I have only shallow understanding so far.
With that in mind, simple kubernetes solutions probably have its place among those who can't or don't want to use managed kubernetes from popular clouds.
I have no idea about those simple kuberneteses though.
My opinion is that vanilla kubernetes is not that hard and you should have some understanding about its moving parts anyway. But if you want easy path, I guess it's something worth considering.
One main reason you'd want to run minikube or kind is also that these clusters are easy to reproduce and don't pollute your system's network namespace and sysctl.
Then add the IP addresses of the nodes as a wildcard DNS.
because they are. they are directly built from the go sources, all are wrappers around the meat of k8s. (which are the various control loops packaged into services like api-server, kubelet, schedulker, controller-manager, etc ... and etcd itself)
minikube does a big monolithic build for convenience. (it can do this because all involved components are in pure go.)
microk8s is also a distribution of k8s.
almost all distributions have convenience features to help you with installation/setup. but all they do is what "the hard way" setups do. fetch/copy binaries, generate/sync keys, setup storage (devicemapper, btrfs volumes, whatever), setup wrappers that then start the binaries with the long list of correct arguments, and set them up to start when the node starts (usually by adding systemd services or something).
so if you want to get the genuine original mainline experience you go to the project's github repo, they have releases, and mention that the detailed changelog has links to the binaries. yeey. (https://github.com/kubernetes/kubernetes/blob/master/CHANGEL... .. the client is the kubectl binary, the server has the control plane components the node binaries have the worker node stuff), you then have the option to set those up according to the documentation (generate TLS certs, specify the IP address range for pods (containers), install dependencies like etcd, and a CNI compatible container network layer provider -- if you have setup overlay networking eg. VXLAN or geneve or something fancy with openvswitch's OVN -- then the reference CNI plugin is probably sufficient)
at the end of this process you'll have the REST API (kube-apiserver) up and running and you can start submitting jobs (that will be persisted into etcd, eventually picked up by the scheduler control loop that calculates what should run where and persists it back to etcd, then a control loop on a particular worker will notice that something new is assigned to it, and it'll do the thing, allocate a pod, call CNI to allocate IP, etc.)
of course if you don't want to do all this by hand you can use a distribution that helps you with setup.
microk8s is a low-memory low-IO k8s distro by Canonical (Ubuntu folks) and they run dqlite (distributed sqlite) instead of etcd (to lower I/O and memory requirements), many people don't like it because it uses snaps
k3s is started by Rancher folks (and mostly still developed by them?),
there's k0s (for bare metal ... I have no idea what that means though), kind (kubernetes in docker), there's also k3d (k3s in docker)
these distributions work by consuming/wrapping the k8s components as go libraries - https://github.com/kubernetes/kubernetes/blob/master/staging...
then there's the whole zoo of various k8s plugins/addons/tools for networking (CNI - https://github.com/containernetworking/cni#3rd-party-plugins), storage (CSI - https://kubernetes-csi.github.io/docs/drivers.html), helm for package management, a ton of security-related things that try to spot errors in all this circus ... and so on.
Given that understanding, could someone please explain what "rootless" would mean? I want to understand these in simpler terms:)
(Thank you in advance)
As simple as possible:
run every command inside container with full root permissions on host
$root-> Docker -> container
run every command as the current user
Maybe take a look here for a better explanation: https://docs.docker.com/engine/security/#docker-daemon-attac...
For example, buildah (the container-building part of podman) is daemonless and can use the fuse-overlayfs storage driver to build containers rootlessly— you appear as root inside the container, but from the outside, those processes and any files created are owned by the original invoking user or some shim UID/GID based on a mapping table.
But critically, this doesn't mean it's possible to just run buildah inside any Kubernetes pod and build a container there, because buildah needs to be able to start a user namespace, and must have the /dev/fuse device mapped in. I believe there continues to be ongoing work in this area (for example Linux 5.11 allows overlayfs in unprivileged containers), but the issue tracking  it is closed without really being IMO fully resolved, since the linked article  from July 2021 is still describing the different scenarios as distinct special cases that each require their own special sets of flags/settings/mounts/whatever.
You can run "capsh --print" to see your current capabilities. And to run a container without any capabilities:
podman run --cap-drop ALL -it fedora capsh --print
podman in rootless mode gets around these by using slirp4netns to create pure-userspace overlay networks, fuse-overlayfs to create pure-userspace overlay filesystems (or a driver that can't deduplicate storage on older kernels), and uid/gid mapping in user namespaces to create the illusion inside of a container that an application is running as root when it isn't really root on the host.
Additionally, podman gets rid of the daemon and just uses normal fork/exec of the ephemeral podman process.
The upsides are:
- podman can run entirely in home directories and doesn't need to globally install config files or the container filesystems, making it easier for many users to share the same server.
- Running a malicious or compromised container won't compromise your host (big caveat here is unless it can exploit a vulnerability in user namespaces).
- Users who don't have root at all can still run containers. Note that while this appeared to be true using Docker because you could just be part of the 'docker' group to write to dockerd's socket, effectively this was giving you root.
The biggest downside is the userspace networks and filesystems are slow compared to their in-kernel counterparts, which is why you typically won't see it in any kind of production setting, but minikube is meant to be used as a small-scale mock of production kubernetes run by developers, so it can be a good fit there.
Note that rootless minikube was actually already possible, but way more convoluted than just using rootless podman as the container runtime.
Docker has two main components. The daemon (you can think of it somewhat like a server) and the client (application you use to run commands).
When you install docker on your machine, it generally installs both. The daemon is a process that runs on your local machine and runs as root.
Rootless refers to the alternative method (used by podman for instance) to run the daemon as a standard user, and delegate root-level tasks to something else, like systemd for instance.
Is the daemon what they call the docker-engine? Is this what's available on Linux natively? Rootless makes sense here bc you wouldn't want one docker image able to interfere with another, or even the Linux system that is running the docker runtime/engine.
For Windows/Mac docker solutions, where does the daemon live/exist/run? Inside a virtualized Linux instance?
As I understand it, most of these alternatives to docker-desktop are all just wrappers around a virtualized Linux image running the docker engine/runtime. That's why many of them require a virtualization engine like Virtual Box. So are these no-commercial solutions just wrappers around one or more virtualized Linux runtimes where the docker engine/runtime is running natively?
If all the above is (approx) correct, then "what" is rootless with this announcement? The docker runtime/engine in the virtualized Linux instance?
I thought the docker engine/runtime on Linux was always able to run rootless docker images. So what is the news here if all these non-commercial solutions are just wrappers around the docker engine/runtime running in a virtualized Linux?
Docker-engine is the daemon built by docker. Podman is an opensource work a like. Docker-engine doesn't support running as a user other than root. Podman does. This announcement says minikube will work with Podman running as not root.
I remember hearing that development of docker-engine was ceasing, but could obviously live on as it was forked. I guess rootless is some of the work that Docker (company) wanted to keep proprietary and out of this open-source project.
Really quite a shame, although understandable from a commercial perspective.
Assuming that these improvements are finding their way back into an open-source project, I'm glad to hear about this work from minikube and Podman.
Rootless mode for Docker is completely FLOSS, and its main contributor (me) has even never worked for Docker (company).
Docker engine does.
You can see some downsides to this when you do the classic developer setup system of having a docker image with your tools and mounting a volume of your source tree into the container for building. When you build, the build products in your filesystem are owned by root because the code was actually running under the daemon. This can cause all sorts of pain.
When you run something like podman, there's no daemon - it's all just processes running as your user (like any other script) so files created end up on your filesystem owned by you.
Doesn't this have more to do with the daemon that the user executing commands ?
Which effectively gives you root on the host.
I'm fairly sure this is only the case on older systems, if your system is up to date then podman should not rely on suid binaries.
cgroup V2 support
Ensure fuse-overlayfs is installed
of course, withouth any of that your containers will be able to do very little (eg: no networking).
Better to hide this feature and promote the rootless docker mode for local use. On servers you won't be adding any unprivileged user to the docker group in any case.
%wheel ALL=(ALL) NOPASSWD: ALL
effectively disabling sudo completely.
I haven't encountered any issues like this personally with rootless podman (although I'm not doing any large scale deployments).
The network performance is caused by slirp (usermode TCP/IP) but it is being resolved too : https://github.com/rootless-containers/bypass4netns
I think Google's distroless container images are worth checking out as a quasi-alternative: https://github.com/GoogleContainerTools/distroless You use them as a base for a docker image and copy in your server code. These images are tailor made to strip out _everything_ that's not necessary to run the software--there's no shell for example. So you're still running a Linux kernel, libc, etc. but there's nothing there for an attacker to use other than your app code. You yourself can't even get into a shell to debug or examine what the state of your app is (which can actually be kind of aggravating in development).
I build my own distroless-like images for personal use using Fedora and RHEL, though I do follow the ubi-micro build steps and include a tiny bit of user space components to enable debugging.
Just run your program as the only process.
As a Linux host with no other software.
No /bin/sh, nothing else in the filesystem.
Simple demo: https://github.com/tv42/alone