Project demonstrating container isolation techniques via namespaces, cgroups, and seccomp. My motivation in creating this is that I have found that I work best with abstractions, Docker in this instance, if I understand the underlying concepts.
| Category | Technology Used |
|---|---|
| Language | C |
| Compiler | GCC |
| Build System | GNU Make |
| Libraries | Standard C Library, Linux kernel headers |
| Operating System | x86_64 Linux |
| License | GNU General Public License V2 |
Creates an isolated execution environment via:
The child process should be added to the cgroup before it is allowed to do anything else because we do not want it to run without those restrictions in place even temporarily. We have to configure the cgroup limits in the parent before having the child add itself to the cgroup, so we needed a way to signal from the parent to the child that configuration is complete. It is considered best practice to have the child add itself to the cgroup since it knows its PID before clone() returns in the parent. I opted to create a pipe in the parent and store its file descriptors in the configuration struct that gets passed to the child process. The child waits to read a single byte from the pipe, which the parent writes when it finishes configuring the cgroup.
Cgroup controllers are the kernel subsystem used to manage a particular resource within a given cgroup. It took an embarrassing amount of time to get this right in my implementation. I created the cgroup before writing to the immediate parent's subtree_control and was confused why it wasn't enforcing. I tried writing to the new cgroup's subtree_control instead and was equally confused that it wasn't enforcing. Here is the actual process. Before creating a cgroup, you must enable the controllers that you will be using in the immediate parent's cgroup.subtree_control file. We write the following to said file to enable the CPU, memory, and PID controllers in Euclid:
+cpu +memory +pids
Like fork(), the clone() syscall creates a child task (a process or thread). The key difference between the two is that clone() allows for fine-grained control over what attributes of the parent get shared to the child. This makes it the natural choice for containerization because it allows us to create new namespaces for the task. Tasks in Linux have their own kernel and user stacks, which adds a bit of complexity to how we manage our configuration variables for the container. clone() does allow for the parent and child to share the same memory space via the CLONE_VM flag, but this would be a security nightmare for a container as we'd be creating a thread rather than an isolated process.
Without the CLONE_VM flag, the child gets a copy of the parent's heap (see Copy-on-write section of Lessons Learned). This allows us to allocate a struct containing our configuration variables on the heap and pass a pointer to that struct to clone(). Since the child's address space gets destroyed entirely on its exit, we only need to free the struct in the parent.
Seccomp-bpf allows for filtering syscalls using bytecode based on Berkeley Packet Filter syntax. While the syntax is not that verbose, the filter became unwieldy at nearly 100 syscalls. Since allowing a syscall consists of two instructions that differ only by the syscall number, I defined a macro to reduce the filter length.
#define ALLOW(syscall)
BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, syscall, 0, 1),
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW)
This translates to:
This is the core concept of containerization. Namespaces were introduced as a mechanism within the Linux kernel back in 2002 to isolate resources that are normally global. While there are currently 8 supported namespace types, Euclid only concerns itself with 5 of them:
There are generally many instances of a namespace type running on a given system. Since container runtimes want their resource usage to be isolated from the rest of the system, the clone() system call is used to create new namespaces for the target application. If we wanted to join a process to an existing namespace, we would use the setns() system call. The only namespace type that setns() does not work on is PID, because PID namespace membership is fixed when a process is created.
This really intrigued me. OverlayFS is a type of union filesystem, meaning that it combines multiple directories into a single view. In the case of OverlayFS, four directories are relevant:
We use OverlayFS in Euclid to give the user the illusion of altering the rootfs while the modifications that they are making are actually being written to the upperdir that we have mounted within a tmpfs.
Another fascinating kernel mechanism. Tmpfs is a filesystem that stores its files within RAM and swap space rather than on disk. It follows that the /tmp directory uses tmpfs, but this is not always true. Some systems, such as those running older distributions or having been configured intentionally to operate as such, use a directory on disk for /tmp.
As indicated in the previous section, the filesystem writes that occur within our container in Euclid are confined to a tmpfs. This choice was made for a few reasons. It serves to isolate the container from the host filesystem, allows for quicker reading and writing because RAM is significantly faster than disk, and the tmpfs disappears when the container exits.
When you run fork() or clone() without the latter using the CLONE_VM flag, the kernel makes use of this fascinating optimization. The kernel marks the pages in the parent's virtual address space, which is shared with the child, as read-only. If either the parent or child attempts to write to one of those pages, a new page is allocated. The primary motivation for implementing this was that part of the way fork() duplicates a process is by copying its entire virtual address space, but is most often followed by the exec() syscall, which replaces the calling process with another: this means that the copying would usually be wasteful.
Every process has a struct of type fs_struct in the kernel, which contains the task's filesystem context including a pointer to the path of the root filesystem as recognized by the process. The chroot syscall changes that pointer to the path specified, which makes it lacking for security purposes because the old pointer could still potentially be accessed.
The pivot_root syscall is quite different from chroot. Rather than working on the process level, it changes the root mount in the mount namespace that it is called within. pivot_root does not make the old root mount inaccessible on its own, it moves it to a specified directory under the new root mount. If we want to hide the old root mount, we must unmount it manually.
I consider this project to be feature complete for what I intended to learn, but I have been considering using what I learned here to write a cgroups wrapper in Python for testing low memory conditions via memory.high (triggers throttling) and memory.max (kills with OOM).