Postmortem

How Players Cleared (and Failed) Level 6

Format: Community retrospective | Reward: +0 XP (Knowledge XP, technically) | Prerequisites: Level 6 attempted

Why we are writing this

We have been watching Level 6: The Boss Level for long enough that the patterns are clear. Some players walk in, work through Phase 1 to Phase 3 in an evening, and submit a clean policy. Most do not. Most of you bring the same five mistakes and ask the same three questions, and we are going to publish the answers here so the next batch of players has a fighting chance.

This is not a solution guide. There is no full policy module in this post. If you are looking for that, close the tab and go back to the level. We are doing this the same way a good game guide does it - hints, common pitfalls, the parts where players reliably get stuck, the bonus objective almost no one finishes. The rest you write yourself.

If you cleared the level without reading this, achievement unlocked. You earned the result. The rest of this post is not for you.

The Phase 1 trap: triaging too quickly

The most common Level 6 failure pattern is not technical. It is procedural. Players see 200+ AVC denials in the audit log and immediately reach for audit2allow -M custom to generate a module that allows everything. The level explicitly tells you not to do that, and roughly forty percent of submissions do it anyway.

The triage step is the level. The policy writing is bookkeeping. Look at what we said in the Mission Briefing one more time:

type=AVC: 41 denials for httpd_t accessing user_home_t (read)

Players who blanket-allowed this one are about to ship a production policy that lets the web server read every user's home directory. That is not a policy gap. That is a misconfigured web application that is trying to serve content out of a place it should not be. The fix is the application config or the file labels, not the policy.

The same logic applies to the httpd_t -> var_log_t (write) denials and the container_t -> etc_t (write) denials. None of those should be allowed. Two of them should be investigated as security incidents. The level was teaching you to recognise the shape of denial that means "the system is telling you something is wrong, do not silence it."

If you wrote allow rules for the container-escape denials, restart Phase 1. If you ran setenforce 0 while debugging, restart everything. We do not need to explain why.

The Phase 2 mistake: writing the module backwards

Every player who submits a working policy gets through Phase 2. The struggles cluster around three specific spots, and we know what they look like because we have watched dozens of in-progress modules go through them.

Mistake 1: Defining types without committing them to a domain. A surprising number of submissions declare type appserver_data_t; and then never associate it with anything. The module loads. The label never sticks. The first time the policy is tested, the application server still cannot read its own data files because the data files are still labeled default_t. The fix is the file-context entry, which has to live in appserver.fc alongside the .te file:

/opt/app/data(/.*)? gen_context(system_u:object_r:appserver_data_t,s0)
/var/log/appserver(/.*)? gen_context(system_u:object_r:appserver_log_t,s0)

Then restorecon -Rv /opt/app /var/log/appserver and the labels actually take. About a third of the failed submissions skipped this step entirely.

Mistake 2: Forgetting that init_daemon_domain is not a free pass. The macro creates the entrypoint transition from init_t to your custom appserver_t, but it does not give your daemon any other capabilities. The submissions that landed at "the app server starts but cannot bind its port" almost always missed corenet_tcp_bind_generic_node as a separate rule. corenet_tcp_bind_http_port(appserver_t) by itself is not enough on RHEL 9 and later.

Mistake 3: Allowing what you should be transitioning. Several submissions added allow appserver_t shell_exec_t:file execute instead of writing a domain transition for the helper script the application calls out to. A blanket execute permission is not security policy. It is a confession. Use domain_auto_trans or type_transition when the helper has its own labelling, even if the helper is a one-line shell script.

The Phase 3 misconception: MLS is not extra targeted policy

We have answered the same question in our inbox roughly a hundred times by now. The question is some variant of: "I implemented MLS but my web server can still read the s2 files. What did I do wrong?"

The answer is almost always that the player did not actually switch to MLS. They left SELINUXTYPE=targeted in /etc/selinux/config, added MLS-style range labels to a few files, and assumed the kernel would do the rest. The kernel will not. MLS is a different policy, not an extension of targeted. The full switch requires:

[root@gamehost ~]# dnf install selinux-policy-mls
[root@gamehost ~]# sed -i 's/SELINUXTYPE=targeted/SELINUXTYPE=mls/' /etc/selinux/config
[root@gamehost ~]# fixfiles -F onboot
[root@gamehost ~]# reboot

If you skipped the package install, you got an unexpected log of "policy mls not found" warnings on the next boot, and the system fell back to targeted silently. We watched several players spend an evening debugging "why are my MLS rules not working" before checking sestatus.

The other common confusion is the constraint direction. In MLS, a process at s0 cannot read a file at s2 (no read up). That part most players got. The part that surprised them was that a process at s2 cannot write to a file at s0 either (no write down). This catches every submission that tried to "make the s2 backup process also write a status file in /var/log" - the status file is at s0, the backup process is at s2, and the kernel says no. The fix is to either label the status file at s2 or pick a different mechanism for status reporting (a syslog facility configured to accept the higher sensitivity is the usual answer).

The bonus objective almost no one finished

The bonus asked for an incident report on the container-escape attempt from Phase 1. It included a timeline reconstructed from audit logs and the exact policy rules that prevented exfiltration. Of the submissions we have seen, fewer than one in twenty included a real bonus answer. Most skipped it. A handful submitted a paragraph of prose and called it done. Two were excellent.

If you want to attempt it now, here is the structure that worked in the strong submissions:

  1. Pull every audit record for the offending PID with ausearch -p <pid> and reconstruct the process tree from the ppid field. The escape attempt almost always shows up as container_t spawning a child with fork + execve on a shell, which is itself denied.
  2. Note the timestamps. The two-minute gap between the first denied etc_t (write) and the second is where the attacker tried a different approach. That gap is your evidence that this was not a misconfigured container - misconfigured containers do not change tactics.
  3. Identify which rule prevented exfiltration. In our reference attack chain, it was container_t -> var_t (search) getting denied, which prevented the attacker from reaching /var/spool/cron. Without that path, the persistence mechanism they were going for was not available.
  4. Write the report as if you are presenting it to an incident-response team. Three paragraphs maximum. The audit log is the evidence; your report is the index.

If your bonus answer reads like a tutorial on how SELinux works, you wrote the wrong document. The audience for an IR report knows what SELinux is. They want to know what happened, when it happened, and why it did not get worse.

The questions we keep getting

"Should I use permissive mode while developing the policy?" Yes, but only on the specific domain you are writing, not system-wide. semanage permissive -a appserver_t puts appserver_t in permissive mode while leaving the rest of the system enforcing. setenforce 0 is the cardinal sin. Domain-targeted permissive is a reasonable development tool.

"What about audit2allow?" Use it as a hint generator, not as a policy generator. The output is a set of allow rules that would silence the denials it sees. Read every line. Strike the ones that should not be allowed. Keep the ones that should. The result is a starting point for your .te file, not a finished module. Players who pipe audit2allow straight into semodule -i are building permission lists, not policy.

"My policy module compiles but the daemon still gets blocked." Run semodule -l | grep appserver to confirm the module is actually loaded. Half the time it is not. The other half, the file context labels are not applied yet because restorecon was not run on the right paths. matchpathcon /opt/app/data/somefile tells you what label SELinux thinks it should have, and ls -Z /opt/app/data/somefile tells you what label it actually has. If those disagree, run restorecon.

"How do I test without breaking production?" Use a clean RHEL or Fedora VM with the same package set. The Fedora SELinux quick docs are the right reference. Dan Walsh's blog has answers to almost every architectural question you might hit, and the SELinux Notebook is the official reference for everything else. Do not develop policy on the box that is also serving customers.

What we are working on next

We are going to publish a Level 7 in the coming weeks that focuses on AVC-denial decoding without reaching for audit2allow. If your Boss Level submission depended on audit2allow, that level is for you. If you also struggled with the labels-versus-policy distinction in this postmortem, take another run at Level 2 first - the muscle memory you build there is what makes Level 6 manageable rather than overwhelming.

For everyone who finished Boss Level cleanly, including the bonus: the achievement plate stands. You can write SELinux policy. We are going to keep raising the ceiling.

POSTMORTEM COMPLETE

You have read the answer key

Knowledge XP gained, leaderboard XP unchanged. The points you earned, you earned by clearing the level - not by reading this. Go back and try the parts you skipped. We will be here.

Return to Level 6: The Boss Level or read more Advanced Guides.