For more half-baked ideas, see the ideas tag.
When your program links to a library, the program and the library run in the same process and so have the same SELinux and other security context (UID, resource limits, etc).
Why can’t libraries have their own context? This would protect your program from bugs in libraries, and allow you to control more precisely what the library is doing, eg. what files it is trying to access.
I originally thought this would be simple: just have the kernel set the context according the userland instruction pointer (ie.
%rip). Program code in certain pages would therefore have a different context when calling into the kernel from library code in other pages. You might also have to be careful with system calls that read and write user data. Anyway, this doesn’t quite work because a library could bypass this by changing the memory protection on some code in another part of the program, overwrite that code, then jump to it. Perhaps if the library’s security context prevented it also from calling mprotect etc it could be made to work? Edit: no it can’t be made to work, see the comments.
Another way to do it would be to (behind the scenes) run the library really in a separate process, turning local calls into RPCs. This is also not totally trivial if you consider global variables (shared memory?) and callbacks, and the performance implications could sink this idea too.
In any case the important thing would be to make any technique as transparent to the programmer as possible, so that programmers don’t have to explicitly split programs up into separate communicating processes in order to gain the security advantages. If no code changes were needed, we could apply the technique to existing programs in order to harden Linux still further.