For more half-baked ideas see the ideas tag
I guess like most people who work a lot with computers, I have a large number of computers in a “server room” (my loft). I’m too cheap to buy actual servers though, so most of these computers lack any sort of remote management / IPMI / BMC etc. I also just bought 6 Intel NUCs and these are also ideal as development servers, but unless you buy very specific (and unobtainium) versions they don’t come with remote management either.
Could we design a small USB device which would turn an ordinary PC into a remotely managed server?
What would we want it to do?
- Reboot remotely (like hitting the power switch, but remote).
- Unattended operating system install and reinstall.
- Modify BIOS settings.
At first the answer is obviously no since you cannot capture the display output from this device, but let’s ignore the display for a moment.
My initial idea was that you could simply have a USB flash drive which would have some kind of minimal remote operating environment. You would have to reconfigure the server BIOS so it always boots off the USB drive, and the USB drive then either chain boots to the operating system or can interrupt the boot process to download a new image. This lets you reinstall an OS, but not much else.
However USB can do a whole lot more, including acting as a keyboard, and it’s possible for a single USB port to provide multiple functions like drive + keyboard. Could we design a USB peripheral which would inject keys (like Ctrl+Alt+Del to soft reboot)? It could also blindly control the BIOS by injecting keys.
Designing a USB peripheral is a bunch of work, but the hardware to do it already exists. The Raspberry Pi Zero can act as a USB device (known as a Linux USB gadget).
So the idea is you’d connect the RPi Zero to the server using a USB OTG cable. It would appear as a keyboard and mass storage device (but unlike above you would not change the server to boot off USB, it boots normally). The RPi Zero could be remotely accessed over wifi and using that you could inject keypresses (like Ctrl+Alt+Del), and present a boot disk to the server, and using blind keypresses negotiate the BIOS menus to select an alternate boot disk when you want to reinstall. You can also act as a virtual serial console gadget allowing simpler interactive use for some operating systems.
One problem here is the RPi is powered from the server, so when it reboots it’s possible it will shutdown (but perhaps not if plugged into the yellow USB ports since I guess those should have constant power).
Another problem is you can’t see the display, so you’re driving the BIOS and other menus blind. But it’ll probably work in most automated cases. USB has a video class but I’m not clear if it would help here (it also wouldn’t be any good unless the server BIOS could use it).
A third problem is you cannot hard reboot the server in case it hangs.
Richard WM Jones 
Rich, nice idea indeed. I think for now, you can use Foreman. The idea in Foreman PXE workflow is simple – set all your servers to only boot from 1) PXE, 2) HDD. Managed servers either boot into installer (Foreman calls this “build” mode), or they are told to carry on booting from HDD (non-build mode). In EFI mode local drive is chainbooted, so only set PXE in EFI. The thing is – you only set boot order once in BIOS/EFI and you keep this for the whole server lifecycle.
And it is more than that, Foreman Discovery plugin allows all servers which has been deleted from the inventory to boot into small live CD (PXE booted) based off Fedora/RHEL and you can remotely do things via Foreman UI/API/CLI. We currently have only reboot and kexec API calls to initiate reinstallation, but in the future we would like to add more (firmware update, BIOS/EFI change). We are also looking into bare-metal image-based (mass) provisioning, sky is the limit since this is full Linux distro (running from memory tho).
I was thinking/hoping you’d comment. This afternoon I asked a question on The Foreman community about whether I could use PXE and I see you’ve answered it, thanks!
The answer above is interesting, but wouldn’t it be possible to do even more with an active USB device of some sort (whether it was based on RPi Zero, or some other technology)?
Frankly, I had the very same idea but I did not even elaborate it to the same degree. There are possibilities but what for? The only thing you care about is really BIOS/EFI and the amount of work to get it working without display is huge. I believe that PXE (or UEFI HTTP) solves the half of the problem and the only device you really need is hard reset (power on/off) which you can achieve with UPS or any kind of smart plug.
That does not mean I don’t think this is an important effort. I hope that time of Open BMC will come, I expect a future security bug will be the trigger and companies like Facebook or Google will start pushing it.
Update: By the way I found The Foreman documentation on the bootdisk to be pretty terrible. I was hoping there’d be some kind of very basic background information about how people actually use it, for example, are there low profile USB flash drives for this purpose?
Would you mind bringing this over to our community forums? In short – bootdisk was a “hack” written by our community member and lots of users felt in love with it. There is indeed lack of docs and also limited set of features. As far as I know, most of our users use bootdisk for booting VMs actually, we do have a mechanism to connect it automatically to hypervisor (only vmware as of summer 2018).
Hey Richard Have you also thought of doing abit of hackery using the RPI Zero either switching the Reset /Power buttons either via a FET buffer o opto isolator meaning as well as controlling the computer via gadget Class Mouse keyboard implementations but also the Reset and Power directly via GPIO
I was hoping it would be possible entirely using off-the-shelf hardware, but as you say it looks like we’d need hardware modifications. Not sure if it’s possible to easily get at the power button on the Intel NUC — I’d have to take one apart to find out.
Yeah shouldn’t cost you more than a quid to have a go .. I did notice that the reset and softpower were on the PCB of my Mini PC(Not NUC BTW) so you’ll probably find something similar in the NUC’s guts some where !