Gigabyte just announced a bunch of full ARM servers, with between 32 and 96 cores. They are based around the Cavium ThunderX processors that we’ve had at Red Hat for a while so they should run RHEL either out of the box or very soon after release.
Tag Archives: gigabyte
I finally got UEFI flashed onto the Gigabyte board so now it is SBSA/SBBR compliant [edit: see note at end] and will just work with RHEL. Instructions here: https://lists.centos.org/pipermail/arm-dev/2016-March/001743.html
Here are the boot messages from TianoCore:
TianoCore 1.20.03 UEFI 2.4.0 Jan 26 2016 18:09:04 CPU: APM ARM 64-bit Potenza Rev B0 2400MHz PCP 2400MHz 32 KB ICACHE, 32 KB DCACHE SOC 2000MHz IOBAXI 400MHz AXI 250MHz AHB 200MHz GFC 125MHz Board: X-Gene Mp30ar0 Board Slimpro FW: Ver: 2.4 (build 01.15.01.00 2015/05/22) TPC: disable AVS: support PMD: 970 mV SOC: 950 mV The default boot selection will start in 1 second
Note: A few people have pointed out that the Gigabyte isn’t SBSA compliant because it lacks the right serial port, RTC and WDT. However it still boots RHELSA fine if you add
IPMI works out of the box. I’m using FreeIPMI to test this (not ipmitool) since FreeIPMI is a lot easier to use.
You need to know that:
- The default user name is
adminand the default password is
- You have to use the dedicated management interface, marked “f” in the software reference guide (the ethernet port above the two USB sockets).
Here are the sensors:
$ ipmi-sensors -h 192.168.0.104 -u admin -p password Caching SDR repository information: /home/rjones/.freeipmi/sdr-cache/sdr-cache-moo.192.168.0.104 Caching SDR record 34 of 34 (current record ID 205) ID | Name | Type | Reading | Units | Event 4 | CPU0_TEMP | Temperature | 49.00 | C | 'OK' 9 | DIMM_P0_A0 | Temperature | N/A | C | N/A 10 | DIMM_P0_A1 | Temperature | N/A | C | N/A 12 | DIMM_P0_B0 | Temperature | N/A | C | N/A 13 | DIMM_P0_B1 | Temperature | N/A | C | N/A 15 | DIMM_P0_C0 | Temperature | N/A | C | N/A 16 | DIMM_P0_C1 | Temperature | N/A | C | N/A 18 | DIMM_P0_D0 | Temperature | N/A | C | N/A 19 | DIMM_P0_D1 | Temperature | N/A | C | N/A 59 | P12V | Voltage | 11.83 | V | 'OK' 60 | P5V | Voltage | 5.11 | V | 'OK' 61 | P3V3 | Voltage | 3.33 | V | 'OK' 62 | P5V_STBY | Voltage | 5.13 | V | 'OK' 64 | P_VBAT | Voltage | 3.07 | V | 'OK' 65 | P_VCCP | Voltage | 0.97 | V | 'OK' 66 | P_1V2_HUB | Voltage | 1.20 | V | 'OK' 67 | P_VDDQ_AB | Voltage | 1.50 | V | 'OK' 68 | P_VDDQ_CD | Voltage | 1.50 | V | 'OK' 71 | P_0V9_VDD | Voltage | 0.96 | V | 'OK' 72 | P_1V5_VDD | Voltage | 1.52 | V | 'OK' 73 | P_2V5_VDD | Voltage | 2.50 | V | 'OK' 74 | P_1V8_VDD | Voltage | 1.82 | V | 'OK' 136 | CPU0_FAN | Fan | 4000.00 | RPM | 'OK' 138 | SYS_FAN1 | Fan | N/A | RPM | N/A 139 | SYS_FAN2 | Fan | N/A | RPM | N/A 140 | SYS_FAN3 | Fan | N/A | RPM | N/A 141 | SYS_FAN4 | Fan | N/A | RPM | N/A 190 | CPU0 | Processor | N/A | N/A | 'Processor Presence detected' 202 | MB_TEMP1 | Temperature | 37.00 | C | 'OK' 203 | MB_TEMP2 | Temperature | 31.00 | C | 'OK' 204 | MB_TEMP3 | Temperature | 28.00 | C | 'OK' 205 | SEL | Event Logging Disabled | N/A | N/A | 'OK'
Turns out that
acpi=off is only needed by the RHEL 7.2 installer kernel. After installation, ACPI works fine. That might be because the installer kernel is older than the current RHEL 7 kernel. dmesg output after the break.
eth0 is the top left connector (the upper connector marked “c” in the motherboard manual).
I burned RHELSA
RHEL-7.2-20151030.0-Server-aarch64-boot.iso to a USB key and inserted it into the machine.
I had to chain-load Tianocore (UEFI firmware) as described here. This requires that you run a TFTP server. You should also set the u-boot
serverip environment variables so that u-boot can contact your TFTP server.
Once in UEFI, do these commands at the UEFI shell:
set MAC0 xx:xx:xx:xx:xx:xx
Then reboot again, go into the UEFI shell again, and:
fs0: cd \EFI\BOOT BOOTAA64
I also had to stop the grub configuration and edit the linux kernel command line, adding
console=ttyS0,115200 acpi=off vnc at the end.
acpi=off is needed because the SBBR requires ACPI support, which the version of TianoCore which APM/Gigabyte have provided does not supply (or it’s broken in some way).
acpi=off is required to boot the installer, I found that RHELSA itself will actually work with ACPI. So you have to remove
acpi=off from the kernel command line after installation.
After that you should see lots of kernel messages, and with luck you’ll arrive in Anaconda and it will print details of the VNC server. Finally, I have a RHELSA installer …
The Gigabyte uses the APM X-gene1, which is an 8 core ARM processor. This is rather old now — I’ve had a Mustang at home with the same processor for a couple of years now and the chip design itself is 3+ years old. It uses Cortex A53 cores. Even APM have a newer X-gene 2. Nevertheless it has two very big advantages over the alternatives:
- You can actually buy it. I had this one shipped overnight from UK supplier Xcase.
- It’s (sort of — see below) server class hardware, unlike the very cheap but ultimately crap development boards based on phone chips.
A particularly annoying thing at the moment is that Gigabyte are claiming this board is SBSA compliant, but then they ship it with u-boot as firmware. However UEFI firmware can be downloaded, see this thread, so it should run RHEL.
Here are the boot messages to the u-boot prompt:
U-Boot 2013.04 (Jun 02 2015 - 10:54:10) REV: 1.15.01-F05 ( uart0 ) CPU0: APM ARM 64-bit Potenza Rev B0 2400MHz PCP 2400MHz 32 KB ICACHE, 32 KB DCACHE SOC 2000MHz IOBAXI 400MHz AXI 250MHz AHB 200MHz GFC 125MHz Boot from SPI-NOR Slimpro FW: Ver: 2.4 (build 01.15.01.00 2015/05/22) TPC: disabled AVS: supported (margin: -0mV) RST: supported PWROFF: supported PMD: 970 mV SOC: 950 mV Board: GIGABYTE MP30AR0 - AppliedMicro APM883408-xNA24SPT Customer Board I2C: ready DRAM: 32 GiB @ 1600MHz... SF: Detected MX25L25635F with page size 64 KiB, total 32 MiB MMC: X-Gene SD/SDIO/eMMC: 0 PCIE0: (RC) link down PCIE2: (RC) X1 GEN-1 link up PCIE3: (RC) link down 00:00.0 - 10e8:e004 - Bridge device 01:00.0 - 1a03:1150 - Bridge device 02:00.0 - 1a03:2000 - Display controller Video: ASPEED VGA Card (1a03, 2000) found @(2:0:0) Mode: 1024x768x32 48kHz 60Hz In: serial Out: vga Err: serial CPUs: 11111111 Net: eth0 USB0: scanning bus 0 for devices... XHCI: WARN: Didn't find a matching TT 3 USB Device(s) found USB1: scanning bus 1 for devices... 2 USB Device(s) found scanning usb for storage devices... 0 Storage Device(s) found XHCI: ep 0x1 - rounding interval to 128 microframes XHCI-ERR: xhci_submit_async_int ! Register 1 keyboards Hit any key to stop autoboot: 0
It was a bit of a saga to get it working. The main problem was the RAM that I bought was faulty, but I didn’t know that at the time since all you get is a blank display. The Brix uses strange low voltage SO-DIMMs and it’s unlikely you’d have any lying around — I certainly did not. Guessing that it could be the RAM, I ordered more which took another two days, and that worked.
Suggestion for Gigabyte: solder a minimal amount of RAM on the motherboard (eg. 256 MB) so that the Brix at least boots to the EFI shell with no SO-DIMM inserted.
Disassembling and reassembling the Brix is fiddly, but I got there in the end.
Without an operating system installed, it boots into the EFI shell. Booting with a RHEL USB key boots straight into the RHEL installer.
The dual core 64 bit Celeron is pretty nice given that the whole system cost me £137.
lstopo looks like this:
/proc/cpuinfo like this:
processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 55 model name : Intel(R) Celeron(R) CPU N2807 @ 1.58GHz stepping : 8 microcode : 0x811 cpu MHz : 981.246 cache size : 1024 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 2 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer rdrand lahf_lm 3dnowprefetch ida arat epb dtherm tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms bogomips : 3166.40 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 55 model name : Intel(R) Celeron(R) CPU N2807 @ 1.58GHz stepping : 8 microcode : 0x811 cpu MHz : 1831.644 cache size : 1024 KB physical id : 0 siblings : 2 core id : 1 cpu cores : 2 apicid : 2 initial apicid : 2 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer rdrand lahf_lm 3dnowprefetch ida arat epb dtherm tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms bogomips : 3166.40 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management:
For about 12 months I’ve been using the ASUS RT-N16 + OpenWRT as my gateway. But you know what? It sucks — I hate that it’s not a real Linux distro, that it’s difficult to upgrade, that it randomly reboots itself, that it only kinda manages IPv6, that it uses a half-assed packaging system, that I can’t run regular tools, and that it’s insecure.
And since I switched over to VDSL @ 80 Mbps which uses PPPoE (ie. no need for DSL terminated at the gateway), it’s time to switch to running a real distro on a real computer.
The first step is to choose a small form-factor PC, crucially with two ethernet ports. Unfortunately Intel still don’t sell NUCs with two ethernet ports. Also Intel NUCs are really expensive. So I’ve settled for a much cheaper alternative:
- Gigabyte Brix (Celeron N2807 version) — £90.99
- 4GB of RAM — £25.84
- USB 3.0 ethernet adapter (AX88179-based, upstream in Linux 3.9) — £21.04
Total cost (including tax and delivery): £137.87
Note that I already have a spare 2.5″ SSD, but you would need to add the cost of a SATA HDD/SSD if you don’t have one already.
It’ll be interesting to see how fast the USB 3.0 ethernet adapter is in real life, because it’s the obvious weak spot. However I only need the router to be able to forward at 80 Mbps, and even a Celeron and a weak ethernet adapter surely should be able to handle that.