Xilinx Virtex 7 FPGA bitstream reverse engineered

While my article on HN is getting no traction I might as well post on here some fantastic news: The Xilinx Virtex 7 FPGA bitstream has been reverse engineered by Clifford Wolf.

For some context this is a very popular and cheap series of FPGA devices. For example you can buy the Arty board which has one of these FPGAs for $99, or the slightly more advanced Nexys 4 DDR for $265.

Currently you must use the Xilinx Vivado tool which is a 40 GB download [no, that isn’t a typo], requires a paid license to unlock the full features, and is generally awful to use.

This work should eventually lead to a complete open source toolchain to program these devices, just like Project IceStore for the Lattice devices.

Playing with PicoRV32 on the iCE40-HX8K FPGA Breakout Board (part 2)

It works!

Press ENTER to continue..
Press ENTER to continue..

  ____  _          ____         ____
 |  _ \(_) ___ ___/ ___|  ___  / ___|
 | |_) | |/ __/ _ \___ \ / _ \| |
 |  __/| | (_| (_) |__) | (_) | |___
 |_|   |_|\___\___/____/ \___/ \____|


SPI State:
  LATENCY 8
  DDR OFF
  QSPI OFF
  CRM OFF
Select an action:

   [1] Read SPI Flash ID
   [2] Read SPI Config Regs
   [3] Switch to default mode
   [4] Switch to Dual I/O mode
   [5] Switch to Quad I/O mode
   [6] Switch to Quad DDR mode
   [7] Toggle continuous read mode
   [9] Run simplistic benchmark
   [0] Benchmark all configs

Command> 9
Cycles: 0x00f3d36d
Instns: 0x0003df2d
Chksum: 0x5b8eb866

In the first part I got my reverse-engineered Lattice iCE40-HX8K FPGA using the completely free Project IceStorm toolchain working. I wrote a simple Verilog demo which flashed the LEDs.

Today I played with Clifford Wolf’s PicoRV32 core. What he’s written is actually a lot more sophisticated than I initially realized. There’s a simple memory mapped serial port, a memory mapped SPI bus, & a bit of interactive firmware so you can test it out (see above).

Rather than using Clifford’s build scripts (which compile the riscv32 cross-compiler and run sudo at various points) I wrote a Makefile to build and program the FPGA on Fedora.

And it works (see above)! Now what we need is a simple 32 bit operating system that could run on it. swapforth seems to be one but as far as I can tell it hasn’t been ported to RV32. Maybe I could port Jonesforth …

Playing with PicoRV32 on the iCE40-HX8K FPGA Breakout Board (part 1)

It’s now possible to get a very small 32 bit RISC-V processor onto the reverse-engineered Lattice iCE40-HX8K FPGA using the completely free Project IceStorm toolchain. That’s what I’ll be looking at in this series of two articles.

I bought my development board from DigiKey for a very reasonable £41.81 (including tax and next day delivery). It comes with everything you need. This FPGA is very low-end [datasheet (PDF)], containing just 7680 LUTs, but it does have 128 kbits of static RAM, and the board has an oscillator+PLL that can generate 2-12 MHz, a few LEDs and a bunch of GPIO pins. The board is programmed over USB with the supplied cable. The important thing is the bitstream format and the probable chip layout have been reverse-engineered by Clifford Wolf and others. All the software to drive this board is available in Fedora:

dnf install icestorm arachne-pnr yosys emacs-verilog-mode

My first job was to write the equivalent of a “hello, world” program — flash the 8 LEDs in sequence. This is a good test because it makes sure that I’ve got all the tools installed and working and the Verilog program is not too complicated.

// -*- verilog -*-
// Flash the LEDs in sequence on the Lattice iCE40-HX8K.

module flash (input clk, output reg [7:0] leds);
   // Counter which counts upwards continually (wrapping around).
   // We don't bother to initialize it because the initial value
   // doesn't matter.
   reg [18:0] counter;
   // This register counts from 0-7, incrementing when the
   // counter is 0.  The output is wired to the LEDs.
   reg [2:0] led_select;

   always @(posedge clk) begin
      counter <= counter + 1;

      if (counter[18:0] == 0) begin
         led_select <= led_select + 1;
      end
   end

   // Finally wire each LED so it signals the value of the led_select
   // register.
   genvar i;
   for (i = 0; i < 8; i=i+1) begin
      assign leds[i] = i == led_select;      
   end
endmodule // flash

It looks like the base clock frequency is 2 MHz.

The fully working example is in this repo: https://github.com/rwmjones/icestorm-flash-leds

In part 2 I’ll try to get PicoRV32 on this board.

Now building Fedora/RISC-V “stage4” disk images

I’m happy to announce that Fedora/RISC-V, the project to bootstrap Fedora on the new RISC-V architecture, has reached a key milestone: We are now releasing clean “stage4” disk images which are built entirely from RPMs (ie. every file except two[1] are managed by RPM).

You can get the latest image from http://oirase.annexia.org/riscv/

To use it, you must enable my RISC-V tools copr:

# dnf copr enable rjones/riscv
# dnf install riscv-qemu riscv-pk

and you can then boot the stage4 directly using this qemu command[2]:

$ qemu-system-riscv -m 4G -kernel /usr/bin/bbl \
    -append vmlinux \
    -drive file=stage4-disk.img,format=raw -nographic

This is an early release and there are a few problems. The main one is that we lack a util-linux package, and thus there is no mount command so the disk image stays read-only after boot. You’ll see lots of errors like this at boot:

/init: line 16: mount: command not found
/init: line 19: mount: command not found
/init: line 20: mount: command not found

I hope to get that fixed soon.

There’s also no actual rpm command in the stage4, again because of a required dependency, and again that’s something that will be fixed real soon.

Many thanks go to David Abdurachmanov and Stefan O’Rear for their huge efforts building packages.

Notes:

[1] Because there is no systemd package yet, currently two extra files are added to the disk image which are not under the control of RPM: /init and /usr/bin/poweroff

[2] For real hardware, read this page

Fedora/RISC-V, steady progress

davidlt has done an amazing job building RISC-V RPMs: https://github.com/rwmjones/fedora-riscv/tree/master/stage3-built-rpms

I also managed to boot our “stage 3” filesystem on the real FPGA hardware. Unfortunately it’s extremely crashy:

# ldconfig /usr/lib64 /usr/lib /lib64 /lib
disk cannot read 4096 bytes @1544056832!

This is in the HTIF / SD-card access layer which we have full source for so at least it can be fixed.

RISC-V on an FPGA, pt. 8

Some thoughts on SiFive Freedom U500 on the low-end Arty Board:

• 256MB (249MB available), vs 128MB for the Nexys 4. However memory is used for the filesystem initramfs / tmpfs (see next point) so there is a trade-off between RAM and storage. Recall the Nexys 4 has a microSD card for storage.
• The root filesystem is the initramfs, presumably loaded off the 16MB of SPI flash included with the board. So it’s more like an embedded target than something you could do any development on.
• Slow – noticeably slower than lowRISC on the Nexys 4. However it’s also under half the price, and this is an FPGA design, and the performance of the real hardware will be completely different.
• I was kind of expecting that ethernet would work, but in any case it doesn’t appear to work for me.
• You can replace their Linux kernel and root image with your own — see section 5 in the documentation. However that step, as well as programming the board, requires the proprietary Vivado software. AFAICT there is no free software alternative.

RISC-V on an FPGA, pt. 7

Today I’m going to try SiFive’s Freedom U500 64 bit RISC-V design on the very low-end $148 Arty Board. If you want more background into what SiFive are up to then I recommend watching this 15 minute video, but in brief they seem to be positioning themselves as a distributor and integrator of RISC-V.

The good thing is they compile everything you need including the Xilinx bitstream. The bad thing is that these are behind a registration wall with a poisonous license agreement (in particular, non-commercial use only). I hope this is only because of the bits of 3rd party proprietary IP they are using for ethernet, flash, UART, etc.

If you want to do this yourself, read the getting started guide here. Assuming you have Xilinx Vivado installed, following the instructions is completely straightforward except for one point: You need to do “Boot from Configuration Memory device” after programming. Anyway you will have a booting Linux/RISC-V in a few minutes.

After the cut, the boot output.

RISC-V on an FPGA, pt. 6

(Click for larger image)

The board on the left is the Digilent Nexys 4 DDR which I was using yesterday to boot Linux. It costs £261 (about $341) including tax and next day delivery. The board on the right is the cheaper Digilent Arty Board, which cost me $148 (two day delivery from the US) + £20 tax.

There are clear differences in the number of connectors, LEDs and buttons. The Nexys has VGA, ethernet, USB, 8 digit LED, many lights, a temperature sensor, and lots of buttons. The Arty has just the bare minimum as you’d expect given the price difference. Both boards use the same Xilinx Artix-7 family of FPGA, but they are not exactly the same. The expensive board on the left uses the XC7A100T which has 100K+ logic cells, the cheap board on the right uses the XC7A35T with 33K cells. What this means practically is that you are more limited in the size of designs which can be programmed on the smaller chip. Does this matter for RISC-V? Yes and no. The untether-v0.2 design I was using yesterday takes about 50K cells so it won’t fit on the smaller board. However SiFive apparently (I have not checked) have a reduced design that will fit. (Note that none of this affects the operating system or available RAM — those are separate issues. However a smaller design will have to cut a few corners, leave out parts of the chip that implement optimizations, etc and so will generally run slower).

Oddly the Arty Board (on the right) has more DDR3 RAM — 256MB vs 128MB. That is an improvement, since doing any real Linux work in 128MB of RAM is tough, but still not massive. The other significant difference is the Arty Board does not have a microSD-card socket. It has (just) 16MB of on-board flash instead. I’m not clear how you get Linux on there, but that’s what I’ll be exploring.

Finally it’s worth saying that both boards are incomplete, although in very minor ways. The Nexys 4 comes with an OTG USB cable, which is all you need to power the board, program it and use the serial port. However it omits a microSD-card which you will need to store Linux / other RISC-V software that you want to run. The Arty comes without any cables and thus requires that you supply an OTG USB cable. As mentioned above there seems to be no microSD-card option at all.

RISC-V on an FPGA, pt. 5

I’ve learned a few things about this process. These are just my random thoughts in no particular order:

• You need the heavily proprietary Xilinx Vivado to compile the Verilog source code to a bitstream. However you can use free tools to write the bitstream to the FPGA (eg. xc3sprog). Edit: There is another option: You can write the bitstream onto the SD-card and switch the jumper JP1 to SD-card. On boot the FPGA will load the bitstream from the SD-card. This is actually much nicer because it means you don’t need to manually reprogram the FPGA every time you power it up.
• The core lowRISC CPU is free, but that’s not everything that goes into the bitstream. Also in the bitstream are some peripherals, such as the UART, and those are Xilinx proprietary IP. So the bitstream isn’t fully free and more importantly isn’t redistributable. (Note, according to this talk there is a plan to fix this).
• This is a nice summary of the v0.2 architecture, esp. page 4

RISC-V on an FPGA, pt. 4

It boots!

lowRISC boot program
=====================================
Load boot into memory
Load 11660 bytes to memory.
Read boot and load elf to DDR memory
Boot the loaded program...
[    0.000000] Linux version 3.14.41-g9a25e8d (rjones@moo.home.annexia.org) (gcc version 5.2.0 (GCC) ) #1 Mon Jul 25 19:07:50 BST 2016
[    0.000000] Available physical memory: 126MB
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x00200000-0x07ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00200000-0x07ffffff]
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 31815
[    0.000000] Kernel command line: root=/dev/htifblk0
[    0.000000] PID hash table entries: 512 (order: 0, 4096 bytes)
[    0.000000] Dentry cache hash table entries: 16384 (order: 5, 131072 bytes)
[    0.000000] Inode-cache hash table entries: 8192 (order: 4, 65536 bytes)
[    0.000000] Sorting __ex_table...
[    0.000000] Memory: 124488K/129024K available (1725K kernel code, 120K rwdata, 356K rodata, 68K init, 211K bss, 4536K reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS:2
[    0.150000] Calibrating delay using timer specific routine.. 20.01 BogoMIPS (lpj=100097)
[    0.150000] pid_max: default: 32768 minimum: 301
[    0.150000] Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
[    0.150000] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes)
[    0.150000] devtmpfs: initialized
[    0.150000] NET: Registered protocol family 16
[    0.150000] bio: create slab  at 0
[    0.150000] Switched to clocksource riscv_clocksource
[    0.150000] NET: Registered protocol family 2
[    0.150000] TCP established hash table entries: 1024 (order: 1, 8192 bytes)
[    0.150000] TCP bind hash table entries: 1024 (order: 1, 8192 bytes)
[    0.150000] TCP: Hash tables configured (established 1024 bind 1024)
[    0.150000] TCP: reno registered
[    0.150000] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    0.150000] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    0.150000] NET: Registered protocol family 1
[    0.150000] futex hash table entries: 256 (order: 0, 6144 bytes)
[    0.150000] io scheduler noop registered
[    0.150000] io scheduler cfq registered (default)
[    0.180000] htifcon htif0: detected console
[    0.190000] console [htifcon0] enabled
[    0.190000] htifblk htif1: detected disk
[    0.190000] htifblk htif1: added htifblk0
[    0.190000] TCP: cubic registered
[    0.190000] VFS: Mounted root (ext2 filesystem) readonly on device 254:0.
[    0.190000] devtmpfs: mounted
[    0.190000] Freeing unused kernel memory: 68K (ffffffff80000000 - ffffffff80011000)
# uname -a
Linux ucbvax 3.14.41-g9a25e8d #1 Mon Jul 25 19:07:50 BST 2016 riscv GNU/Linux