Fortinet Firmware Fiasco - Part 1 - FAP-221-B

Preamble

Fortinet does not like you, they like your business. Fortinet says “Buy our access points, they integrate nicely with our firewall devices, it’s a single pane of glass view!”. Fortinet omits the statement “Our access points are literally useless without one of our firewall devices OR a cloud subscription service! Oh, and don’t think about registering an old decommissioned device to a new account, we don’t want to know!”. I found this out after inheriting some FortiAP devices and struggled to even find firmware images for them. Though, truthfully, I know Fortinet aren’t the only ones who hold binaries hostage like this - I just happen to have a bunch of their devices, so they’re who I’m calling out.

So what’s a budding enthusiast to do? Acquire a firmware image from a “legitimate” source, reverse engineer the format, and port OpenWrt to the device of course! This series will focus on porting efforts for as many different Fortinet devices as I can get my hands on.

The Board

Insert an image of the board on both sides, describe anything I can identify.

The device is based on an Atheros AR9344 Rev 2 SOC (datasheet), which contains a MIPS 74KC processor running at 560MHz. This particular board implements an AR9340 for 5GHz and an AR9380 for 2.4GHz. Providing NOR flash to the device to boot from is a MX25L12805D. The board can be powered via 12V DC through a barrel jack, or 48V power-over-ethernet.

The Bootloader

The installed u-boot is heavily locked down. Upon booting the board, while connected to UART, you are prompted to interrupt boot. Here you’re greeted with a menu with exactly three options:

1. [G]: Get OS image from TFTP server.

2. [Q]: Quit menu and continue to boot with default OS.

3. [H]: Display this list of options.

I wish I was joking. I also wish I’d happened upon the work of Michael Pratt before embarking on this journey - for two reasons. First, they figured out a sequence of commands that would escape this hellish boot menu, and drop down to a real u-boot menu. Second, because they were working on the device at the same time as me - I kid you not, they submitted their pull request less than 24 hours before I started authoring mine - and, honestly, they did a much better job than I did. Upon reflection over the last few days working on this project, it’s apparent to me that much of this journey was valuable mostly for the purposes of learning about the OpenWrt code base, rather than any real contribution to the OpenWrt project. That’s not to take away from the experience, though, I have other devices which I’m looking to port to - this can only be a benefit to me in that respect.

As it turns out, the restrictive menu can be escaped by entering “k” into the menu. The user is prompted for a password, which is simply “1”. To go to the effort of securing a device to this degree only to skimp on the passwords used to secure it, is nothing but comical in my opinion.

There is something interesting about the u-boot partition on this device, which I don’t think has yet been figured out outside of Fortinet. It ends with a pair of SSL certificates and an RSA private key. I’m yet to figure out exactly what these are for. The OEM firmware makes mention of certificates being “insecure” upon boot, and switches to using a self signed certificate - perhaps some SSL-based interface is served on the device that would otherwise use the certificates burned into the device? I know SSH is open by default. Perhaps it goes deeper and some element of u-boot is encrypted? Though, to leave private keys in plain text alongside would be particularly weak security. For now, it remains a mystery. Perhaps I’ll figure it out after examining a few more Fortinet devices.

OEM firmware

The layout of the flash as used by the OEM firmware image is as follows:

Partition start	Partition size	Label
0x000000	0x40000 (256KiB)	u-boot
0x040000	0x900000 (9216KiB)	rootfs
0x940000	0x1B0000 (1728KiB)	kernel
0xAF0000	0x500000 (5120KiB)	reserved
0xFF0000	0x010000 (64KiB)	caldata

OEM flash layout

This layout might look fairly typical for an average MIPS-based SoC, though the “reserved” partition is an interesting one. Upon further inspection, one can observe an ASCII string “KERN CRASH LOG”. I’m not sure how much Fortinet expected these devices to crash, but 5MiB of storage dedicated to kernel crash logs feels excessive. The first device I dumped had a single crash log from 2016, about 3 years after its manufacture date, which consumed less than 64KiB.

Talk about dumping the firmware through SSH, and how janky it was.

Firmware Image Format

It took some time to find an OEM firmware image, as none of the Fortinet devices I own can be registered to an account that I control. I spent a fair amount of time trying to brute-force figure out the format in which u-boot expected a firmware image. Consider that this was before I had learned the trick to break out of u-boot, I really knew very little about the device.

Once I had found an OEM firmware image, reverse engineering the format was exciting. The binwalk tool was the most valuable tool I could’ve asked for. I noticed two interesting things about the output from binwalk when run on the OEM image.

Insert binwalk output - image.out

First, it’s just a gzip archive. Second, the name reported by the gzip header differs from the filename. Fortinet compresses their firmware images and the device decompresses them at the point of flashing. However, the name reported by the gzip header has significance - it is parsed by both the u-boot environment and restore binary on the OEM firmware. The u-boot checks the first part of the name, to confirm the board name is identical to its own. Actually, that’s all the u-boot checks - if that matches, it just goes ahead and starts flashing the image. The restore binary seems to check a bit more of the information given. How much? I’m not sure, I haven’t tested. In OpenWrt builds, all the values are provided (although obviously bogus), so the image should be accepted regardless. So now I knew I needed to feed Gzip’d data, what format does that data need to be in? Running binwalk on the decompressed data revealed all.

Insert binwalk output - image, decompressed

It can be observed that the image contains a jffs2 filesystem, immediately succeeded by a u-boot uImage. It can also be noted that the offset of the uImage is exactly the offset into the flash memory that the stock kernel is located, with respect to the end of the u-boot partition. To test this theory, I concatenated the rootfs and kernel partitions that I’d previously dumped from the device, named the output the same as the stock image I’d found, and Gzip’d it. Upload to a TFTP server, feed the u-boot the details of my TFTP server and the file name… low and behold, it’s flashing! This was a massive breakthrough in the process, I had a method to soft unbrick the device - which meant I could be more daring and try to compile custom firmware.

Before moving onto some of the OpenWrt specifics, there is an interesting difference between the u-boot environment and the restore binary. u-boot does not care for partition boundaries. If you pass the TFTP flash script an image, it will overwrite flash for as many bytes as there are bytes in the uncompressed image. This was particularly valuable to me, as I had already identified that the reserved partition that succeeds the kernel is just for crash logs, and the kernels being produced by the OpenWrt build system were larger than the mere 1728KiB provisioned by default, the u-boot didn’t care and overwrote the reserved partition. When it came time to boot the kernel, u-boot still didn’t care about partition boundaries - it just treats the flash chip as a contiguous memory space. Which should be unsurprising, really, since that’s what it is.

OpenWrt

I spent a lot of time learning about how firmware is built in OpenWrt, the same as many others, from example. In this instance, the first thing I looked at was the OpenWrt documentation and porting guide. After that, the DTS file for a board with the same chipset. For those who don’t know, a DTS file (or Device Tree file) is a file format designed to describe hardware. It gets compiled to a binary format, where an operating system kernel can parse it and figure out what I/O address the UART is at, for example. It’s not limited to Linux, and I can assure you that it’s much more comprehensive I’ve described. Of course, I’m barely scratching the surface of this stuff myself - the documentation is by far the best place to go to get any real details on the format.

The FAP-221-B is based on an Atheros AR9344 SoC. Many devices supported by OpenWrt run on this chipset. To build a working OpenWrt image I copied a DTS that looked promising (that’s a real scientific metric, honest), and started making changes where I knew they were necessary. Things like the partition layout were easy to figure out. The GPIO pins that each of the LEDs are connected to? Less so. I ended up pasting a script into the UART console to figure out which GPIO was connected to which LED by brute-force, avoiding the GPIOs known to touch the SPI interface by default on the datasheet.

cd /sys/class/gpio
for i in 0 1 2 3 4
do
    echo GPIO $i

    # Export the GPIO via sysfs, and capture its current state
    echo $i > export
    dir=$(cat gpio$i/direction)
    val=$(cat gpio$i/value)

    # Set the GPIO direction and value
    echo out > gpio$i/direction
    echo 1 > gpio$i/value
    echo Set to 1
    sleep 2
    echo 0 > gpio$i/value
    sleep 2

    # Restore the GPIO pin to its previous settings, and unexport it
    echo $val > gpio$i/value
    echo $dir > gpio$i/direction
    echo $i > unexport

    # Echo a new line for no reason other than to make tidier output
    echo

done

Talk about the odd problem where the device stopped booting from squashfs, and how lowering the SPI frequency for the flash chip seemed to resolve it.

It seems as though routers and access points have two sets of MAC addresses. The MAC addresses burned into some of the ethernet/WiFi chips on the board, and the MAC addresses assigned to the device by the manufacturer. To make a device behave similarly to its stock firmware, one should aim to use the MAC addresses assigned by the manufacturer. Who knows if the chip OEMs are reusing addresses in the knowledge that the firmware images ultimately used will override them anyway. I digress. Most manufacturers store the base MAC address that they assign to a device in the flash storage somewhere. Fortinet chose to write the base MAC address in ASCII, but without colons, in the u-boot partition. This is perfectly acceptable, though it might be the first device of its kind to do so - there is a function in the OpenWrt code base to grab a MAC address from a partition, in ASCII, in the form aa:bb:cc:dd:ee:ff. But no such function to read ASCII in the form aabbccddeeff - it is hard coded to read 17 characters. That doesn’t work in this particular instance, the 17th character from the offset of the MAC address is the serial number for the board. So trying to use the function unaltered returns nothing, sanitising the MAC address simply fails as an extra character is found in the stream. While building my own images, I made a copy of the function that reads 12 characters instead - the original passed the resulting text to a sanitiser anyway! Though Michael Pratt had a much more elegant solution than I, they modified the function to take an additional “length” variable, which defaults to 17 characters so as not to break compatibility with existing uses of the function.

I did run into an odd problem during my porting efforts. At some point, the device stopped recognising the squashfs part of the root filesystem. For NOR flash based systems, OpenWrt defaults to logically partitioning the root filesystem into a squashfs image, which contains pre-installed files, and a jffs2 filesystem immediately succeeding it to hold user changes. The two filesystems are merged using overlayfs, and can even be accessed individually at /rom and /overlay. I digress. When I encountered the problem, I sought the help of the wonderful folks at the #openwrt-devel IRC channel. Almost thankfully, it was not a trivial problem - nobody could see anything immediately wrong. A helpful member of the channel suggested lowering the frequency in which the SPI NOR flash chip was being driven as a hail mary - and it worked! Though I have no idea why. I was lowering it from 25MHz to 12.5MHz - yet the chip’s datasheet boasts capabilities of up to 50MHz. Michael Pratt’s port allows the SPI NOR flash chip to run at speeds up to 40MHz - and it’s seemingly fine. It’ll be one to keep an eye on, for sure - perhaps something about the frequencies I chose didn’t resonate properly with the other clock signals? It’s a mystery.

As eluded to throughout the article, Michael Pratt has done a much better job of supporting this board than I. It appears as though the board is produced by a company named Senao, who license the design to other OEMs. Through Michael’s work, however, I learned of the “loader” - a small binary that can be placed at the address that u-boot tries to find a uImage, which in turn looks for a uImage starting from a pre-defined offset, and then executes it. That could be handy later. However, ultimately, my porting efforts for this device shall not be submitted to OpenWrt.

My next efforts look at the FAP-221-C, which is based on a similar Atheros chipset. Until then, folks.