Sometimes it can be useful to have an out of band way to monitor and/or SSH into your Raspberry Pi - for example when the normal internet connection has issues.

Waveshare has an interesting NB-IoT / Cat-M Pi Hat using a SIM7000E module for LTE wireless connectivity. While this doesn't give you amazing data rates, it is enough to perform some monitoring and do manual SSH debugging.

The setup I wanted to go for is a Wireguard tunnel that can use the LTE connection if available. The way this works is via a PPP connection like any other modem. For this, we need to apt install ppp pppconfig wireguard. You can run pppconfig to set up some default config - it doesn't really matter what settings you choose as we will overwrite them again.

PPP Config

First up is /etc/ppp/options. My provider does not require any authentication (if yours does, you can find plenty of PPP config examples on the internet):

ttyAMA0
115200
lock
nocrtscts
modem
passive
novj
nodefaultroute
noipdefault
noauth
hide-password
persist
holdoff 10
maxfail 0
debug

We'll also change /etc/ppp/peers/provider to use these options and a chatscript:

file /etc/ppp/options
connect "/usr/sbin/chat -v -f /etc/chatscripts/provider"

The chatscript goes in /etc/chatscripts/provider:

# Abortstring - responses that indicate errors
ABORT BUSY ABORT 'NO CARRIER' ABORT VOICE ABORT 'NO DIALTONE' ABORT 'NO DIAL TONE' ABORT 'NO ANSWER' ABORT DELAYED

# Send AT so the modem autobauding can detect the baud rate
'' AT

# Set result mode
'OK' 'ATQ0'

# Reset settings to defaults
'OK-AT-OK' 'ATZ'

# Query some modem and network debug info for the log
OK ATI;+CSUB;+CSQ;+COPS?;+CGREG?;&D2

# Set Cat-M mode
OK AT+CMNB=1

# Set LTE only
OK AT+CNMP=38

# Set the APN
OK AT+CGDCONT=1,"IP","internet",,0,0

# Dial
OK ATD*99#

Since my provider only supports Cat-M but not NB-IoT, I'm forcing the modem to use it. You can also remove the corresponding line or change it to NB-IoT (see the SIM7000E datasheet for valid values)

You should now be able to run pon to establish a connection. Log messages appear in /var/log/messages. After the connection is established, you should see a ppp0 interface.

Test your connection:

# ping -I ppp0 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.4.73.208 ppp0: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=297 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=82.9 ms

Transfer Speed

The configuration above is relatively simple and just uses the maximum autobaud speed of 115200 baud. This will likely limit your throughput below what LTE can do. The SIM7000E supports higher baud rates, but you have to manually switch the rate through AT commands.

For example, you could send AT+IPR=921600 to the modem and update /etc/ppp/options accordingly to achieve higher speeds. You can't do this through the chatscript though as it would have to switch speeds in the middle.

Autostart on Boot

You can for example use a custom systemd unit to start the connection on boot.

Wireguard

The wireguard setup is pretty much standard with the addition of an FwMark, so my configuration looks something like this in /etc/wireguard/wg0.conf:

[Interface]
PrivateKey = YOURKEY
Address = 192.168.2.2/24
FwMark = 51821

[Peer]
PublicKey = YOUR_SERVER_PUB_KEY
AllowedIPs = 192.168.2.1/32
Endpoint = YOUR_SERVER_IP:51821
PersistentKeepalive = 600

What the FwMark lets us do is selectively route the encrypted Wireguard packets through ppp0 when it's up (and via the normal gateway otherwise) by using a new kernel routing table and sending packets with the mark there.

PPP has hooks we can use to activate the config when the interface goes up/down.

/etc/ppp/ip-up.d/wireguardroute:

#!/bin/sh

# Reset routing table 10 to empty
ip route flush table 10

# Table 10 should send per default via ppp0
ip route add default table 10 dev ppp0

# And we'll use the fwmark to send wireguard traffic there
ip rule add fwmark 51821 table 10

/etc/ppp/ip-down.d/wireguardroute:

#!/bin/sh
ip rule del fwmark 51821 table 10
ip route flush table 10

Use wg-quick as usual to manage the interface.

Resetting the SIM7000E

If you have issues during testing or want to do some sort of watchdog reset, you can avoid having to unplug the whole thing by using the on/off control pin wired through to the Pi. For example in Python:

#!/usr/bin/env python
import time
import RPi.GPIO as G

G.setmode(G.BCM)
G.setup(4, G.OUT)
G.output(4, True)
time.sleep(2)
G.output(4, False)

From poking around earlier, we can make an educated guess that the graphical output works via frame buffer as there is a device node /dev/fb0. fbset can tell us a bit more about its format:

<root@rigol>fbset

mode "1024x600-0"
        # D: 0.000 MHz, H: 0.000 kHz, V: 0.000 Hz
        geometry 1024 600 1024 600 16
        timings 0 0 0 0 0 0 0
        accel false
        rgba 5/11,6/5,5/0,0/0
endmode

So we have 1024 * 600 pixels formatted as 16bit RGB values with 5, 6 and 5 bits respectively dedicated for the colors.

Let's try dumping the frame buffer (cp /dev/fb0 /tmp/screenshot) and transfer it over to our more beefy Ubuntu VM. There, we can use ffmpeg to convert it to PNG:

$ ffmpeg -vcodec rawvideo -f rawvideo -pix_fmt rgb565 -s 1024x600 \
         -i screenshot -f image2 -vcodec png screenshot.png

Looking at the resulting file, we however just see the RIGOL logo with the loading progress bar at 100%, so there seems to be more to it. The easiest way to figure out how the frame buffer works would be to look at the source - fortunately, Olliver Schinagl managed to get kernel sources from Rigol and put them on GitLab.

The frame buffer driver

Looking at .config we find that the only enabled frame buffer driver is CONFIG_FB_XILINX. This driver can be found in /drivers/video/xilinxfb.c. After the normal header includes, this driver also seems to just straight up include /drivers/video/dpu.c.

An interesting place to start is probably the ioctls this driver supports as there might be some special ones Rigol put in. And we can indeed find some custom commands in xilinx_fb_ioctl:

• DPU_SET_LAYER_RST: Resets the driver
• DPU_SET_LAYER_ID: Changes the layer ID
• DPU_GET_LAYER_ID: Returns the current layer ID
• DPU_SET_TRACE_MASK: ???
• DPU_SET_TRACE_COLOR: ???
• DPU_SET_TRACE_DATA: ???
• DPU_SET_LAYER_ALPHA: ???
• DPU_SET_LAYER_OPEN: ???
• DPU_SET_LAYER_CLOSE: ???
• DPU_SET_WAVE_XY: Set the X and Y coordinate of the wave layer
• DPU_SET_HDMI: ???
• DPU_SCR_PRINT: Instruct the hardware to do a printscreen

The command numbers start with 0x0F000000 for DPU_SET_LAYER_ID and then just increment in the order defined in the enum in /drivers/video/dpu.h (note that the order is slightly different than above / in the ioctl code!).

Layers

So it seems we have a stack of layers. drvDPUInit provides a good idea of what these layers are. The layer numbers are defined via an enum:

enum
{
    DPU_Layer_Back,  // layer 0
    DPU_Layer_Wave,  // layer 1
    DPU_Layer_Eye,   // layer 2
    DPU_Layer_Fore,  // layer 3
    DPU_Layer_Print, // layer 4
    DPU_Layer_Comp,  // layer 5
    DPU_Layer_All,
    DPU_Layer_Logo = DPU_Layer_All
};

DPU_Layer_All does not seem to be a layer itself but is used as the number of layers (of which there are 6).

How do we get those layers? The memory that is mapped in xilinx_fb_mmap is chosen based on drvdata->opAddr which indexes to the layer currently chosen via the DPU_SET_LAYER_ID ioctl. We can thus use said ioctl to select which layer we want to mmap.

Dumping layers

To dump layers, we have to switch the layer via ioctl, mmap the file and then just write out the bytes. We can again use ffmpeg to convert the raw pixel array to a PNG file. You can find my rust implementation for the dumper on Github. It also contains code to directly output PNG, so you can even skip ffmpeg.

The background (0) and foreground (3) layers contain about what we'd expect, namely the backing grid and the UI elements respectively. The wave layer has a different color format and dimension - the code indicates 1000 * 480 in R8G8B8 format. The reality seems to look different though if we look at an excerpt from hexdump:

000e6b90  cc cc cc 00 00 45 45 00  cc cc cc 00 00 45 45 00  |.....EE......EE.|
000e6ba0  00 6a 6a 00 cc cc cc 00  cc cc cc 00 00 53 53 00  |.jj..........SS.|

We know that the transparency color is 0xCCCCCC so we have 4 bytes with the 4th always being zero. Additionally, the trace is yellowish, but the first byte is zero for opaque pixels. So this looks more like a BGR format with 1 byte each + a zero-byte per pixel.

Taking screenshots

From the code, we can also deduct that there is a hardware-assisted print screen function. The implementation can be found in the printSCR function. It seems to set a register to request the operation and then wait until completion is signalled via another register or the timeout expires.

We can trigger this by using command 0x0F00000C and passing a pointer to an integer - this will contain 0 to signal success and 1 for failure.

Putting it all together

With the separate layers, we can do custom stackups. The following image is the back layer with the wave layer on top as two images. Since we have transparency preserved in the PNG files, this works as intended and shows the background grid under the traces:

In the last post, we got NFS set up - that should make it quite easy to experiment with cross-compiling our own code for the scope and run it directly off of an NFS share.

I'm using the same Ubuntu 20.10 VM as always for this. The goal this time is to see if we can write some code in rust and get it running. To follow along, you should have rustup installed. You can find instructions here.

Creating a hello world binary

First, we're going to need the ARM target for Rust as well as the ARM cross compiler:

$ sudo apt install gcc-arm-linux-gnueabihf
$ rustup target install armv7-unknown-linux-musleabihf

Note that the target is using musl (a small C standard library) instead of glibc. The reason for this is mainly that on the scope, we have glibc 2.4 and we'd need a toolchain built against the same version in order to support dynamic linking. It therefore seems easier to use musl and just link everything statically for now.

Let's create a new rust project:

$ cargo new hello
     Created binary (application) `hello` package
$ cd hello

Feel free to use your favorite editor to change the hello world message into whatever you like in src/main.rs. We can tell cargo what linker to use and build a binary for our target:

$ export CARGO_TARGET_ARMV7_UNKNOWN_LINUX_MUSLEABIHF_LINKER=/usr/bin/arm-linux-gnueabihf-gcc
$ cargo build --target=armv7-unknown-linux-musleabihf --release

Unfortunately this results in a 3.6 MB executable for me, which seems a bit excessive. We can squash this in half: In Cargo.toml, add a [profile.release] section with the option lto = true and build again.

A slight detour: Let's make it smaller

1.5 MB still seems excessive, so let's see if we can't make this even smaller. The nightly rust releases seem to have some more features:

• Install nightly: rustup default nightly
• Install the rust source for nightly: rustup component add rust-src --toolchain nightly
• You might have to install the target again for nightly: rustup target install armv7-unknown-linux-musleabihf

Applying some more tricks, we get this Cargo.toml:

cargo-features = ["strip"]

[package]
name = "hello"
version = "0.1.0"
edition = "2018"

[dependencies]

[profile.release]
lto = true
strip = "symbols"
codegen-units = 1
opt-level = "z"

for a binary size of about 260 KB. Still large for just printing "hello world" but much better.

Running it on the actual scope

You will need SSH access to your scope to test it - refer to the fist post in this series on how to get that.

You can either use the NFS share from last time or copy our binary over to the scope:

$ scp target/armv7-unknown-linux-musleabihf/release/hello root@IPOFYOURSCOPE:/tmp/

and login via SSH and run it:

<root@rigol>/tmp/hello
Hello, world!