How I bricked then recovered my reMarkable 2
Published: 27 Sep 2021 11:16

Let me start by saying that the reMarkable 2 is a great device. I’ve wanted something like this for a long time: a relatively open device that runs Linux with an eInk screen and the ability to write on that screen with a pen. The reMarkable 2 delivers: I can ssh into it, and there’s a whole bunch of people writing code for it. I find I’m using it a lot.

Unfortunately, the fact that I can ssh into it leads to the first part of this post.

How I bricked my reMarkable 2

(note: I’ll use the abbreviation rm2 from now on).

I applied to work at Oxide Computer Company earlier this year and in the first part of that interview I was given access to their RFD repo. I grabbed the PDF renders, concatenated them together as one giant PDF, and loaded that onto my rm2.

After reading and making notes on it for a week, I wanted to extract those annotations onto my computer and organize them. My goal was to try out the remarks project: it extracts your annotations and turns that into a modified version of the original PDF.

I could have moved files from the rm2 to my computer with scp (which is installed on the rm2 by default) but I wanted to back up everything on the device using rsync. Fortunately there’s another project called toltec that provides a “community-maintained repository of free software for the reMarkable tablet”. I could simply opkg install rsync if I wanted to install the installer, but that would require connecting my rm2 to WiFi which I want to avoid.

Could I extract rsync from the project and install it myself?

I cloned the toltec repo and had a look. After digging into scripts/toltec/builder.py, it looks like the toltec tools extract from a toolchain repo:

# Prefix for all Toltec Docker images
IMAGE_PREFIX = "ghcr.io/toltec-dev/"

# Toltec Docker image used for generic tasks
DEFAULT_IMAGE = "toolchain:v1.3.1"

So I tried to do this using docker myself. I didn’t want to install opkg and figured that what I was doing was functionally equivalent to

going through the toltec install process
opkg install rsync
figuring out how to uninstall what a step 1 installed.

The project uses the “wget | bash” method of installing software (after verifying checksums), but I’m not a fan of that because of the auditing required, and I didn’t see an easy way from that of uninstalling what it installed without tracing everything the script does.

The plan was to run a container, grabbing the necessary binaries from the top-most layer:

$ docker run --rm -it ghcr.io/toltec-dev/toolchain:v1.3.1
# opkg update
# opkg install rsync

Docker uses layers to construct its file system. By examining the top layer, I can find what files are changed in from this container and the base image it was started on:

$ rsync -avAX \
    "$(docker inspect 404dcbc40ce8 | jq -r .[0].GraphDriver.Data.UpperDir)"/ \
    diff/
$ tree diff/
diff/
├── opt
│   └── x-tools
│       └── arm-remarkable-linux-gnueabihf
│           └── arm-remarkable-linux-gnueabihf
│               └── sysroot
│                   ├── opt
│                   │   ├── bin
│                   │   │   └── rsync
│                   │   ├── etc
│                   │   │   ├── nsswitch.conf
│                   │   │   └── xattr.conf
│                   │   └── lib
│                   │       ├── ld-2.27.so
│                   │       ├── ld-linux.so.3 -> ld-2.27.so
│                   │       ├── libacl.so -> libacl.so.1.1.2301
│                   │       ├── libacl.so.1 -> libacl.so.1.1.2301
       <snip>
│                   │       ├── libssp.so.0.0.0
│                   │       ├── libutil-2.27.so
│                   │       ├── libutil.so.1 -> libutil-2.27.so
│                   │       ├── libz.so -> libz.so.1
│                   │       ├── libz.so.1 -> libz.so.1.2.11
│                   │       ├── libz.so.1.2.11
│                   │       ├── libzstd.so -> libzstd.so.1
│                   │       ├── libzstd.so.1 -> libzstd.so.1.4.9
│                   │       └── libzstd.so.1.4.9
       <snip>

Even though I didn’t specify the arm architecture when running the container, these files seem to be meant for the reMarkable:

$ file lib/libgcc_s.so.1
./lib/libgcc_s.so.1: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV),
dynamically linked, stripped

reMarkable: ~/ file /lib/libgcc_s.so.1
/lib/libgcc_s.so.1: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV),
dynamically linked, BuildID[sha1]=cb70486621a3ffe493c562640c6c5963701ffe59,
stripped

Cool! I could simply move these over with scp and bypass the install process. I wrote a small bash script to move each file found in the diff folder over to the rm2’s root:

$ ./push.sh 2>&1 | tee push.out
bin
./sysroot/opt/bin/rsync
etc
./sysroot/opt/etc/nsswitch.conf
./sysroot/opt/etc/xattr.conf
lib
./sysroot/opt/lib/ld-2.27.so
scp: /lib//ld-2.27.so: Text file busy
./sysroot/opt/lib/ld-linux.so.3
./sysroot/opt/lib/libacl.so
./sysroot/opt/lib/libacl.so.1
./sysroot/opt/lib/libacl.so.1.1.2301
./sysroot/opt/lib/libanl-2.27.so
./sysroot/opt/lib/libanl.so.1
./sysroot/opt/lib/libattr.so
./sysroot/opt/lib/libattr.so.1
./sysroot/opt/lib/libattr.so.1.1.2501
./sysroot/opt/lib/libc-2.27.so
client_loop: send disconnect: Broken pipe
lost connection
./sysroot/opt/lib/libcidn-2.27.so
ssh: connect to host 10.11.99.1 port 22: Connection refused
lost connection
./sysroot/opt/lib/libcidn.so.1
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C
- exit 130

Shit.

I never thought I could cause a system to reboot by copying over /lib files, but now I know you can.

After I thought about it, I realized I had made the mistake of copying the files over to / instead of /opt. Now my rm2 refused to boot. It would flash the screen off and on, show “reMarkable 2”, sit there for a bit, then repeat.

How I recovered my reMarkable 2

Luckily all was not lost. The rm2 has pogo pins on the side that one can plug a USB keyboard into, but also is part of a recovery process detailed at ddvk/remarkable2-recovery:

connect the USB-C breakout board
pull down B8 (connect B8 over a 10K resistor to B12)
connect the pogo pins to the tablet and to the host usb
if the device does not start on its own, press the Power Button
you should see a new device like USB HID v1.10 Device [Freescale SemiConductor Inc SE Blank ULT1]
remove the pulldown resistor (disconnect B8)
run sudo ./imx_usb or add the udev rules to use it without sudo
should see USB Mass Storage device detected
should see a new mountable block device

jcs recently even tweeted about a very similar scenario:

I was messing around with systemd files on my Remarkable tablet and put it into a reboot loop.

So I ordered the parts and tried it myself:

The USB-C breakout board with the pull down resistor (not actually) connected on the bread board

The micro USB breakout with soldered on headers. Note the medicore soldering :)

Note: B8 is SBU2, and is only on one side of the USB-C connector:

If you see the following when booting the rm2:

[277649.783822] usb 2-1-port5: Cannot enable. Maybe the USB cable is bad?
[277652.451828] usb 2-1-port5: Cannot enable. Maybe the USB cable is bad?
[277653.323858] usb 2-1-port5: Cannot enable. Maybe the USB cable is bad?
[277653.323956] usb 2-1-port5: attempt power cycle
[277654.515990] usb 2-1-port5: Cannot enable. Maybe the USB cable is bad?
[277655.391970] usb 2-1-port5: Cannot enable. Maybe the USB cable is bad?
[277655.392094] usb 2-1-port5: unable to enumerate USB device

You may just need to rotate the USB-C cable:

[ 1418.674499] usb 2-1.6: new high-speed USB device number 35 using ehci-pci
[ 1418.784014] usb 2-1.6: New USB device found, idVendor=15a2, idProduct=0076, bcdDevice= 0.01
[ 1418.784018] usb 2-1.6: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 1418.784020] usb 2-1.6: Product: SE Blank ULT1
[ 1418.784022] usb 2-1.6: Manufacturer: Freescale SemiConductor Inc
[ 1418.785658] hid-generic 0003:15A2:0076.0005: hiddev1,hidraw4: USB HID v1.10 Device [Freescale SemiConductor Inc  SE Blank ULT1] on usb-0000:00:1d.0-1.6/input0

Or you need to make sure the USB micro breakout header is firmly pressed against the rm2’s pogo pins.

Another note: before I thought to rotate the cable, I bought a USB-C 3.1 gen 2 cable, thinking that the other USB-C connector cable I had didn’t have the secondary bus. I don’t know if this is required for this recovery to work or not but I would be remiss if I didn’t mention it.

I cloned the boundarydevices/imx_usb_loader repo and built ./imx_usb myself, and copied the following files from ddvk/remarkable2-recovery:

imx_usb.conf
mx7_usb_work.conf
u-boot-ums.imx

Once the Freescale SemiConductor Inc SE Blank ULT1 was attached, I followed the guide: remove the pull down, run sudo ./imx_usb:

$ sudo ./imx_usb 
config file <.//imx_usb.conf>
vid=0x15a2 pid=0x0076 file_name=mx7_usb_work.conf
config file <.//mx7_usb_work.conf>
parse .//mx7_usb_work.conf
Trying to open device vid=0x15a2 pid=0x0076
Interface 0 claimed
HAB security state: development mode (0x56787856)
== work item
filename u-boot-ums.imx
load_size 0 bytes
load_addr 0x00000000
dcd 0
clear_dcd 0
plug 0
jump_mode 3
jump_addr 0x00000000
== end work item
header_max=10400

loading binary file(u-boot-ums.imx) to 877effd4, skip=0, fsize=9f02c type=aa

<<<651308, 651308 bytes>>>
succeeded (security 0x56787856, status 0x88888888)
jumping to 0x877f08e4

and after a few moments it reattaches as a USB mass storage device with 4 partitions:

[65935.111114] usb 1-1.6: new high-speed USB device number 66 using ehci-pci
[65935.220804] usb 1-1.6: New USB device found, idVendor=0525, idProduct=a4a5, bcdDevice= 2.21
[65935.220807] usb 1-1.6: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[65935.220809] usb 1-1.6: Product: USB download gadget
[65935.220811] usb 1-1.6: Manufacturer: FSL
[65935.220812] usb 1-1.6: SerialNumber: 00000224ec37afea
[65935.222125] usb-storage 1-1.6:1.0: USB Mass Storage device detected
[65935.222439] usb-storage 1-1.6:1.0: Quirks match for vid 0525 pid a4a5: 10000
[65935.222534] scsi host8: usb-storage 1-1.6:1.0
[65936.252071] scsi 8:0:0:0: Direct-Access     Linux    UMS disk 0       ffff PQ: 0 ANSI: 2
[65936.252458] sd 8:0:0:0: Attached scsi generic sg2 type 0
[65936.253225] sd 8:0:0:0: [sdc] 14942208 512-byte logical blocks: (7.65 GB/7.13 GiB)
[65936.253593] sd 8:0:0:0: [sdc] Write Protect is off
[65936.253595] sd 8:0:0:0: [sdc] Mode Sense: 0f 00 00 00
[65936.253987] sd 8:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[65936.274983]  sdc: sdc1 sdc2 sdc3 sdc4
[65936.278496] sd 8:0:0:0: [sdc] Attached SCSI removable disk

Now that the rm2 was showing up as a USB mass storage device, I first made an image of the whole volume:

$ sudo dd if=/dev/sdc of=rm2.raw bs=4096 status=progress
<it failed here and I didn't record the shell output>

$ sudo dd if=/dev/sdc of=rm22.raw bs=4096 status=progress skip=898076
3969245184 bytes (4.0 GB, 3.7 GiB) copied, 326 s, 12.2 MB/s
969700+0 records in
969700+0 records out
3971891200 bytes (4.0 GB, 3.7 GiB) copied, 326.592 s, 12.2 MB/s

$ cat rm2.raw rm22.raw > rm.raw

Make sure to backup rm.raw, unless you want to repeat the process.

This took two tries because it was difficult for me to hold the USB micro breakout headers against the pogo pins, and any slip caused a disconnect and the volume would disappear. For this reason, whenever Ubuntu auto-mounted the partitions, I would unmount them right away. I didn’t want any file system corruption on top of the bad libraries.

I mounted the image, and verified that the whole volume imaged ok by mounting and unmounting each file system, and performing some basic integrity checks:

# losetup --show --find rm.raw
/dev/loop22
# partprobe /dev/loop22
# ls /dev/loop22*
/dev/loop22  /dev/loop22p1  /dev/loop22p2  /dev/loop22p3  /dev/loop22p4

Partition layout:

boot stuff
root
another root
the /root/ homedir, plus some extra files like log.txt

If your goal was just to recover your data, it would be enough to copy partition 4.

I looked at partition 1’s uboot files and found that partition 2 looked like the “active” one:

$ sudo mount /dev/loop22p1 /mnt/
$ strings /mnt/uboot.env | grep active_partition
#active_partition=2
<snip>
$ sudo umount /mnt/

Next I mounted partition 2 and 3, and copied them to respective directories:

$ sudo mount /dev/loop22p2 /mnt/
$ sudo rsync -avAX --delete /mnt/ bad_remarkable_p2/
$ sudo umount /mnt

$ sudo mount /dev/loop22p3 /mnt/
$ sudo rsync -avAX --delete /mnt/ bad_remarkable_p3/
$ sudo umount /mnt

I figured that because partition 3 looked like a copy, and because it was not active, it would contain good libraries. If I copied those over to partition 2 it should restore partition 2 to a good state and the rm2 would boot successfully again.

To test this, I booted the partition 2 root files in an arm32v7/debian:stretch-slim container:

$ cat docker.sh
#!/bin/bash
vols=""
for fi in bad_remarkable_p2/*;
do
    if [[ "$(basename ${fi})" == "proc" ]] || [[ "$(basename ${fi})" == "dev" ]] || [[ "$(basename ${fi})" == "tmp" ]];
    then
        continue
    fi
    vols="${vols} -v ${PWD}/${fi}:/$(basename ${fi})"
done

set -x
exec docker run \
    --rm -ti \
    ${vols} \
    -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static \
    --tmpfs /tmp -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
    --entrypoint '/sbin/init' \
    arm32v7/debian:stretch-slim

$ ./docker.sh
+ exec docker run --rm -ti
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/bin:/bin
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/boot:/boot
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/etc:/etc
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/home:/home
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/lib:/lib
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/lost+found:/lost+found
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/media:/media
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/mnt:/mnt
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/postinst:/postinst
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/run:/run
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/sbin:/sbin
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/sys:/sys
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/uboot-postinst:/uboot-postinst
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/usr:/usr
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/var:/var
    -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static
    --tmpfs /tmp
    -v /sys/fs/cgroup:/sys/fs/cgroup:ro
    --entrypoint /sbin/init
    arm32v7/debian:stretch-slim
WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested
/sbin/init: error while loading shared libraries: /lib/libacl.so.1: internal error
- exit 127

Awesome! libacl.so.1 was one of the files transfered by push.sh earlier, so this makes sense. What about partition 3? I switched the directory (and entrypoint) in docker.sh:

$ ./docker.sh
+ exec docker run --rm -ti
    -v /home/jwm/src/myremarkable/bad_remarkable_p3/bin:/bin
    -v /home/jwm/src/myremarkable/bad_remarkable_p3/boot:/boot
    -v /home/jwm/src/myremarkable/bad_remarkable_p3/etc:/etc
    -v /home/jwm/src/myremarkable/bad_remarkable_p3/home:/home
    -v /home/jwm/src/myremarkable/bad_remarkable_p3/lib:/lib
    -v /home/jwm/src/myremarkable/bad_remarkable_p3/lost+found:/lost+found
    -v /home/jwm/src/myremarkable/bad_remarkable_p3/media:/media
    -v /home/jwm/src/myremarkable/bad_remarkable_p3/mnt:/mnt
    -v /home/jwm/src/myremarkable/bad_remarkable_p3/postinst:/postinst
    -v /home/jwm/src/myremarkable/bad_remarkable_p3/run:/run
    -v /home/jwm/src/myremarkable/bad_remarkable_p3/sbin:/sbin
    -v /home/jwm/src/myremarkable/bad_remarkable_p3/sys:/sys
    -v /home/jwm/src/myremarkable/bad_remarkable_p3/uboot-postinst:/uboot-postinst
    -v /home/jwm/src/myremarkable/bad_remarkable_p3/usr:/usr
    -v /home/jwm/src/myremarkable/bad_remarkable_p3/var:/var
    -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static
    --tmpfs /tmp
    -v /sys/fs/cgroup:/sys/fs/cgroup:ro
    --entrypoint journalctl
    arm32v7/debian:stretch-slim
WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested
No journal files were found.
-- No entries --

Awesome! This confirms the hypothesis that partition 2 was the active one (partition 3 had no entries in the journal), and that the library transfer caused the rm2 not to be able to boot.

I tried restoring partition 3’s library files to partition 2:

$ sudo rsync -avAX --delete bad_remarkable_p3/lib/ bad_remarkable_p2/lib/
sending incremental file list
deleting libattr.so.1.1.2501
deleting libattr.so
deleting libacl.so.1.1.2301
deleting libacl.so
deleting ld-linux.so.3
./
libacl.so.1.1.0
libanl-2.27.so
libattr.so.1.1.0
libc-2.27.so
deleting systemd/system/remarkable-qa.service
depmod.d/
firmware/
firmware/brcm/
modprobe.d/
modules/
modules/4.14.78/
modules/4.14.78/modules.alias
modules/4.14.78/modules.alias.bin
modules/4.14.78/modules.builtin.bin
modules/4.14.78/modules.dep
modules/4.14.78/modules.dep.bin
modules/4.14.78/modules.devname
modules/4.14.78/modules.softdep
modules/4.14.78/modules.symbols
modules/4.14.78/modules.symbols.bin
modules/4.14.78/kernel/
modules/4.14.78/kernel/crypto/
modules/4.14.78/kernel/drivers/
modules/4.14.78/kernel/drivers/crypto/
modules/4.14.78/kernel/drivers/crypto/virtio/
modules/4.14.78/kernel/drivers/dma/
modules/4.14.78/kernel/drivers/i2c/
modules/4.14.78/kernel/drivers/i2c/algos/
modules/4.14.78/kernel/drivers/input/
modules/4.14.78/kernel/drivers/input/mouse/
modules/4.14.78/kernel/drivers/input/serio/
modules/4.14.78/kernel/drivers/net/
modules/4.14.78/kernel/drivers/net/usb/
modules/4.14.78/kernel/drivers/net/wireless/
modules/4.14.78/kernel/drivers/net/wireless/ath/
modules/4.14.78/kernel/drivers/net/wireless/ath/ath6kl/
modules/4.14.78/kernel/drivers/net/wireless/broadcom/
modules/4.14.78/kernel/drivers/net/wireless/broadcom/brcm80211/
modules/4.14.78/kernel/drivers/net/wireless/broadcom/brcm80211/brcmfmac/
modules/4.14.78/kernel/drivers/net/wireless/broadcom/brcm80211/brcmutil/
modules/4.14.78/kernel/drivers/rpmsg/
modules/4.14.78/kernel/drivers/usb/
modules/4.14.78/kernel/drivers/usb/class/
modules/4.14.78/kernel/drivers/usb/misc/
modules/4.14.78/kernel/drivers/usb/serial/
modules/4.14.78/kernel/fs/
modules/4.14.78/kernel/fs/fat/
modules/4.14.78/kernel/fs/isofs/
modules/4.14.78/kernel/fs/nls/
modules/4.14.78/kernel/fs/udf/
modules/4.14.78/kernel/lib/
systemd/
systemd/network/
systemd/system-generators/
systemd/system-preset/
systemd/system/
systemd/system/dbus.target.wants/
systemd/system/graphical.target.wants/
systemd/system/local-fs.target.wants/
systemd/system/multi-user.target.wants/
systemd/system/poweroff.target.wants/
systemd/system/reboot.target.wants/
systemd/system/rescue.target.wants/
systemd/system/sockets.target.wants/
systemd/system/sysinit.target.wants/
systemd/system/timers.target.wants/
udev/
udev/hwdb.d/
udev/rules.d/

sent 1,561,964 bytes  received 893 bytes  3,125,714.00 bytes/sec
total size is 19,007,362  speedup is 12.16

I tried running journalctl --list-boots:

$ ./docker.sh
+ exec docker run --rm 
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/bin:/bin 
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/boot:/boot 
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/etc:/etc 
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/home:/home 
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/lib:/lib 
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/lost+found:/lost+found 
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/media:/media 
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/mnt:/mnt 
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/postinst:/postinst 
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/run:/run 
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/sbin:/sbin 
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/sys:/sys 
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/uboot-postinst:/uboot-postinst 
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/usr:/usr 
    -v /home/jwm/src/myremarkable/bad_remarkable_p2/var:/var 
    -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static
    --tmpfs /tmp 
    -v /sys/fs/cgroup:/sys/fs/cgroup:ro
    --entrypoint journalctl
    arm32v7/debian:stretch-slim
    --list-boots
WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested
-1 b606cefd9a91430399e96bee2f90a790 Fri 2020-12-18 11:54:09 UTC—Thu 2020-12-24 05:03:39 UTC
 0 f5920cb0a50b492eadcf90f76366cb46 Thu 2020-12-24 14:25:10 UTC—Sun 2021-01-17 14:47:07 UTC

Nice! I also tried /sbin/init, and it looked like it started successfully (bypassing the ‘internal error’ seen before) then failed to do basically everything because it was running in a container.

Next, I wrote a script to restore to the actual rm2’s second partition after actually mounting it with the recovery method:

$ cat restore.sh
#!/bin/bash
set -x
sudo umount /dev/sdc*
sudo mount /dev/sdc2 /mnt
sudo rsync -avAX bad_remarkable_p2/lib/*.so* /mnt/lib/
sudo umount /mnt

And it worked!

Note: I took it apart trying to find serial headers but there aren't any obvious ones on the top of the board. That's why the left side isn't on :)

Final thoughts

Vendors: please just install rsync on things, it’s 2021, there’s space for it. Also, thank you for providing recovery methods for people like me.

Use /opt people!

Discuss on Hacker News

Edit 1: I clarified my motivation for extracting rsync from the Docker layer over just installing opkg.