Setup was (on Xubuntu 18.04):
* VM client 1 (Debian9)
* VM client 2 (Debian10)
Transferring files using rsync-over-ssh was slowly crawling around 300-600 KB/s.
[SOLUTION]
The Debian9 VM client had an "Intel PRO/1000" adapter assigned, whereas the Debian10 VM was using "virtio-net" (Paravirtualized Network).
I've set both to virtio, and now we're back to normal speeds of >100MB/s.