I had the very same issue as you using original Apple adapter USB-A to USB-C.
Luckily I had USB-C/HDMI adapter, also Apple Original (USB-C Digital AV Multiport Adapter) and then I realized it only works with this adapter.
Tested on all USB-C ports using the USB-C/HDMI adapter and it works fine.
My conclusion: Original Apple USB-A to USB-C does not support this type of connection.
(Please correct me if I’m mistaken)