I'll answer myself ;-)
I copied the /lib/udev/rules.d/69-libmtp.rules to /etc/udev/rules.d, added the line
ATTR{idVendor}=="0403", ATTR{idProduct}=="6001", GOTO="libmtp_rules_end"
That prevents the M5 devices to be recognized as MTP devices.
Then they're recognized as Serial ports, and Arduino IDE can access them.

So we don't need the driver, just a change in udev configuration.

Oh cool, thats for working out how to do this. I can continue my plans of adding an RPI to my project bench to programme the M5Stacks and Sticks.
Thanks you.