Turns out the calibration values stored on the MiniJoyC were all out of whack.
I created programs to show and set the calibration values usin raw i2c commands
Show: https://uiflow2.m5stack.com/?pkey=4b2548481e1644aa8a0ccc88ca8f7859
Set: https://uiflow2.m5stack.com/?pkey=165277eeb4364ab9be0d843ede823559