@felmue @mlindholm
I recently had the same problem with I2C on an M5STAMP C3U.
The appropriate solution seems to be the @mlindholm one, but its sample code was a bit incomplete
Here is a working example in the context of a DS3231 RTC
A TwoWire structure needs to be inititalised specifically, then referenced by the used I2C library (if any ; RTC library in my case).
No need for pullup resistor (in my case)

#include <Wire.h>

// Declare alternate I2C pins
#define I2C_SDA 5
#define I2C_SCL 4

// Setup the TwoWire instance (static memory !)
TwoWire I2CM5 = TwoWire(0);

void setup ()
{
...
// Initialise the TwoWire instance
I2CM5.begin(I2C_SDA, I2C_SCL, 400000);

// Ask the library to use this setting
// (most I2C libraries accept a TwoWire object as argument of begin)
rtc.begin(&I2CM5);
...