Found a solution, so this is now closed.
The fix for me was enabling wifi at the start of my code, I knew I needed wifi for the RTC functions but assumed it wouldn't affect positioning of objects, but it did. I added a check loop to make sure wifi is connected before proceeding.
Screenshot 2024-08-17 075257.png