So, I think something is rather wrong with my board or the flash scrips. Im using the latest from here: https://github.com/efabless/caravel_board/tree/main/firmware