I think I was thinking more significant changes than your were. I'd go so far as to re-write it as 4 colossi (not counting the finale.) I'd be sacrilegious enough to combine elements of them along the feature-sets I grouped them into. For instance, the first of the four main fights could introduce the puzzle solving against a biped. Use elements of the sword-carrier mostly, probably. The smaller four-legged beast colossi could group well; it could chase Wander around some ruins, and end up fighting in a dilapidated arena. For the flying one, I'd have it start as fighting in a cave, going in the sand there, before bursting out of the cave and flying into the air along the beach. Then the final fight for Wander has to be against the final Colossus that you fight AA Wander; you have to hit that story best with Agro and get emotional resolution for it, which perfectly sets up the finale.

Basically for me the fights are an opportunity to showcase visual spectacle in both colossi and the world as he combats them, in addition to overcoming the challenge both in plot and the eyes of the viewer. I imagine the 4 fights being probably 15m or less each. (The first being longer, to lay out the rules, then shorter and shorter.) That's less than an hour of hard action.

That leaves you an hour for the other three significant parts of the film: a) serene beautiful shots of Wander and Agro in this world to establish their bond: hunting lizards, camping, climbing trees, showing Wander getting less healthy, and showing the two of them bonding. b) Intercut that with Lord Emon and his men traveling and camping with Emon explaining their mission to the men. Add in a little foreshadowing of how they might have to face "the colossus" when they get there. c) Then the finale and resolution.

Lowery

Dude killed it with The Green Knight. It wasn't even the scene with the giants that made me think of him, but the pacing and quiet solemnity that made parts of it feel like a storybook. That whole part with him coming up to the battlefield was great. The stop at the other castle felt like a whole other movie, yet still felt perfect. It was great. He could make a compelling story about a man and his horse. And I think that's the part everyone assumes will be "the challenge" of the film, and makes me doubtful with almost any director.

I felt the same about the serenity and visual elements of The Assassination of Jesse James by the Coward Robert Ford, if you haven't seen it. The cast may not have agreed gracefully (Brad Pitt, Casey Affleck,) but the movie they made is a masterpiece as far as show deliberate pacing with moderate amounts of silence go. Dominik was great for this. Add in Roger Deakins and it's worth a watch and a rewatch, imo.