One of the bigger reasons has to do with the square cube law - as the size of something increases, surface area increases by a factor of 2 but mass increases by a factor of 3, so little fishes have a surface area-to-mass ratio that is quite a bit higher than a larger fish, and they're more susceptible to abrupt changes in temperature.

Kinda like how an ice cube will melt a lot faster than a big slab of ice, the core temperature of some small fish like a goldfish is gonna change more rapidly than the core temperature of a big fish like a trout so they tend to be a lot more finnicky in regard to significant and instantaneous changes to temperature and stuff. A larger fish might shrug off a significant change because it affects them more slowly, but that might be a totally wild an overwhelming experience for a little fish to go through