I think it would be harmful if defederation became commonplace and fragment the fediverse. On the other hand, I don't see any issue with there being "niche" instances which have a specific demographic of users they want to serve, and I don't see any harm in them defederating with other niche instances. Lemmygrad and exploding heads seem like a good example, I've heard they've defederated each other. I don't think that affects the overall fediverse too much because the users of those instances would probably block or brigade the other instance anyway. There wouldn't be any positive interaction between them, so nothing of value has been lost.

Basically, I think it would be ideal if it's only the "niche" instances that use defederation as a moderation tool, and most instances only defederate for spam or illegal content.