I don’t want to tell you how to play your game, but I will say that diamond is well worth the effort, even if you don’t want to get it the easy way with villagers. The amount of time you will save using diamond will more than make up for the time spent mining, and make you resent all the time you’ve wasted using stone. Just dig a tunnel down to y -59 and strip mine, you never need to see a mob or get lost in a cave if you don’t want to. A normal level 30 enchant with efficiency 4 and unbreaking 3 will last a very long time, and can be repaired infinitely if you get mending on it.

I would compare it to something like drinking instant coffee all the time and finally tasting a properly brewed, high quality coffee. Or only buying cheap shoes all the time and then investing in a proper pair of very comfortable and well made ones. Or playing video games on a 5 year old hand me down Mac then upgrading to a decent gaming pc.