I have been DMing for a while, but I have recently come across a method of worldbuilding that has transformed how I create campaigns. This method also allows me to make shorter campaigns that have a end-point trajectory, rather than one that is open ended, and helps me to create satisfying stories. Additionally, this helps to reduce the amount of effort that needs to be performed from session-to-session, by frontloading the most important parts of the campaign preparation before the start of the game.
In short, the process has 3 phases
Phase 1
You get the idea for the kind of world the adventure will take place in. The general story themes and concepts you would like to play around with as well. Write a 1 page treatment for what the world is like and what the broad locations in the world appear to be, and send it to your players. 1 page is important, because it forces you to distill down what is unique and interesting about this world.
For example, in my most recent campaign prep I made a treatment of what the factions in the world are, the virtues of the different areas of the world, and a statement of what is common historical knowledge versus what are secrets that are not commonly known. I ended with a statement of what the themes are going to be for this world as well. NOTE: You should not know what the plot of the actual game is at this stage.
Phase 2
After the players have read your brief 1 page treatment of the world, ask them what kinds of characters they want to play. Work with them on their backstories to make sure they bookend with eachother for a matter of convenience, and ask them to provide information about their families and what aspirations and goals their characters have, or what kinds of stories they want for their characters.
This allows players to either lean into, or purposefully away from the elements of your world. As long as they do one or the other, this process has worked, as they can be part of the established culture of the world, or part of the counter-culture. Either way, you have material and character motivation to work with. The only way this does not work, is if they clearly did not actually engage with the world in some fashion during their character creation. If you notice that, try to gently encourage it with additional requests for information from them.
Phase 3
Use all the information your players gave you to worldbuild and plot out your campaign. Did you have a player who wrote a lot? GREAT! The NPCs will become major plot NPCs, the events depicted will become common knowledge, with some hidden truths that player didn’t know. But before you know what your players want to do, you can't know what kind of plot will engage them. This is where the actual plot for your campaign is written.
This makes your players into collaborators for your story, which will make them invested in the goings on. Additionally, if you know exactly who their characters are and what they want, then you simply need to place obstacles within the plot between them and their desires. Or better yet, present them with interesting decisions or bespoke antagonists that challenge their sense of self. This allows you to be INCREDIBLY detailed and plan out plot beats ahead of time because you are essentially building a railroad to exactly where the players want to go.
Railroading is bad because it usually doesn’t honor the choices of the players. This is railroading, but it honors the players’ decisions.
Finally, I should credit where this idea came from, as this is the method that Brennan Lee Mulligan of Dimension 20 fame has described in multiple instances. My latest games have actually been the easiest to prep from session to session, because I have done a lot of the hardest work prior to the start of the campaign.
And as for the game-to-game prep, I have to credit Slyfourish. His 'Lazy DM Guide' series also keeps my 'procrastinating' work down to a minimum as I focus on what is the most important for the game sessions.
As with many of my other posts, I have crossposted this to the /r/dndnext subreddit. Feel free to upvote it there for higher visibility.
TBH, my first campaign struggled largely because I tried too hard to make a sandbox. There was a little intro adventure, set up some lore and conflict, then I told them they could go wherever. They decide to travel to a nearby town thay mysteriously cut contact with the village they saved. Oh, there's a cult and supernatural plague here? Cool, lets dip and wander to the active battlefield to the west, run into an imperial patrol and get drafted into the army. The enemy army is made up of strangely changed soldiers that refuse to stay dead. Spooky. Lets desert and go back to the first village, and investigate the strangely coordinated goblins that had them under threat. >> This all happened over the course of about five sessions. Their ability to run into, and then immediately drop, plot threads was unparalleled. :P
It helps a lot if you can understand the character's motivation ahead of starting the campaign. You can even tell them during character creation "write into your backstory a reason why your character cares what happens to XYZ".
I'm starting my first big Pathfinder campaign this weekend, and I instructed all of the players to write into their backstories:
a connection they have to the large city (which they will learn during the adventure faces an existential threat),
a reason why their character finds themself in the small town where the adventure begins.
This means that I've effectively pre-railroaded them. But rather than railroading their character's decisions or the choices they have in play, I can railroad their character concepts, which in my experience players are far more likely to be okay with. Once the story starts they can do whatever they want, but they will be aware that their character should want to tackle the primary objective of the adventure.