My instinctive response is that it's a terrible idea. While having no expectation to mask is great, it seems to me that gathering a group of people who generally struggle to take care of themselves and their environment and who have very low tolerance for certain environmental stimuli and a deep need for other environmental stimuli is a recipe for chaos.
I attend a local autistic adults zoom group every other week, and it's great for support and understanding, but if I had to be in the same room as one of the other members their stims would give me a meltdown. I over-empathise emotionally, an autistic friend has almost no emotional empathy, as a result some of our interactions do not go as intended. Multiply these kinds of issues with having to effectively live with eachother and I just don't see it going well.
This is why I concluded that I can't live in shared housing. Thankfully my social phobia isn't triggered by living with a partner or I'd be fucked as far as affording housing.
I don't have any useful advice or way to help, but you're not a burden and you're not the only one to feel this way. Sorry if that's not exactly comforting.