Far from making them optional, I think making them expire after a certain amount of turns would help, it would also eliminate the issue where all of them cluster in one spot and make the mechanic a bit more dynamic.
I play insane only and you pretty much never want to fight parties on insane+, and each time I come around to daikara or SWL I have to do this waiting game hoping that the parties will disperse or let me through. This might be worth checking since they swarm those locations in particular.
If this is already a thing that I'm unaware of then maybe it should be way lower
