This is the type of situation that agents like to avoid, which is why I explain to my buyers and sellers that my services don't start at finding the house or the buyer. Broker services are for marketing, putting the deal together, getting the word out, and helping resolve any issues. What happens if your buyer doesn't qualify or finds a problem with the house before or after they close? Just because the buyer looks like they have enough money to close, doesn't mean that they actually can. Normally, a broker would help you resolve these types of issues, but who are you going to turn when you've decided to go out on your own?
When a seller or buyer tries to cut their broker out of the deal, it shows they don't value the job that agent is doing. The broker already put time, effort, and money into selling your house and now you're ready to push them out because the buyer wants to work alone with you. You should talk to the broker honestly to 1) see what your listing agreement says about this situation and 2) see if they can perhaps be a little more flexible with the commission structure.