Hard to respond to your individual situation without knowing a lot more facts but here are some general facts that may help.
Here's what a really smart,
real estate law firm has to say about the practice you're describing.
As a general rule you have the right to shop around and get your mortgage loan anywhere you choose and it's usually illegal to "make" you use a particular firm with any federally regulated loan (and most loans are federally regulated in some way).
RESPA, which stands for "The Real Estate Settlement and Procedures Act" regulates mortgages, some real estate, and most all real estate closing firms.
Same thing goes for title companies (they're the ones that do the paperwork at real estate closings)
You might try asking your broker if he's certain all of his actions are RESPA compliant.
Be interesting if you would come back and tell us his answer.
Best of luck with your home purchase.