First, I wonder if we are talking about an agent as apposed to the managing broker? Anyway, the first sign of trouble was when he told you
"[you] had to sign a agency agreement and that all brokers worked with an agreement"
him telling you this, the fact you where not given a copy of the document and that you where told that a signed agreement was a formality all point to a
very unethical agent.
I don't know the Canadian system, but I suspect that you can file a grievance with the Province's licencing body. If you where in fact not working with the managing broker of the brokerage, you might want to give that person a call and threaten a compliant against the agent and the broker.
You should definitely contact an attorney.