Shane, absolutely not crazy. I did pretty much the same thing 7 years ago. Got tired of sitting in front of a computer all day and realizing at the end of the day that the day had just passed me by. I wanted to be a part of the day and what was going on. I up and quit my job (I don't totally recommend this) and didn't know what I was going to do and just fell into real estate. I started near the end of a similar market to where we are now, got some knowledge under my belt and some sales and was ready when the upswing happened.
I'm glad I started in a slow market as it taught me how to "work" for my leads and sales. So when the market slowed down I just went back to old tactics and knew where to focus.
The profession may be flooded with agents, but some of those started at the peak of the market and haven't figured out how to work in a down market and may not have faced reality yet.
If you're good at what you do, you can make it in any market, it's about being adaptable and seeing the change when it's coming and adjusting accordingly.
I now have my broker's license and decent sphere of influence and am expanding my knowledge and skills in a new direction.
Just set your mind to it and you can do it!
_________________________