I have known people that have been very successful doing RE part time, so I know it can work. You just have to be smart at your prospecting. I am just starting out as a part time agent myself, so I really can't speak from experience yet. No, I don't expect to be a top producer, but I do expect to lay the foundation to go full time in a few years.
Most RE work with clients is in evenings and on weekends. And at first, most of your time will be prospecting anyway. I figure if I can do ok during this down cycle, then by building a decent contact list before I go fulltime should make it that much easier.
I think a lot of "fulltime" agents (maybe not most) don't really just do sales either, they are into other things, whether it is real estate investment, title company, appraisals, property management, etc.