Can someone explain to me the importance or advantages of having robots or spyders visiting your website?

I know that they are necessarily "good" to have but I don't fully understand the mechanics behind it.

This week already, I have had 12 robots; MSNbots and Googlebots visit my website. My website is fairly new and I would like to understand more about these bots and how I can attract them more.

Thank you for your Help,

Search engines send out spiders that "crawl" the internet's web pages often getting to one web page by following a link found on another webpage. The information on the found web page can then be evaluated using the particular criteria set by the search engine's ranking algorithm. This information together with off-page factors such as link popularity enable the search engine to assign a rank to the web page and that determines where you will appear in the results when someone performs a search.

Having search engine spiders visit your site is good because it means that search engines are aware of your pages and have the information required for your site to appear for keyword searches. You have no chance of appearing in the results of keyword searches if search engines are not aware of your web pages

You don't really have to watch your log files for the coming and going of spiders. Usually once a spider has visited a page once it will keep on returning - no encouragement is usually necessary. In fact, they will often keep looking for a page years after the page no longer exists!

Thank you Doug for your help. I'm getting a better idea now. Just this morning when I checked my log files... I noticed 5 more bots/spiders visited my site overnight. I'm glad that they are aware of me!

I might be wrong, but ranking does not neccessarily equate to position. You'll find that spiders/bots will take interest in you, if someone else is taking interest in you (typically links posted elsewhere).

agreed

In that case, how does one move up in positioning other than pay-per-click? What are the cost-effective ways?

uniqueness, content, and visitors.

Uniqueness - segment your market and post content that speaks their language.

Content - Know what your segment is seeking, and provide it.

Visitation - advertise to your market segment(your webiste address on all materials) that you have what the're looking for.

Around the world daily, millions of search phrases are submitted to Google. Google takes note and says, now that we know what people are looking for, lets send out our spiders to locate quality content so that we can respons with quality content.

For many years, Many tried to trick Google into thinking that their content was higher value than it's face value. Google responded by saying not, we will fight you to maintain our high quality product. Today, anyone that has web skills, can go to Google and view their do's and don'ts.

The content remarks are on the mark. Robots can visit your site a hundred times a day, but if your content isn't what the engine wants to see, you'll never get decent positioning. For a primer and some really good SEO info, check out:

http://www.technologyforrealestate.com/web_site_presence/seo_and_ppc/seo-and-ppc.htm

I find also that if you are adding and changing content on a regular basis then the bots and spiders will visit more often.

Try to make sure that each of your pages is very specifically about one subject so that it is more relevent in a search. eg: cityname and fsbo or cityname and condo info so that when Google, Yahoo, or MSN visit they will see this as a relevent page and it will rank high for that term.

Don't try to have the best condo page just make sure it's the best in your city, remember that you want to find buyers and sellers in your area not buyers and sellers across the country. The search engines recognize this focus if you do so it is much better to try to get traffic from buyers in your area than just buyers anywhere.