Early this month Google announced a new user agent for the robots.txt file to direct the Google News bot on what to do. The new user agent, Googlebot-News, is used just the same way you use the Googlebot agent. To make things clear though here are some of the examples given by Google on the use of the new user agent.

Include pages in Google web search, but not in News:
User-agent: Googlebot
Disallow:

User-agent: Googlebot-News
Disallow: /

Include pages in Google News, but not Google web search:
User-agent: Googlebot
Disallow: /

User-agent: Googlebot-News
Disallow:

Block different sets of pages from Google web search and Google News:
User-agent: Googlebot
Disallow: /latest_news

User-agent: Googlebot-News
Disallow: /archives

According to Google “The pages blocked from Google web search and Google News can be controlled independently. This robots.txt file blocks recent news articles (URLs in the /latest_news folder) from Google web search, but allows them to appear on Google News. Conversely, it blocks premium content (URLs in the /archives folder) from Google News, but allows them to appear in Google web search.” Note that you can do this for any specific page.

Stop Google web search and Google News from crawling pages:
User-agent: Googlebot
Disallow: /

In this case since Googlebot is disallowed and there is no specific instruction on what to do with Google News, the news bot will play it safe and simply not crawl the page.

Happy New Year!

Bookmark to:

Got any questions on SEO? Who better to answer them than Matt Cutts?

If you have questions that need answering head on to the Google Moderator Video Questions Page that Matt Cutts created early this month and submit it there.

Note though that Moderator screens questions and won’t accept your question if it is already too similar to another question submitted. If this happens just vote for the question similar to yours and cross your fingers that others will find the question relevant as well making it popular enough for Matt Cutts to take notice. Matt Cutts will also answer questions he finds interesting; whether popular or not and SEO-related or not.

Matt Cutts will be answering questions via video. It would make sense if he lumps together similar questions in one video, though for sure he’ll need to record multiple videos. Don’t know either if he’ll be mentioning the “questioner” but am sure everyone pretty much wants to be name-dropped by Matt Cutts in his video.

So far top ranking questions are:

“Links from relevant and important sites have always been a great way to get traffic & acceptance for a website. How do you rate links from new platforms like Twitter, FB to a website?” ~ Mani, Delhi

“Are stats from Analytics (bounce rate, time on site) a factor in a page’s or domain’s ranking?” ~ A, San Francisco

“Organic Link Building, according to me is one of the most difficult tasks for SEO’s of SME’s. Can you please list 5 effective ways of organic link building other than building great content?” ~ Pulkit Agrawal, Ahmedabad, India

So far no non-SEO related on the top voted questions. Do submit them though because Matt Cutts seems to genuinely like getting a break from talking about purely SEO stuff.

Bookmark to: