Friends, I welcome you again! Now we will analyze what search robots are and talk in detail about the google search robot and how to be friends with them.
First you need to understand what search robots are, they are also called spiders. What kind of work do search engine spiders do?
These are programs that check sites. They look at all the posts and pages on your blog, collect information, which they then transfer to the base of the search engine for which they work.
You do not need to know the entire list of search robots, the most important thing is to know that Google now has two main spiders called "panda" and "penguin". They fight low-quality content and junk links, and you need to know how to fend off their attacks.
The google search robot "panda" was created in order to promote only high-quality material in the search. All sites with low-quality content are demoted in search results.
The first time this spider appeared in 2011. Before its appearance, it was possible to promote any site by publishing a large amount of text in articles and using a huge amount keywords... Taken together, these two techniques did not bring high-quality content to the top of search results, but good sites dropped in the search results.
"Panda" immediately put things in order by checking all the sites and put everyone in their deserved places. Although it fights against low-quality content, it is now possible to promote even small sites with high-quality articles. Although previously such sites were useless to promote, they could not compete with the giants whose a large number of content.
Now we will figure out how you can avoid the "panda" sanctions. You must first understand what she doesn't like. I already wrote above that she is struggling with bad content, but what kind of text is bad for her, let's figure it out so as not to publish such on her website.
The google search robot strives to provide only high-quality materials for applicants in this search engine. If you have articles in which there is little information and they are not attractive in appearance, then urgently rewrite these texts so that the "panda" does not get to you.
High-quality content can be both large and small, but if a spider sees a long article with big amount information means it will benefit the reader more.
Then it should be noted duplication, or in other words plagiarism. If you think that you will rewrite other people's articles on your blog, you can immediately put an end to your site. Copying is severely penalized by the imposition of a filter, and plagiarism is checked very easy, I wrote an article on the topic how to check texts for uniqueness.
The next thing to notice is keyword overload. Whoever thinks that he will write an article from some keys and take first place in the search results is very much mistaken. I have an article on how to check pages for relevance, be sure to read it.
And another thing that can attract a "panda" to you is old articles that are outdated morally and do not bring traffic to the site. They must be updated.
There is also a google search robot "penguin". This spider fights spam and junk links on your website. It also calculates purchased links from other resources. Therefore, in order not to be afraid of this search robot, you should not engage in buying links, but publish high-quality content so that people themselves link to you.
Now let's formulate what needs to be done to make the site look perfect through the eyes of a search robot:
- In order to make quality content, do your research first before writing an article. Then you need to understand that people are really interested in this topic.
- Use specific examples and pictures, this will make the article lively and interesting. Break your text down into small paragraphs to make it easy to read, like if you open a page with jokes in a newspaper, which ones will you read first? Naturally, each person first reads short texts, then longer, and last but not least, long footcloths.
- The panda's favorite quibble is not the relevance of an article that contains outdated information. Stay tuned for updates and change the texts.
- Keep track of the density of keywords, how to determine this density I wrote above, in the service I told you about, you will get the exact required number of keys.
- Do not plagiarize, everyone knows that you cannot steal other people's things or texts - this is the same thing. You will be responsible for the theft by falling under the filter.
- Write texts of at least two thousand words, then such an article will look informative through the eyes of search engine robots.
- Stay on top of your blog topic. If you run a blog on making money on the Internet, then you do not need to print articles about air guns. This can lower the rating of your resource.
- Design articles beautifully, divide them into paragraphs and add pictures to make it pleasant to read and do not want to quickly leave the site.
- When purchasing links, make them to the most interesting and useful articles that people will actually read.
Well, now you know what kind of work search engine robots do and you can be friends with them. And the most important thing is the google search robot and "panda" and "penguin" you have studied in detail.
Guys, we put our soul into the site. Thank you for
that you discover this beauty. Thanks for the inspiration and the goosebumps.
Join us at Facebook and In contact with
Robots, or just bots, are little personal assistants on your gadget. They are programmed with numerous functions and are incredibly useful in the most different areas our life.
- @iVideoBot - this is the easiest way invented by mankind to download YouTube videos. You just need to send the link to the bot, choose the format and size from the options offered. And voila! Audio or video is already on your gadget.
- @utubebot and @ytaudiobot - 2 more ways to download materials from YouTube in one click.
- @SaveVideoBot- this bot can download videos from all other platforms, including even - attention! - Instagram.
- @auddbot- a bot that is analogous to the Shazam application. He guesses the song from the passage. It is enough just to send him by voice message a few seconds of the melody - and you will get its coveted name.
- @ImageSearchBot will find any image. You just need to enter a word or phrase and select the quality.
- @joinstabot winds up likes on Instagram. It works properly, however, it is a little unclear why this is needed, but lovers of vanity should keep in mind that from a sharp cheat of more than 1,000 likes, your account can be blocked indefinitely.
- @topdf_bot- unrealistically necessary and cool bot. It converts various files to PDF format - just send the file.
- @pollbot- with this guy, you can easily conduct a poll or vote in any chat. Moreover, you yourself prescribe the answer options.
- @MyTeleCloudBot is a limitless cloud in Telegram. You can store and categorize absolutely any files. Well, isn't it a miracle?
- @temp_mail_bot- this helper creates mailbox for 10 minutes if you suddenly need to quickly register on any site.
- @voicybot- the perfect bot for the lazy and tired who can no longer type messages. All you have to do is dictate the message out loud, and he will provide you with it in text form.
- @uber_promo_bot periodically sends promotional codes to taxi Uber.
- @storebot - this is a bot of bots. It will help you find an assistant for every taste.
- @Chess_Bot- you can play chess with this bot.
- @saytextbot- This funny bot will convert your text message to an audio file. The male voice has voice acting like in films - you can entertain friends with such messages.
- @strangerbot arranges a chat with a randomly selected user of the same bot. Who knows, what if you meet your destiny or a good friend? Or you just have nothing to do.
- @PandaQuizBot is an entertaining quiz with over 25 thousand questions. Good way while away the time in line.
- @zodiac_bot- if you believe in horoscopes, you should pay attention to this operational bot. Its developers guarantee, if not the veracity of predictions, so stability and daily alerts for sure.
- @PokerBot- a poker bot. You won't make money with him, but the game is pretty gambling. Your rivals are 4 "random" players - users of this channel.
- @delorean_bot - send yourself a message to the future! Or just a reminder.
- @magic_sticker_ball_bot- the bot will help you make a decision. He will answer your questions and doubts with the phrases of the same American ball of fate # 8.
Yandex / 1.01.001 (compatible; Win16; I)
- the main indexing robot
Yandex / 1.01.001 (compatible; Win16; P)
- indexer pictures
Yandex / 1.01.001 (compatible; Win16; H)
- a robot that determines mirrors sites
Yandex / 1.02.000 (compatible; Win16; F)
- robot indexing favicons sites
Yandex / 1.03.003 (compatible; Win16; D)
- a robot accessing a page when adding it via "Add URL" form
Yandex / 1.03.000 (compatible; Win16; M)
- a robot that addresses when opening a page following the link “ Found words»
YaDirectBot / 1.0 (compatible; Win16; I)
- a robot that indexes pages of sites involved in Yandex Advertising Network
YandexBlog / 0.99.101 (compatible; DOS3.30; Mozilla / 5.0; B; robot)
- robot blog search indexing post comments.
There are many IP addresses from which the Yandex robot "walks", and they can change. We do not disclose the list of addresses.
In addition to robots, Yandex has several “tapping” agents that determine whether a site or document is currently available, which is linked to in the corresponding service.
Yandex / 2.01.000 (compatible; Win16; Dyatel; C)
- "rattling" Yandex.Catalog... If the site is unavailable for several days, it is removed from publication. As soon as the site starts responding, it automatically appears in the Directory.
Yandex / 2.01.000 (compatible; Win16; Dyatel; Z)
- "rattling" Yandex.Bookmark... Links to inaccessible sites are grayed out.
Yandex / 2.01.000 (compatible; Win16; Dyatel; D)
- "rattling" Yandex.Direct... It checks the correctness of links from ads before moderation. No automatic action is taken.
Yandex / 2.01.000 (compatible; Win16; Dyatel; N)
- "rattling" Yandex.News... She generates statistical reports for the content manager and informs him about possible problems from partner-suppliers of news
Source: help.yandex.ru
Google robots
Mozilla / 5.0 (compatible; Googlebot / 2.1; + http: //www.google.com/bot.html) - Google crawler.
Googlebot-Image (Google) Googlebot-Image / 1.0
- robot-indexer of pictures.
Directives addressed to this robot are written to remove images from Google Images, for example, to prohibit indexing of images in news (in this case, illustrations for news are placed in the folder / news / img /):
User-agent: *
Disallow: / news
User-agent: Googlebot-Image
Disallow: / news / img /
Mediapartners-Google- Adsense analyzer robot.
Directives addressed to this robot are written to prohibit indexing of pages while preserving the display of AdSense ads, for example:
User-agent: *
Disallow: / news
User-agent: MediaPartners-Google
Allow: / news
Googlebot-Mobile (compatible; Googlebot-Mobile / 2.1; + http: //www.google.com/bot.html)
- a robot that indexes websites for mobile devices.
Google Search Appliance (Google) gsa-crawler
- search robot of the new hardware-software complex Search Appliance (GSA 6.0).
AdsBot-Google (+ http: //www.google.com/adsbot.html)
- Evaluating the quality of AdWords landing pages.
Rambler Robot
StackRambler / 2.0 (MSIE incompatible) - Rambler search robot.StackRambler / 2.0- Rambler search robot.
Aport robots
Aport- search robot Aporta
AportCatalogRobot / 2.0- Robot Aport catalog.
Yahoo! robots
Mozilla / 5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
- search robot Yahoo!
Mozilla / 5.0 (compatible; Yahoo! Slurp / 3.0; http://help.yahoo.com/help/us/ysearch/slurp)
- the new robot Yahoo! 3rd generation.
Yahoo-MMCrawler / 3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)
- robot-indexer of pictures.
Yahoo-Blogs / v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html)
is a blog search robot.
MSN robots
msnbot / 1.1 (+ http: //search.msn.com/msnbot.htm) is the main robot MSN.msnbot-media / 1.0 (+ http: //search.msn.com/msnbot.htm) - Robot indexer of multimedia files for images.live.com.
msnbot-media / 1.1 (+ http: //search.msn.com/msnbot.htm) - robot-indexer of multimedia files.
msnbot-news (+ http: //search.msn.com/msnbot.htm) - a robot that indexes news.
msnbot-NewsBlogs / 1.0 (+ http: //search.msn.com/msnbot.htm)
- ensures the relevance of news and blogs for search.live.com/news
If the search engine robot tries to access your website more than once every few seconds, you can increase the delay between hits and adjust the minimum frequency (in seconds) by using the parameter Crawl-delay
in your robots.txt file, for example:
User-agent: msnbot
Crawl-delay: 120
msnbot-Products / 1.0 (+ http: //search.msn.com/msnbot.htm) - indexing for product search and shopping products.live.com
msnbot-Academic / 1.0 (+ http: //search.msn.com/msnbot.htm) - performs academic search in academic.live.com
Alexa robot
ia_archiver (+ http: //www.alexa.com/site/help/webmasters; [email protected])
- Alexa robot.
ia_archiver-web.archive.org
- Alexa robot. Alexa robots are useful for indexing websites for web.archive.org
SAPE.BOT is watching you! - scans sites for the SAPE.ru exchange
How to view information about site visits by robots can be found on the page
Removing the entire site
To remove a site from search engines and prevent all robots from crawling it in the future, place a robots.txt file with the following content in the root directory of the server:
User-agent: *
Disallow: /
To remove a site only from Google and prevent the Google crawler from crawling it in the future, place a robots.txt file with the following content in the server root directory:
User-agent: Googlebot
Disallow: /
Each port must have its own robots.txt file. In particular, if the http and https protocols are used, each of them will require separate files robots.txt. For example, to allow the Google crawler to index all http pages and not crawl https, your robots.txt files should look like this.
For the http protocol ( http://yourserver.com/robots.txt):
User-agent: *
Allow: /
For the https protocol ( https://yourserver.com/robots.txt):
User-agent: *
Disallow: /
If the robots.txt file remains in the root directory of the web server, then Google will not crawl the site or its directories. If you do not have access to the root directory of the server, you can place your robots.txt file at the same level as the files you want to remove. After you do this and use the system automatic deletion URL, the site will be temporarily, for 180 days, removed from the Google index, regardless of whether the robots.txt file is removed after the request is processed. (If you leave your robots.txt file at the same level, you will need to remove the URL using automatic system every 180 days.)
Removing part of the site
Option 1. Robots.txt
To remove directories or individual pages on your site, you can place a robots.txt file in the root directory of your server. For information on how to create a robots.txt file, see The Robot Exclusion Standard. Keep the following points in mind when creating your robots.txt file. When deciding which pages to crawl on a particular host, the Google crawler acts according to the first entry in the robots.txt file, where the User-agent parameter begins with the word "Googlebot". If there is no such entry, the first rule in which the User-agent is "*" is executed. In addition, Google allows you to use the robots.txt file more flexibly by using asterisks. In ban patterns, the "*" character can mean any sequence of characters. The pattern may end with "$", which denotes the end of the name.
To remove all pages in a directory (for example, "lemurs"), add the following entry to your robots.txt file:
User-agent: Googlebot
Disallow: / lemurs
To remove all files of a specific type (for example, .gif), add the following entry to your robots.txt file:
User-agent: Googlebot
Disallow: /*.gif$
To delete dynamically created pages, add the following entry to your robots.txt file:
User-agent: Googlebot
Disallow: / *?
Option 2. Meta tags
Another standard, more convenient for working with pages, provides for the use of a meta-tag on the page in HTML format, which prohibits robots from indexing the page. This standard is described on the page.
To prevent all robots from indexing a website page, add the following meta tag to the section of this page:
To prohibit indexing of the page only by Google robots, and allow the rest, use the following tag:
To allow robots to index the page, but prevent crawling by external links, use the following tag:
Note. If your request is urgent and it is impossible to wait for the next Google crawl, use the automatic URL removal system. To start this automatic process, the webmaster must first insert the appropriate meta tags into the HTML page code. After that, directories will be temporarily, for 180 days, removed from the Google index, regardless of whether you remove the robots.txt file or meta tags after processing the request.
Removing fragments (snippets)
A snippet is a text that appears under the page title in the search results list and describes the content of the page.
To prevent Google from displaying snippets from your page, add to the section
next tag:Note. Deleting fragments also deletes the pages stored in the cache.
Deleting Cached Pages
Google automatically creates and archives a snapshot of every page it crawls. Having these cached versions allows end users to find pages even if they are not available (due to a temporary problem on the server hosting the page). Users see cached pages exactly as they were when they were crawled by Google. A message is displayed at the top of the page stating that this is a cached version. To access such a page, the user must select the Cached Cached link on the search results page.
To prevent all search engines from displaying this link on your site, add to the section
next tag:Note. If your request is urgent and it is impossible to wait for the next crawling session of the site by Google, use the automatic URL removal system. To start this automatic process, the webmaster must first insert into the code HTML pages corresponding meta tags.
Removing an image from the Google image search engine
To remove an image from the Google Image Index, place a robots.txt file in the server root directory. (If this is not possible, place it at the directory level).
Example: If you want to remove from the index Google image sobaki.jpg posted on your website at www.vash-sajt.ru/kartinki/sobaki.jpg, create a page www.vash-sajt.ru/robots.txt and add the following text to it:
User-agent: Googlebot-Image
Disallow: /images/sobaki.jpg
To remove all images on the site from the index, place a robots.txt file with the following content in the server root directory:
User-agent: Googlebot-Image
Disallow: /
This is the standard protocol that most scanners follow; it allows you to remove a server or directory from the index. Additional Information about robots.txt is presented on the page
In addition, Google allows you to use the robots.txt file more flexibly through the use of asterisks. In ban patterns, the "*" character can mean any sequence of characters. The pattern may end with "$", which denotes the end of the name. To delete all files of a certain type (for example, to keep images in .jpg format and delete in .gif format), add the following entry to your robots.txt file:
User-agent: Googlebot-Image
Disallow: /*.gif$
Note. If your request is urgent and it is impossible to wait for the next crawling session of the site by Google, use the automatic URL removal system. To start this automatic process, the webmaster must first create a robots.txt file and place it on the appropriate site.
If the robots.txt file remains in the root directory of the web server, Google will not further crawl the site or its directories. If you do not have access to the root directory of the server, you can place your robots.txt file at the same level as the files you want to remove. After you do this and use the automatic URL removal system, directories specified in the robots.txt file will be temporarily removed from the Google index for 180 days, regardless of whether you remove the robots.txt file after processing the request. (If you leave the robots.txt file at the same level, the URL will need to be removed using an automated system every 180 days.)
Search robot called special program any search engine, which is designed to enter into the database (indexing) sites found on the Internet and their pages. The names are also used: crawler, spider, bot, automaticindexer, ant, webcrawler, bot, webscutter, webrobots, webspider.
Principle of operation
A crawler is a browser-based program. He constantly scans the network: visits indexed (already known to him) sites, follows the links from them and finds new resources. When a new resource is found, the robot of procedures adds it to the search engine index. The search robot also indexes updates on sites, the frequency of which is recorded. For example, a site that is updated once a week will be visited by a spider with this frequency, and content on news sites can be indexed within minutes after publication. If no link from other resources leads to the site, then in order to attract search robots, the resource must be added through a special form (Center Google webmasters, Yandex webmaster panel, etc.).
Types of search robots
Yandex Spiders:
- Yandex / 1.01.001 I - the main bot that deals with indexing,
- Yandex / 1.01.001 (P) - indexes pictures,
- Yandex / 1.01.001 (H) - finds site mirrors,
- Yandex / 1.03.003 (D) - determines whether the page added from the webmaster's panel matches the indexing parameters,
- YaDirectBot / 1.0 (I) - indexes resources from the Yandex advertising network,
- Yandex / 1.02.000 (F) - indexes favicons of sites.
Google Spiders:
- Googlebot is the main robot,
- Googlebot News - crawls and indexes news,
- Google Mobile - indexes sites for mobile devices,
- Googlebot Images - searches and indexes images,
- Googlebot Video - indexes videos,
- Google AdsBot - checks the quality of the landing page,
- Google Mobile Adsense and Google adsense- indexes the sites of the Google advertising network.
Other search engines also use several types of robots, functionally similar to those listed.
Instructions
