Google Webmasters - News & Updates

  • Thread starter Bapun
  • Start date
  • Replies: Replies 9
  • Views: Views 1,176
Status
Not open for further replies.

Bapun

Staff member
Community Manager
Joined
3 Nov 2010
Messages
26,551
Solutions
7
Reaction score
35,614
Source

In this episode of Google Search News, John Mueller goes over recent developments in the Google Search world.

Google Search News is a news show for webmasters, publishers, and SEOs! Join us every few weeks for updates on what’s going on around Google Search.
 
Robots.txt Specification Test

First, we are releasing a testing framework for robots.txt parser developers, created by Andreea. The project provides a testing tool that can validate whether a robots.txt parser follows the Robots Exclusion Protocol, or to what extent. Currently there is no official and thorough way to assess the correctness of a parser, so Andreea built a tool that can be used to create robots.txt parsers that are following the protocol.

Java robots.txt parser and matcher

Second, we are releasing an official Java port of the C++ robots.txt parser, created by Ian. Java is the 3rd most popular programming language on GitHub and it's extensively used at Google as well, so no wonder it's been the most requested language port. The parser is a 1-to-1 translation of the C++ parser in terms of functions and behavior, and it's been thoroughly tested for parity against a large corpora of robots.txt rules. Teams are already planning to use the Java robots.txt parser in Google production systems, and we hope that you'll find it useful, too.

New open source robots.txt projects
 
Status
Not open for further replies.
Back
Top Bottom
AdBlock Detected

We get it, advertisements are annoying!

Sure, ad-blocking software does a great job at blocking ads, but it also blocks useful features of our website. For the best site experience please disable your AdBlocker.

I've Disabled AdBlock