LinuxWebLog.com

robot.txt file is your invitation to the search engines when they send a spider to crawl through the pages of your web-site. These spiders are out and about on the web gathering information that contributes to your search engine ranking.

There are different kinds of robots/spiders/crawlers, some are just taking inventory, some are looking for what's new, but whatever kind of robot they are, when they hit your site, you have some control over what they do, and you need to give them some instructions.

You can block bots, you can tell them which pages they can and can't look at and you can tell them when you'd like them to return.

There are probably a thousand free robot.txt generators out there and here is one you can use: 1-hit.com

If you want to generate your own file read on...

4images is a powerful web-based image gallery management system. Features include comment system, user registration and mangagement, password protected administration area with browser-based upload, HTML templates for page layout and design, search engine and a lot more. 4images is free for private and non-commercial use.

We are pleased to announce that “Hermann Hesse Gesellschaft –Nepal

Accessibility-Checking - A large collection of bookmarklets to assist Web developers to ensure more accessible sites. Many are useful as design tools.

Conditional Get - Simple guide on how to implement HTTP's Conditional GET mechanism, with regards to producers and consumers of RSS feeds.

XML and CSS editor - The Morphon CSS-Editor and the XML-editor is now being offered under a free license, there are no strings attached. Morphon Technologies would like to increase the user base of the Editors. The software is fully functional with no crippling features, no time limit to usage.

zvon.org - XML tools, tutorials, references, download...

OpenSource CMS - Well organised list of Open Source Content Management Systems.

Script-based Website Maintenance - This website describes a method of automatic HTML generation and management using a combination of several common UNIX utilities.