Northcote Internet are a company which provides data for investors on a wide range of publicly quoted companies and has been used by the stockbroker, The Share Centre to provide these services to its members.
The Northcote Link Finder is based upon an internet robot created by RWAP Services (c) 2003-04, written in Perl 5.
The Link Finder was developed to find new links to be added to the database of links to the company information which is maintained by Northcote Internet. It is easy for this database of links to rapidly become out of date and therefore the Link Spider's job is to crawl a list of websites, looking for company information which falls within certain categories.
As well as spotting the standard web page errors and re-directions, the Link Finder is able to list questionable pages, based upon a list of user-defined phrases which may appear in the title or body of a given webpage.
The Northcote Link Finder is designed to follow the standard robots exclusion protocol rules, and is identified by the following user-agent: NorthcoteFinder. If you want to exclude the Link Finder from any part of your website, you will need to follow the robots exclusion protocol and using this identifier to either include a statement in the robots.txt file on your website, or within the meta tag: