MACROMEDIA COLDFUSION MX 61 - CONFIGURING AND ADMINISTERING COLDFUSION MX Manual page 117

-nofollow
Type: Web crawling only
Syntax:
-nofollow "exp"
Specifies that Verity Spider cannot follow any URLs that match the exp expression. If you do not
specify an exp value for the
documents are followed.
You can use wildcard expressions, where the asterisk (*) is for text strings and the question mark
(?) is for single characters. Always encapsulate the exp values in double-quotation marks to ensure
that they are properly interpreted.
If you use backslashes, you must double them so that they are properly escaped; for example:
C:\\test\\docs\\path
To use regular expressions, also specify the
Earlier versions of Verity Spider did not allow the use of an expression. This meant that for each
starting point URL, only the first document would be indexed. With the addition of the
expression functionality, you can now selectively skip URLs, even within documents.
See also
-regexp
-norobo
Type: Web crawling only
Specifies to ignore any robots.txt files encountered. The robots.txt file is used on many websites to
specify what parts of the site indexers should avoid. The default is to honor any robots.txt files.
If you are re-indexing a site and the robots.txt file has changed, Verity Spider deletes documents
that have been newly disallowed by the robots.txt file.
Use this option with discretion and extreme care, especially in conjunction with the
option.
See also and http://info.webcrawler.com/mak/projects/robots/norobots.html.
-nodocrobo
-pathlen
Syntax:
-pathlen num_pathsegments
Limits indexing to the specified number of path segments in the URL or file system path. The
path length is determined as follows:
•
The host name and drive letter are not included; for example, neither www.spider.com:80/ nor
C:\ would be included in determining the path length.
•
All elements following the host name are included.
•
The actual filename, if present, is included; for example, /world.html would be included in
determining the path length.
•
Any directory paths between the host and the actual filename are included.
option, Verity Spider assumes a value of "*", where no
-nofollow
-regexp
option.
Path and URL options
-cgiok
117