MACROMEDIA COLDFUSION MX 61 - CONFIGURING AND ADMINISTERING COLDFUSION MX Manual page 113

-header
Type: Web crawling only
Syntax:
-header string
Specifies an HTTP header to add to the spidering request; for example:
-header "Referer: http://www.verity.com/"
Verity Spider sends some predefined headers, such as Accept and User-Agent, by default. Special
headers are sometimes necessary to correctly index a site.
For example, earlier versions of Verity Spider did not support the Host header, which is needed
for Virtual Host indexing. Also, a Proxy-authentication header was needed to pass a username
and password to a proxy server.
In Verity Spider V3.7, the Host header is supported by default, and the
available for proxy server authentication. Therefore, the
backwards compatibility and possible future enhancements.
Note: Misuse of this option causes spider failure. If this happens, rerun the indexing task with
modified values.
-header
-hostcache
Syntax:
-hostcache num_hostnames
Specifies the number of host names to cache to avoid DNS lookups. Without this option, the
host cache continues to grow.
The default value is 256.
-noflowctrl
Type: Web crawling only
Disables round-robin indexing of websites with network flow control.
By default, Verity Spider uses round-robin indexing of websites to avoid overwhelming a web
server and to improve indexing performance. Verity Spider connects to each web server in a
round-robin manner, using up to the value for the
URL is fetched from each web server, in turn.
Note: Using the
-noflowctrl
-noproxy
Type: Web crawling only
Syntax:
-noproxy name_1 [name_n] ...
Used in conjunction with the
directly access the hosts whose names match those specified. By default, when you specify the
option, Verity Spider first tries to access every host with the proxy information. To
-proxy
improve performance, use the
proxy host. For the name variable, you can use the asterisk (*) wildcard for text strings; for
example:
'*.verity.com'
option can result in a significant drop in performance.
option, the
-proxy -noproxy
option for the hosts you know can be accessed without a
-noproxy
-proxyauth
option is maintained only for
-header
option. This means that one
-connections
option specifies that Verity Spider
Networking options
option is
113