About Verity Spider Syntax; The Verity Spider Command - MACROMEDIA COLDFUSION MX 61 - CONFIGURING AND ADMINISTERING COLDFUSION MX Manual

Multithreading
Since version 3.1, Verity Spider has separated the gathering and indexing jobs into multiple
threads for concurrence. Verity Spider V3.7 can create concurrent connections to web servers for
fetching documents, and have concurrent indexing threads for maximum utilization. This
translates to an overall improvement in throughput. In previous releases, work was done in a
round-robin manner, so that at any given time, only one job was running. Spider attends to the
websites within an indexing job in a round-robin manner.
Efficient DNS lookups
Verity Spider V3.7 significantly reduces DNS lookups, which means great improvements to
spidering throughput. If spidering is limited by domain or host, then no DNS lookups are made
on hosts that fall outside of that range. In earlier versions, DNS lookups were made on all
candidate URLs.
Proxy handling efficiency
To allow for greater flexibility when dealing with indexing jobs that involve proxy servers and
firewalls, use the following options:
•
-noproxy
To reduce proxy checking for certain hosts
•
-proxyauth
Note: Information Server V3.7does not support retrieving documents for viewing through secure
proxy servers. Do not use the
Information Server V3.7.

About Verity Spider syntax

Before you create an indexing task for a new collection, make copies of the relevant default style
files to ensure that you have a set of template style files in a known, stable state.
Running multiple simultaneous Verity Spider jobs on the Information Server host can cause
performance problems for searches. This does not mean that you should never run indexing jobs
when users might be searching, because your collections are available for searching even while
indexing jobs are running. To optimize performance, try staggering your indexing jobs to avoid
overloading your server.

The Verity Spider command

The vspider executable, which starts the vspider application, is located in the
cf_root\lib\_nti40\bin directory in Windows, and in the cf_root/lib/platform/bin directory on
UNIX.
In these pathnames, cf_root refers to the ColdFusion MX root directory. In Windows, this is
typically C:\CFusionMX; on UNIX, this is typically /opt/coldfusionmx. On UNIX, platform
refers to the UNIX version of the server that runs ColdFusion: _solaris, _hpux11, or _ilnx21.
To authenticate on proxy servers
-proxyauth
option for indexing documents that you will view through
About Verity Spider syntax 103