Configuring the Data Crawler

Input adapter

• class - Internal use only; defines the Data Crawler input adapter class. The default value is: com.ibm.watson.crawler.connectorframeworkinputadapter.Crawl

• config - Internal use only; defines the connector framework configuration. The default configuration key within this block to pass to the chosen input adapter is: connector_framework

  The connector framework is what allows you to talk to your data. It could be internal data within the enterprise, or it could be external data on the web or in the cloud. The connectors allow access to a number of different data sources, while connecting is actually controlled by the crawling process.

  Data retrieved by the Connector Framework Input Adapter is cached locally. It is not stored encrypted. By default, the data is cached to a temporary directory that you must clear after performing a reboot and it must be readable only by the user who executed the crawler command.

  There is a chance that this directory can outlive the crawler if the connector framework is removed before it can clean up after itself. Consider the location for your cached data. You can put data on an encrypted filesystem, but that might have performance implications. Pick the appropriate balance between speed and security for your crawls.

• crawl_config_file - The configuration file to use for the crawl. Default value is: connectors/filesystem.conf
• crawl_seed_file - The crawl seed file to use for the crawl. Default value is: seeds/filesystem-seed.conf
• id_vcrypt_file - Keyfile used for data encryption by the Crawler; the default key included with the crawler is id_vcrypt. Use the vcrypt script in the bin folder if you need to generate a new id_vcrypt file.
• crawler_temp_dir - The Crawler temporary folder for connector logs. Default value, tmp, is provided. If it doesn't already exist, the tmp folder is created in the current working directory.

• extra_jars_dir - Adds a directory of extra JARs to the connector framework classpath.

  Relative to the connector framework lib/java directory.

  • This value must be database when using the Database connector.

  You can leave this value empty (i.e., empty string "") when using other connectors.

• urls_to_filter - Blocklist of URLs that must not be crawled, in regular expression form. The Data Crawler does not crawl URLs that match any of the regular expressions provided.

  The domain list contains the domains that cannot be crawled. Add to it if necessary.

  The filetype list contains the file extensions that the Orchestration Service does not support.

  Remove any supported filetypes from the regular expressions.

  Ensure that your seed URL domain is allowed by the filter. Use an empty filter for allow everything behavior.

  Ensure that your seed URL is not excluded by a filter, or the Crawler might hang.

• max_text_size - The maximum size, in bytes, that a document can be before it is written to disk by the Connector Framework. Adjusting this higher decreases the amount of documents written to disk, but increases the memory requirement. Default value is 1048576

• extra_vm_params - Allows you to add extra Java parameters to the command used to launch the Connector Framework.

• bootstrap_logging - Writes connector framework startup log; useful for advanced debugging only. Possible values are true or false. Log file is written to crawler_temp_dir

• read-timeout - Sets the time in seconds that the crawler waits for a response from the connector framework. The default value is 5 seconds.

Output adapter

• class - Defines the Data Crawler output adapter class.

• config - Defines which configuration key to pass to the output adapter. The string must correspond to a key within this configuration object. In the following code example:

  class - "com.ibm.watson.crawler.discoveryserviceoutputadapter.DiscoveryServiceOutputAdapter"
  
  config - "discovery_service"
  
  discovery_service {
      include "discovery/discovery_service.conf"
    }
  

  the configuration key is discovery_service.

You must select an output adapter by specifying its class parameter and config key.

• Discovery Service Output Adapter - Uploads crawled documents to the IBM Watson™ Discovery Service. Select this adapter by setting the class parameter and config key as follows.

class - "com.ibm.watson.crawler.discoveryserviceoutputadapter.DiscoveryServiceOutputAdapter"

config - "discovery_service"

discovery_service {
            include "discovery/discovery_service.conf"
          }

• Test Output Adapter - The Test Output Adapter writes a representation of the crawled files to disk in a specified location. Select this adapter by setting the class parameter and config key as follows.

  An additional parameter, output_directory, selects the directory to which the representation of the crawled data must be written.

class - "com.ibm.watson.crawler.testoutputadapter.TestOutputAdapter"

config - "test"

output_directory - "/tmp/crawler-test-output"`

• retry - Specifies the options for retry in case of failed attempts to push to the output adapter.

  • max_attempts - Maximum number of retry attempts. Default value is 10
  • delay - Minimum amount of delay between attempts, in seconds. Default value is 2
  • exponent_base - Factor that determines the growth of the delay time over each failed attempt. Default value is 2

  The formula is:

  **`d(nth_retry) - delay * (exponent_base ^ nth_retry)`**
  

  For example, the default settings with a delay of 1 second and an exponent base of 2 means that the second retry, or the third attempt, is delayed 2 seconds. The third retry is delayed 4 seconds, and so on.

  `d(0) - 1 * (2 ^ 0)` - 1 second
  `d(1) - 1 * (2 ^ 1)` - 2 seconds
  `d(2) - 1 * (2 ^ 2)` - 4 seconds
  

  So, with the default settings, a submission is attempted up to 10 times, waiting up to approximately 1022 seconds - a little more than 17 minutes. This time is approximate because there is additional time added to avoid having multiple resubmissions execute simultaneously. This "fuzzed" time is up to 10%, so the last retry in the previous example could delay up to 7.7 seconds. The wait time does not include the time spent connecting to the service, uploading data, or waiting for a response.

  The output_timeout value takes precedence over the wait time here; if the total retry wait time exceeds that setting, the submission fails, even if it should have been retried.

Additional crawl management options

• full_node_debugging - Activates debugging mode; possible values are true or false.

  This puts the full data of every document crawled into the logs.

• logging.log4j.configuration_file* - The configuration file to use for logging. In the sample crawler.conf file, this option is defined in logging.log4j and its default value is log4j_custom.properties. This option must be similarly defined whether using a .properties or .conf file.

• shutdown_timeout - Specifies the timeout value, in minutes, before shutting down the application. Default value is 10.
• output_limit - The highest number of indexable items that the Crawler attempts to send simultaneously to the output adapter. This can be further limited by the number of cores available to do the work. It says that, at any given point, there is no more than "x" indexable items sent to the output adapter waiting to return. Default value is 10.
• input_limit - Limits the number of URLs that can be requested from the input adapter at one time. Default value is 30.

• output_timeout - The amount of time, in seconds, before the Data Crawler gives up on a request to the output adapter, and then removes the item from the output adapter queue to allow more processing. Default value is 1200.

  Consider the constraints imposed by the output adapter, as those constraints might relate to the limits defined here. The defined output_limit only relates to how many indexable objects can be sent to the output adapter at once. Once an indexable object is sent to the output adapter, it is "on the clock," as defined by the output_timeout variable. It is possible that the output adapter itself has a throttle preventing it from being able to process as many inputs as it receives. For instance, the orchestration output adapter might have a connection pool, configurable for HTTP connections to the service. If it defaults to 8, for example, and if you set the output_limit to a number greater than 8, then there are processes, on the clock, waiting for a turn to execute. You might then experience timeouts.

• num_threads - The number of parallel threads that can be run at one time. This value can be either an integer, which specifies the number of parallel threads directly, or it can be a string, with the format "xNUM", specifying the multiplication factor of the number of available processors, for example, "x1.5". The default value is "30"