Overview of Cloud Pak for Data data sources

Data source requirements

The following requirements and limitations are specific to IBM Watson® Discovery:

• The individual file size limit is 32 MB per file, which includes compressed archive files (ZIP, CZIP, TAR). When decompressed, the individual files within compressed files cannot exceed 32 MB per file. This limit is the same for collections in which you upload your own data.

• Depending on the type of installation (starter or production mode), the number of collections you can ingest simultaneously varies. A starter installation includes one crawler pod, which allows three collections to be processed simultaneously. A production installation includes two crawler pods, which can process six collections simultaneously.

  If you are running a starter installation and you want to process more than three collections simultaneously, you must increase the number of crawler pods by running the following commands:

  oc patch wd wd --type=merge --patch='{"spec": {"ingestion": {"crawler": {"replicas": <number-of-replicas> } } } }'
  

  In a starter installation, the maximum number of simultaneous collections that can crawl an external data source is 3. If you start a fourth, that collection does not start to process until the prior three crawls finish.

  Each number-of-replicas allows 3 simultaneous crawls, so number-of-replicas=2 increases the replicas to 6, and number-of -replicas=3 increases them to 9.

Crawler plug-in settings

When you deploy one or more crawler plug-ins, you can configure your collection to use one of the plug-ins.

These settings are only available when crawler plug-ins are deployed.

• For more information about building a plug-in, see Building a Cloud Pak for Data crawler plug-in.
• For more information about deploying a crawler plug-in, see Commands and options for managing your crawler plug-ins.

When you are ready to configure a collection to use a crawler plug-in that was created by using the scripts/manage_crawler_plugin.sh script, you can see a Plug-in settings section with the following options:

• Enable plug-in: The switch is set to Off. Enable this option if you want to use a crawler plug-in to process documents.
• Plug-in: Lists the names of available crawler plug-ins. Select a plug-in to use.

Supporting document-level security

If document-level security is activated, you can use the security settings from your source documents to control the search results that are returned to different users.

Discovery supports prefiltering only. To prefilter, Discovery replicates the document's source access control list (ACL) at crawl time into the index. The search engine must compare user credentials to the replicated document ACLs. Discovery is faster when documents are prefiltered and when you control which documents you add to the index. However, it is difficult to model all of the security policies of the various data sources in the index and implement comparison logic uniformly. Also, prefiltering is not as responsive to changes that occur in the source ACLs after the most recent crawl.

Document-level security is supported by the following data source types:

• Box
• FileNet P8
• HCL Notes
• Microsoft SharePoint Online
• Microsoft SharePoint On Prem
• Microsoft Windows File System

When you query collections where document-level security is enabled, no results are returned if the users associated with your Discovery instance are not present in the source system. For more information about querying these collections, see Querying with document-level security enabled.

To enable document-level security, you must complete the following steps:

1. Create Discovery users that match the users available on the source system.
2. Associate users with your Discovery instance. For more information, see Giving users access to a Watson Discovery instance.
3. Enable document-level security for the data source when you connect to it.