--- name: discovery-data-connector-lfs-cp4d title: Local File System description: Crawl documents that are stored in a local file system. last-updated: 2025-02-13 --- > ## Documentation Index > The table of contents for this documentation set is at https://cloud.ibm.com/docs/discovery-data?format=markdown > The index for all IBM Cloud docs is at: https://cloud.ibm.com/docs/llms.txt > Use these files to discover more information as needed. # Local File System {: #connector-lfs-cp4d} Crawl documents that are stored in a local file system. {: shortdesc} [IBM Cloud Pak for Data]{: tag-cp4d} [IBM Software Hub]{: tag-teal} This information applies only to installed deployments. {: note} ## What documents are crawled {: #connector-lfs-cp4d-docs} - Only file types that are supported by Discovery in your file path are crawled; all others are ignored. For more information, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections&format=markdown#supportedfiletypes). - Only files in the `/mnt` directory or one of its subdirectories can be accessed by the crawler. - Only files with file extensions that match the file extension filter rules that you specify are crawled. *Added with the 4.7.0 release.* - When a source is recrawled, new documents are added, updated documents are modified to the current version, and deleted documents are deleted from the collection's index. - All Discovery data source connectors are read-only. Regardless of the permissions that are granted to the crawl account, Discovery never writes, updates, or deletes any content in the original data source. ## Prerequisite steps {: #connector-lfs-cp4d-prereq} Before you connect to the Local File System data source, complete the following step: - [Create a persistent volume claim on the crawler pod](#mount-persistent-volume) The service uses Portworx storage by default. However, if you are using Network File System (NFS) storage, see [Prerequisite steps for NFS storage](#connector-lfs-cp4d-prereq-nfs) instead. ### Creating and mounting a persistent volume claim on the crawler pod {: #mount-persistent-volume} Before you can crawl a local file system, you must create a persistent volume claim and mount it on the `crawler` pod. You also need to copy the files that you want to crawl to the Discovery cluster that you are working on. If you have multiple Discovery clusters, you must copy the files along with the `crawler-pvc-portworx.yaml` file that you will create in this task to each cluster. Complete the following steps: 1. Enter the following command to check the `storageclass` name of the Portworx provisioner: ```bash oc get storageclass | grep portworx-gp3-sc ``` {: pre} You might see output similar to the following: ```bash NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE portworx-gp3-sc kubernetes.io/portworx-volume Retain Immediate true 51d ``` {: codeblock} 1. Create a file named `crawler-pvc-portworx.yaml` to define the persistent volume claim (PVC) with the following content: ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi storageClassName: portworx-gp3-sc ``` {: codeblock} Replace `` with the name of your dynamic Portworx persistent volume claim. For example, `jdoe-pvc-portworx` 1. Enter the following command to create the persistent volume claim: ```bash oc create -f crawler-pvc-portworx.yaml ``` {: pre} A message is displayed: ```bash persistentvolumeclaim/jdoe-pvc-portworx created ``` {: codeblock} 1. Enter the following command to mount the persistent volume claim to the `crawler` pod: ```bash oc patch wd wd --type=merge \ --patch='{"spec": {"ingestion": {"crawler": {"mount": {"enabled": true, "persistentVolumeClaimName": "" } } } } }' ``` {: pre} Replace `` with the name of your dynamic Portworx persistent volume claim. For example, `jdoe-pvc-portworx`. 1. Enter the following command to copy the files that you want to crawl to your dynamic Portworx persistent volume claim. You only need to run this command one time against one of the existing `crawler` pods. The persistent volume claim is shared among all `crawler` and `ingestion-api` pods. Replace the variables in the command with the appropriate information. ```bash oc rsync :/mnt ``` {: pre} You mounted the persistent volume claim (PVC) and copied the files that you want to crawl to the PVC. ## Connecting to a local file system data source {: #connector-lfs-cp4d-task} From your Discovery project, complete the following steps: 1. From the navigation pane, choose **Manage collections**. 1. Click **New collection**. 1. Click **Local File System**, and then click **Next**. 1. Name the collection. 1. If the language of the documents that you want to crawl is not English, select the appropriate language. For a list of supported languages, see [Language support](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support&format=markdown). 1. **Optional**: Change the synchronization schedule. For more information, see [Crawl schedule options](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections&format=markdown#crawlschedule). 1. In the *Specify what you want to crawl* section, enter the file path that you want to crawl in the **Path** field, and then click **Add**. The file path is case-sensitive. Remember, only files in the `/mnt` directory or one of its subdirectories can be accessed by the crawler. 1. Optionally, add more file paths. 1. If you want to limit the types of files to add to the collection, you can list the file extensions for file types to either include or exclude. For a list of supported file types, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections&format=markdown#supportedfiletypes). Support for this option was added with the 4.7.0 release. {: note} 1. If you want the crawler to extract text from images in documents, expand *More processing settings*, and set **Apply optical character recognition (OCR)** to `On`. When OCR is enabled and your documents contain images, processing takes longer. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections&format=markdown#ocr). {: note} 1. Click **Finish**. The collection is created quickly. It takes more time for the data to be processed as it is added to the collection. If you want to check the progress, go to the Activity page. From the navigation pane, click **Manage collections**, and then click to open the collection. ## Prerequisite steps for NFS storage {: #connector-lfs-cp4d-prereq-nfs} Choose one of the following methods to enable the `crawler` pod to access the file system: - [Configure an external NFS server](#use-external-nfs) - [Configure dynamic provisioning with an NFS storage class](#dyn-prov-nfs) ### Configuring an external NFS server {: #use-external-nfs} If the local file system files or folders that you want to crawl are stored in an external Network File System (NFS), you can use the external NFS server to create the persistent volume claim. 1. Create a file named `crawler-pv-nfs.yaml` with the following content: ```yaml apiVersion: v1 kind: PersistentVolume metadata: name: labels: pv-name: spec: capacity: storage: 10Gi accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain nfs: server: path: ``` {: codeblock} Replace references to `` with the name of your persistent volume. For example, `jdoe-nfs-pv` and add the missing external NFS details. 1. Enter the following command to create the persistent volume claim: ```bash oc create -f crawler-pv-nfs.yaml ``` {: pre} The following message is displayed: ```bash persistentvolume/jdoe-nfs-pv created ``` {: codeblock} 1. Create a file called `crawler-pvc-nfs.yaml` with the following content: ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi selector: matchLabels: pv-name: ``` {: codeblock} Replace the following variables: - ``: Specify the name of your persistent volume claim. For example, `jdoe-nfs-pvc`. - ``: Specify the name of your persistent volume. For example, `jdoe-nfs-pv`. 1. Enter the following command to create the persistent volume claim: ```bash oc create -f crawler-pvc-nfs.yaml ``` {: pre} The following message is displayed: ```bash persistentvolumeclaim/jdoe-nfs-pvc created ``` {: codeblock} 1. Enter the following command to mount the persistent volume claim to the `crawler` pod. This command also mounts the persistent volume claim to all `ingestion-api` pods. Replace `` with the name of your persistent volume claim. For example, `jdoe-nfs-pvc`. ```bash oc patch wd wd --type=merge \ --patch='{"spec": {"ingestion": {"crawler": {"mount": {"enabled": true, "persistentVolumeClaimName": "" } } } } }' ``` {: pre} ### Configuring dynamic provisioning with an NFS storage class {: #dyn-prov-nfs} If you want to crawl your local file system files or folders but you do not want to prepare an extra NFS server to store those files or folders, you can configure dynamic storage by using an NFS storage class. For more information about storage providers that Discovery supports and for storage comparisons, see [Storage considerations](https://www.ibm.com/docs/SSQNUZ_4.6.x/cpd/plan/storage_considerations.html){: external}. Before you complete this task, copy the files that you want to crawl to the Discovery cluster that you are working on. If you have multiple Discovery clusters, you must copy the files along with the `crawler-pvc-dynamic.yaml` file that you create in this task to each cluster. Complete the following steps: 1. Enter the following command to check the `storageclass` name of the NFS provisioner: ```bash oc get storageclass ``` {: pre} A message is displayed. ```bash NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE nfs-client cluster.local/innocence-nfs-client-provisioner Delete Immediate true 177m ``` {: codeblock} 1. Create a file that is named `crawler-pvc-dynamic.yaml` and add the following content to it: ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi storageClassName: nfs-client ``` {: codeblock} Replace `` with the name of your dynamic NFS persistent volume claim. For example, `jdoe-dynamic-pvc`. 1. Enter the following command to create the persistent volume claim: ```bash oc create -f crawler-pvc-dynamic.yaml ``` {: pre} A message is displayed. ```bash persistentvolumeclaim/jdoe-dynamic-pvc created ``` {: codeblock} 1. Enter the following command to mount the persistent volume claim to the `crawler` pod. This command also mounts the persistent volume claim to all `ingestion-api` pods. ```bash oc patch wd wd --type=merge \ --patch='{"spec": {"ingestion": {"crawler": {"mount": {"enabled": true, "persistentVolumeClaimName": "" } } } } }' ``` {: pre} Replace `` with the name of your dynamic NFS persistent volume claim in the previous step. For example, `jdoe-dynamic-pvc`. 1. Enter the following command to copy the files that you want to crawl to your dynamic NFS persistent volume claim. You must run this command only one time against one of the existing `crawler` pods. The persistent volume claim is shared among all `crawler` and `ingestion-api` pods. Replace the variables in the command with the appropriate information. ```bash oc rsync :/mnt ``` {: pre} You mounted the persistent volume claim (PVC) and copied all of the files that you want to crawl to the PVC.