Creating text indexes

When you create a single text index, it's a good practice to use the default values, but some useful index attributes can be modified.

A text index can be partitioned or global; this option is set by using the partitioned field.

For Full Text Indexes (FTIs), type must be set to text.

The name and ddoc attributes are for grouping indexes into design documents. Use the attributes to refer to index groups by using a custom string value. If no values are supplied for these fields, they're automatically populated with a hash value.

If you create multiple text indexes in a database, with the same ddoc value, you need to know at least the ddoc value and the name value. Creating multiple indexes with the same ddoc value places them into the same design document. Generally, you must put each text index into its own design document.

For more information, see the more about text indexes.

See the following example of a JSON document that requests a partitioned index creation:

{
    "index": {
        "fields": [
            {
                "name": "Movie_name",
                "type": "string"
            }
        ]
    },
    "name": "Movie_name-text",
    "type": "text",
    "partitioned": true
}

See the following example of JSON document that requests a global index creation:

{
    "index": {
        "fields": [
            {
                "name": "Movie_name",
                "type": "string"
            }
        ]
    },
    "name": "Movie_name-text",
    "type": "text",
    "partitioned": false
}

See the following example of a JSON document that requests creation of a more complex partitioned index:

{
    "type": "text",
    "name": "my-index",
    "ddoc": "my-index-design-doc",
    "index": {
        "default_field": {
            "enabled": true,
            "analyzer": "german"
        },
        "selector": {},
        "fields": [
            {"name": "married", "type": "boolean"},
            {"name": "lastname", "type": "string"},
            {"name": "year-of-birth", "type": "number"}
        ]
    },
    "partitioned": true
}

The index field

The index field contains settings specific to text indexes.

To index all fields in all documents automatically, use the simple syntax:

"index": {}

The indexing process traverses all of the fields in all the documents in the database.

In the example movies' demo database, you can see an example of a text index that contains all fields and all documents in a database.

Take care when you index all fields in all documents for large data sets as it might be a resource-consuming activity.

See the following example of a JSON document that requests creation of an index of all fields in all documents:

{
	"type": "text",
	"index": { }
}

The default_field field

The default_field value specifies how the $text operator can be used with the index.

The default_field includes two keys:

The analyzer key in the default_field specifies how the index analyzes text. Later, the index can be queried by using the $text operator. For more information, see the Search documentation for alternative analyzers. You might choose an alternative analyzer when documents are indexed in languages other than English, or when you have other special requirements for the analyzer, such as matching email addresses.

If the default_field isn't specified, or is supplied with an empty object, it defaults to true and the standard analyzer is used.

The fields array

The fields array includes a list of fields that must be indexed for each document. If you know an index queries only on specific fields, then this field can be used to limit the size of the index. Each field must also specify a type to be indexed. The acceptable types are shown in the following list:

• "boolean"
• "string"
• "number"

The index_array_lengths field

IBM Cloudant Query text indexes have a property that is called index_array_lengths. If the property isn't explicitly set, the default value is true.

If the field is set to true, the index requires extra work. This work contains a scan of every document for any arrays, and creating a field to hold the length for each array found.

You might prefer to set the index_array_lengths field to false in the following situations:

• You don't need to know the length of an array.
• You don't use the $size operator.
• The documents in your database are complex, or not completely under your control. As a result, it's difficult to estimate the impact of the extra processing that is needed to determine and store the array lengths.

The $size operator requires that you set the index_array_lengths field to true. Otherwise, the operator can't work.

See the following example JSON document with suggested settings to optimize performance on production systems:

{
	"default_field": {
		"enabled": false
	},
	"index_array_lengths": false
}

The partitioned field

This field determines whether the created index is a partitioned or global index.

The default follows the partitioned setting for the database:

Advanced: text index internals

The basic premise for full text indexes is that a document is "expanded" into a list of key:value pairs that are indexed by Lucene. This expansion enables the use of Lucene's search syntax as a basis for the query capability.

This technique supports enhanced searches, but does have certain limitations. For example, it might not always be clear whether content for an expanded document came from individual elements or an array.

The query mechanism resolves this uncertainty by preferring to return "false positive" results. In other words, if a match was found because of a search for either an individual element, or an element from an array, then the match is considered a success.

Like IBM® Cloudant® for IBM Cloud® Search indexes, IBM Cloudant Query indexes of type: text are limited to 200 results when queried.

{
	"age": {
		"$gt": 5
	}
}

(age_3anumber:{5 TO Infinity])

{
	"$or": [
		{
			"age": {
				"$gt": 5
			}
		},
		{
			"twitter": {
				"$exists":true
			}
		},
		{
			"type": {
				"$in": [
					"starch",
					"protein"
				]
			}
		}
	]
}

(
	# Search for age > 5
	(age_3anumber:{5 TO Infinity])

	# Search for documents that contain the twitter field
	(($fieldnames:twitter_2e*) OR ($fieldnames:twitter_3a*))

	# Search for type = starch
	(
		((type_3astring:starch) OR (type_2e_5b_5d_3astring:starch))

		# Search for type = protein
		((type_3astring:protein) OR (type_2e_5b_5d_3astring:protein))
	)
)