IBM Cloud Docs
Working with Milvus

Working with Milvus

Creating a collection in Milvus

In Milvus, collections are used to store and manage entities. A collection in Milvus is equivalent to a table in a relational database management system (RDBMS).

See Create a Collection for creating collection in Milvus.

Inserting data in Milvus

Milvus supports default values for scalar fields except a primary key field. You can keep some fields empty during data inserts.

For more information, see Insert entries.

It is recommended to insert your data in batches due to the following reasons:

  • The number of vectors that can be ingested in a single GRPC API call is limited by the maximum message size that is allowed by Kafka. In IBM Cloud, the maximum limit of message size is limited to 1 MB.

  • The maximum number of rows that can be inserted at a time depends on the total size of the data you are trying to ingest. The exact number decreases with the increase in the dimensions of the vector and the presence of non-vector fields in the row.

Use the bulk insert API for inserting the data sets larger than 500,000 vectors. The bulk insert API performs better than the batch insert API when ingesting larger data sets. If you are using batch insert API, manually flush the collection after every 500,000 rows. For more information, see Bulk Insert API.

Creating indexes in Milvus

Create an index before you conduct the Approximate Nearest Neighbor (ANN) search in Milvus.

IBM officially supports the following indexes:

Indexes that are not listed in this list might work, but are not validated by IBM.

  • HNSW
  • SCANN
  • FLAT
  • IVF_FLAT
  • IVF_PQ

You can create the index by specifying the vector field name and index parameters. For more information, see Index Vector Fields.

Conducting a vector similarity search in Milvus

In Milvus, you can conduct a vector similarity search after you prepare the parameters for your search scenario.

Conducting a query based on scalar filtering in Milvus

In Milvus, you can conduct a query based on scalar filtering. For more information, see Get & scalar query.

You can do the following search types:

  • Range search: To find vectors within a specific distance range from the query vector. For more information, see Range search.
  • Grouping search: To get results based on a specific field to ensure diversity in the results. For more information, see Grouping search.

When running a search query with scalar filtering on a large data set, it is important to adjust the limit parameter to manage query results effectively. Use the following approach to set the limit parameter:

hello_milvus.query(expr=query_condition, output_fields=["random", "embeddings"], limit=100,offset=0)

The sum of the values of limit and offset parameters must be in the range of [1, 16384].

Deleting entities from Milvus by using primary key

In Milvus, you can delete the entities by using primary key. Prepare the Boolean expression that filters the entities to delete.

For more information, see Delete Entities.

Dropping a collection from Milvus

Dropping a collection from Milvus is irreversible. You cannot recover the deleted data.

Add the following to your .ipynb or Python script to drop a collection..

from pymilvus import utility
utility.drop_collection(<collection name>)

Best practices

Following are some best practices:

  • If there are long Varchar fields (greater than 256 characters), keep non-vector fields outside of Milvus. You can keep them in a COS bucket or storage bucket and perform a referential search.
  • Make sure that the PyMilvus version is 2.4.0 or later. Milvus 2.4.0 or later versions support sparse vector search, hybrid search (sparse_dense), and multi vector search.
  • While loading collections by using batch insert, follow the pattern: Insert in a batch, release the memory, and then repeat the process.
  • Don't ingest in parallel to all of the collections at once. Ingest sequentially and flush between ingests.
  • Each collection's maximum size should be corresponding to the maximum number of vectors supported in a T-shirt size.
  • If you have multiple collections or partitions loaded, ensure that the sum of vectors in all loaded entities does not exceed the limit of the T-shirt size. You can still store more vectors than the T-shirt size limit as long as the number of vectors that are loaded is within the limit.

During Milvus upgrade, there can be a slight delay in response for about 20 seconds. On-going searches and upsert queries might fail. You can retry the queries immediately.