Release notes for Natural Language Understanding

Service API versioning

Current API version: 2022-08-10

API requests require a version parameter that takes the date in the format version=YYYY-MM-DD. Send the version parameter with every API request.

When we change the API in a backwards-incompatible way, we release a new minor version. To take advantage of the changes in a new version, change the value of the version parameter to the new date. If you're not ready to update to that version, don't change your version date.

Entity types (Version 1) deprecation announcement

Entity types (Version 1) is deprecated. As of 11 July 2023, the v1 Entities type system will no longer be available. Consider switching to the v2 Entities type system. By using the more generic v2 Entities type system together with other features, such as Concepts and Categories, you can achieve similar outcomes with more flexibility. For more information about v2 entity types, see Entity types (Version 2).

9 June 2023

Retired Custom Sentiment Feature

The Custom Sentiment feature is retired. Custom sentiment models can no longer be created nor can they be used with Analyze API calls. To ensure we continue providing our clients with robust and powerful text classification capabilities, IBM recently announced the general availability of a new single-label text classification capability. This new feature includes extended language support and training data customizations suited for building a custom sentiment classifier.

16 February 2023

Updated training process for Custom Classifications

Updates were made to the Custom Classifications training process including improvements in the preprocessing stage, updating underlying libraries, and fixing minor bugs.

2 February 2023

Sentiment support for additional languages

Support for sentiment is now available, for all public service instances, for the following languages: Czech, Danish, Finnish, Hebrew, Hindi, Norwegian Bokmål, Norwegian Nynorsk, Polish, Romanian, Slovak, Swedish, Turkish. For details, see Language support.

11 January 2023

Improved English and Korean Entities Model

Fixed a bug for English and Korean models using Version 2 Entity type system where not all entities are returned properly.

3 November 2022

Improved Error Handling and Validation for Sentiment and Custom Sentiment

If a request contains both an error in the sentiment feature and a valid feature request for another feature, the response returns a 200 with a warning for the sentiment feature.

Requests for Custom Sentiment that include at least one target found in the document will now return a 200 with sentiment analysis for the found targets.

The error log has changed from sentiment request must specify at least one of: document, targets to sentiment request must specify at least one of: document or targets.

1 September 2022

Improved Categories and Custom Categories (beta) Models

The model for Categories and Custom Categories (beta) now supports more contemporary vocabulary. For example, newer words such as "Covid-19" can be interpreted.

10 August 2022

Entities support for additional languages

Support for entities is now available, for all public and premium service instances, for the following languages: Czech, Danish, Finnish, Hebrew, Hindi, Norwegian Bokmål, Norwegian Nynorsk, Polish, Romanian, Slovak, Turkish. For details, see Language support.

Version 2 entity support for Russian and Swedish

Version 2 entity type support is now available, for all public and premium service instances, for the following languages: Russian and Swedish. For details, see Entity type systems.

3 August 2022

Support for Single Label Classifications

Classifications now allows users to pass in a model_type training parameter when creating or updating a model, in order to train a single label classifier. See Classifications training parameters for more details.

11 July 2022

Improved custom classifications model - Japanese

Japanese custom classifications models now train and perform inference faster, with improved model results.

Improved custom categories model

Model contains improved word filtering, resulting in better category determination.

7 April 2022

Categories bug fix

Fixed a bug in the Version 2 Categories type system.

14 February 2022

Emotion support

Support for emotion is now available, for all public service instances, for French. For details, see Language support.

Improved error handling and validation for emotion

If a request contains both an error in the emotion feature and a valid feature request for another feature, the response returns a 200 with a warning for the emotion feature.

The error log has changed from emotion request must specify at least one of: document, targets to emotion request must specify at least one of: document or targets.

Requests that include the emotion feature with any nonstring elements in the targets list returns an error.

1 December 2021

Tone analytics

Tone analytics is now available, for English and French languages only. The tone analytics feature detects excited, frustrated, impolite, polite, sad, satisfied, and sympathetic tones from text.

12 October 2021

Keyword and syntax support for additional languages

Support for keywords and syntax is now available, for all public and premium service instances, for the following languages: Hindi, Romanian, and Turkish. In addition, syntax is available for all supported languages. For details, see Language support.

15 August 2021

Custom classification feature

The custom classifications feature is now generally available (GA).

Categories stock model updated

The categories stock model has been updated to use the IAB Tech Lab 2.0 taxonomy.

5 May 2021

Advanced Rules beta feature deprecated

The advanced rules beta feature is deprecated. As of June 10, 2021, you will not be able to deploy advanced rules models to Natural Language Understanding; existing models will keep running. After June 24, 2021, no advanced rules models will run in Natural Language Understanding.

25 March 2021

Custom classifications beta feature

The custom classifications (Beta) feature allows you to train a multi-label text classifier (English-only) by using your own labeled data.

18 February 2021

Custom categories beta feature

The custom categories (Beta) feature allows you to train custom categories models (English only) with service instances deployed in the Dallas location.

December 2020

November 2020

September 2020

August 2020

8 July 2020

Arabic keyword support

Support for Arabic keywords is now available, for all public and premium service instances. For details, see Language support.

10 April 2020

Custom Sentiment (Beta)

Custom Sentiment (Beta) is now available. The Beta supports English and is available only for service instances in the Dallas location. For details, see Creating custom sentiment models.

March 2020

December 2019

November 2019

10 October 2019

Improved Sentiment response times

Improved response times for Japanese and Korean sentiment requests.

September 2019

29 August 2019

Improved Sentiment support for English

Improved accuracy for English sentiment.

12 July 2019

The following changes are activated when you use the version date 2019-07-12 or later.

New English entities model with improved accuracy.

Confidence scores are returned in English entities results.

June 2019

May 2019

25 April 2019

Entity mentions for rule-based custom models

Enabled entity mentions for rule-based custom models. To view mentions in your results, set the mentions entities option to true.

Explanations for English categories results

Released explanations for English categories results. To see relevant text from the source that contributed to each result, set the explanation categories option to true.

19 March 2019

Custom categories models created with Knowledge Studio

Introduced experimental support for custom categories models created with Knowledge Studio.

Keywords and concepts improved for Spanish

Improved Spanish keywords and concepts.

10 January 2019

Requests with no version date parameter return error

List models and Delete model requests that don't include a version date parameter now return 400 errors instead of successful responses.

Entites performance improved

Improved entities performance for languages other than English.

December 2018

November 2018

30 October 2018

IAM used for service instances

As of 30 October 2018, new service instances created in the Germany and US South regions use Identity and Access Management (IAM) authentication.

21 September 2018

Language support improvements

Added support for French relations and Portuguese concepts.

The targeted sentiment option is now supported for French and Portuguese keywords.

Improved French keywords.

Improved Korean sentiment.

Improved Portuguese keywords and sentiment.

Released a new Portuguese entities model with the latest entity type system. You can learn about the latest type system on the Entity types and subtypes (Version 2) page. When your application is compatible with the new type system, change the version date parameter in your requests to 2018-09-21 to use the new model.

June 2018

May 2018

April 2018

16 March 2018

Language support improvements

Added support for German categories, relations, and semantic roles.

A new relation type system is used for German relations. To view details, see the Relation types (Version 2) page.

Improved German keywords and sentiment.

Added support for Japanese categories and concepts.

Language detection improvements.

Improved webpage cleaning.

Improved French and German entities models. The models use a new entity type system. Check out the new entity types and subtypes on the Entity types and subtypes (Version 2) page. When your application is compatible with the new type system, change the version date parameter in your requests to 2018-03-16 to use the new model. The following are the differences between the Version 1 type system and the Version 2 type system.

• New entity types:
  • Date
  • Duration
  • Measure
  • Money
  • Number*
  • Percent*
  • PhoneNumber*
  • Ordinal
  • Time
  • URL*
• Removed entity types:
  • Anatomy
  • Award
  • Broadcaster
  • Company
  • Crime
  • Drug
  • HealthCondition
  • Movie
  • MusicGroup
  • NaturalEvent
  • PrintMedia
  • Quantity
  • Sport
  • SportingEvent
  • TelevisionShow
  • Vehicle
• New entity subtype:
  • Quantity

* This entity type is not yet detectable in French text.

January 2018

15 December 2017

Language support improvements

Dutch custom model support is now available for entities and relations.

French language support now includes concepts.

The French sentiment model has been improved to deliver better results.

The language detection model is faster and detects more languages overall. For the complete list of languages, see Detectable languages.

• The following languages are new additions to the list:

  • Belarusian (be)
  • Bihari (bh)
  • Dhivehi (dv)
  • Galician (gl)
  • Ganda (lg)
  • Inuktitut (iu)
  • Javanese (jv)
  • Kannada (kn)
  • Khmer (km)
  • Kinyarwanda (rw)
  • Laothian (lo)
  • Malayalam (ml)
  • Marathi (mr)
  • Oriya (or)
  • Punjabi (pa)
  • Scots Gaelic (gd)
  • Sinhalese (si)
  • Tamil (ta)
  • Telugu (te)
  • Yiddish (yi)

• The following languages are no longer detectable:

  • Breton (br)
  • Chamorro (ch)
  • Esperanto (eo)
  • Faroese (fo)
  • Hausa (ha)
  • Ndebele (nr)
  • Ojibwa (oj)

November 2017

28 November 2017

Entity mentions

Get the locations of entity mentions in your entities requests by adding the option "mentions": true.

• Example POST /analyze request body:

  {
    "text": "Intel planned to announce Monday a laptop-computer chip that combines an Intel processor and an AMD graphics unit.",
    "features": {
      "entities": {
        "mentions": true
      }
    }
  }
  

• Example response:

  {
    "usage": {
      "text_units": 1,
      "text_characters": 114,
      "features": 1
    },
    "language": "en",
    "entities": [
      {
        "type": "Company",
        "text": "Intel",
        "relevance": 0.33,
        "mentions": [
          {
            "text": "Intel",
            "location": [
              0,
              5
            ]
          },
          {
            "text": "Intel",
            "location": [
              73,
              78
            ]
          }
        ],
        "disambiguation": {
          "subtype": [
            "OperatingSystemDeveloper",
            "ProcessorManufacturer",
            "SoftwareDeveloper"
          ],
          "name": "Intel",
          "dbpedia_resource": "http://dbpedia.org/resource/Intel"
        },
        "count": 2
      },
      {
        "type": "Company",
        "text": "AMD",
        "relevance": 0.33,
        "mentions": [
          {
            "text": "AMD",
            "location": [
              96,
              99
            ]
          }
        ],
        "disambiguation": {
          "subtype": [
            "ProcessorManufacturer"
          ],
          "name": "Advanced Micro Devices",
          "dbpedia_resource": "http://dbpedia.org/resource/Advanced_Micro_Devices"
        },
        "count": 1
      }
    ]
  }
  

17 November 2017

Korean language support improved

Korean language support has expanded to include the following features:

• Entities
• Keywords
• Semantic Roles

30 July 2017

Limit number of characters processed

You can control cost by using the optional limit_text_characters parameter to limit the number of characters that are processed. For example,

{
  "text": "The United States of America, commonly known as the United States (U.S.) or America, is a federal republic composed of 50 states, a federal district, five major self-governing territories, and various possessions. Forty-eight of the fifty states and the federal district are contiguous and located in North America between Canada and Mexico. The state of Alaska is in the northwest corner of North America, bordered by Canada to the east and across the Bering Strait from Russia to the west.",
  "features": {
    "entities": {}
  },
  "limit_text_characters": 50,
  "return_analyzed_text": true
}

• Each character counts as one character, regardless of whether the character is a single-byte character or a multibyte character.

• For metering, one Natural Language Understanding Item continues to be one feature (also known as an enrichment) per one text unit. One text unit is 10,000 characters or less.

• For detailed pricing information, see Natural Language Understanding in the IBM Cloud® Catalog.

In addition to adding the limit_text_characters parameter, the following changes were made to text size limits and truncation:

• All text greater than 50,000 characters is truncated before processing. Previously, truncation was based on kilobytes, where one kilobyte equaled 1024 bytes.

• An informational message is returned in the response when text beyond 50,000 characters is truncated.

• Text limit size for custom models has increased from 10,000 characters to 50,000 characters.

• Usage information is added to the response for clarity around number of Natural Language Understanding Items used for each request. For example,

{
  "usage": {
    "text_units": 5,
    "text_characters": 41186,
    "features": 1
  },
  ...
}

8 May 2017

Emotion tone score updates

Updated the emotion tone score model. The training data set was expanded and feature engineering was altered and as a result, the model has higher precision on the IBM benchmark data set.

March 2017

27 February 2017

Service GA

The Natural Language Understanding service is now GA.