Configuration reference

PDF

If you configure your collection using Smart Document Understanding, the PDF and Word conversion settings listed are not used, so changing these conversion settings are ignored.

The pdf conversion object defines the conversion from PDF to HTML and has the following structure:

"pdf": {
  "heading": {
    "fonts": [
      {
        "level": 1,
        "min_size": 24,
        "max_size": 80,
        "bold": false,
        "italic": true,
        "name": "arial"
      },
      {
        "level": 2,
        "min_size": 18,
        "max_size": 24,
        "bold": true,
        "italic": false,
        "name": "ariel"
      }
    ]
  }
},

When converting PDF files, headings in those files can be identified and converted into an appropriate HTML "h" tag by identifying the size, font, and style of each heading level. Heading levels can be specified multiple times, if necessary, to correctly identify all relevant sections. HTML heading levels are important to identify, if you plan on extracting the contents, using CSS selectors, or if you intend to split the document, using document splitting. The heading object contains the fonts array and each item in that array specifies a heading level, using the following parameters:

• "level": INT - required - the HTML h level that text identified with these parameters are converted into.
• "min_size": INT - optional - the smallest font size that is identified as this heading level.
• "max_size": INT - optional - the largest font size that is identified as this heading level.
• "bold": boolean - optional - When true only bold fonts are identified as this heading level.
• "italic": boolean - optional - When true only italic fonts are identified as this heading level.
• "name": "string" - optional - The name of the font that is identified as this heading level.

For an area of text to be identified as a heading it must match all of the parameters defined in the specific array item. If the parameters defined are too flexible, you might have more headings identified than you anticipated. Your results are better if you strictly define each heading level multiple times to ensure all heading level variations are included without invalid matches.

Word

If you configure your collection using Smart Document Understanding, the PDF and Word conversion settings listed are not used, so changing these conversion settings are ignored.

The word conversion object defines how to convert PDF documents into HTML and has the following structure:

"word": {
  "heading": {
    "fonts": [
      {
        "level": 1,
        "min_size": 24,
        "bold": true,
        "italic": false,
        "name": "arial"
      },
      {
        "level": 2,
        "min_size": 18,
        "max_size": 23,
        "bold": true,
        "italic": false
      }
    ],
    "styles": [
      {
        "level": 1,
        "names": [
          "pullout heading",
          "pulloutheading",
          "header"
        ]
      },
      {
        "level": 2,
        "names": [
          "subtitle"
        ]
      }
    ]
  }
},

The Microsoft Word conversion object works in a similar way to the PDF conversion object. However, there are two different arrays that can be specified inside the heading object when extracting headings from Microsoft Word documents. You can use either or both heading extraction arrays to extract heading level elements from your Microsoft Word documents.

Each item in the fonts array specifies a heading level using the following parameters using font characteristics:

• "level": INT - required - the HTML h level that text identified with these parameters are converted into.
• "min_size": INT - optional - the smallest font size that is identified as this heading level.
• "max_size": INT - optional - the largest font size that is identified as this heading level.
• "bold": boolean - optional - When true only bold fonts are identified as this heading level.
• "italic": boolean - optional - When true only italic fonts are identified as this heading level.
• "name": "string" - optional - The name of the font that is identified as this heading level.

For an area of text to be identified as a heading in the font array it must match all of the parameters defined in the specific array item. If the parameters defined are too flexible, you might have more headings identified than you anticipated. Your results are better if you strictly define each heading level multiple times to ensure all heading level variations are included without invalid matches.

Each item in the styles array specifies a heading level from the Microsoft Word styles that are applied to that paragraph.

• "level": INT - required - the HTML h level that text identified with these parameters are converted into.
• "names": array - required - a comma separated array of style names that is identified as this heading level.

HTML

"html": {
  "exclude_tags_completely": [
    "script",
    "sup"
  ],
  "exclude_tags_keep_content": [
    "font",
    "em"
  ],
  "exclude_content": {
    "xpaths": [
      "//*[@id='list-old']",
      "//*[@id='unstable']"
    ]
  },
  "keep_content": {
    "xpaths": [
      "//*[@id='footer']",
      "//*[@id='header']"
    ]
  },
  "exclude_tag_attributes": [
    "EVENT_ACTIONS"
  ],
  "keep_tag_attributes": [
    "id"
  ],
  "extracted_fields": {
    "{field_name}": {
      "css_selector": "{CSS_selector_expression_1}",
      "type": "{field_type}"
    }
  }
},

"{field_name}": {
  "css_selector": "{CSS_selector_expression_1}",
  "type": "{field_type}"
}

Segment

The segment object is a set of configuration options that split ingested documents into one or more segments based on the identified HTML headings (h1, h2).

"segment": {
  "enabled": true,
  "selector_tags": ["h1", "h2", "h3", "h4", "h5", "h6"]
}

• "enabled": boolean - required - must be set to true to enable document segmentation.
• "selector_tags": array - required - a comma separated array of HMTL h tags on which to split documents.

As an overview, when document segmentation is enabled the following cannot be specified:

• json_normalizations cannot be specified as part of the configuration.
• normalizations cannot be specified as part of the configuration.
• The extracted_fields option of the html conversion cannot be specified as part of the configuration.

For detailed information, see Performing segmentation.

JSON

You can perform pre-enrichment normalization of the ingested JSON by defining operation objects in the json_normalizations array.

"json_normalizations": [
  {
    "operation": "remove",
    "source_field": "header"
  },
  {
    "operation": "copy",
    "source_field": "title",
    "destination_field": "title_old"
  },
  {
    "operation": "move",
    "destination_field": "content",
    "source_field": "body"
  },
  {
    "operation": "merge",
    "source_field": "synopsis",
    "destination_field": "preamble"
  },
  {
    "operation": "remove_nulls"
  }
]

Element Classification enrichments

The Element Classification enrichment is deprecated and will no longer be available, effective 10 July 2020.

When you use the Element Classification, each elements enrichment object must contain an "options": {} object with the following parameters specified:

• "model" : string - required - The element extraction model to be used with on this document. Currently supported models are: contract

When you use the elements enrichment, it is important to follow the guidelines specified in Element Classification. Specifically, only PDF files can be ingested when this enrichment is specified.

Natural Language Understanding enrichments

When you use the Natural Language Understanding, each object within the enrichments array must also contain an "options": { "features": { } } object that contains one or more of the following enrichments:

categories

The categories enrichment identifies any general categories in the ingested document. This enrichment has no options and must be specified as an empty object "categories" : {}

concepts

The concepts enrichment finds concepts with which the input text is associated, based on other concepts and entities that are present in that text.

• "limit" : INT - required - The maximum number of concepts to extract from the ingested document.

emotion

The emotion enrichment evaluates the overall emotional tone (for example anger) of entire document or specified target strings in the entire document. This enrichment can only be used with English content.

• "document" : boolean optional - When true the emotional tone of the entire document is evaluated.
• "targets" : array optional - A comma-separated array of target strings of which to evaluate the emotional state within the document.

entities

The entities enrichment extracts instances of known entities such as people, places, and organizations. Optionally, a Knowledge Studio custom model can be specified to extract custom entities.

• "sentiment" : boolean - optional - When true, sentiment analysis is performed on the extracted entity in the context of the surrounding content.
• "emotion" : boolean - optional - When true, emotional tone analysis is performed on the extracted entity in the context of the surrounding content.
• "limit" : INT - optional - The maximum number of entities to extract from the ingested document. The default is 50.
• "mentions": boolean - optional - When true, the number of times that this entity is mentioned is recorded. The default is false.
• "mention_types": boolean - optional - When true, the mention type for each mention of this entity is stored. The default is false.
• "sentence_location": boolean - optional - When true, the sentence location of each entity mention is stored. The default is false.
• "model" : string - optional - When specified, the custom model is used to extract entities instead of the public model. This option requires a Knowledge Studio custom model to be associated with your instance of Discovery. See Integrating with Watson Knowledge Studio for more information.

keywords

The keywords enrichment extracts instances of significant words within the text. To understand the difference between keywords, concepts, and entities, see Understanding the difference between Entities, Concepts, and Keywords.

• "sentiment" : boolean - optional - When true, sentiment analysis is performed on the extracted keyword in the context of the surrounding content.
• "emotion" : boolean - optional - When true, emotional tone analysis is performed on the extracted keyword in the context of the surrounding content.
• "limit" : INT - optional - The maximum number of keywords to extract from the ingested document. The default is 50.

semantic_roles

The semantic_roles enrichment identifies sentence components such as subject, action, and object within the ingested text.

• "entities" : boolean - optional When true, entities are extracted from the sentence components.
• "keywords" : boolean - optional When true, keywords are extracted from the sentence components.
• "limit" : INT - optional - The maximum number of semantic_roles objects to extract (sentences to parse) from the ingested document. The default is 50.

sentiment

The sentiment enrichment evaluates the overall sentiment level of entire document or specified target strings in the entire document.

• "document" : boolean optional - When true the sentiment of the entire document is evaluated.
• "targets" : array optional - A comma-separated array of target strings to evaluate sentiment of within the document.

relations

The relations enrichment extracts known relationships between identified entities within the document. Optionally, a Knowledge Studio custom model can be specified to extract custom relationships.

• "model" : string - optional - When specified, the custom model is used to extract relations instead of the public model. This option requires a Knowledge Studio custom model to be associated with your instance of Discovery. See Integrating with Watson Knowledge Studio for more information.