Chain of thought reasoning

Chain of thought (CoT) reasoning is a different approach to generating answers that is available with the OpenAI gpt‑oss‑120b option. With CoT, watsonx BI generates more thoughtful responses with step-by-step reasoning, which allows it to handle complex and more nuanced questions that are beyond simple queries.

Why use CoT

CoT's structured answers, which reflect the AI’s reasoning path, are helpful when you want to see how watsonx BI built the query or queries that were necessary to answer your question.

Another advantage of using CoT is that it can answer complex or multi-part questions that might require multiple queries or visualizations. An example of a multi-part query that benefits from CoT reasoning is Show me how many units of the Extreme brand were sold in Canada by product, and identify which countries sold more units.

This question spans multiple dimensions and regions and cannot be answered in a single chart. Instead, it requires two distinct visualizations:

  • Total quantity of Extreme brand sold by product in Canada

  • A comparison of units sold by country

CoT reasoning ensures that the LLM interprets the relationships, organizes the data accurately, and presents the results in a coherent, structured way.

How CoT works

When a response is generating in watsonx BI, you can view the reasoning steps that are in progress. You can also click Reasoning... to view more details about how chain of thought is deriving the answer.

After the response is generated, you can view the reasoning steps and explanation by clicking Show AI steps.

Detailed reasoning displays in the AI steps panel
Detailed reasoning displays in the AI steps panel

Reasoning steps and explanation can include:

  • An interpretation of the question and how the LLM decided to answer.

  • The data source that the AI chose to answer the question. The LLM picks the best asset it thinks can answer the question, but the option is available to select an alternate asset if other choices were found.

    You might see a repetition of this step if the LLM decides that the asset chosen cannot answer the question effectively and goes back to find other assets that can.

  • Intermediate calculations and filters that feed into the reasoning.

  • The exact SQL that was generated and executed.

  • Live data for the query, which allows you to see the same logic replayed on the current data state. This means that the live data might differ from the original data used if the underlying data has changed.

A key differentiator with CoT is that the steps needed to answer a question might vary. You see only those steps that were required to answer the question and you might see the same step more than once if watsonx BI determines that it needs to re-evaluate a previous choice.

Additionally, CoT can also reuse information from a previous question.

With CoT, watsonx BI also determines the best way to present the results as a visualization, a text response, or both.

Disambiguating filter names and values

Sometimes a question can be ambiguous, meaning more than one column could match your request. For example, if you ask for sales and the table includes SALES_COUNT and REVENUE, or if you ask about activity in 2025 and the table includes SALE_DATE, SHIPMENT_DATE, and PAYMENT_DATE.

In these cases, you can check which column the AI selected in the reasoning steps. If the choice isn’t what you intended, ask the AI to reconsider. For example, if you ask, How many units were sold in 2025? and the AI uses SALE_DATE, you can correct it by saying:

  • What about by payment date?

  • Use payment date instead of sale date

  • I mean payment date, not sale date

You can also include more context in your original question:

  • How many units were paid for in 2025?

Current response behaviour

CoT introduces advanced reasoning capabilities, and its responses might differ from those of other LLMs in the following ways:

  • Insights with visualizations - CoT does not generate insights alongside charts.

  • Chart customization - You cannot customize chart attributes through conversation but you can ask for a chart type. For example, you can ask watsonx BI to Create a bar chart with the sales trend for this year.

  • Response presentation - The way responses display can vary based on the AI’s reasoning. For example, the AI determines whether to include a chart, which chart type best answers your question, and how the accompanying text is structured.

  • Visualization titles - Titles are generated automatically as part of AI reasoning.

  • Data availability question - When you ask What data do I have available to me?, CoT provides a more detailed response. It provides a list of metrics and tables in the project, along with descriptions for each to help you understand your data context.