Qualitative Inputs

Qualitative vs. Quantitative Inputs

Portfolio Navigator relies heavily on well-defined numeric inputs used in financial models. However, there are other inputs important in the decision-making process that are qualitative and descriptive.  These inputs are especially important in innovation and early stage projects for which a detailed financial model is premature.

• Some examples: “How well does this project fit with corporate direction?”, “How well are development tasks in technical, marketing, and production coordinated?”, “Do we have internal commitment to the implications of this project?”, “In what countries will the product be marketed?”.

Qualitative inputs for a project are a choice between discrete descriptive values.  These choices should be:

• Mutually Exclusive: You can choose only one value for a project.
• Collectively Exhaustive: Every project can be described by a value that appears in the list of possible choices.
  • Default Value: When a new project is created, what is the default value assigned? This is a surprisingly difficult question. If there is no appropriate default value, some way of dealing with “Value Not Yet Entered” needs to be defined.

There is a three-fold challenge in dealing with qualitative inputs.

• Definition: The input must be defined carefully. It should have enough clarity and detail that it is readily understood by all the people involved.
• Universality: The input must apply uniformly over all the projects in the analysis. The purpose of the input is to help compare and prioritize projects.
• Usefulness: There needs to be a way to summarize and analyze results to facilitate project comparisons. Defining a scale or level of measurement is often (but not always) possible.

Scale of Measure

If the qualitative inputs are to be more than meaningless words, they need to be tied to some measurement scale. “Scale of measure” or “level of measurement” is a classification that describes the nature of information within the inputs. One meaningful classification is the following:

• Categorical/Nominal: The inputs specify some attribute or (meta)-category. There is no meaningful order in the choices in the inputs, just membership in a group.
  • Example: “Region of Sales” possible values are “US”, “EU”, “US+EU”
• Ordinal: There is an order to the choices in the inputs.
  • Example: “Phase of Development” choices are “Phase I”, “Phase II”, and “Phase III”.  Phase II occurs after Phase I and Phase III after Phases I and II.
• Interval: A scale determines the difference between the choices. 
  • Example: “Year Development Phase I Terminates” might have choices of calendar year 2024, 2025, and 2028 – the difference between the 2024 and 2025 choices is 1 year and the difference between the 2025 and 2028 choices is 3 years.
• Ratio: The scale which determines the difference between the choices has a true 0. Very informally, many ratio scales can be described as specifying "how much" of something (i.e. an amount or magnitude) or "how many" (a count).
  • Example: “Duration of Development Phase I” could have choices of 1 year, 2 years, 4 years – the duration of the third choice is twice that of the second choice (4/2) and four times that of the first choice (4/1). Note that the ratios of the calendar year choices in the Interval example above are not very meaningful because of the lack of a true, non-arbitrary zero in the scale.

Portfolio Navigator

Portfolio Navigator and “Scale of Measure”

• In Portfolio Navigator, Categories function at the Categorical/Nominal level.  In the following example, Product A is marketed in Europe and Product B in the US. Note the implications of mutually exclusive (no product can be marketed in more than one of these choices) and collectively exhaustive (only four regions are possible).

              

• In Portfolio Navigator, most templates treat implicitly treat Phases as Ordinal choices.  The template model assigns values such as duration and cost to the phases, but the ordering is implicit.  Other ordinal inputs specify a project’s Maturity Matrix.  Occasional inputs in Portfolio Navigator occur in Scoring evaluations and Rate and Weight evaluations. These inputs sometimes start as Ordinal, but usually become Interval or Ratio inputs in later evaluations.
• In Portfolio Navigator, most inputs are numeric (2 years, $7 million, 30% probability, etc.) which can be regarded as the limit of Interval or Ratio choices. 

Assessment and Process

The process in assessing qualitative inputs:

• The context and definition of the variables needs to be succinctly defined. The assessments are usually stand-alone items, and it is important that everyone involved in the assessment share a common understanding.
  • In an analytic evaluation, the assessed values and their effects are easily tested and reviewed using sensitivity analysis (e.g., tornado diagram). This process contributes to clarity. These tools are not available for survey or questionnaire type assessments, so clarity in the definition is required for quality of the assessment.
• The form of the input should not predetermine the answer.  It is surprisingly easy to create input choices that predetermine the answers.
  • The effect of biases in assessment (discussed elsewhere) is exacerbated by the inclusion of data in the definition of the inputs and choices.
  • The number of choices provided, and their scale can affect the responses.
• Moving from Ordinal to Interval or Ratio inputs should be done with care.
  • Example:  Ordinal Cost inputs with choices “Low Cost” and “High Cost” can be better specified (e.g., “$0-200” and “>$200”) and then specified as an input (e.g., “$100” and “$350”) thus turning them into an Interval/Ratio measure.
  • Example: An innovative project may require an input “Market Segment” with choices “Current”, “Adjacent”, and “New”.  These choices may be ordered in terms of various measures like difficulty of entry, cost of new salesforce, potential profit, or probability of success. To move to an Interval/Ratio input, you need to choose a meaningful measure and then link the choices to values of the measure.

Design of Input

The design of input forms or questionnaires requires care.

• If the qualitative input is in the form of a few discrete choices, five to seven choices works well.  There should be an odd number of choices so that there is a central choice.
  • It is claimed that increasing the number of choices does not improve the quality of the assessment and may worsen it.
• Information on this topic and references to more material can be found at https://en.wikipedia.org/wiki/Level_of_measurement/ and https://conversionxl.com/blog/survey-response-scales/