How do you support your decision making when dealing with uncertainty, unprecedented events or lack of adequate data?
Despite its increased accessibility, data is often insufficient and can therefore impact quantitative assessments regarding future events, risks or opportunities. In this course you will learn to apply a tried and tested tool to use expert opinions to quantify uncertainty.
Any decision or risk analysis can benefit from the use of expert judgement in cases when:
- a decision process is impacted by substantial uncertainty. Validated expert opinions can supplement existing information in projects where the quantification of parameter uncertainty is crucial;
- data is incomplete, uninformative or conflicting. Historical data, used by data-driven models for prediction, is not always sufficiently representative of the problem at hand, especially in highly volatile settings;
- you need to build rational consensus. That is, you acknowledge the diversity of expert opinions and want to use a rigorous method that objectively evaluates and aggregates diverse expert assessments.
Structured Expert Judgement (SEJ)
SEJ provides a transparent and performance-driven approach to using expert opinions in decision-making processes. It gives you an accessible and easy-to-use tool to forecast the uncertainty of specific quantities of interest, enabling you to evaluate and aggregate the uncertainty assessments offered by a pool of experts.
This rigorous and mathematically sound method can be used by any professional interested in supporting the assessment and prediction of risks and opportunities, from portfolio analysts and consultants to risk analysts and managers.
In this course you will learn how to use SEJ in a specific project. You will be able to use your own case and receive personalized support and feedback throughout.
After taking this course you will be able to:
- use SEJ to quantify uncertainty to support your decision making
- design a complete expert judgment process
- carry out a structured protocol during which experts provide their assessments and rationales
- analyse expert data and produce a report of the findings
The method taught in this course was developed at TU Delft by Roger Cooke. It has been used by institutions and companies such as NASA, RIVM, WHO, Airbus, Shell and KLM, and a Procedure Guide has been published by the European Commission.
Overall, the SEJ method has been used for over 30 years in areas as diverse as disaster management, epidemiology, public health, safety, aeronautics/aerospace, finance, risk management, climate change, environment, engineering, and many others.
In practice: Predicting electricity prices
Electricity prices are important to suppliers due to their purchase strategy. Portfolio managers buy and sell electricity on the spot market, but they can also buy volumes of the commodity years in advance on the exchange or over the counter.
“Forecast models for electricity prices are therefore of great interest to portfolio managers. However, the considerable uncertainty of these models – due to the extreme price volatility caused by the economically non-storable nature of electricity, the constant balance between consumption and production, weather effects and other factors –, presented challenges to predict average day ahead spot prices for 2025, 2030 and 2035. Our strategy was to combine data-driven models with expertise of traders using SEJ to subject the process of soliciting such expert advice to a transparent, traceable and validated methodology.” – Ashni Bachasingh, Electricity Portfolio Analyst
Is this course for you?
This course is addressed to both professionals and researchers with any academic background who consider applying structured expert judgment to their field. Performing an entire structured expert judgment study will be your focus throughout the course.
How Structured Expert Judgment works
The main aim of the SEJ method is to quantify uncertainty. It therefore measures the performance of experts as uncertainty assessors. This method asks experts to quantify uncertainty both with regard to questions of interest, and for questions that are used to objectively evaluate experts’ own performance, known as calibration questions. Through the (elicitation) process, experts’ assessments are used to compute two complementing performance measures that infer on experts’ statistical accuracy and informativeness.
In constructing the aggregated model, experts are treated as statistical hypotheses. Their assessments are combined in such a way that the statistical accuracy and informativeness of the aggregated model, known as the decision maker, is maximised. Finally, the decision maker is used to obtain the estimates and uncertainty intervals for the desired quantities of interest.
The SEJ method is built upon a number of principles, including scrutability (all data and all processing tools are open to peer review and results must be reproducible by competent reviewers); fairness (experts are not pre-judged); neutrality (methods of elicitation and processing do not bias results); and performance control (quantitative assessments are subjected to empirical quality controls).
Part 1 - Design the elicitation process
In the first two weeks, you will choose your topic (from your own context or from an available list of topics) and define the questions of interest. You will discuss data sources for calibration questions and develop your elicitation process. You will be supported in designing the questions of interest and the calibration questions. Feedback on pre-selected questions will be discussed in one-to-one meetings with the course team. The course team will also advise on preparing practical details such as contacting the experts, setting up the process and the format of the process, ensuring that experts are given sufficient background information, etc.
Part 2 - Expert elicitation
In this week, you will perform the expert elicitation.
Part 3 – Data Analysis & Reporting
In the last two weeks, you will use Excalibur, a software developed at TU Delft, to analyse with your expert data and report your findings. You will share and discuss preliminary results with your peers. You will be given supporting information on how to analyse the data and what is important to include in your reporting. And you will receive very detailed and personalized feedback on your reports.
If you successfully complete this course you will earn a professional education certificate and you are eligible to receive 2.0 Continuing Education Units (CEUs).
This course is primarily geared towards working professionals.
- Basic concepts in Probability Theory and Statistics. To support your learning, you will have access to videos introducing these concepts.
If you have any questions about this course or the TU Delft online learning environment, please visit our Help & Support page.