Learn to apply the most rigorous method to support your decision making in the absence of data and under uncertainty. In this course you will be able to gather, assess the performance of and combine expert opinion for your own study.
Structured Expert Judgment (SEJ) is a technique that enables you to appropriately account for uncertainty when there is no data or no appropriate data available. An increasing number of organizations are making use of panels of experts in order to provide assessments over key uncertainties. Since the use of multiple experts in decision making results in a significant reduction in risk, SEJ has become a critical part of their decision-making processes for complex issues.
But how can data be properly elicited from qualified experts? And how can we guarantee that experts' assessments of uncertainty are evaluated as objectively as possible?
In this course you will learn to design a successful expert judgment study using the most rigorous and mathematically sound method known as the Classical Model (CM). This method has been used by institutions and companies such as NASA, RIVM, WHO, Airbus, Shell and KLM, to support their (data-driven) decision-making.
"The choice is not whether to use expert judgment, but rather to do it well or do it badly."
A correct application of this method requires sound knowledge and often external support to, for instance, perform an expert elicitation or evaluate and combine experts' assessments.
This course is aimed at professionals who wish to learn all the practical aspects of the CM and perform structured expert judgment in a specific project: from gathering and analyzing expert data, to developing and implementing your own expert elicitation protocol and carry out the analysis and report the findings of the study.
Together we will work through an elicitation study using CM, and you will be able to use your own case to design your own expert judgment study.
After taking this course you will be able to:
- Perform a study within your own context, using CM
- Design a complete expert judgment elicitation
- Perform an expert judgment elicitation
- Analyse expert data gathered during elicitation and develop a report with findings
Decisions made with limited or contradictory information, or in volatile settings due to economic or health crises, can jeopardize the success of your projects. In this practical course you will learn how expert opinion can be used for uncertainty quantification in a rigorous manner.
The Classical Model (CM) or Cooke's method, developed at TU Delft by Roger Cooke, has been used for over 30 years with successful applications in areas as diverse as disaster management, epidemiology, public and global health, aeronautics/aerospace, finance, risk management, civil engineering, safety, climate change, environment and ecology, engineering and many others.
Is this course for you?
This course is the advanced version of our MOOC, which will allow you to design and carry out your own study. It builds on the knowledge that you gained (or already have) about The Classical Model. The MOOC is not a mandatory prerequisite but previous knowledge of the CM is an advantage. If you did not participate in our MOOC, support material will be provided throughout the course. Performing an entire structured expert judgment study will be your focus throughout your course activity.
This course is addressed to both researchers and professionals from any academic background who consider applying structured expert judgment to their field by using the most mathematically rigorous method available.
Part 1 - Design the elicitation protocol
In the first two weeks you will choose your topic (from your own context) and define questions of interest. You will discuss data sources for calibration questions and develop your elicitation protocol.
Part 2 - Expert elicitation
In this week you will perform an expert elicitation study.
Part 3 – Data Analysis & Reporting
In the last two weeks you will analyze with your expert data, in Excalibur, and report your findings.
If you successfully complete this course you will earn a professional education certificate and you are eligible to receive 2.0 Continuing Education Units (CEUs).
This course is primarily geared towards working professionals.
- Basic concepts in Probability Theory and Statistics (links to videos introducing the concepts will be provided).
- Introductory concepts of CM (links to relevant videos or materials from the MOOC will be provided).
If you have any questions about this course or the TU Delft online learning environment, please visit our Help & Support page.