Measuring the Efficacy of EORE: Odds Ratios, and an Evidence-Based Solution

By Dr Robert Keeley, RK Consulting (EOD) Ltd

In a recent publication by the Geneva International Centre for Humanitarian Demining (GICHD), it was noted that there is still a need to develop evidence-based methods to assess the effectiveness of explosive ordnance risk education (EORE).1

The aim of this paper is to set out one such method, drawn from the discipline of epidemiology, that has been applied in two different mine action programs. This method is called a case-control study, and this paper will demonstrate how this objective, quantitative, and established epidemiological tool can be utilized in EORE, especially as part of a knowledge, attitude, and practice (KAP) study.

BACKGROUND: FOCUS GROUP DISCUSSIONS AND THE “STATED PREFERENCE PROBLEM

Currently, it seems that most EORE projects do attempt to assess the efficacy of their work, but do so in a qualitative manner, often via focus group discussions (FGD). FGD are a qualitative technique and as such provide context and assist researchers with understanding the nature of the issue they are studying. However, the problem with FGD and other such face-to-face interviews is that there is a significant risk of the respondent providing what they think is the right answer. This is known to economists as a stated preference problem. See Figure 1.2

Figure 1. Definitions of revealed and stated preferences.3

BACKGROUND: EPIDEMIOLOGY

The science of epidemiology dates to the 1854 London cholera epidemic and the work of John Snow4 to identify the source of the outbreak. The World Health Organization (WHO) definition of epidemiology is set out in Figure 2.5

Using this definition and recognizing an injury from mines and other explosive ordnance (EO) as a health- related event, it is appropriate to apply relevant epidemiological tools to interpreting EO casualty data.

Figure 2. The World Health Organization definition of epidemiology.

CASE-CONTROL STUDIES

While the principles behind case-control studies were being discussed in the nineteenth century, the first modern case-control study was Janet Lane-Claypon’s study of breast cancer in 1926. The next major use was in 1950, when four published case-control studies linked smoking and lung cancer.6 Case-control studies have since become an established study design in epidemiology and the social sciences.

A case-control study is a type of observational study commonly used to look at factors associated with diseases or outcomes.7 The case-control study starts with a group of cases, which are the individuals who have the outcome of interest. The researcher then tries to construct a second group of individuals called the controls, who are similar to the case individuals but do not have the outcome of interest. The researcher then looks at historical factors to identify if some exposure(s) is/are found more (or less) commonly in the cases than the controls.

Odds ratios and protective effect. The key means to analyze results in case-control studies is the odds ratio (OR). The OR is the odds of having a disease (or outcome) with the exposure versus the odds of having the disease without the exposure.

The OR tells us how strongly the exposure is related to the disease state. An OR of greater than one implies the disease is more likely with exposure. An OR of less than one implies the disease is less likely with exposure and thus the exposure may be protective.

This paper will now demonstrate, using hypothetical data, how a case control study can be used to assess the protective effect of EORE.

BUILDING A CASE-CONTROL STUDY FOR EXPLOSIVE ORDNANCE RISK EDUCATION

Case-control studies are retrospective: the researcher works backward in time, using existing data sets for casualties. The best way of understanding how one is built is by following the steps outlined in Figure 3.

Figure 3. The structure of a case-control study.

Step 1. Identify the target population

The target population can be the entire population of a country, or perhaps just one province, or indeed any other subgroup. If the case-control study is only conducted in one province, it may be difficult to draw conclusions about other provinces, but of course the bigger the study, the more resources are required.

Step 2. Identify cases

This is easier in countries with existing victim information systems (VIS) but beware if the VIS has not been routinely updated. Extra time might be needed to work with community leaders to find additional cases if those in the VIS have since died or moved out of the community.

It is recommended that a study selects cases by age and gender in the same ratio as they appear in the overall VIS. For example, if 70 percent of the casualties in the target country are adult males, ensure that 70 percent of the cases selected are adult males. See Figure 4.

Figure 4. Identifying controls. The control group (right) must reflect the demographics of the cases (left) and should also reflect the overall demographics of casualties in the country.

Step 3. Identify controls

Controls – people who are not EO casualties – should be selected as randomly as possible but the overall control group must reflect the same demographic makeup as the cases. It’s important to ensure that the controls are as identical as possible to the cases in terms of age, gender, and location. This will help remove any potential confounding variables.

Step 4. Interview cases and controls

Once the cases and controls have been identified, they can be interviewed. For the purposes of the case-control study, there is only one question to ask: “Did you receive EORE training (before your accident)?” Of course, other questions relating to a wider KAP study can be asked at the same time.

There are a number of survey ethics questions which must also be addressed at this time. Firstly, the interviewees must give informed consent. Many academic institutions require a signed declaration to this extent which can be problematic in places with low literacy rates. In such circumstances it can be possible to seek oral permission with a community leader signing to witness that consent was given.

Secondly, there is a risk of disturbing a casualty who is suffering from post-traumatic stress disorder (PTSD). Data-gatherers must be trained on how to conduct the interviews in a comfortable location, and to recognize and react to symptoms of PTSD.

Finally, there is the question of interviewing children. It is unlikely that a group of data gatherers conducting the interviews will be trained in dealing with child casualties, and under these circumstances it will be better to find adult survivors who were children at the time of their accident.

CALCULATING THE ODDS RATIO

The interviews will identify four groups of people: cases who received EORE (before their accident); cases who did not receive EORE (before their accident); controls who received EORE; and controls who did not receive EORE. It’s common to set out these data in an exposure matrix, and here they are represented by the letters a, b, c, and d as shown in Table 1.

Table 1. Exposure matrix example

The table has been populated with some hypothetical results. The OR is then calculated by using the following equation:

Figure 5. OR Equation.

Substituting the hypothetical results set out in Table 1, this gives a result for the OR as 0.32 as shown in Figure 6.

Figure 6. Calculating the odds ratio.

It is also possible to use an online odds ratio calculator8 to do this calculation. See the example in Figure 7. This can help prevent errors, but it is also useful for assessing the statistical significance of results. An explanation of the statistical tests automatically done by the calculator can be found online as they are too long to explain here.9


Figure 7. Using an online odds ratio calculator.

What if there is a problem with statistical significance? The most likely reason for an issue may be a situation where the sample size was too small to be confident in a very small protective effect. It’s possible to have case-control studies with very small samples, but wherever resources allow, the larger the sample the better.

REAL-WORLD RESULTS

This method has been tried twice by the author: the first in Cambodia in 2020 and the second in Laos10 in 2023. The results are set out in Table 2.

Table 2. Real-world results.

The odds ratio calculations for both studies are set out in Figure 8.

Figure 8. Odds ratio calculations for Cambodia (left) and Laos (right).

Whilst both demonstrated a protective effect (i.e., where the OR < 1), the calculator shows that the results for Cambodia were not statistically significant. This can be attributed to the relatively small sample size. The lesson learned from this is that for EORE, a suitable minimum sample size would be around 120 in total.

ADVANTAGES AND DISADVANTAGES OF CASE-CONTROL STUDIES

Case-control studies have several advantages and disadvantages:

Advantages of case-control studies. Firstly, case-control studies are less susceptible to the stated preference system commonly found within the mine action sector.
Secondly, case-control studies can be conducted in situations of rare events, i.e., with relatively small sample sizes compared to other quantitative methods.

Disadvantages of case-control studies. The main disadvantage with case-control studies in the context of EORE is that they are a retrospective study. As such it is hard to use them in very new conflict settings. Similarly, it’s harder (but not impossible) to conduct a case-control study in circumstances where there is no functioning VIS. Under such circumstances it’s necessary to allow more time to seek out cases by making inquiries amongst local officials.
Another disadvantage – though more of a limitation – is that the results are aggregated. The study might reveal that EORE in a particular province is effective, but on its own it won’t reveal which vector for EORE has been the most (or least) effective. This is why it’s useful to embed a case-control study within a KAP so that other research methods can be used to provide other insights.

CONCLUSIONS

Case-control studies have been used for nearly a century to examine the quantitative relationship between exposure to a particular variable and a medical outcome. As such, they are a standard epidemiological tool, are not particularly complicated, and can easily be incorporated into a KAP. Furthermore, they produce objective, quantitative results, unlike FGD.

However, based on the examples set out in this paper, case-control studies require a minimum sample size (at least 120, divided equally between cases and controls). Also, in regions where there is more than one means to deliver EORE, a case-control study cannot by itself inform on the relative efficacy of those different EORE means. Using the case-control study as part of a wider KAP can resolve this.

Anyone wishing to know more about how to conduct a case-control study is welcome to contact the author. ■

Footnotes:
  1. Explosive Ordnance Risk Education (EORE): Sector mapping and needs analysis, Geneva International Centre for Humanitarian Demining, 14 October 2024, https://bit.ly/40riLSS.
  2. Wolffram, Elsner, Torsten Heinrich, and Henning Schwardt. 2015. “Critiques of Neoclassical “Perfect Market” Economy and Alternative Price Theories.” In The Microeconomics of Complex Economies. University of Bremen. https://bit.ly/4fU0kez.
  3. Ibid.
  4. Begun, Fahema. Mapping disease: John Snow and Cholera. 9 December 2016. Royal College of Surgeons of England. https://bit.ly/3ZzmpJJ.
  5. The Global Health Observatory. https://bit.ly/3OVT8Tr.
  6. Paneth, Nigel & Susser, Ezra & Susser, Mervyn. (2004). Origins and early development of the case-control study. 10.1007/978-3-0348-7603-2_16.
  7. Steven, Tenny, Connor C. Kerndt, Mary R. Hoffman. “Case Control Studies.” National Library of Medicine. https://www.ncbi.nlm.nih.gov/books/NBK448143/
  8. MedCalc Software Ltd. Odds ratio calculator. https://www.medcalc.org/calc/odds_ratio.php (Version 23.0.5; accessed October 17, 2024)
  9. Frost, Jim. “Odds Ratio: Formula, Calculating & Interpreting. Statistics by Jim. https://bit.ly/3Zw2OtN.
  10. Knowledge, Attitudes, and Practices Study.” 15 October 2024. UNDP, Lao PDR. https://bit.ly/3OyCEjI.

     

ABOUT THE AUTHOR

Dr. Robert Keeley is a former British Army Bomb Disposal Officer active in humanitarian and commercial mine action and explosive ordnance disposal (EOD) since 1991. He has worked in numerous countries and for several governments and international organisations. He specializes in project design, evaluation, and quality assurance of all aspects of mine action and has helped shape the emerging humanitarian improvised explosive device (IED) sector. Dr. Keeley is a member of the Institute of Explosives Engineers, a Member of the International Association of Bomb Technicians and Investigators, and is a Fellow of the Royal Geographical Society. He holds a PhD in Applied Environmental Economics; his thesis was on “the Economics of Landmine Clearance.”
Contact Information
Dr. Robert Keeley, Director, RK Consulting (EOD) Ltd Ashford, Kent, United Kingdom
[email protected]
www.rk-consulting.net

Download PDF: 71-76 Robert Keeley article – Measuring the Efficacy of EORE – COUNTER-IED REPORT, Autumn 2025

Counter-IED Report, Autumn 2025