Evaluating Quality in Qualitative Research

Carrying on from Ben’s ‘Levels/Typology of Evidence’ post, some recent interesting work on appraising qualitative research and evaluation has been afoot in the UK Cabinet Office (part of the UK. Govt.’s push to encourage evidence informed policy).

Given the importance of qualitative in addition to quantitative evidence for HIA, this UK document is a good introduction into the can of worms that is, ‘What is the quality in qualitative methods?’ Or in other words, given the vast amount of both well and poorly conceived and reported qualitative work out there, how do we choose what is the best available qualitative evidence to assist our HIA?

The original UK document can be downloaded from http://www.strategy.gov.uk/downloads/su/qual/index.htm

At 170 pages the report is (ironic given its subject matter!) very long, but here are its key elements, followed by some caveats and qualifications.

The suggested framework for considering quality in qualitative research is based around: (continue reading)

Four guiding principles - that the research should be:
– contributory in advancing wider knowledge or understanding;
– defensible in design by providing a research strategy which can address the evaluation questions posed;
– rigorous in conduct through the systematic and transparent collection, analysis and interpretation of qualitative data;
– credible in claim through offering well-founded and plausible arguments about the significance of the data generated.

And eighteen appraisal questions:

1. How credible are the findings?
2. How has knowledge or understanding been extended by the research?
3. How well does the evaluation address its original aims and purpose?
4. How well is the scope for drawing wider inference explained?
5. How clear is the basis of evaluative appraisal?
6. How defensible is the research design?
7. How well defended are the sample design/target selection of cases/ documents?
8. How well is the eventual sample composition and coverage described?
9. How well was the data collection carried out?
10. How well has the approach to, and formulation of, analysis been conveyed?
11. How well are the contexts of data sources retained and portrayed?
12. How well has diversity of perspective and content been explored?
13. How well has detail, depth and complexity (i.e. richness) of the data been conveyed?
14. How clear are the links between data, interpretation and conclusions – i.e. how well can the route to any conclusions be seen?
15. How clear and coherent is the reporting?
16. How clear are the assumptions/ theoretical perspectives/values that have shaped the form and output of the evaluation?
17. What evidence is there of attention to ethical issues?
18. How adequately has the research process been documented?

There are also some caveats and qualificationsthat need to be considered:

“The framework is designed to aid the informed judgement of quality, but not to be prescriptive or to encourage the mechanistic following of rules.”

What this means is that qualitative research is a value based exercise, and thus assessment of its quality is necessarily value based also. Use the framework to help you make a judgement about the research, but be flexible and don’t let the framework take over from your personal viewpoint!!

“Most of the items included in the framework are heavily recurrent in the wider literature.”

True, and for those interested it will be invaluable to check out the ‘The bible’ of qualitative methods: Denzin, N. K. and Lincoln, Y. S. (Eds.) (2000). Handbook of qualitative research. 2nd edition. Thousand Oaks, CA: Sage. In particular look at some of the earlier Chapters which go into vital values and perspectives stuff. It will also be useful to search out journals such as ‘Qualitative Health Research’, of ‘Qualitative Inquiry’, and ‘Qualitative Research’. Look directly at how the equivalent of ‘Methods’ and ‘Results’ sections are reported and try to put the criteria to the test!

“Conceptions of quality are influenced by the various philosophical assumptions which underpin different approaches to qualitative research. These epistemological and ontological positions are diverse and span issues such as the nature of reality, the relationship between the researcher and the researched, the relationship between facts and values, the nature of knowledge, and appropriate methods of research.”

Long winded, but true – see the two points above on how to dissect this statement.

Happy hunting!!

By Patrick Harris, CHETRE