Hierarchy of Evidence
- Meta analysis
- Systematic reviews
- Cohort Studies
- Case-control studies
- Case studies / reports
- Background Info / Expert Opinion
“Process of systematically locating, appraising and synthesising evidence from scientific studies in order to obtain a reliable overview” - Helen Worthington
“Systematic reviews aim to identify, evaluate and summarise the findings of all relevant individual studies, thereby making the available evidence more accessible to decision makers.” - CRD (2009) p.V
“A systematic review can be defined as “a review of a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyze and summarize the results of the included studies.” - http://www.cochrane.org/glossary
Why might a review be needed?
Is there a need to summarise existing information using an unbiased method?
Are more general conclusions possible to be drawn from a SR over multiple single studies?
Is evidence needed to inform a research project or answer previous unanswered questions in a topic area?
Points to Consider Beforehand
Why is this question being asked?
Do we already know the answer e.g. what other evidence is there related to this?
Is a review the best method?
Does the research question now need to be refined?
Narrative Review vs Systematic Review
- Often more broad, less or undefined method of searching, critiquing and synthesising data (unlikely have meta analysis)
- Focused research question, rigorous method, searching, appraising, critiquing and then synthesising data (qual and quant)
Pros and Cons
- Reduces Bias
- Can provide understanding of conflicting findings across studies
- Helps inform decision making
- Can combine data from studies to perform a meta-analysis to determine effect of interventions
- Can still lead to inconclusive results and recommendations for more evidence
- May have an empty or very limited review
- Study quality may be poor
- Can take a long time for evidence to impact practice/policy
Has the review or a similar review already been conducted?
- If so when/where – is it in need of updating/different location?
Has it been conducted using one method but there is scope to use a different one (e.g. qual/quant or review of reviews)
Are there likely to be studies in the review
Has the title or a similar study been registered as a protocol (e.g. on PROSPRO)
Provides a framework – ensure transparency and reproducibility
Provides a brief background and rationale for the review
Describes how papers will be identified, selected and evaluated
Sets out the objectives
Sets out the selection criteria
- What types of studies and why?
- Types of participants and rationale
- If relevant types of interventions and comparisons –> what criteria must they meet (e.g. setting, time, who deliver)
- Types of outcomes measures
- Describe how studies will be identified, selected and evaluated
What is the title of your review?
- should be a statement not a question
Who are the participants/target audience your interested in and why?
- (e.g. age, gender, ethnicity, pre existing conditions)
Intervention/programme characteristics of importance?
Comparisons interested in? (e.g. intervention vs. control)
Outcomes likely to be used within papers and interested in?
Any studies you know of that would fit in the review?
Background and rationale for the study
- key words, data bases with years, any scoping
- inclusion & exclusion, how criteria will be applied how ensure bias is kept to a minimum
- how will this be checked and why that tool
- how will the details needed to answer research objectives be taken from studies
How synthesis planned
- any expected analysis and type of analysis (qual or quant or meta analysis), any sub groups interested in
What and who you are aiming to contribute to through conducting the review
A provisional timeline
Should relate to specific patient/topic problem
Likely to be linked to an intervention, aetiology, diagnosis or prognosis
E.g To estimate the accuracy of … or To determine the effectiveness of …
May be helpful to use the PICOC acronym
- Outcome/evaluative measures
Other acronyms include:
- PEO, SPICE, SPIDER, PICo, PEO
Is it a broad or narrow question?
- e.g. Do behaviour change interventions work in reducing back pain in office workers?
- Is CBT effective in treating moderate depression in females aged 30-40 years old?
- Formulate question
- Develop search strategy
- Conduct comprehensive search
- Title and abstract screening - using predesigned questions
- Full text screening of possible included papers and unclear papers
- Final set of papers obtained
- Quality assessment of papers and data extraction
- Synthesis of data
Piece 1: Develop and Conduct comprehensive search
- MeSH (Medical Sub-Headings)
- Subject Headings
Piece 2: Screen Titles, Abstracts and Full Papers
- Inclusion and Exclusion criteria
Websites to explore:
Systematic Review vs Non Systematic Review
- Description of how data will be synthesised
- It is objective and reproducible
- The question is clearly defined
- Criteria for inclusion of studies are stated
- The literature search is comprehensive
- Studies included are appraised for quality
- Provides an overview or update on a topic
- Criteria for inclusion are not stated
- It may be a subjective view from an expert
- No appraisal criteria are stated
- The search is not explained and/or is limited
- There is no systematic synthesis
Different Types of Systematic Reviews
|SR with or without meta analysis||Qualitative/Narrative reviews||Review of reviews||Behavioural intervention reviews||Rapid SR|
|Aims to reduce bias||Focus on synthesis of findings||Combine outcomes multiple reviews||Aim to identify aspects to improve int. design||Streamlined approach|
|Follow standard structure||Look at experience/implementation||Search limited to other types of reviews||Tend to look at how can combine multiple behaviours||Evidence more timely|
|Best available evidence for a specific question||More likely to include grey literature||Helps provide overarching review of evidence||Same as SR process with specific requirements||May limit sources, search|
|Synthesis data from multiple studies||Themes identified across studies||Used to synthesis findings for evidence briefs||Influenced by literature and need to streamline interventions||More descriptive summaries, categories|
|Transparent||Can help provide a deeper understanding of an area||Can combine meta analysis||May involve multiple searches of different behaviours||Findings more limited/cautious|
|Detail procedure in advance||Aid dialogue, reflection and use of narrative||Aims to maintain rigour|
|Explicit search, inclusion/exclusion criteria||No minimum reporting standards|
|May include meta analysis||Come out of clinical/tech/time requirements|
SR With and Without Meta-Analysis
Qualitative/Narrative systematic reviews
Review of reviews
Behavioural intervention reviews
NOTE - If you wanted to undertake this type of review for your dissertation you would need to first speak to you supervisor due to their complexity and more advanced nature of the review.
Rapid systematic reviews
NOTE – this type of review is NOT suitable for your dissertation.
Steps of Review Process
Why do we do Data Extraction?
Studies are often published in different formats and using different reporting styles
Allows information needed to be standardised, helps us know about the main elements of interest, can help us start to see patterns and things relevant for analysis.
Aims to extract the findings from each study in a consistent way to allow synthesis
Once complete, you should require minimal reference to the original paper during data synthesis
Data Extraction Forms
By this stage, you will have identified which papers contain evidence relevant for your review question.
Process of taking relevant data from included papers and putting it into a single usable format.
All forms likely to be different in some way – needs to work for you and your review (what data do you want to extract).
Think of the order of information in articles when designing the form (tip record p. no for ease of referencing).
Interested in two things – the descriptive data and the analytical outcome data
What you extract will be informed by your research question/aim
CRD on Data Extraction
“The ﬁrst stage of any data extraction is to plan the type of analyses and list the tables that will be included in the report”
“Standardised data extraction forms can provide consistency in a systematic review, whilst reducing bias and improving validity and reliability”
“The form should be unambiguous and easy to use in order to minimize discrepancies.”
Things to consider for data extraction
- What is your objective? What do you want to know?
- What was your inclusion criteria?
- What was the intervention? (time, session, setting, content etc)
- Evaluation tools and outcomes.
- Adverse events
- Participant characteristics
Suggested Sections of the Form
- Methods – design, length intervention or study, groups, blinding, attrition rates etc
- Participants – n= for control and intervention, age, geographical location, gender, setting, inclusion/exclusion criteria etc
- Intervention/main area of interest – design/pilot, intervention description, target, delivered by, involvement, aspects relating to specific factors etc
- Outcomes – Key statistics of interest or the quotes and interpretations, conclusions etc
- Notes – any linked papers or background papers, sponsors etc
Developing your data Extraction Tool
“A systematic research synthesis is attempting to answer a question using the findings of research evidence. It is therefore important that the evidence is both trustworthy and relevant.” EPPI (2010)
“Recording the strengths and weaknesses of included studies provides an indication of whether the results have been unduly inﬂuenced by aspects of study design or conduct (essentially the extent to which the study results can be ‘believed’). Assessment of study quality gives an indication of the strength of evidence provided by the review and can also inform the standards required for future research.” CRD (2009)
Risk of Bias and Quality Assessment
Taken from Cochrane review training:
- Quality = “did they do the best they could?”
- Bias = “should I believe the results?”
Consider using a pre-validated tool, for example
- Cochrane risk of Bias tool
Tools likely to consider
- Appropriateness of study design
- Choice of outcome measure and quality of reporting
- Quality of intervention delivery and reporting
Choosing a Tool - Considerations
- Study design – qual/quant, RCT etc
- Level of detail required
- Ease of the tool and generalisability of tool
- Previous use of the tool in similar style studies
- May need to use a batch of tools if different study types
- CASP Randomised Controlled Trial
- CASP Systematic Review
- CASP Cohort Study
- CASP Case Control Study
- CASP Qualitative Research
- CASP Economic Evaluations
- CASP Diagnostic Test
- CASP Clinical Prediction Rule
Example Qualitative Tools
CDR critical appraisal for reviews
- “Was the review question clearly deﬁ ned in terms of population, interventions, comparators, outcomes and study designs (PICOS)?
- Was the search strategy adequate and appropriate? Were there any restrictions on language, publication status or publication date?
- Were preventative steps taken to minimize bias and errors in the study selection process?
- Were appropriate criteria used to assess the quality of the primary studies, and were preventative steps taken to minimize bias and errors in the quality assessment process?
- Were preventative steps taken to minimize bias and errors in the data extraction process?
- Were adequate details presented for each of the primary studies?
- Were appropriate methods used for data synthesis? Were differences between studies assessed? Were the studies pooled, and if so was it appropriate and meaningful to do so?
- Do the authors’ conclusions accurately reflect the evidence that was reviewed?”
- To provide added value beyond just descriptive mapping and reporting and providing a bibliography
- Provide a synthesis of the studies = collation, combination and summary of individual studies into one analysis
Looking to work through three steps:
- The analysis
- Producing the narrative, numerical, thematic or graphical presentation
- Finalising the report
Provide a PRISMA Flow diagram
Use tables to present data e.g. table of included studies, to make it easier that lots of text for the reader
For your dissertation - Provide an extract of your title and abstract screening process in the appendices
Present initial scene setting data e.g. countries and number of participants
Provide a synthesis of the outcomes and analysis
Presenting your results in a written manner (but not just repeating what is in tables)
Used to report overall summarised findings and explained in words
Should help you present similarities and differences in your results
It is not a meta-analysis: narrative synthesis can support a meta-analysis or be used when meta-analysis is not possible
In this section you are only presenting (not discussing) your results
Results and Data Synthesis
- Results of studies
- Summary assessment methodology and interventions
- Summary of results
- Meta analysis if possible
- Aims to look for consistencies and differences in qual. literature
- Can identify gaps in themes
- Steps in analysis can be less prescribed than qualitative reviews
- May use framework analysis or thematic analysis
Discussion and Conckusion Points to Consider
- Adapted from Boland, Cherry & Dickson (2014)
- Did you find the evidence you thought your would?
- Were you able to answer your objectives
- How do your findings fit with previous literature
- Did the studies have any particular strengths and weaknesses
- Did the review process have any particular strengths and weaknesses
- How generalizable are the findings
- What conclusions are you able to make from your review
- What recommendations and what are the implications of your review for practice, future research and policy?