Systematic Review 学习（二）

1 The Review Process

1.1 Planning the review（策划评审）

• Identification of the need for a review 确定需要进行审查

• Commissioning a review 委托进行审查

• Specifying the research question(s) 指定研究问题

• Developing a review protocol 制定审查协议

• Evaluating the review protocol 评估审查协议

1.2 Conducting the review（进行评审）

• Identification of research 鉴定研究

• Selection of primary studies 初等研究的选择

• Study quality assessment 学习质量评价

• Data extraction and monitoring 数据提取和监测

• Data synthesis 数据综合

1.3 Reporting the review（汇报评审）

• Specifying dissemination mechanisms 指定传播机制

• Formatting the main report 格式化主报表

• Evaluating the report 评估报告

 

我们认为上述所有阶段都是强制性的，但是：

• Commissioning a review which depends on whether or not the systematic review is being done on a commercial basis.

委托进行审查，这取决于系统审查是否是在商业基础上进行的。

• Evaluating the review protocol and Evaluating the report which are optional and depend on the quality assurance procedures decided by the systematic review team (and any other stakeholders).

评估审查协议和评估报告这两项是可选的，取决于系统评审组（和任何其他利益攸关方）决定的质量保证程序。

 

上面列出的阶段看起来可能是连续的，但重要的是要认识到许多阶段涉及迭代。特别是，许多活动实在协议制定阶段发起的，并在审查本身进行时加以改进。比如：

• The selection of primary studies is governed by inclusion and exclusion criteria. These criteria are initially specified when the protocol is drafted but may be refined after quality criteria are defined.

根据包容和排除标准选择的初级研究。这些标准最初是在起草协议时指定的，但可以在定义质量标准之后加以改进。

• Data extraction forms initially prepared during construction of the protocol will be amended when quality criteria are agreed.

在协议制定过程中最初编写的数据提取表格将在商定质量标准之后进行修订。

• Data synthesis methods defined in the protocol may be amended once data has been collected.

协定书中规定的数据综合方法可在数据收集完毕后加以修正。

 

2 Planning

Prior to undertaking a systematic review it is necessary to confirm the need for such a review. In some circumstances systematic reviews are commissioned and in such cases a commissioning document needs to be written. However, the most important pre-review activities are defining the research questions(s) that the systematic review will address and producing a review protocol (i.e. plan) defining the basic review procedures. The review protocol should also be subject to an independent evaluation process. This is particularly important for a commissioned review.

在进行系统评审之前，有必要确认进行这种审查的必要性。在某些情况下，委托进行系统审查，在这种情况下，需要编写委托文件。然而，最重要的审查前置活动是确定系统审查将解决的研究问题，并制定确定基本审查程序的审查协议（即计划）。审查协议书还应接受独立的评估程序。这对于委托进行的审查尤其重要。

2.1 The need for a systematic review

The need for a systematic review arises from the requirement of researchers to summarise all existing information about some phenomenon in a thorough and unbiased manner. This may be in order to draw more general conclusions about some phenomenon than is possible from individual studies, or may be undertaken as a prelude to further research activities.

需要进行系统审查的需求来自于研究人员要求以彻底和公正的方式总结有关某些现象的所有现有信息。这可能是为了就某些现象得出比个别研究可能更多的一般性结论，也可能是作为进一步研究活动的前奏。

Prior to undertaking a systematic review, researchers should ensure that a systematic review is necessary. In particular, researchers should identify and review any existing systematic reviews of the phenomenon of interest against appropriate evaluation criteria.

在进行系统审查之前，研究人员因确保系统审查是必要的。特别是，研究人员应根据适当的评价标准确定和审查对感兴趣现象的任何现有系统审查。

CRD suggests the following checklist:
• What are the review’s objectives?
• What sources were searched to identify primary studies? Were there any restrictions?
• What were the inclusion/exclusion criteria and how were they applied?
• What criteria were used to assess the quality of primary studies?
• How were quality criteria applied?
• How were the data extracted from the primary studies?
• How were the data synthesised?
• How were differences between studies investigated?
• How were the data combined?
• Was it reasonable to combine the studies?
• Do the conclusions flow from the evidence?

The CRD Database of Abstracts of Reviews of Effects (DARE) criteria (http://www.york.ac.uk/inst/crd/crddatabases.htm#DARE) are even simpler. They are based on four questions:
1. Are the review’s inclusion and exclusion criteria described and appropriate?
2. Is the literature search likely to have covered all relevant studies?
3. Did the reviewers assess the quality/validity of the included studies?
4. Were the basic data/studies adequately described?

From a more general viewpoint, Greenlaugh suggests the following questions:
• Can you find an important clinical question, which the review addressed? (Clearly, in software engineering, this should be adapted to refer to an important software engineering question.)
• Was a thorough search done of the appropriate databases and were other potentially important sources explored?
• Was methodological quality assessed and the trials weighted accordingly?
• How sensitive are the results to the way that the review has been done?
• Have numerical results been interpreted with common sense and due regard to the broader aspects of the problem?

2.2 Commissioning a review

Sometimes an organisation requires information about a specific topic but does not have the time or expertise to perform a systematic literature itself. In such cases it will commission researchers to perform a systematic literature review of the topic. When this occurs the organisation must produce a commissioning document specifying the work required.

有时，一个组织需要一个特定主题的信息，但没有时间和专业知识来进行系统文献的本身。这种情况下，它将委托研究人员对这一主题进行系统的文献回顾。当这种情况发生时，组织必须出示一份委托文件，具体说明所需的工作。

A commissioning document will contain or consider the following items (adapted from the CRD guidelines )
• Project Title
• Background
• Review Questions
• Advisory/Steering Group Membership (Researchers, Practitioners, Lay members, Policy Makers etc)
• Methods of the review
• Project Timetable
• Dissemination Strategy
• Support Infrastructure
• Budget
• References

The commissioning document can be used both to solicit tenders from research groups willing to undertake the review and to act as a steering document for the advisory group to ensure the review remains focused and relevant in the context.

委托文件既可用于向愿意进行审查的研究小组征求投标书，也可作为咨询小组的指导文件，以确保审查的重点和相关性。

The commissioning phase of a systematic review is not required for a research team undertaking a review for their own needs or for one being undertaken by a PhD student. If the commissioning stage is not undertaken then the dissemination strategy should be incorporated into the review protocol. As yet, there are no examples of commissioned SLRs in the software engineering domain.

系统审查的委托阶段不要求研究小组根据自己的需要进行审查，也不需要博士生进行审查。如果没有进行委托阶段，则应将传播战略纳入审查议定书。到目前为止，在软件工程领域还没有委托SLRs的例子。

2.3 The research questions

Specifying the research questions is the most important part of any systematic review. The review questions drive the entire systematic review methodology:
• The search process must identify primary studies that address the research questions.
• The data extraction process must extract the data items needed to answer the questions.
• The data analysis process must synthesise the data in such a way that the questions can be answered.

2.3.1 Questions types

The most important activity during planning is to formulate the research question(s). The Australian NHMR Guidelines identify six types of health care questions that can be addressed by systematic reviews:
1. Assessing the effect of intervention.
2. Assessing the frequency or rate of a condition or disease.
3. Determining the performance of a diagnostic test.
4. Identifying aetiology and risk factors.
5. Identifying whether a condition can be predicted.
6. Assessing the economic value of an intervention or procedure.

In software engineering, it is not clear what the equivalent of a diagnostic test would be, but the other questions can be adapted to software engineering issues as follows:
• Assessing the effect of a software engineering technology.
• Assessing the frequency or rate of a project development factor such as the adoption of a technology, or the frequency or rate of project success or failure.
• Identifying cost and risk factors associated with a technology.
• Identifying the impact of technologies on reliability, performance and cost models.
• Cost benefit analysis of employing specific software development technologies or software applications.

2.3.2 Questions structure

Medical guidelines recommend considering a question about the effectiveness of a treatment from three viewpoints:
• The population, i.e. the people affected by the intervention.
• The interventions, which are usually a comparison between two or more alternative treatments.
• The outcomes, i.e. the clinical and economic factors that will be used to compare the interventions.

Population
In software engineering experiments, the populations might be any of the following:
• A specific software engineering role e.g. testers, managers.
• A category of software engineer, e.g. a novice or experienced engineer.
• An application area e.g. IT systems, command and control systems.
• An industry group such as Telecommunications companies, or Small IT companies.

Intervention
The intervention is the software methodology/tool/technology/procedure that addresses a specific issue, for example, technologies to perform specific tasks such as requirements specification, system testing, or software cost estimation.

Comparison
This is the software engineering methodology/tool/technology/procedure with which the intervention is being compared. When the comparison technology is the conventional or commonly-used technology, it is often referred to as the “control” treatment. The control situation must be adequately described. In particular “not using the intervention” is inadequate as a description of the control treatment. Software engineering techniques usually require training. If you compare people using a technique with people not using a technique, the effect of the technique is confounded with the effect of training. That is, any effect might be due to providing training not the specific technique. This is a particular problem if the participants are students.

Outcomes
Outcomes should relate to factors of importance to practitioners such as improved reliability, reduced production costs, and reduced time to market. All relevant outcomes should be specified. For example, in some cases we require interventions
that improve some aspect of software production without affecting another e.g. improved reliability with no increase in cost.

Context
For Software Engineering, this is the context in which the comparison takes place (e.g. academia or industry), the participants taking part in the study (e.g. practitioners, academics, consultants, students), and the tasks being performed (e.g. small scale,
large scale). Many software experiments take place in academia using student participants and small scale tasks. Such experiments are unlikely to be representative of what might occur with practitioners working in industry. Some systematic reviews might choose to exclude such experiments although in software engineering, these may be the only type of studies available.

Experimental designs
In medical studies, researchers may be able to restrict systematic reviews to primary studies of one particular type. For example, Cochrane reviews are usually restricted to randomised controlled trials (RCTs). In other circumstances, the nature of the question and the central issue being addressed may suggest that certain study designs are more appropriate than others. However, this approach can only be taken in a discipline where the large number of research papers is a major problem. In software engineering, the paucity of primary studies is more likely to be the problem for systematic reviews and we are more likely to need protocols for aggregating information from studies of widely different types.

2.4 Developing a review protocol

A review protocol specifies the methods that will be used to undertake a specific systematic review. A pre-defined protocol is necessary to reduce the possibility of researcher bias. For example, without a protocol, it is possible that the selection of individual studies or the analysis may be driven by researcher expectations. In medicine, review protocols are usually submitted to peer review.

The components of a protocol include all the elements of the review plus some additional planning information:
• Background. The rationale for the survey.
• The research questions that the review is intended to answer.
• The strategy that will be used to search for primary studies including search terms and resources to be searched. Resources include digital libraries, specific journals, and conference proceedings. An initial mapping study can help determine an appropriate strategy.
• Study selection criteria. Study selection criteria are used to determine which studies are included in, or excluded from, a systematic review. It is usually helpful to pilot the selection criteria on a subset of primary studies.
• Study selection procedures. The protocol should describe how the selection criteria will be applied e.g. how many assessors will evaluate each prospective primary study, and how disagreements among assessors will be resolved.
• Study quality assessment checklists and procedures. The researchers should develop quality checklists to assess the individual studies. The purpose of the quality assessment will guide the development of checklists.
• Data extraction strategy. This defines how the information required from each primary study will be obtained. If the data require manipulation or assumptions and inferences to be made, the protocol should specify an appropriate validation process.
• Synthesis of the extracted data. This defines the synthesis strategy. This should clarify whether or not a formal meta-analysis is intended and if so what techniques will be used.
• Dissemination strategy (if not already included in a commissioning document).
• Project timetable. This should define the review schedule.

2.5 Evaluating a review protocol

The protocol is a critical element of any systematic review. Researchers must agree a procedure for evaluating the protocol. If appropriate funding is available, a group of independent experts should be asked to review the protocol. The same experts can later be asked to review the final report.

PhD students should present their protocol to their supervisors for review and criticism.

The basic SLR review questions discussed in Section 5.1 can be adapted to assist the evaluation of a systematic review protocol. In addition, the internal consistency of the protocol can be checked to confirm that:
• The search strings are appropriately derived from the research questions.
• The data to be extracted will properly address the research question(s).
• The data analysis procedure is appropriate to answer the research questions.

2.6 Lessons learned for protocol construction

Brereton et al. identify a number of issues that researchers should anticipate during protocol construction:
• A pre-review mapping study may help in scoping research questions.
• Expect to revise questions during protocol development, as understanding of the problem increases.
• All the systematic review team members need to take an active part in developing the review protocol, so they understand how to perform the data extraction process.
• Piloting the research protocol is essential. It will find mistakes in the data collection and aggregation procedures. It may also indicate the need to change the methodology intended to address the research questions including amending the data extraction forms and synthesis methods.