Moving structures (for example, lung tumours) can produce artefacts and blurring that affect the accuracy of radiotherapy verification when using cone beam CT (CBCT).^1,2 This led to the development of four-dimensional CBCT (4D-CBCT) where the series of projections are binned into different phases of the breathing cycle which minimizes blurring.² The disadvantage is that the amount of data for image reconstruction of each phase is reduced compared with conventional three-dimensional CBCT (3D-CBCT). This leads to streak artefacts and some loss of image quality.³

One solution is to slow down the gantry speed so that more frames can be acquired.³ However, this leads to additional dose to the patient, given that the dose per frame is the same and the frame acquisition rate is constant. The longer acquisition time reduces patient throughput and increases the possibility that the patient may move.^3,4

Alternative reconstruction algorithms to reduce streaking, different gantry speeds or multiple rotations have been studied with the aim of reducing dose and image acquisition time.^3,5–10 These used quantitative methods for image quality to compare techniques using phantoms or very small numbers of patients (up to four).

Rit et al¹ used accuracy of verification (measurement of patient-positioning errors) alongside quantitative methods in a study involving patients, by combining a different reconstruction algorithm with two gantry speeds. The use of accuracy allows the effect of real patient factors to be included (principally tumour movement and location, and patient size).

Dose optimization within radiotherapy departments is encouraged, and any image taken should be of sufficient quality for the purpose.¹¹ Investigating image quality by observer methods can complement quantitative methods (for example, signal-to-noise ratio or contrast) and accuracy of verification and can also incorporate the effect of real patient factors. Visual grading analysis (VGA) involves observers reviewing the images and rating the visibility (or clarity) of a given list of structures and rating the overall image acceptability.^12,13 Structures are also identified when verification is carried out in radiotherapy, therefore VGA is a good fit for a study of verification image quality. The alternative observer-based method (receiver operating characteristics) tends to be used to determine the accuracy of a diagnostic technique.¹² A hybrid between the two methods is also available.¹⁴

VGA has been used in radiotherapy verification by Sweeney et al¹⁵ when observers were asked to match 4D-CBCT scans using both four-dimensional (4D) and three-dimensional matching methods. They also found that 4D-CBCT was more accurate than 3D-CBCT because of improved visualization of small tumours with large motion amplitudes and of tumours close to the diaphragm.

Five other radiotherapy studies used VGA to compare portal film with electronic portal imaging.^16–20 Three later studies also used VGA but compared other imaging methods.^21–23 These included replacing the standard target in the linear accelerator by an aluminium target,²¹ a modified computed radiography (CR) cassette²² and using two different methods of processing CR images.²³ In all the above studies (which covered a range of body sites including the lung), observers were asked to rate certain aspects of the images such as ease of verification, visibility of selected anatomical structures, comparison of one modality with the other or preference for one modality over the other. One study also included accuracy of verification.¹⁸

In the study reported here, an investigation of image quality and acceptability of 4D-CBCT for radiotherapy was carried out using VGA to evaluate the effect of three different presets with varying acquisition times and dose.

The Elekta (Crawley, UK)-recommended preset in XVI software v. 4.5 (Elekta) for the acquisition of 4D-CBCT (known here as Preset 1) takes 4 min to acquire and has a high nominal [volume CT dose index (CTDIvol)] dose. CTDIvol is a standardized method of measuring dose using a cylindrical phantom.²⁴ Two additional presets had been created: Preset 3 (identical to the Elekta-recommended 3D-CBCT preset for lung treatment verification) and Preset 2 (designed to have the same dose as Preset 3 and to have a faster delivery than Preset 1) (Table 1). The likely reduction in image quality due to undersampling from the lower number of projections could render Presets 2 and 3 unsuitable for verification. Any increase in noise due to reduction in mAs is likely to be less significant.^7,25 All three presets use 120-kVp X-rays and are delivered over 200° of rotation (start angle 340° and stop angle 180°) with a 27-cm diameter field of view.

Approval for this study as a service evaluation was given by the Trust's Research and Development Department. Verification images for this study came from ten patients receiving radical conventionally fractionated radiotherapy between August 2013 and July 2014. These patients had all consented to the use of their images for research purposes as part of their consent to radiotherapy treatment. They were a convenience sample, and VGA was carried out retrospectively. One patient (Patient 6) was excluded from the study because his tumour had been excised, leaving nine patients in the study. All (except Patient 1) were treated supine with arms above their heads, supported by a standard lung board and with a knee rest. Patient 1 was treated prone with head and arms supported by a prone pillow, with a pad beneath his ankles, since a stable supine position had not been achievable without collision with the gantry. All had 4D-CT planning scans acquired (Phillips Brilliance big bore with 2-mm slice thickness). The patients had been unable to comply with the Assisted Breathing Coordinator™ (Elekta) normally used for radical treatment of patients with lung cancer.²⁶

4D-CBCT images were acquired on the first three factions using Presets 1, 2 and 3 at Fractions 1, 2 and 3, respectively. Example screen shots are shown in Figure 1.

Four volunteer observers were recruited from the pool of radiographers with advanced competencies in verification for stereotactic lung radiotherapy (which includes the use of 4D-CBCT). The other radiographers would require training in the use of the software. Each observer reviewed all 27 images. For each patient, the images were reviewed from Preset 3 first, then from Preset 2 and finally from Preset 1. In this order, observers would not have prior knowledge of what they might see in the poorer quality images. Blinding the observers was not possible, and the lack of blinding could be a confounding factor.

The observers independently rated the clarity of the edges of the tumour and of certain other thoracic structures.^12,16–23 Each of the six edges of the tumour were rated since some edges may be clearer to see than others. The other thoracic structures were those commonly used as surrogate structures when reviewing images. The list was based on Yin et al²⁰ and was chosen because it was a comprehensive list without being too onerous. These structures were the vertebrae, chest wall and ribs, clavicle, apex of lung, diaphragm, and trachea, bronchi and carina (the last three structures were combined together).

The clarity scale was chosen to have only four steps (“very clear”, “clear”, “unclear” and “not visible”) to avoid observers choosing the middle option.²⁷ Other VGA studies in radiotherapy have used an odd number of steps. A fifth option (“not imaged”) was included for the thoracic structures since some would not appear in all of the images.

Next, the observers performed a 4D dual registration match (XVI software v. 4.5) (bone match to a large volume, then 4D grey match to a volume around the tumour) and recorded the translation and rotation patient-positioning errors and rated the acceptability of the match. The acceptability scale was based on the European standard for CT image quality¹³ with changes in wording (“very acceptable”, “acceptable”, “acceptable with reservations” and “not acceptable”).

Guidelines were given with examples of images with very clear, clear and unclear tumours, and definitions of acceptability were added. The basis of these definitions is given below:

Very acceptable: The observer encountered no difficulties with the automatic match.

The location of tumour, size of tumour, amplitude of tumour motion and size of the patient were also recorded since they may affect clarity or acceptability.^1,2,28 The patient separation (anterior/posterior and left/right), tumour volume and overall tumour size (gross tumour volume, at end exhale) were measured from the planning scans in the treatment planning system. Separation was taken to be the maximum separation on a CT slice containing the planning target volume. The 4D-CBCT software measures the peak-to-peak amplitude of the breathing motion. One factor that can affect image quality is the breathing rate.⁷ The user does not have access to this information without additional equipment or processing of the raw data, therefore it was not included.

The data from the observers were carefully checked to identify errors of completeness in accordance with best practice.²⁹ There were nine instances of missing clarity values, which were “missing completely at random” since they were overlooked.²⁹ They were generally replaced by the mode of the values chosen by the same observer for the other structures from the same scan. (A measure of central tendency is an acceptable method of replacing a missing value.²⁹) The choices made by the other observers were also considered since, in two cases, “not imaged” was an appropriate entry.

The analysis was carried out in SPSS^® v. 22 (IBM Corp., New York, NY; formerly SPSS Inc., Chicago, IL) using binary logistic regression by converting the scales to binary values (the divisions were made between “clear” and “unclear”, and between “acceptable” and “acceptable with reservations”). “Not imaged” was treated as missing data for the analysis. Logistic regression was chosen because the clarity and acceptability data were non-parametric.³⁰ Ordinal logistic regression could have been used³¹ but distinguishing between, say, “unclear” and “not visible” did not make clinical sense in this study. To estimate model goodness of fit, Nagelkerke's pseudo R² (which has a potential range from 0 to 1, where 1 indicates a perfect model) was calculated.³²

There was a clinical requirement (set by a consultant clinical oncologist) that a preset should get acceptability ratings of “acceptable” or higher and clarity ratings of “clear” or higher on at least 75% of images for it to be clinically useful. This was based on the requirement that an image should have sufficient quality for its purpose.¹¹

The data on patient size, tumour size and tumour location are presented in Table 2. Only Patient 9 was obese (body mass index >30), and none were severely obese. Scatter plots of patient-positioning errors and box plots of patient breathing motion are shown in Figures 2 and 3. The VGA results from the observers are summarized in Figures 4 and 5.

Clarity was tested against preset, patient, observer, structure, lung and lobe of lung, and against patient size, size of tumour and breathing motion. Each variable was tested individually, and those that were significant (p < 0.05) were placed together in a backwards stepwise binary logistic regression. The final model is shown in Table 3. Variables were removed from the final model if grossly not significant (p > 0.1). Two terms that had been significant in an individual analysis but were removed from the final model (because their significance had increased to >0.1) were breathing motion amplitude in the superior/inferior direction [p = 0.002, odds ratio (OR) = 6.912] and breathing motion amplitude in the anterior/posterior direction (p = 0.022, OR = 1.442).

Acceptability was tested against preset, patient, observer, lung and lobe of lung, and against patient size, size of tumour, breathing motion and the magnitude of the positioning errors. The same logistic regression method was used. Acceptability could not be tested against clarity because clarity was assessed 13 times in each scan while acceptability was assessed once. The result showing the final model is shown in Table 4. Two terms that had been significant individually were removed from the final model; they were anterior/posterior dimension of the gross tumour volume (p = 0.03, OR = 1.262) and the pitch rotation positioning error (p = 0.049, OR = 1.246).

Both clarity and acceptability could not be tested by patient in the final analyses because the degrees of freedom were reduced. The individual logistic regression results are shown in Tables 5 and 6.

Preset 3 only achieves 44% for clarity (ignoring the “not imaged” results) and 58% for acceptability, and therefore fails the clinical requirement (Figures 4a and 5a). Preset 2 has 78% for clarity and 86% for acceptability, whereas Preset 1 has 89% for clarity and 97% for acceptability, therefore both presets meet the clinical requirement.

Clarity and acceptability are both significantly affected by preset (Tables 3 and 4). Preset 1 has the best results, whereas Preset 3 has the least acceptable (lower OR values than either Preset 1 or 2). Preset 2 has the same dose as Preset 3 but has better image quality because more projections are taken thereby reducing undersampling.⁷

The patient has a statistically significant effect on clarity (Table 5, Figure 4b). Those patients who had a statistically significant effect (Patients 5, 7 and 8) all had tumours that were close to other structures so that the edges were unclear (Figure 4c). These three patients were responsible for most of the “unclear” and “not visible” results for Presets 1 and 2. In clinical practice, images such as these would be reviewed by a clinician and radiographer. Patient 3 is also just statistically significant (p = 0.048), and this is due to poor visibility of tumour edges with Preset 3, which had been commented on by the observers. The OR values show that these patients have lower clarity than Patient 1. The Nagelkerke's R² value is low (0.075), suggesting that the patients did not influence clarity strongly (other factors may have more influence).³⁰

Acceptability is not dependent upon patients but Patient 4 has a significantly poorer acceptability than Patient 1 (Table 6, Figure 5b), and this is due to the high translation and rotation positioning errors seen in the images for this patient (Figure 2). These errors were independent of preset since they occurred on all 3 days. Patient 4 also had a separate nodal planning target volume which made verification decision-making difficult. The positioning errors for this patient are probably responsible for the significance of the anterior/posterior translation error in the final model for acceptability (Table 4). SPSS was unable to find a solution for one of the test statistics leading to the calculation of the goodness of fit parameters.

Despite providing examples of images where the tumours were “very clear”, “clear” and “unclear” in the instructions, there was still significant variation between observers for clarity (Table 3). Observer 1 was significantly more likely to select “clear” or “very clear” than the other three (their ORs are similar and <1). The variation existed despite providing sample images, but it is likely that this variation cannot be completely removed. This agrees with the findings of other studies.^16,20,23 Defining terms for acceptability did reduce variation between observers.

The significant structures (Table 3) for clarity are mostly those with many instances of “not imaged”, so their significance probably is artificially raised and the low numbers are often below or close to the requirement for 10 events for binary logistic regression.²⁹ However in certain patients (particularly, Patients 5, 7 and 8), some tumour edges were difficult to distinguish from other adjacent structures which probably accounts for the significant result and low OR for the right edge of tumour. All the instances of “acceptable with reservations” for Presets 1 and 2 were associated with Patients 5 and 8, with Patient 4 (due to high positioning errors).

The patient-dependent variables that were thought to affect clarity and acceptability were generally not significant and not included in the final analyses. Further study of more patients would be required to investigate the effect of patient-dependent variables. Right lung had a higher OR than left lung and was significant for both clarity and acceptability. Patients 5, 7 and 8 (with tumours adjacent to other structures) and Patient 4 (with the large positioning errors) had left lung tumours, thereby reducing clarity and acceptability for the patients with left lung tumours. The maximum anterior/posterior patient separation was a significant term in the final model for clarity but with an OR >1. This suggests that clarity improves as the patient separation increases, which does not make clinical sense.²⁸ Patients 5 and 7 (with low clarity levels) had small anterior/posterior separations which may be one explanation for this result.

The Nagelkerke's R² values suggest that the terms in the models are reasonable predictors of acceptability (0.503) and clarity (0.446).³⁰ The events per variable for acceptability is low at 7 (a total of 21 “not acceptable” or “acceptable with reservations” results and 3 variables in the final model) while the ideal is 10 or more. This could be improved by more patients or more observers. This suggests that the final model has some uncertainty. The events per variable for clarity is 72 (361 “not visible” or “unclear” results and 5 variables in the final model) which is acceptable and suggests a more reliable model.

The logistic regression models show that clarity varies with preset and observer and that acceptability varies with preset. The presence of other significant terms in the models seems to depend more on whether the tumour is attached/not attached to other structures than the actual values of the terms (e.g. left or right lung).

A prospective study, where the presets could be applied in a random order, may avoid any bias from possible patient anxiety. Patients could also be selected to reduce variability, for example, by excluding those with unusual immobilization, nodal disease or with tumours attached to other structures. Other studies have excluded patient groups such as those with breathing motion <5 mm¹ or those with nodal disease.⁴ The effect of breathing rate could also be investigated if an external method of measuring could be added.

An initial recommendation of Preset 2 could be made, with Preset 1 in reserve for when image quality is insufficient. By choosing Preset 2, the nominal dose is reduced by 18% and the acquisition time is reduced by almost half compared with Preset 1. In certain circumstances, Preset 1 could be preferred. This may include imaging a tumour where the edges are hard to distinguish. Another circumstance might include where the patient has a large separation (larger than the patients in this study), but there was insufficient evidence to state this definitely.

Preset 3 is not recommended since the clarity of the structures and acceptability of the matches was not of sufficient quality to be used for verification.

Future work could include investigating patient breathing motion from both 4D-CT and 4D-CBCT to predict patients who would benefit most from 4D-CBCT. Comparing accuracy of three dimensional and 4D-CBCT matches may also contribute to the selection of patients.

The views expressed in this publication are those of the authors and not necessarily those of the NHS Executive.

This work was supported by The Institute of Cancer Research (ICR) and Cancer Research UK grant numbers C46/A3970 and C33589/A19727 to the ICR Section of Radiotherapy. This work was undertaken in The Royal Marsden NHS Foundation Trust which received a proportion of its funding from the NHS Executive. The authors acknowledge NHS funding to the NIHR Biomedical Research Centre at The Royal Marsden and The Institute of Cancer Research.