Author + information
- Received September 25, 2014
- Revision received December 4, 2014
- Accepted December 17, 2014
- Published online February 1, 2015.
- Maik J. Grundeken, MD∗,
- Yuki Ishibashi, MD, PhD†,
- Philippe Généreux, MD‡,
- Laura LaSalle, MPH‡,
- Javaid Iqbal, MD†,
- Joanna J. Wykrzykowska, MD, PhD∗,
- Marie-Angèle Morel, BSc§,
- Jan G. Tijssen, PhD∗,§,
- Robbert J. de Winter, MD, PhD∗,
- Chrysafios Girasis, MD§,‖,
- Hector M. Garcia-Garcia, MD, PhD†,§,
- Yoshinobu Onuma, MD, PhD†,§,
- Martin B. Leon, MD‡ and
- Patrick W. Serruys, MD, PhD¶∗ ()
- ∗Amsterdam Heart Center, Academic Medical Center, Amsterdam, the Netherlands
- †Thoraxcenter, Erasmus Medical Center, Rotterdam, the Netherlands
- ‡Cardiovascular Research Foundation, New York, New York
- §Cardialysis B.V., Rotterdam, the Netherlands
- ‖First Cardiology Department, Onassis Cardiac Surgery Center, Athens, Greece
- ¶International Centre for Circulatory Health, National Heart and Lung Institute, Imperial College London, London, United Kingdom
- ↵∗Reprint requests and correspondence:
Dr. Patrick W. Serruys, International Centre for Circulatory Health, NHLI, Imperial College London, Royal Brompton Campus, London SW7 2AZ, United Kingdom.
Objectives This study sought to evaluate inter–core lab variability in quantitative coronary angiography (QCA) analysis of bifurcation lesions.
Background QCA of bifurcation lesions is challenging. To date there are no data available on the inter–core lab variability of bifurcation QCA analysis.
Methods The randomized Tryton IDE (Tryton Pivotal IDE Coronary Bifurcation Trial) compared the Tryton Side Branch Stent (Tryton Medical, Durham, North Carolina) with balloon angioplasty as side branch treatment. QCA was performed in an angiographic subcohort (n = 326) at 9-month follow-up. Inter-core lab variability of QCA analysis between the Cardiovascular Research Foundation and the Cardialysis core labs was evaluated before and after alignment of the used QCA methodology using angiographic data derived from this angiographic follow-up cohort.
Results In the original analysis, before alignment of QCA methodology, the mean difference between the core labs (bias) was large for all QCA parameters with wide 95% limits of agreement (1.96 × SD of the bias), indicating marked variability. The bias of the key angiographic endpoint of the Tryton trial, in-segment percentage diameter stenosis (%DS) of the side branch, was 5.5% (95% limits of agreement: –26.7% to 37.8%). After reanalysis, the bias of the in-segment %DS of the side branch reduced to 1.8% (95% limits of agreement: –16.7% to 20.4%). Importantly, after alignment of the 2 core labs, there was no longer a difference between both treatment groups (%DS of the side branch: treatment group A vs. group B: 34.4 ± 19.4% vs. 32.4 ± 16.1%, p = 0.340).
Conclusions Originally, a marked inter–core lab variability of bifurcation QCA analysis was found. After alignment of methodology, inter–core lab variability decreased considerably and impacted angiographic trial results. This latter finding emphasizes the importance of using the same methodology among different core labs worldwide. (Tryton Pivotal Prospective, Single Blind, Randomized Controlled Study to Evaluate the Safety & Effectiveness of the Tryton Side Branch Stent Used With DES in Treatment of de Novo Bifurcation Lesions in the Main Branch & Side Branch in Native Coronaries [TRYTON]; NCT01258972)
- bifurcation lesions
- percutaneous coronary interventions
- quantitative coronary angiography
- Tryton Side Branch Stent
Ever since the late 1970s, visual estimation of the severity of a stenosis on coronary angiography has been regarded as unreliable due to a marked intra- and interobserver variability (1,2). Therefore, quantitative coronary angiography (QCA) was introduced in the mid-1980s to provide an objective and reproducible quantification of coronary lesions (3,4). QCA parameters have been widely used as primary and secondary endpoints in numerous randomized clinical trials evaluating the efficacy of new technologies in percutaneous coronary interventions and the effect of new pharmaceutical agents on coronary artery disease progression/regression (5–7).
Due to the fractal geometry of the coronary tree, there is a natural tapering of the bifurcation, with differences in reference vessel diameter (RVD) among the proximal main branch, distal main branch, and side branch (8,9). Due to this natural tapering, the interobserver variability of visual estimation of lesion severity increases even more in bifurcation lesions (10). Furthermore, conventional QCA algorithms have the limitation of being inaccurate in bifurcation lesions because they have been developed and validated in a single straight coronary segment (11). To improve the accuracy of QCA in bifurcation lesions, dedicated bifurcation algorithms were developed, which subsequently have been used in recent clinical trials on bifurcation treatment (12–16).
To eliminate the potential bias stemming from the investigators, QCA analysis in clinical trials is usually performed at independent core laboratories (core labs). These core labs aim to provide unbiased and reproducible results by using validated QCA software and by using standard operating procedures during QCA analysis. Although intraobserver and interobserver variability of bifurcation QCA algorithms have been investigated before (14,16,17), to date no data are available on the differences in bifurcation QCA measurements between core labs. This study aimed to examine inter–core lab variability by comparing the QCA results of 2 core labs using data from the 9-month angiographic follow-up cohort of the randomized trial on the Tryton Side Branch Stent (Tryton Medical, Durham, North Carolina).
Tryton IDE (Tryton Pivotal IDE Coronary Bifurcation Trial), an investigational device exemption (IDE) randomized trial, compared the Tryton Side Branch Stent with side branch balloon angioplasty, both in combination with a regular drug-eluting stent in the main branch, for the treatment of de novo true coronary bifurcation lesions. The primary endpoint (powered for noninferiority), at 9-month follow-up, was the difference in the occurrence of target vessel failure, defined as the composite of cardiac death, Q-wave or non–Q-wave target vessel myocardial infarction (>3× the upper limit of normal of creatine kinase isoenzyme), and target vessel revascularization. The key secondary endpoint (powered for superiority) was in-segment percentage diameter stenosis (%DS) of the side branch in a pre-specified subgroup of 374 subjects (with an expected loss to follow-up of 15%) undergoing planned repeat angiography at 9 months (the angiographic follow-up cohort).
Two core labs were assigned to perform different types of analyses in the angiographic follow-up cohort of the Tryton IDE trial. The Cardiovascular Research Foundation (CRF, New York, New York) was assigned to perform 2-dimensional QCA analysis of the complete angiographic follow-up cohort. Cardialysis B.V. (Rotterdam, the Netherlands) was assigned to perform 3-dimensional QCA and intravascular ultrasound analyses and for this purpose 9-month follow-up angiograms of 130 subjects included in the angiographic follow-up cohort were available at Cardialysis. Besides 3-dimensional QCA and intravascular ultrasound analyses, Cardialysis also performed 2-dimensional QCA analysis in this subgroup. The inter–core lab variability of the 9-month 2-dimensional QCA analysis between the 2 core labs was investigated in these 130 subjects. This initial analysis indicated diverging angiographic results between the 2 core labs (Figure 1A). Thereafter, both core labs disclosed and shared their QCA analysis plans to unravel potential explanations for these differences. Both core labs decided to perform a reanalysis of the total angiographic follow-up cohort using an identical QCA analysis plan, which they had agreed on (Table 1).
Initial QCA analysis plan of CRF
At the start of the Tryton IDE trial, the dedicated bifurcation QCA algorithms were not yet validated against precision phantoms. Therefore, the initial QCA analysis plan of the trial, approved by the U.S. Food and Drug Administration and used for the main publication, included the use of a conventional single-vessel QCA algorithm (QAngio XA, version 7.2.34, Medis Medical Imaging Systems, Leiden, the Netherlands). For each bifurcation, 2 analyses were performed: one from the proximal main branch to the distal main branch, and the other from the proximal main branch to the side branch. First, calibration was performed using the guiding catheter and then, the region of interest was defined by indicating its proximal (i.e., in the proximal main branch) and distal (i.e., in the side branch or distal main branch) boundaries (Figure 2B). Hereafter, the QCA software automatically detected the vessel contour based on the change in brightness of the pixels (Figure 2C) (4). Then, the analysts were allowed to manually edit the vessel contour whenever the contour did not appeared to be smooth or appropriately delineated (Online Figure 1). Because conventional single-vessel QCA software does not recognize the side branch origin, segmentation of the bifurcation lesion was performed manually, with the carinal point as the beginning of the side branch segment (Figure 2E). For the side branch, a single point 5 mm distal to the balloon/stent edge was taken as the reference vessel diameter (RVD) (Figure 2F). The %DS was calculated as follows: (RVD of the side branch – minimal lumen diameter [MLD] of the side branch) / RVD of the distal side branch (Figure 2F). For the main branch, the RVD was defined as the average of the reference diameter in the “normal” segments proximal and distal to the stent. The %DS in the main branch was calculated by using the MLD and the averaged RVD. QCA measurements were performed on a single “worst” projection (i.e., the projection in which the stenosis looks most severe).
Initial QCA analysis plan of Cardialysis
Cardialysis used a dedicated bifurcation software algorithm for their QCA analysis on bifurcation lesions (Coronary Angiography Analysis System [CAAS], version 5.9, Pie Medical Imaging, Maastricht, the Netherlands) (13,18). For each bifurcation, only 1 analysis needed to be performed. The bifurcated region of interest—including the proximal main branch, distal main branch, and side branch—was defined by indicating its proximal (i.e., in the proximal main branch) and distal (i.e., in both the side branch and the distal main branch) boundaries (Figure 3A). After the automatic vessel contour detection (Figure 3B), analysts were not allowed to manually edit the vessel contour, except in the cases where the vessel contour was erroneously detecting a side branch or other overlapping contours instead of the vessel contour itself (Online Figure 2). The point of bifurcation is automatically determined by the software and is defined as the mid-point of the largest circle that can be fitted in the bifurcation area, touching all 3 contours (Figure 3C) (19). The centerlines of each of the 3 segments meet at the point of bifurcation (Figure 3C). Segmentation of the bifurcation in 3 individual segments was performed automatically using the point of bifurcation and centerlines as previously described (19). The %DS was automatically calculated using the interpolated RVD at the MLD site of each segment (Figures 3E to 3G). All QCA measurements were performed on at least 2 projections (if available), and the average from all projections were reported.
Alignment of QCA analysis plans between both core labs for the reanalysis
After establishing the differences in QCA plans, the 2 core labs agreed to perform a post-hoc reanalysis. It was decided to use the dedicated CAAS bifurcation software for reanalysis, considering the superior accuracy and precision of this method compared with single-vessel algorithms (12). After delineating the region of interest and the automatic detection of the vessel contours, the analysts were not allowed to manually edit the vessel contour, except in cases were the vessel contour was erroneously not following the vessel contour, as previously described (Online Figure 2). Segmentation of the bifurcation was performed automatically by the software and the %DS was calculated using the interpolated RVD at the site of the MLD in each segment. All QCA measurements were performed on a single worst projection. Both core labs performed their QCA on exactly the same frame, which was selected by the CRF core lab.
The individual signed differences of both core labs were averaged; the mean of these signed differences (bias) was used as a measure of accuracy and the standard deviation as a measure of precision (20). The agreement between both core labs with regard to the measurement of %DS of the side branch (in-segment) was evaluated by nonparametric orthogonal regression analysis using the Passing-Bablok method (21). The differences in %DS of the side branch between core labs were also displayed using Bland-Altman plots: the mean of both core lab measurements were plotted on the horizontal axis against the individual signed differences of both core labs on the vertical axis. The 95% limits of agreement (mean difference [bias] ± 1.96 SD of the bias) were determined as the measure of variability. For the Passing-Bablok regression analysis, STATA (version 12, StataCorp LP, College Station, Texas) was used. All other statistical analyses were performed using the SPSS software package (version 21.0, IBM, Chicago, Illinois).
Inter–core lab variability
Inter–core lab variability of the initial QCA analysis of the 130 subjects is shown in Table 2. Core lab A systematically measured larger RVD, MLD, and %DS in both main branch and side branch compared with Core Lab B. Furthermore, the 95% limits of agreement were wide for all QCA parameters, indicating a marked variability between both core labs. The average difference in the in-segment %DS of the side branch (the key secondary endpoint of the Tryton IDE trial, powered for superiority) was 5.54%, with the 95% limits of agreement between –26.74% and 37.82% (Figure 4). Passing-Bablok orthogonal linear regression analysis showed systematic as well as proportional bias between both core labs with regard to the measurement of in-segment %DS of the side branch, with an intercept of 7.1% (95% confidence interval [CI]: 4.5% to 10.0%) and a slope of 0.62 (95% CI: 0.50 to 0.74) (Figure 4).
From the 374 patients included in the pre-specified angiographic cohort, 326 patients returned for repeat angiography (87%). From these 326 patients, 311 matched cases were used to assess the inter–core lab variability after QCA reanalysis. In 15 cases, angiography was not available for both core labs due to missing or corrupt CDs. After reanalysis of the total angiographic cohort using the same QCA analysis plan with the same frame and QCA bifurcation software, less systematic bias was observed (Table 2). The side branch in-segment QCA measurements were almost identical between both core labs; only small average differences were observed with regard to the MLD (–0.0297 mm; 95% limits of agreement: –0.4683 to 0.4089 mm) and the RVD (0.0227 mm; 95% limits of agreement: –0.4193 to 0.4647 mm), resulting in an average difference in %DS of 1.8% (95% limits of agreement: –16.7% to 20.4%) (Figure 4). Finally, orthogonal linear regression analysis showed a marked improvement in reproducibility between core labs with regard to the measurement of in-segment %DS of the side branch. The intercept of the orthogonal regression line was close to 0%, with 0% being enclosed in the 95% CI (intercept: 0.8%, 95% CI: –0.8% to 2.5%), suggesting there was no systematic bias. The slope of the orthogonal regression line was closer to the identity line with the identity line almost being enclosed in the 95% CI, suggesting only a minimal proportional bias between both core labs after reanalysis (slope: 0.94, 95% CI: 0.88 to 0.99) (Figure 4).
Initial QCA analysis in the subgroup of 130 patients showed diverging results between both core labs with regard to the Tryton IDE trial key secondary endpoint of in-segment %DS of the side branch (Figure 1A). However, after complete reanalysis using contemporary bifurcation QCA software without the routine use of manual contour editing and manual segmentation, both core labs showed similar results with regard to in-segment %DS of the side branch (Figure 1B).
Our study, for the first time, has highlighted a marked inter–core lab variability in bifurcation QCA measurements due to the use of different QCA software and methodology. After using the same QCA bifurcation software and aligning the methodology of the QCA core labs, the inter–core lab variability improved considerably. Furthermore, after aligning the QCA methodology, we have demonstrated that the trial results were affected. These findings emphasize the importance of standardization of QCA methodology among the different core labs.
The steering committee did acknowledge the methodological differences between core labs that have been raised retrospectively after the database have been locked for the IDE submission to the U.S. Food and Drug Administration. During the designing phase of the trial, however, the validation of the CAAS bifurcation QCA algorithm against a precision bifurcation phantom model was not yet published (14), and therefore the bifurcation algorithm not yet integrated in the core lab responsible for the baseline and follow-up QCA analyses. It was decided to report the data according to the initial approved methodological plan and analysis in the main paper, while reporting the post-hoc QCA results using the bifurcation software in the Online Appendix (22).
Use of single-vessel versus dedicated bifurcation software
One of the potential explanations for the differences in the initial analysis was the use of a single-vessel QCA algorithm, which is less accurate than a dedicated bifurcation algorithm is and introduces a systematic bias for several reasons (8,12). Due to the fractal geometry of the epicardial coronary tree, there is a natural tapering of the coronaries with different vessel diameters proximal to and distal from each bifurcation (9). When using a single-vessel algorithm, the interpolated reference is determined by the diameters of the proximal and distal branches, which are by definition unequal, resulting in a systematic underestimation of the RVD and %DS in the proximal main branch, and an overestimation of the RVD and %DS in the distal main branch and side branch (8). In addition, if the reference diameter is selected from a normal segment distal to the balloon/stent in the side branch to calculate the %DS, this can introduce a subjective random error when compared with the interpolated reference diameter based on the vessel contour of the complete region of interest encompassing the side branch (Figure 2). Furthermore, in the so-called polygon of confluence, the single-vessel software is not able to define the vessel contour automatically and often requires manual editing, potentially introducing another factor of random error (Figure 2D, Online Figure 1). Finally, single-vessel algorithms do not recognize the origin of the distal branches, requiring manual segmentation, which may introduce another bias (Figures 2E and 2F).
Manual contour editing
Another potential explanation for the differences in the initial analysis was the difference in allowance for manual corrections. Lack of smoothness of the automatic contour delineation, observed when the angiogram is of suboptimal quality (23), should be accepted as such. Manual editing will introduce subjectivity (Online Figure 1) and should be allowed only in those cases where the algorithm erroneously detects other structures than the vessel contour (side branches, for example). Analysts should use the “restriction function” to exclude an area for the automatic contour detection instead of completely redrawing it themselves (Online Figure 2). All corrections and adjustments made after completion of the QCA analysis should be captured by audit trails and final approval of the QCA should be electronically signed to fulfill the Code of Federal Regulation Title 21, part 11 (CFR 21, part 11) guideline of the U.S. Food and Drug Administration, which defines the criteria for reliable electronic data capturing.
Remaining inter–core lab variability compared with the accuracy and precision of the bifurcation software
Although the reproducibility improved remarkably after aligning the core lab methodology, some systematic bias remained. This remaining inter–core lab bias for the in-segment side branch QCA parameters was 0.0227 mm (RVD), –0.0297 mm (MLD), and 1.84% (%DS) (Table 2). We believe this is acceptable, considering that the intrinsic accuracy of the CAAS bifurcation software, calibrated on a precision phantom model, was comparable to this bias: −0.032 mm (side branch RVD), −0.017 mm (side branch MLD), and 0.88% (side branch %DS) (12).
Inter–core lab variability (standard deviation of the bias) after reanalysis was 0.2255 mm for the in-segment side branch RVD, 0.2238 mm for the in-segment side branch MLD, and 9.45% for the in-segment %DS. This variability is a little higher as would have been expected from the phantom validation, which showed precision of the bifurcation software of 0.075 mm (side branch RVD), 0.123 mm (side branch MLD), and 5.35% (side branch %DS) (12). This somewhat higher variability might be explained by differences in calibration. Whereas the calibration in the phantom validation study was performed using a grid in a static model, the calibration in the current study was performed using the guiding catheter in close proximity of the beating heart. Although calibration is preferably performed in the same frame as the QCA analysis, sometimes this is not possible and calibration in another frame is needed, which may introduce extra variability. Given the fact that most angiograms have a calibration factor of ∼0.2000 mm/pixel, a precision with regard to measuring the MLD and RVD being around this number can be considered as acceptable.
Although core lab analysts are ideally blinded to the treatment groups to minimize bias, this is not always possible. This is a major limitation, which is not restricted to the current Tryton IDE trial, but also applies to other trials such as trials comparing metallic stents with bioresorbable scaffolds or different types of transcatheter aortic valves. This is important to realize because device and pharmaceutical companies cover the costs of the analysis, which makes core labs not completely independent. Therefore, it is vital to use a methodology in which the analysis is performed as automatically as possible to minimize human subjectivity, with manual corrections restricted to a minimum.
It seems to remain challenging to aim for standardization of QCA methodology in various core labs worldwide (11). Our study has highlighted the importance of using the same QCA methodology among different core labs, using software validated on the same high-resolution calibrated bifurcation phantom model (10,14,16,24), to ensure reproducibility and objectivity. Although standard operating procedures are often not shared because they are considered intellectual property of the core lab, we believe it is important to share at least the key factors of the QCA methodology to ensure reproducible and generalizable results. Considering cross-validation, using 2 distinct core labs, of a pre-specified proportion of patients enrolled might even be a valid option to ensure identification of any potential issues precluding data accuracy and reproducibility.
Because the agreed QCA reanalysis plan included the use of exactly the same angiographic view and frame, selected by 1 of the core labs, this study did not investigate the role of frame selection on the inter–core lab variability. Although it has been shown that differences in frame selection from the same angiographic view does not influence the accuracy and variability to a large extent (25), differences in selection of the angiographic view itself is a major determinant of variability in QCA (26).
We found a marked inter–core lab variability in QCA of bifurcation lesions when different QCA methodologies were used, including difference in software use (single-vessel vs. dedicated bifurcation software), differences in allowance for manual vessel contour correction, and differences in the method for segmentation (automatic vs. manual). However, when the same methodology was used, inter–core lab variability decreased considerably. More importantly, QCA results of the trial were affected following alignment of the methodology. This latter finding emphasizes the importance of using the same QCA methodology among different core labs worldwide.
Dr. Généreux has received speaking fees from Abbott Vascular; and consults for Cardiovascular Systems Inc. Dr. Wykrzykowska has received speaking and consulting fees from Abbott, Tryton Medical, and Stentys. Dr. Leon is member of the Scientific Advisory Board for Abbott Vascular, Boston Scientific, and Medtronic; and is the principal investigator of the Tryton Bifurcation Study. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- Abbreviations and Acronyms
- percentage diameter stenosis
- confidence interval
- Cardiovascular Research Foundation
- investigational device exemption
- minimal lumen diameter
- quantitative coronary angiography
- reference vessel diameter
- Received September 25, 2014.
- Revision received December 4, 2014.
- Accepted December 17, 2014.
- American College of Cardiology Foundation
- Zir L.M.,
- Miller S.W.,
- Dinsmore R.E.,
- Gilbert J.P.,
- Harthorne J.W.
- Reiber J.H.,
- Serruys P.W.,
- Kooijman C.J.,
- et al.
- Garg S.,
- Serruys P.W.
- Lichtlen P.R.,
- Hugenholtz P.G.,
- Rafflenbeul W.,
- Hecker H.,
- Jost S.,
- Deckers J.W.,
- the INTACT Group Investigators
- Keane D.,
- Haase J.,
- Slager C.J.,
- et al.
- Ishibashi Y.,
- Grundeken M.J.,
- Nakatani S.,
- et al.
- Reiber J.H.
- Généreux P.,
- Kumsars I.,
- Lesiak M.,
- et al.
- Reiber J.H.,
- van der Zwet P.M.,
- Koning G.,
- et al.
- Reiber J.H.,
- van Eldik-Helleman P.,
- Kooijman C.J.,
- Tijssen J.G.,
- Serruys P.W.