Introduction

In January 2020, the World Health Organization (WHO) announced a new coronavirus known as “severe acute respiratory syndrome coronavirus-2” (SARS-CoV-2), which had not been previously seen in humans, causing coronavirus disease 2019 (COVID-19), and recognized this outbreak as a pandemic in March 2020 [1,2,3]. The first case in Turkey was confirmed on 11 March 2020.

Inhalation of respiratory droplets and interaction with infected surfaces are the major transmission paths [4]. To prevent the spread of this pandemic, early identification of infected patients and contact isolation are critical [4]. Real-time fluorescence reverse transcription-polymerase chain reaction (RT-PCR) and chest imaging are the current diagnostic tools for COVID-19 [5]. RT-PCR is the preferred initial and reference diagnostic test for COVID-19 [6]. A positive RT-PCR confirms the diagnosis of COVID-19 and its specificity is high [7]. However, the sensitivity of RT-PCR is variable and false negativeness can be seen [8]. In the clinical setting, the sensitivity of these tests is probably dependent on the form and nature of the specimen, the duration of the disease at the time of testing, and exactly which one was used [9, 10]. Various degrees of sensitivity, ranging from 37 to 71%, were reported for RT-PCR tests performed in China during the first period of the outbreak. The pooled sensitivity for RT-PCR was reported as 89% in a meta-analysis [10].

Computed tomography (CT) plays an important role in the diagnosis, treatment management, and determination of complications of COVID-19 [3, 5, 11,12,13,14,15]. It has been reported that CT was more sensitive than RT-PCR testing in the early phase of the COVID-19 outbreak [5, 10]. The pooled sensitivity for chest CT was found as 94% in a meta-analysis [10]. In Turkey, chest CT was used frequently in the diagnosis and treatment management of asymptomatic and symptomatic patients with suspected COVID-19 in the early stages of the outbreak due to factors such as low RT-PCR test capacity, the widespread use of CT, and fast results. In addition, responsible physicians determined treatment plans according to pulmonary involvement. The WHO suggests that chest imaging be used in the diagnosis of COVID-19 in cases where RT-PCR tests are not available, results are delayed, and the initial RT-PCR test is negative, but the suspicion of COVID-19 remains [16].

Although the Radiological Society of North America (RSNA) does not recommend the use of chest CT to diagnose COVID-19 as a screening or first-line test, they suggest using a structured reporting system to help radiologists identify results, reduce reporting variability, reduce ambiguity in reporting findings that are possibly due to this infection, and improve the understanding of these radiologic findings by the referee [17]. Various structured reporting systems have been proposed to standardize CT findings in patients with COVID-19 [12, 17,18,19]. However, the most accepted and widely used is the expert consensus statement proposed by the RSNA.

This study aimed to evaluate the diagnostic performance of the RSNA structured reporting language for chest CT findings in patients COVID-19.

Materials and methods

This retrospective, single-center study was approved by the Ethics Committee of our institution (2020/294) and by the Ministry of Health of the Republic of Turkey, COVID-19 Scientific Research Committee. The requirement for informed consent was waived because of the retrospective study design.

Study population

Patients who were admitted to the emergency department or outpatient clinics with suspected COVID-19 in the early period of the epidemic in Turkey between March 16th and May 31st, 2020, were evaluated consecutively. The reason for choosing this time interval was that chest CT is used as a screening tool in addition to RT-PCR tests in all patients with suspected of COVID-19 because it provides fast results and is easily accessible. The use of chest imaging as a screening tool was discontinued with the increase in RT-PCR testing capacity and the publication of the WHO guideline for the use of chest CT in the diagnosis and management of COVID-19 [16].

The inclusion criteria for the study were as follows: (1) symptomatic or asymptomatic patients with suspected COVID-19; (2) patients with at least one chest CT; (3) patients who underwent at least one RT-PCR test within 3 days of CT screening. COVID-19 was suspected in asymptomatic patients due to a history of close contact with a RT-PCR positive patient. The exclusion criteria were as follows: (1) patients aged under 18 years; (2) motion artifacts in CT screening; (3) interval between RT-PCR and CT scan longer than 3 days. A flow diagram representing the patient population is summarized in Fig. 1.

Fig. 1
figure 1

Flow diagram of the research

Full size image

Sociodemographic features (sex, age, and comorbidities), clinical follow-up, CT scan, and RT-PCR results were obtained from patients’ health records. Symptoms and duration of symptoms were ignored in the study. Interval times were noted in patients with serial CT scans.

Reference standard test

As the first reference standard, the RT-PCR test was used. One or more RT-PCR tests were performed on patients according to their close contact history and clinical status. At least one RT-PCR confirmed the positivity of COVID-19. In addition, patients with a history of close contact, clinical features consistent with COVID-19, and typical radiologic findings were considered clinically positive with a multidisciplinary decision even if one or more RT-PCR tests were negative. The close contact criterion was determined as living or working with at least one person with a positive RT-PCR test. The total number of clinically positive patients with the RT-PCR test was used as the second reference standard test because RT-PCR may be a false negative (RT-PCR + /Clinical +), similar to the study by Prokop et al. [12].

CT protocol

All images were obtained using a 16-detector CT scanner (Somatom Scope, Erlangen, Germany, Siemens Healthineers). To prevent cross-contamination, the CT scanner was reserved only for patients with suspected COVID-19, the surfaces in the CT room were decontaminated with sodium hypochlorite solution after the examination, and the CT room was ventilated for at least 20 min between examinations. Images were acquired in the supine position, during the inspiration phase from the thoracic inlet to the diaphragm without intravenous contrast agent administration. The CT protocol was as follows: 110 kVp tube voltage, automatic tube current modulation (Care Dose 4D with an “average” modulation strength), 1.2 mm collimation width, 0.8–1.25 pitch factor. Reconstructions were made using a sharp B80f kernel with a slice thickness of 3 mm and matrix of 512 × 512.

Image analysis

In the first session, the CT scans were independently analyzed by two radiologists who were blinded to the RT-PCR tests and clinical information. Chest CT findings were categorized into four patterns as “typical”, “indeterminate”, “atypical,” and “negative,” according to the reporting language proposed by the RSNA (Table 1) [17]. The first observer was a radiologist (R) with 5 years’ experience, and the second observer was a senior radiology resident (RR) (Figs. 2, 3, 4, 5, 6).

Table 1 RSNA structured reporting language [17]
Full size table
Fig. 2
figure 2

“Typical” pattern in patients with COVID-19. a A 39-year-old female, characterized by bilateral, peripheral GGO. b A 45-year-old male, characterized by multifocal GGO of rounded morphology. c A 42-year-old male, characterized by bilateral, peripheral, GGO and reverse halo sign (arrow)

Full size image
Fig. 3
figure 3

“Typical” pattern in patients with non-COVID-19. A 52-year-old female with breast cancer, characterized by bilateral rounded GGO with surrounding consolidation (reverse halo sign). Also, note the prosthesis on the left of the breast. Similar imaging findings were seen in the patient's previous CT scans, and the RT-PCR test was negative three times. Cryptogenic organizing pneumonia was the final diagnosis, based on a multidisciplinary consensus

Full size image
Fig. 4
figure 4

“Indeterminate” pattern in patients with COVID-19. a A 69-year-old female with single, peripheral GGO (arrow) in the right upper lobe. b A 36-year-old male with single, round GGO (arrow) in the left lower lobe. c A 39-year-old female, characterized by bilateral GGO lacking a specific distribution. d A 40-year-old male, characterized by isolated very small GGO in the left lower lobe (circle)

Full size image
Fig. 5
figure 5

“Indeterminate” pattern in patients with non-COVID-19. a An 83-year-old female, characterized by bilateral, diffuse GGO with pleural effusion consistent with alveolar edema. b A 37-year-old male with granulomatosis with polyangiitis, characterized by bilateral, perihilar, diffuse consolidation with pleural effusion consistent with alveolar hemorrhage. c A 42-year-old male, characterized by single, peripheral GGO in the left lower lobe. COVID-19 was ruled out due to a lack of close contact history and three negative RT-PCR tests. Although the infectious agent could not be identified, the patient responded well to broad-spectrum antibiotics. The final diagnosis was atypical pneumonia, based on a multidisciplinary consensus. d A 52-year-old male, characterized by single round consolidation with air bronchogram in the left upper lobe. COVID-19 was ruled out after two negative RT-PCR tests, and the patient responded well to broad-spectrum antibiotic therapy. With multidisciplinary consensus, the final diagnosis was round pneumonia, but the infectious organism could not be identified

Full size image
Fig. 6
figure 6

“Atypical” pattern in patients with non-COVID-19. a A 21-year-old female, characterized by segmental consolidation with air bronchograms in the right middle lobe, reflecting lobar pneumonia, caused by Streptococcus pneumoniae. Furthermore, the PCR test was negative twice. b A 58-year-old male, characterized by a small centrilobular nodule in bilateral upper lobes consistent with respiratory bronchiolitis. c A 39-year-old female, characterized by cavitation and surrounding consolidation in the apicoposterior segment of left upper lobe consistent with tuberculosis. d A 28-year-old male, characterized by bilateral smooth interlobular septal thickening with pleural effusion consistent with interstitial edema

Full size image

The overall decision was reached by consensus in another session with the first two radiologists and a third radiologist with 8 years’ thoracic imaging experience. In this session, CT findings suggestive of imaging patterns according to the RSNA reporting language were also classified as subgroups. “Single, round ground-glass opacities (GGO) (with or without consolidation)” and “single, peripheral GGO (with or without consolidation),” which do not exactly fit in the RSNA reporting language, were evaluated in the indeterminate pattern (Fig. 4a, b).

Statistical analysis

The data were analyzed using the Statistical Package for the Social Sciences, version 21.0 (IBM Corporation, Armonk, NY, USA) and MedCalc for Windows, version 19.2.6 (MedCalc Software Ltd., Mariakerke, Belgium) software packages.

Descriptive statistics are presented as mean ± standard deviation (SD) or median (range) for continuous numerical variables and counts (percentage) for categorical variables.

Fisher’s exact test was used to compare groups (categorical variables). To correct for multiple comparisons, the Bonferroni correction was used. Cohen’s Kappa test was used to determine the inter-observer agreement. The kappa values were obtained by comparing each observer with the overall assessment.

The diagnostic performance of the RSNA structured reporting language for chest CT findings in patients with COVID-19 was evaluated using the area under the receiver operating characteristics (ROC) curve (AUC) for each observer and overall assessment. The sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of the RSNA reporting language were calculated for each pattern. Comparisons among AUC values were assessed using the Delong method. In all statistical analyses in this study, p values of < 0.05 were used to denote statistical significance.

Result

A total of 1186 patients were enrolled in the study (650 males, 536 females) with a mean age of 46.71 ± 18.66 (18–98) years.

The demographic data and clinical characteristics of the patient cohort are summarized in Table 2. More than one RT-PCR test was performed in 573 of 1186 patients for reasons such as persistent clinical suspicion in patients with negative initial RT-PCR tests and treatment management in patients with positive RT-PCR tests; the median was 1 (1–5). Of the 1186 patients, 141 had multiple CT scans, 102 were positive in RT-PCR, and 39 were negative in RT-PCR. The median time interval for the CT follow-up scan was 5 (3–16) days. There was at least 1 comorbid disease in 256 of the patients.

Table 2 Summary of patient characteristics (n = 1186)
Full size table

Of the 1186 patients, 379 had positive RT-PCR tests and 807 had negative tests. In addition, nine patients were clinically diagnosed as having COVID-19 with a multidisciplinary decision, even though more than one RT-PCR test was negative. In total, 388 patients were diagnosed as having COVID-19 based on RT-PCR tests and clinical features (RT-PCR + /clinical +).

Chest CT results

Chest CT results based on the structured reporting language proposed by the RSNA for COVID-19 are given in Table 3.

Table 3 Summary of overall consensus CT results according to the RSNA structured reporting language (n = 1186)
Full size table

Of the 256 patients with "typical" chest CT findings, 248 were confirmed as having COVID-19 (RT-PCR + /clinical +). The final diagnoses of eight patients with typical CT findings were as follows: other pneumonia (n = 3), organizing pneumonia (n = 3), pulmonary infarct (n = 1), and drug toxicity (n = 1) (RT-PCR−/clinical−). At least two (2–5) RT-PCR tests were performed in patients without COVID-19 with “typical” chest CT findings and all negative RT-PCR tests.

COVID-19 (RT-PCR + /clinical +) was diagnosed in 77 of 163 patients whose CT findings were in the “indeterminate” pattern. We performed a further evaluation by analyzing the findings of the chest CT in an “indeterminate” pattern. Among the “indeterminate” chest CT findings “single, peripheral GGO” or “single, round GGO (with or without consolidation)” were most frequently detected in COVID-19, while “diffuse, multifocal, perihilar or unilateral GGO (with or without consolidation, non-rounded and non-peripheral distribution)” were found in non-COVID-19 (p < 0.001). In patients without COVID-19 with “indeterminate” chest CT findings, at least two (2–4) RT-PCR tests were conducted and all RT-PCR tests were negative.

In the patients with COVID-19, no “atypical” CT results were observed. The final diagnoses of patients with “indeterminate” and “atypical” chest CT findings are given in Online Resource 1.

COVID-19 (RT-PCR + /clinical +) was diagnosed in 63 of 613 patients with “negative” chest CT findings.

Chest CT findings differed during the control scan in 25 of 141 patients with COVID-19. On the baseline CT scan, 99 of the 141 patients had "typical" findings, while 32 had "indeterminate" findings, and 11 had negative. There were “typical” CT findings on the follow-up scan in six patients classified as “negative” and 19 patients classified as “indeterminate” with baseline CT scan. Of these 19 patients, five had single, peripheral GGO (with or without consolidation), and 14 had single, round GGO (with or without consolidation).

“Typical” chest CT findings were more frequent in patients with COVID-19 with comorbid diseases or those aged over 50 years, although “negative” findings were less common (p < 0.001). There was no difference in the distribution of “indeterminate” chest CT findings according to age and comorbid diseases (p > 0.05) (Table 4).

Table 4 Distribution of chest CT results by RSNA structured reporting language in patients with COVID-19 by age and comorbid disorders
Full size table

The inter-observer agreement of COVID-19 diagnosis was almost perfect for the radiologist and substantial for the senior resident (K = 0.822, K = 0.773, respectively) compared with the overall consensus CT results. There was a perfect agreement for “typical” (86.7 and 85.9%) and “negative” (99.0 and 98.7%) CT findings, and substantial agreement for “atypical” (79.7 and 72.5%) CT findings (R and RR, respectively). Less moderate agreement was observed for “indeterminate” (59.3 and 49.0%) CT findings (R and RR, respectively).

Diagnostic performance

The diagnostic performance of the RSNA structured reporting language for chest CT findings in patients with COVID-19 is given in Table 5.

Table 5 ROC analysis results and diagnostic performance of RSNA structured reporting language for chest CT findings in patients with COVID-19 (RT-PCR results and clinical diagnosis as standard of reference)
Full size table

The mean AUC was 0.878 (95% CI 0.852–0.903) for the diagnosis of patients with COVID-19 (p < 0.001). When the CT results were re-arranged according to the follow-up CT scan, the sensitivity, specificity, and accuracy of “typical” findings for COVID-19 were 70.4, 99.0, and 89.6, respectively, and the average AUC values increased to 0.891 (95% CI 0.866–0.915) (p = 0.002).

The highest specificity (99.0%) and PPV (96.9%) were achieved when the “typical” pattern was assumed to be CT-positive. When the “typical” and “indeterminate” patterns were considered CT-positive together, the highest sensitivity (83.8%) and NPV (91.8%) were achieved. In addition to the "typical" pattern, the highest diagnostic accuracy (92.2%) was achieved when the “single, peripheral GGO” and “single, round GGO (with or without consolidation)” in the “indeterminate” pattern were considered to be CT-positive.

Discussion

CT plays an important role in the diagnosis, treatment management, and determination of complications of COVID-19 [3, 5, 11,12,13,14]. In the early stages of the epidemic, the sensitivity of chest CT for COVID-19 was reported as 98% by Fang et al. [20], 97% by Ai et al. [5], and 97% by Long et al. [21]. The pooled sensitivity for chest CT was found as 94% in a meta-analysis [10]. The justifications for the overestimation of sensitivity in these studies were the bias in the selection of patients and the lack of information on the definition of positive CT findings [22, 23].

In COVID-19 pneumonia, there are frequently seen characteristic chest CT findings, but these findings may overlap with other infectious and non-infectious diseases. Various structured reporting systems have been proposed to standardize chest CT findings in patients with COVID-19, but the expert consensus statement defined by the RSNA has been the most widely accepted and used. Ciccarese et al. [13] reported the sensitivity of the “typical” chest CT findings per the RSNA structured reporting language for COVID-19 pneumonia as 71.6%, but only symptomatic patients were enrolled in the study. In a similar study by Som et al. [14], the sensitivity was reported as 86% for “typical” chest CT findings, but chest CT was used as a clinical problem-solving method in this study, rather than for COVID-19 screening or primary diagnosis. Although the CT findings in these two studies were well defined, the sample did not exactly reflect the characteristics of the COVID-19 population.

Our experience is not compatible with the literature. The sensitivity was found as 63.9% for the “typical” chest CT findings and as 83.8% for the combination of “typical” and “indeterminate” chest CT findings. The most significant explanation for this discrepancy is that our patient cohorts differ from other studies. Chest CT was performed with RT-PCR in all symptomatic or asymptomatic patients in our institution until the WHO guidelines on the use of CT in the diagnosis and management of COVID-19 were published in the early stages of the epidemic. Previous studies showed that normal CT findings were more common in asymptomatic patients and early stages of disease [24,25,26]. Poor chest CT sensitivity could be attributed to asymptomatic patients and early stages of disease in our research, but the patients’ symptoms and duration of symptoms were not investigated in the study. However, “typical” chest CT findings were subsequently observed in 25 patients with COVID-19 undergoing follow-up scans.

The specificity of chest CT was 99.0% for the “typical” findings and 88.3% for the combination of “typical” and “indeterminate” findings and was higher than in previous studies [5, 10, 13, 14, 20, 21]. This is possibly due to patient cohorts and the classification of chest CT findings according to the RSNA structured reporting language. Although the “typical” pattern is a widely documented imaging feature of COVID-19 pneumonia, the differential diagnosis involves organizing pneumonia, influenza pneumonia, connective tissue diseases, and drug toxicity, which may cause a similar imaging pattern [17]. Compared with COVID-19, the lower prevalence of these diseases included in the “typical” pattern differential diagnosis during the pandemic may have led to an overestimation of our chest CT specificity.

In the RSNA Expert Consensus Statement, the findings that mostly cause confusion are “single, peripheral GGO” and “single, round GGO (with or without consolidation).” These single GGOs that can be seen in the early stages of COVID-19 pneumonia [27, 28] do not fully meet both the “typical” and “indeterminate” patterns. These lesions do not fit with the “typical” pattern because they are not multifocal and bilateral, the “indeterminate” pattern because they are round and peripheral, and the “atypical” pattern because they have GGOs. As a result, we evaluated “single GGO (peripheral or round)” as separate CT findings within the “indeterminate” pattern. Afterward, we made further analysis by creating subgroups according to CT findings in the RSNA imaging pattern. In the “indeterminate” pattern, “single GGO (peripheral or round)” was more common in patients with COVID-19. In contrast, “diffuse, multifocal, perihilar or unilateral GGO” and “few, very small GGO (with non-rounded or non-peripheral)” were more frequent in non-COVID-19 patients. Additionally, the highest diagnostic accuracy was obtained by combining “single GGO (peripheral or round)” with typical findings. Therefore, we believe that it would be better to consider “single GGO (peripheral or round)” as high probability subgroup within the “indeterminate” pattern. Similarly, in the COVID-19 Reporting and Data System (CO-RADS) prepared by the Dutch radiology association, “unilateral, peripheral GGOs” are classified as highly suspicious (CO-RADS 4) [12]. Small GGO is another confusing finding in the consensus classification. This definition is still subjective. The opacities below the pulmonary acinus size known as 6 mm were considered to be small in our research [29].

“Typical” and “atypical” patterns are more easily understood in the RSNA structured reporting language, whereas the “indeterminate” pattern is more complex. “Indeterminate” pattern is mostly built on exclusion findings, with the emphasis on non-round morphology and non-peripheral distribution in the absence of “typical” findings. Therefore, the inter-observer agreement was found to be lower for the “indeterminate” pattern compared with other patterns, similar to previous studies [13, 14]. However, our overall inter-observer agreement was higher than in other studies [13, 14, 30]. This can be explained by the fact that observers at our institution are familiar with the RSNA structured reporting language in daily practice.

COVID-19 is frequently severe in patients with advanced age and comorbid conditions [31]. When comparing chest CT findings and age groups, there are conflicting results in the literature [5, 11, 13, 32,33,34]. It has been reported that comorbid diseases are risk factors for chest CT involvement, but there are few studies on this subject [34]. We observed “typical” findings more frequently in patients aged over 50 years and those with comorbid diseases.

There are several limitations in our research. First, the lower prevalence of diseases in the differential diagnosis of COVID-19 affected the results of our study. Second, the RT-PCR used as a reference test could be false negative, so at least two tests were performed in patients with “typical” and “indeterminate” CT findings. In addition, very few patients were clinically accepted as COVID-19 with a multidisciplinary decision; so as not to create further limitations, care was taken to ensure that these patients had contact with at least one person with a positive the RT-PCR test where they lived or worked. Third, patients’ symptoms and duration of symptoms were disregarded. Finally, other pneumonias were clinically diagnosed by ruling out COVID-19, and the infected agent could not be identified in the majority of these patients.

In conclusion, the RSNA structured reporting language has helped radiologists understand findings, decrease reporting differences, and properly integrate physicians during the pandemic. The RSNA structured reporting language provides important diagnostic performance for the identification of COVID-19 pneumonia, but chest CT findings that do not exactly fit the RSNA structured reporting language identified in this study should be taken into account to improve diagnostic accuracy in future updates.