Int J Med Sci 2019; 16(5):630-635. doi:10.7150/ijms.30739 This issue Cite

Research Paper

Clostridium difficile Infection Risk with Important Antibiotic Classes: An Analysis of the FDA Adverse Event Reporting System

Chengwen Teng1,2, Kelly R. Reveles1,3, Obiageri O. Obodozie-Ofoegbu1,2, Christopher R. Frei1,4 Corresponding address

1. Pharmacotherapy Division, College of Pharmacy, The University of Texas at Austin, San Antonio, TX, USA
2. Pharmacotherapy Education and Research Center, Long School of Medicine, University of Texas Health-San Antonio, San Antonio, TX, USA
3. South Texas Veterans Health Care System, San Antonio, TX, USA
4. University Health System, San Antonio, TX, USA

Citation:
Teng C, Reveles KR, Obodozie-Ofoegbu OO, Frei CR. Clostridium difficile Infection Risk with Important Antibiotic Classes: An Analysis of the FDA Adverse Event Reporting System. Int J Med Sci 2019; 16(5):630-635. doi:10.7150/ijms.30739. https://www.medsci.org/v16p0630.htm
Other styles

File import instruction

Abstract

Introduction: Antibiotic use is an important risk factor for Clostridium difficile infection (CDI). Prior meta-analyses have identified antibiotics and antibiotic classes that pose the greatest risk for CDI; however, CDI epidemiology is constantly changing and contemporary analyses are needed.

Objectives: The objective of this study was to evaluate the association between CDI and important antibiotic classes in recent years using the FDA Adverse Event Report System (FAERS).

Methods: FAERS reports from January 1, 2015 to December 31, 2017 were analyzed. The Medical Dictionary for Regulatory Activities (MedDRA) was used to identify CDI cases. We computed the Reporting Odds Ratios (RORs) and corresponding 95% confidence intervals (95%CI) for the association between antibiotics and CDI. An association was considered statistically significant when the lower limit of the 95%CI was greater than 1.

Results: A total of 2,042,801 reports (including 5,187 CDI reports) were considered, after inclusion criteria were applied. Lincosamides (e.g., clindamycin) had the greatest proportion of CDI reports, representing 10.4% of all lincosamide reports. CDI RORs (95%CI) for the antibiotic classes were (in descending order): lincosamides 46.95 (39.49-55.82), monobactams 29.97 (14.60-61.54), penicillin combinations 20.05 (17.39-23.12), carbapenems 19.16 (15.52-23.67), cephalosporins/ monobactams/carbapenems 17.28 (14.95-19.97), cephalosporins 15.33 (12.60-18.65), tetracyclines 7.54 (5.42-10.50), macrolides 5.80 (4.48-7.51), fluoroquinolones 4.94 (4.20-5.81), and trimethoprim-sulfonamides 3.32 (2.03-5.43).

Conclusion: All antibiotic classes included in the study were significantly associated with CDI. Lincosamides (e.g., clindamycin) had the highest CDI ROR among the antibiotics evaluated in this study.

Keywords: Clostridium difficile, adverse drug events, antibiotics, antimicrobial stewardship

Introduction

Clostridium difficile infection (CDI) is a great public health concern in hospital and community settings. In the first decade of the twenty-first century, United States hospitals noted a profound increase in CDI incidence [1]. Since then, national standards required hospitals to implement effective infection control interventions and antimicrobial stewardship programs to prevent CDI. Nationally-representative studies now indicate that CDI rates among hospitalized patients might be declining [2]. With the decline in CDI incidence in hospitals, there appears to have been a concurrent shift to community-onset CDI [3].

A rich and diverse intestinal microbiota prevents CDI; disruption of microbiota, especially due to antibiotic use, can lead to loss of colonization resistance and proliferation of C. difficile [4,5]. Antibiotic exposure is the most important risk factor in both hospital and community-onset CDI [6-8]. In previous meta-analyses conducted between 1988 and 2009, clindamycin, fluoroquinolones, and cephalosporins had the highest CDI risks [6-8].

Given the change in CDI epidemiology in recent years, more recent data are needed to evaluate the current CDI associations with various antibiotics. The FDA Adverse Event Reporting System (FAERS) provides recent data on CDI and antibiotics [9]. The objective of this study is to evaluate CDI associations with antibiotics using FAERS data from 2015 to 2017.

Methods

Data Source

FAERS is a publicly available database organized into Quarterly Data Files, which contain adverse event reports that were submitted to United States Food and Drug Administration (FDA) [9]. FAERS data include patient demographic information (age and sex), drug information (drug name, active ingredient, route of administration, and drug's reported role in the event), and reaction information. Each report lists a primary suspected drug with one or more adverse reactions and may include other drugs. Clinical outcomes, such as death and hospitalization, may also be reported.

Study Design

FAERS data from January 1, 2015 to December 31, 2017 were obtained from the FDA. Some adverse event reports were submitted multiple times with updated information. Therefore, duplicate reports were removed by case number, with the most recent submission included in the study. Reports containing drugs which were administered in oral, subcutaneous, intramuscular, intravenous, and parenteral routes were included in the study, while other routes of administration were excluded.

Drug Exposure Definition

Each antibiotic was identified in the FAERS drug files by generic and brand names listed in the Drugs@FDA Database [10]. Only drugs with a reported role coded as “PS” (Primary Suspect Drug) or “SS” (Secondary Suspect Drug) were included in this study [11]. Antibiotics with less than three CDI reports were excluded from the data analysis [12].

Adverse Drug Reaction Definition

FAERS defines adverse drug reactions using Preferred Terms from the Medical Dictionary for Regulatory Activities (MedDRA). MedDRA includes a hierarchy of terms, which are (from the highest to the lowest) System Organ Classes (SOC), High Level Group Term (HLGT), High Level Term (HLT), Preferred Term (PT), and Lowest Level Term (LLT). Standardised MedDRA Queries (SMQs) are groupings of MedDRA terms, usually at the PT level, which relate to an adverse drug reaction. Pseudomembranous colitis (SMQ), including Preferred Terms “Clostridial infection”, “Clostridial sepsis”, “Clostridium bacteraemia”, “Clostridium colitis”, “Clostridium difficile colitis”, “Clostridium difficile infection”, “Clostridium test positive”, “Gastroenteritis clostridial”, and “Pseudomembranous colitis” were used to identify CDI cases [13]. “Clostridium difficile sepsis”, which is a Lowest Level Term, was also used in the study.

Statistical Analysis

A disproportionality analysis was performed by calculating Reporting Odds Ratios (RORs) and corresponding 95% confidence intervals (95%CI) for the association between CDI and each antibiotic class or individual antibiotic [14]. ROR was calculated as the ratio of the odds of reporting CDI versus all other events for a given drug, compared with these reporting odds for other drugs present in FAERS [14]. An association was considered to be statistically significant if the 95%CI did not include 1.0 (see Table 1 for the calculation of ROR and CI) [14]. A higher ROR suggests a stronger association between the antibiotic and CDI. A subgroup analysis was performed on patients who were 65 years or older and patients less than 65 years old. The Cochran-Armitage Trend Test was used to assess a change in the trend of CDI reports in patients who took fluoroquinolones from 2004 to 2017. Data analysis was performed using Microsoft Access 2016, Microsoft Excel 2016 (Microsoft Corporation, Redmond, WA), SAS 9.4, and JMP Pro 13.2.1 (SAS Institute, Cary, NC).

 Table 1 

A two by two contingency table for a drug (A) - ADR (X) combination

ADR (X)Other ADRsTotal
Drug (A)aba+b
Other drugscdc+d
Totala+cb+da+b+c+d

† ADR = adverse drug reaction; ROR = (a/b)/(c/d); 95% Confidence Interval (CI) = eln(ROR)±1.96√(1/a+1/b+1/c+1/d)

Results

After inclusion and exclusion criteria were applied and duplicate reports were removed, FAERS contained a total of 2,042,801 reports from January 1, 2015 to December 31, 2017. There were 5,187 CDI reports from 2015 to 2017, which were included in the data analysis. Female patients represented 61% of CDI patients who had gender information. CDI patients who had age information had a median age (IQR, interquartile range) of 62 (27) years. Please see Table 2 for the gender and age information of patients who were taking various antibiotics.

The lincosamide class had the highest CDI ROR (46.95, 95%CI: 39.49-55.82) among all antibiotic classes included in the study (Figure 1). Clindamycin was the only antibiotic in the lincosamide class which met the inclusion criteria. The monobactam class (including aztreonam only) demonstrated the second highest CDI ROR (29.97, 95%CI: 14.60-61.54). The CDI ROR of the trimethoprim-sulfonamides class was the lowest (3.32, 95%CI: 2.03-5.43).

Among patients who took penicillin combinations, carbapenems, cephalosporins, tetracyclines, macrolides, fluoroquinolones, and trimethoprim-sulfamethoxazole, patients who were 65 years or older had a higher CDI ROR than those less than 65 years old (Figure 2). Among patients who took lincosamides, patients who were 65 years or older had a lower CDI ROR than those less than 65 years old.

 Table 2 

Gender and age information for patients on antibiotics

Antibiotic Class/Antibiotic% FemaleMedian age (IQR)
Lincosamides (clindamycin)5858 (28)
Monobactams (aztreonam)5655 (35)
Penicillin combinations4862 (28)
Piperacillin-tazobactam3964 (25)
Amoxicillin-clavulanate5460 (32)
Ampicillin-sulbactam4667 (33)
Carbapenems4463 (29)
Meropenem4461 (32)
Ertapenem4469 (24)
Imipenem-cilastatin4663 (27)
Cephalosporins, monobactams, and carbapenems4763 (34)
Third/fourth-generation cephalosporins4962 (39)
Cefepime4464 (20)
Cefotaxime3944 (61)
Ceftriaxone5163 (41)
Tetracyclines6051 (35)
Tetracycline6026 (32)
Doxycycline6051 (35)
Macrolides6154 (35)
Erythromycin6355 (26)
Clarithromycin6155 (33)
Azithromycin5948 (41)
Fluoroquinolones5858 (27)
Ofloxacin4368 (24)
Ciprofloxacin5757 (28)
Levofloxacin6159 (24)
Moxifloxacin5555 (26)
Trimethoprim-sulfamethoxazole4560 (28)

† IQR = interquartile range

 Figure 1 

Reporting Odds Ratios (RORs) for Clostridium difficile infection with antibiotics. † CI = confidence interval; CDI = Clostridium difficile infection

Int J Med Sci Image
 Figure 2 

Reporting Odds Ratios (RORs) for Clostridium difficile infection with antibiotics stratified by age. † CI = confidence interval; CDI = Clostridium difficile infection; yrs = years

Int J Med Sci Image

The Cochran-Armitage Trend Test demonstrated that there was a significant relationship between the proportion of CDI reports in patients who took fluoroquinolones and the year of reporting (p<0.0001). From 2004 to 2010, 2.3% of fluoroquinolone reports had CDI. From 2011 to 2017, 1.7% of fluoroquinolone reports had CDI.

Discussion

Our antibiotic CDI association rank order was similar to previous meta-analyses [6-8]. Our results demonstrated significant CDI associations (from strongest to weakest) with lincosamides, monobactams, penicillin combinations, carbapenems, cephalosporins, tetracyclines, macrolides, fluoroquinolones, and trimethoprim-sulfonamides.

In a prior meta-analysis of antibiotics and the risk of community-associated CDI (CA-CDI), the risks from the highest to the lowest were: clindamycin, fluoroquinolones, CMCs, macrolides, trimethoprim- sulfonamides, and penicillins, with no effect of tetracycline on CDI risk [6]. In another prior meta-analysis of CA-CDI and antibiotics, the risks from the highest to the lowest were: clindamycin, fluoroquinolones, cephalosporins, penicillins, macrolides, and trimethoprim-sulfonamides, while no association was found between tetracyclines and CDIs [7]. Regarding hospital-acquired CDI (HA-CDI), a prior meta- analysis indicated that the associations from the strongest to weakest were: third-generation cephalosporins, clindamycin, second-generation cephalosporins, fourth-generation cephalosporins, carbapenems, trimethoprim-sulfonamides, fluoroquinolones, and penicillin combinations [8]. FAERS data do not specify whether CDI is community-associated or hospital-acquired; therefore, our results are likely a mixture of CA-CDI and HA-CDI.

The higher CDI RORs associated with clindamycin, penicillin combinations, and carbapenems may be due to their activity against anaerobes and disruption of gut flora [15]. Clindamycin had the highest CDI ROR in our study, which is consistent with the highest CDI risks associated with clindamycin in prior meta-analyses [6,7]. Piperacillin- tazobactam had the second highest ROR in our study; the reasons might include the broad-spectrum antimicrobial activity of piperacillin-tazobactam and the great extent of gut flora disruption as a result [16,17]. Trimethoprim-sulfonamides had the lowest CDI ROR among the antibiotic classes included in our study. In previous meta-analyses, trimethoprim-sulfonamides also had one of the lowest CDI risks [6-8].

Our results demonstrated that fluoroquinolones had a weaker association with CDI compared with most of the antibiotic classes included in the study, except for trimethoprim-sulfonamides. Prior meta- analyses have implicated fluoroquinolones as one of the highest risk antibiotics for CDI [6,7]; however, these studies used data during the CDI epidemic that was associated with the fluoroquinolone-resistant ribotype 027 Clostridium difficile strain [18,19]. A more recent meta-analysis by Vardakas et al. did not implicate fluoroquinolones as one of the highest risk antibiotics, which is consistent with our findings [20]. Given that ribotype 027 strains are now endemic in healthcare settings, our data suggest that fluoroquinolones might not be as important of a CDI risk factor as before considering the recent changes in CDI epidemiology [21]. A recent article published in 2017 demonstrated that a concomitant decline in inpatient fluoroquinolone use and the NAP1/027 strain may have contributed to the decrease in the incidence rate of long-term-care facility-onset CDI from 2011 to 2015 [22]. Our results from the Cochran-Armitage Trend Test also indicated that there was a trend of decrease in CDI risk with fluoroquinolones from 2004 to 2017.

In the subgroup analysis, the CDI ROR rank order in both subgroups (< 65 years old and ≥ 65 years old) were similar to that in all patients. Our results showed that older patients had a higher CDI ROR among most of the antibiotic classes analyzed (Figure 2). It is known that CDI risk is higher in patients 65 years or older [23].

Knowledge of the CDI risk associated with antibiotic classes has important implications for antimicrobial stewardship. Therapeutic interchanges could be identified, especially for those patients who have a high baseline risk for CDI (e.g., elderly, frequent hospitalizations, and comorbid conditions). For example, to treat non-severe purulent skin and skin structure infections in patients with a high risk of CDI, trimethoprim-sulfamethoxazole could be preferred to clindamycin, considering the much lower CDI ROR of trimethoprim-sulfamethoxazole [24].

Limitations

A causal relationship between a drug and an adverse drug reaction (ADR) cannot be established by FAERS. The spontaneous and voluntary reporting of ADRs may lead to significant bias due to underreporting and lack of overall drug use data [25,26]. The association between a drug and an ADR is confounded by concomitant drugs and comorbidities. Media attention and recent drug approval might affect the reporting behaviors. Furthermore, epidemiological shift in the circulating C. difficile strains in the United States might account for the weaker association between fluoroquinolones and CDI in our study; however, the FAERS study design does not permit us to investigate this hypothesis. Therefore, we believe the next step in this line of research will be to confirm these findings in a future case-control or cohort study.

Conclusions

All antibiotic classes evaluated in the study were significantly associated with CDI. Lincosamides (e.g., clindamycin) had the highest CDI ROR and trimethoprim-sulfonamides had the lowest CDI ROR of all the antibiotic classes investigated in this study. Results from FAERS should be interpreted with caution in the context of data limitations. Antibiotic stewardship is needed to prevent CDI and to improve health outcomes.

Abbreviations

ADR: adverse drug reaction; CMC: cephalosporins, monobactams, and carbapenems; CDI: Clostridium difficile infection; CA-CDI: Community- associated CDI; HA-CDI: Hospital-acquired CDI; FDA: Food and Drug Administration; FAERS: FDA Adverse Event Reporting System; CI: confidence interval; IQR: interquartile range; MedDRA: Medical Dictionary for Regulatory Activities; ROR: Reporting Odds Ratio; SOC: System Organ Classes; HLGT: High Level Group Term; HLT: High Level Term; PT: Preferred Term; LLT: Lowest Level Term; SMQ: Standardised MedDRA Queries.

Acknowledgements

No funding was sought for this research study. Dr. Frei was supported, in part, by a NIH Clinical and Translational Science Award (National Center for Advancing Translational Sciences, UL1 TR001120, UL1 TR002645, and TL1 TR002647) while the study was being conducted. Dr. Reveles was supported, in part, by a NIH Clinical Research Scholar (KL2) career development award (National Institute on Aging, P30 AG044271) while the study was being conducted. The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs, the National Institutes of Health, or the authors' affiliated institutions. The FAERS data are freely accessible to the public and do not contain patient identifier information. Therefore, this work is not considered to be human research.

Authors' contributions

Study concept and design: Teng and Frei. Statistical analysis: Teng. Interpretation of data: Teng, Reveles, and Frei. Drafting of the manuscript: Teng. Critical revision of the manuscript for important intellectual content: All authors. Study supervision: Frei.

Competing interests

Dr. Frei has received research grants, to his institution, for investigator-initiated cancer and infectious diseases research, from Allergan (formerly Forest), Bristol Myers Squibb, and Pharmacyclics, in the past three years.

References

1. Reveles KR, Lee GC, Boyd NK, Frei CR. The rise in Clostridium difficile infection incidence among hospitalized adults in the United States: 2001-2010. Am J Infect Control. 2014;42(10):1028-32

2. Centers for Disease Control and Prevention. Healthcare-associated Infections in the United States, 2006-2016: A Story of Progress. Available from: https:// www.cdc.gov/hai/surveillance/data-reports/data-summary-assessing-progress.html. Accessed July 24. 2018

3. Reveles KR, Pugh MJV, Lawson KA. et al. Shift to community-onset Clostridium difficile infection in the national Veterans Health Administration, 2003-2014. Am J Infect Control. 2018;46(4):431-5

4. Rupnik M, Wilcox MH, Gerding DN. Clostridium difficile infection: new developments in epidemiology and pathogenesis. Nat Rev Microbiol. 2009;7(7):526-36

5. Abt MC, McKenney PT, Pamer EG. Clostridium difficile colitis: pathogenesis and host defense. Nat Rev Microbiol. 2016;14(10):609-20

6. Brown KA, Khanafer N, Daneman N, Fisman DN. Meta-analysis of antibiotics and the risk of community-associated Clostridium difficile infection. Antimicrob Agents Chemother. 2013;57(5):2326-32

7. Deshpande A, Pasupuleti V, Thota P. et al. Community-associated Clostridium difficile infection and antibiotics: a meta-analysis. J Antimicrob Chemother. 2013;68(9):1951-61

8. Slimings C, Riley TV. Antibiotics and hospital-acquired Clostridium difficile infection: update of systematic review and meta-analysis. J Antimicrob Chemother. 2013;69(4):881-91

9. Food and Drug Administration. FDA Adverse Event Reporting System (FAERS). Available from http://www.fda.gov/Drugs/GuidanceCompliance RegulatoryInformation/Surveillance/AdverseDrugEffects/default.htm. Accessed July 24. 2018

10. Food and Drug Administration. Drugs@FDA: FDA Approved Drug Products. Available from https://www.accessdata.fda.gov/scripts/cder/daf/index. cfm. Accessed July 24. 2018

11. McConeghy KW, Soriano MM, Danziger LH. A quantitative analysis of FDA adverse event reports with oral bisphosphonates and Clostridium difficile. Pharmacotherapy. 2016;36(10):1095-101

12. Evans SJW, Waller PC, Davis S. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf. 2001;10(6):483-6

13. MedDRA MSSO. Introductory Guide for Standardised MedDRA Queries (SMQs) Version 21.0. Available from http://www.meddra.org/sites/default/ files/guidance/file/smq_intguide_21_0_english.pdf. Accessed July 24. 2018

14. Bate A, Evans SJW. Quantitative signal detection using spontaneous ADR reporting. Pharmacoepidemiol Drug Saf. 2009;18(6):427-36

15. Brook I, Wexler HM, Goldstein EJ. Antianaerobic antimicrobials: spectrum and susceptibility testing. Clin Microbiol Rev. 2013;26(3):526-46

16. Modi SR, Collins JJ, Relman DA. Antibiotics and the gut microbiota. J Clin Invest. 2014;124(10):4212-8

17. Jump RL, Polinkovsky A, Hurless K. et al. Metabolomics analysis identifies intestinal microbiota-derived biomarkers of colonization resistance in clindamycin-treated mice. PLoS One. 2014;9(7):e101267

18. McDonald LC, Killgore GE, Thompson A. et al. An epidemic, toxin gene-variant strain of Clostridium difficile. N Engl J Med. 2005;353(23):2433-41

19. Denève C, Bouttier S, Dupuy B, Barbut F, Collignon A, Janoir C. Effects of subinhibitory concentrations of antibiotics on colonization factor expression by moxifloxacin-susceptible and moxifloxacin-resistant Clostridium difficile strains. Antimicrob Agents Chemother. 2009;53(12):5155-62

20. Vardakas KZ, Trigkidis KK, Boukouvala E, Falagas ME. Clostridium difficile infection following systemic antibiotic administration in randomised controlled trials: a systematic review and meta-analysis. Int J Antimicrob Agents. 2016;48(1):1-10

21. He M, Miyajima F, Roberts P. et al. Emergence and global spread of epidemic healthcare-associated Clostridium difficile. Nat Genet. 2013;45(1):109-13

22. Guh AY, Mu Y, Baggs J. et al. Trends in incidence of long-term-care facility onset Clostridium difficile infections in 10 US geographic locations during 2011-2015. Am J Infect Control. 2018;46(7):840-2

23. Lessa FC, Mu Y, Bamberg WM. et al. Burden of Clostridium difficile infection in the United States. N Engl J Med. 2015;372(9):825-34

24. Stevens DL, Bisno AL, Chambers HF. et al. Practice guidelines for the diagnosis and management of skin and soft tissue infections: 2014 update by the Infectious Diseases Society of America. Clin Infect Dis. 2014;59(2):e10-52

25. Moore N, Thiessard F, Begaud B. The history of disproportionality measures (reporting odds ratio, proportional reporting rates) in spontaneous reporting of adverse drug reactions. Pharmacoepidemiol Drug Saf. 2005;14(4):285-6

26. Montastruc JL, Sommet A, Bagheri H, Lapeyre-Mestre M. Benefits and strengths of the disproportionality analysis for identification of adverse drug reactions in a pharmacovigilance database. Br J Clin Pharmacol. 2011;72(6):905-8

Author contact

Corresponding address Corresponding author: Christopher R. Frei, PharmD, FCCP, BCPS, Director, Pharmacotherapy Education and Research Center, Long School of Medicine, University of Texas Health-San Antonio, 7703 Floyd Curl Dr., MSC-6220, San Antonio, TX 78229; email: freicedu


Received 2018-10-16
Accepted 2019-2-8
Published 2019-5-7


Citation styles

APA
Teng, C., Reveles, K.R., Obodozie-Ofoegbu, O.O., Frei, C.R. (2019). Clostridium difficile Infection Risk with Important Antibiotic Classes: An Analysis of the FDA Adverse Event Reporting System. International Journal of Medical Sciences, 16(5), 630-635. https://doi.org/10.7150/ijms.30739.

ACS
Teng, C.; Reveles, K.R.; Obodozie-Ofoegbu, O.O.; Frei, C.R. Clostridium difficile Infection Risk with Important Antibiotic Classes: An Analysis of the FDA Adverse Event Reporting System. Int. J. Med. Sci. 2019, 16 (5), 630-635. DOI: 10.7150/ijms.30739.

NLM
Teng C, Reveles KR, Obodozie-Ofoegbu OO, Frei CR. Clostridium difficile Infection Risk with Important Antibiotic Classes: An Analysis of the FDA Adverse Event Reporting System. Int J Med Sci 2019; 16(5):630-635. doi:10.7150/ijms.30739. https://www.medsci.org/v16p0630.htm

CSE
Teng C, Reveles KR, Obodozie-Ofoegbu OO, Frei CR. 2019. Clostridium difficile Infection Risk with Important Antibiotic Classes: An Analysis of the FDA Adverse Event Reporting System. Int J Med Sci. 16(5):630-635.

This is an open access article distributed under the terms of the Creative Commons Attribution (CC BY-NC) license (https://creativecommons.org/licenses/by-nc/4.0/). See http://ivyspring.com/terms for full terms and conditions.
Popup Image