PDF

Int J Med Sci 2015; 12(4):312-321. doi:10.7150/ijms.10703 This issue Cite

Research Paper

Evaluation of Dabigatran- and Warfarin-Associated Hemorrhagic Events Using the FDA-Adverse Event Reporting System Database Stratified by Age

Junko ABE^1,2, Ryogo UMETSU¹, Yamato KATO¹, Natsumi UEDA¹, Yoko NAKAYAMA¹, Yukiya SUZUKI¹, Toshiyuki SUZUKI¹, Hideko NAGASAWA³, Yasutomi KINOSADA⁴, Mitsuhiro NAKAMURA¹

1. Laboratory of Drug Informatics, Gifu Pharmaceutical University
2. Medical Database Co., LTD
3. Laboratory of Pharmaceutical and Medical Chemistry, Gifu Pharmaceutical University
4. Department of Biomedical Informatics, Gifu University Graduate School of Medicine, JAPAN

✉ Corresponding author: Mitsuhiro Nakamura, Laboratory of Drug Informatics, Gifu Pharmaceutical University, 1-25-4, Daigaku-Nishi, Gifu, 501-1196, JAPAN, Tel: +81-58-230-8100, Fax: +81-58-230-8105, E-mail: mnakamuraac.jp

Abstract

Dabigatran and warfarin are oral anticoagulant drugs widely used for the prevention of stroke in patients with atrial fibrillation. The objective of this study was to evaluate the interaction between aging and dabigatran- and warfarin-induced gastrointestinal (GI) and nervous system hemorrhage using data available in the FDA Adverse Event Reporting System (FAERS) database.

We analyzed reports of hemorrhagic events in the GI and nervous system recorded in the FAERS database between 2004 and 2014 using an adjusted reporting odds ratio (ROR).

We demonstrated that dabigatran-associated GI hemorrhage was significantly increased in patients over the age of 80 years. The RORs of dabigatran increased with increasing age, although aging had little effect on warfarin-associated GI hemorrhage. The ROR for anticoagulant-associated nervous system hemorrhage was not significantly affected by aging, as compared to GI hemorrhage.

Our results indicate that the excretion of dabigatran may be affected by aging, as compared to warfarin, likely due to renal function decline. Our results emphasize the need for physicians to closely monitor GI bleeding in aging patients, because it is closely related to renal function deterioration.

Keywords: dabigatran, warfarin, hemorrhagic events, adverse event reporting system

Introduction

Dabigatran is a new oral anticoagulant drug used widely for the prevention of stroke and systemic embolism in patients with nonvalvular atrial fibrillation [1]. The Food and Drug Administration (FDA) approved dabigatran based on the results the phase III, prospective, randomized, open-label multi-national Randomized Evaluation of Long-Term Anticoagulation Therapy (RE-LY) clinical trial [2]. In the RE-LY trial, the rate of stroke and systemic embolism was similar between dabigatran and warfarin [2]. Major bleeding was significantly reduced with dabigatran (110 mg), as compared to warfarin, whereas 150 mg showed an effect similar to warfarin. Furthermore, dabigatran was associated with lower rates of major bleeding and intracranial bleeding than warfarin treatment [2].

Dabigatran is primarily excreted unmetabolized by the kidneys [3]. It was reported that dabigatran concentrations increase approximately two- to three-fold in patients with moderate renal impairment, as compared to patients with normal renal function [4]. Indeed, gastrointestinal (GI) bleeding was increased with 150 mg dabigatran, but not 110 mg dabigatran [2]. Thus, higher blood concentrations of dabigatran may increase the risk of GI bleeding [5].

The effects of dabigatran on GI bleeding are of interest in older patients, because atrial fibrillation is largely a disease of the elderly, and the risk of stroke and bleeding increases with advanced age. In the RE-LY trial, Eikelboom et al. reported that, in patients older than 75, the risk of intracranial bleeding was lower, but the risk of extra-cranial bleeding was similar or higher with both doses (150 mg and 110 mg) of dabigatran, as compared to warfarin [5]. Because ischemic strokes and systemic embolisms have greater clinical significance than nonfatal bleeding, such as GI bleeding, higher doses of dabigatran are more favorable in elderly patients [2]. However, acute and chronic GI bleeding has a negative effect on a patient's quality of life.

The FDA Adverse Event Reporting System (FAERS) database, a spontaneous reporting system, is the primary tool used for pharmacovigilance. The FAERS is a rich resource, and data mining indices provide a powerful means to identify potential associations between drugs and adverse events. Dabigatran is a direct oral thrombin inhibitor, and is administered in a fixed dose, without laboratory monitoring [6]. Initially, it was expected to be an alternative therapy to warfarin; however, reports of serious and fatal bleeding events associated with dabigatran use increased in the FAERS database after approval [7]. Thus, the effects of dabigatran use on internal bleeding remain unclear. In the RE-LY study, patients with severe renal impairment were excluded [2, 6]. In contrast, the FAERS database contains information on patients with varying renal function, ranging from normal to severe dysfunction. Thus, evaluation of GI bleeding events using the FAERS database is valuable, because it reflects the realities of clinical practice.

Recently, data mining algorithms have been developed for use in spontaneous adverse event reporting databases, such as the FAERS database, to identify drug-associated adverse events by disproportionality analysis [8, 9]. The crude reporting odds ratio (ROR) is used by the Pharmaceuticals and Medical Devices Agency in Japan and the Netherlands Pharmacovigilance Center [10]. The crude ROR is an applicable technique that allows for adjustments through logistic regression analyses and control of covariates [11]. We hypothesized that it may be possible to adjust for the above-mentioned reporting bias using this approach.

The effects of aging on dabigatran- and warfarin-induced bleeding have not yet assessed using RORs adjusted by logistic regression analyses. The purpose of this study was to evaluate the relationship between aging and dabigatran-associated hemorrhage, and to compare the data with that obtained from warfarin using the FAERS database.

Methods

Data sources

The FAERS database, which covered the period from January 2004 to March 2014, was obtained from the FDA website (www.fda.gov). The FAERS structure complies with the international safety reporting guidelines, ICH E2B. The adverse events are coded according to the terminology preferred by the Medical Dictionary for Regulatory Activities (MedDRA) [12].

The drugs selected for this investigation were dabigatran and warfarin. The FAERS database permits contributors to register drugs under any name, including a trade name and an abbreviation. The DrugBank database contains drug information used globally, including 1,447 FDA-approved small molecule drugs [13], and was utilized as a dictionary for the batch conversion and compilation of drug names. For duplicate entries, we followed the FDA's recommendation as described on the FAERS website, to adopt the most recent case number to identify duplicate reports from the same patient and excluded them from the analysis.

Definition of hemorrhage events

This study relied on definitions provided by MedDRA version 17.1. To evaluate dabigatran- and warfarin-associated hemorrhagic events in the GI system, we utilized the Standardized MedDRA Query (SMQ) for hemorrhages events (SMQ code: 20000038) and the System Organ Class (SOC) for gastrointestinal disorder, and selectively extracted reports that met both criteria. The number of selected preferred terms for hemorrhages, limited by the SOC (gastrointestinal disorder), was 71. Furthermore, to evaluate dabigatran- and warfarin-associated hemorrhagic events in the nervous system, such as intracranial hemorrhage, we utilized 35 preferred terms that matched the SMQ for hemorrhages events (SMQ code: 20000038) and the SOC (nervous system disorder).

Analysis

Using established pharmacovigilance indices, we evaluated the reporting odds ratio (ROR) to establish the effects of dabigatran and warfarin on “hemorrhagic events.” “Cases” were defined as patients who reported “hemorrhagic events,” while “non-cases” consisted of patients associated with all other reports. The reporting odds ratio (ROR) is the ratio of the odds of reporting adverse events versus all other events associated with dabigatran or warfarin compared to the reporting odds for all other drugs present in the database. To compare the “cases” and “non-cases,” we calculated the RORs as (a:c)/(b:d). RORs were expressed as point estimates with a 95% confidence interval (CI). To evaluate the effect of age on “hemorrhagic events,” the reports were stratified into age groups: 0-29, 30-39, 40-49, 50-59, 60-69, 70-79, 80-89, and greater than 90.

For signal detection, general qualitative judgments were used. The detection of a signal was dependent on the signal indices exceeding a predefined threshold. ROR values < 1 indicated no exposure-event association, and estimates > 1 indicated exposure-event safety signals. Safety signals are considered significant when the ROR estimates and the lower limits of the corresponding 95% CI are ≥ 2. [10]

We refined the signal with a dedicated correction to detect possible confounders present in the database using logistic regression analysis. After adjusting for gender, reporting year, and stratified age groups, the RORs were calculated using logistic regression analysis. To construct the logistic model, the drugs and stratified age groups were coded. The following logistic model was used for analysis:

Log (odds) = β₀ + β₁G + β₂Y + β₃D + β₄A + β₅D*A  (1) 

(G = gender, Y = reporting year, D = drug (dabigatran or warfarin), and A = stratified age group)

The adjusted RORs were calculated using the 40-49 year old group as a reference group. This model can be compared with a model in which no interaction term is present. A likelihood ratio test can be used to evaluate the effect of adding this term. Because the difference in -2 log likelihood follows a chi-square distribution with one degree of freedom after adding the interaction term, a probability (p) value of 0.05 or less was considered statistically significant. Data analyses were performed using JMP, version 11.0 (SAS Institute Inc., Cary, NC, USA).

Results

The FAERS database contains 5,597,297 reports from the first quarter of 2004 through the end of the first quarter of 2014. After excluding duplicates according to the FDA recommendation and extracting reports that contained the age and the gender of the patients, 2,143,443 reports were analyzed. The RORs of dabigatran and warfarin are summarized in Table 2. The RORs (95% CI) of hemorrhage associated with dabigatran, limited by the SOC (GI disorders), in patients age 40-49, 70-79, 80-89, and ≥ 90 were 4.88 (3.26-7.31), 13.55 (12.79-14.35), 19.34 (18.30-20.44), and 26.18 (23.05-29.74), respectively. The RORs for gastrointestinal hemorrhage increased with advancing age after dabigatran treatment (Figure 1). The RORs (95% CI) of hemorrhage associated with warfarin, limited by the SOC (GI disorders), in patients age 40-49, 70-79, 80-89, and ≥ 90 were 2.95 (2.55-3.41), 4.74 (4.46-5.03), 5.80 (5.42-6.20), and 5.39 (4.42-6.57), respectively. The ROR signal for GI hemorrhage in elderly patients treated with dabigatran was higher than in patients treated with warfarin.

The RORs (95% CI) of hemorrhage associated with dabigatran, limited by the SOC (nervous system disorders), in patients age 40-49, 70-79, 80-89, and ≥ 90 were 3.54 (1.46-8.57), 9.57 (8.54-10.72), 10.44 (9.31-11.71), and 10.11 (7.63-13.40), respectively (Table 3). The RORs (95% CI) of hemorrhage associated with warfarin, limited by the SOC (nervous system disorders), in patients age 40-49, 70-79, 80-89, and ≥ 90 were 2.79 (2.09-3.72), 4.92 (4.40-5.50), 6.58 (5.85-7.41), and 6.14 (4.34-8.69), respectively. The RORs for nervous system hemorrhage had no significant correlation with age.

 Figure 1 

Adjusted reporting odds ratios and 95% confidence intervals for dabigatran- and warfarin- associated hemorrhagic events, limited by gastrointestinal disorders. Open circles, dabigatran; triangles, warfarin; filled circles, control.

 Figure 2 

Adjusted reporting odds ratios and 95% confidence intervals, for dabigatran- and warfarin-associated hemorrhagic events, limited by nervous system disorders. Open circles, dabigatran; triangles, warfarin; filled circles, control.

The number of GI hemorrhage cases and crude RORs (95% CI) are summarized in Table 4. The crude RORs (95% CI) for GI hemorrhage in dabigatran-treated patients aged 80-89 and ≥ 90 were 10.61 (9.98-11.28) and 13.22 (11.42-15.30), respectively. The likelihood ratio test of the interaction terms dabigatran*60-69, dabigatran*80-89 and dabigatran*≥ 90 were statistically significant (Table 6). The adjusted RORs for dabigatran*60-69, dabigatran*80-89, and dabigatran*≥ 90, were 16.66 (95% CI, 11.01-25.23), 31.36 (95% CI, 20.81-47.26), and 45.14 (95% CI, 29.30-69.53), respectively. In contrast, the crude RORs (95% CI) for warfarin did not increase with advancing age (Table 5). The likelihood ratio test of the interaction term warfarin*70-79, warfarin*80-89, and warfarin*≥ 90 were statistically significant (Table 6). The adjusted RORs for warfarin*70-79, warfarin*80-89, and warfarin*≥ 90, were 7.33 (95% CI, 6.23-8.62), 9.21 (95% CI, 7.80-10.87), and 9.02 (95% CI, 6.96-11.69).

The crude ROR for dabigatran- and warfarin-associated nervous system hemorrhage did not increase with advancing age. Further, the likelihood ratio test of the interaction term was not statistically significant (Table 6).

Discussion

Bleeding is the most common complication associated with the use of anticoagulant drugs. We examined the association between hemorrhagic events in the GI and nervous system and anticoagulant drugs (dabigatran and warfarin) after stratification by age. In this study, we demonstrated that dabigatran-associated GI hemorrhage was significantly increased in patients over the age of 80 (Table 6). Evaluation of GI hemorrhage revealed that the adjusted RORs of dabigatran increased with advancing age, whereas aging had little effect on warfarin (Table 6 and Figure 1). The adjusted ROR for dabigatran-associated GI hemorrhage was higher than the adjusted ROR of warfarin. Our study supports the results of the RE-LY trials and the safety announcements issued by regulating authorities. In contrast, the RORs of dabigatran- and warfarin-associated nervous system hemorrhage were less affected by aging (Figure 2, Table 6). Since dabigatran is primarily excreted by the kidney and warfarin is metabolized in the liver by cytochrome P450 [14], the effect of dabigatran on GI hemorrhage may be affected by changes in kidney function due to aging.

The RE-LY trial indicated that dabigatran is associated with a reduced risk of intracranial hemorrhage, as compared to warfarin [5]. Furthermore, new retrospective post-marketing studies also indicate that dabigatran is associated with a lower risk of intracranial hemorrhage [15, 16]. In contrast, the adjusted dabigatran RORs for nervous system hemorrhage did not indicate lower adjusted RORs compared to warfarin in our study (dabigatran: 5.61 [95% CI 1.99-12.22]; warfarin: 4.83 [95% CI 3.55-6.41]) (Table 6). Our results showd that the 95% CI of the adjusted dabigatran RORs was broad and not significant. We do not have a conclusive explanation for these data. We adjusted the crude ROR by coding the terms of gender, reporting year, drug, and stratified age groups in the logistic mode. However, our results from the FAERS database using this logistic model could not account for our observations. This contradiction could be considered the result of unobserved bias. Alternatively, differences in the definition of hemorrhagic adverse events in our study, the Preferred Terms (PTs) from MedDRA, and other studies could cause this effect. Furthermore, the manufacturer recommends that high-risk elderly patients (over 75 years of age) and those with chronic kidney disease should be given a lower dose of dabigatran; however, we could not determine whether dabigatran doses were lowered.

The pharmacokinetic profile of dabigatran can be affected by concomitant administration of several drugs. Dabigatran etexilate is a substrate for p-glycoprotein; thus, drugs that inhibit or induce p-glycoprotein could potentiate or attenuate the anticoagulant effect of dabigatran [3]. The effects of this drug-drug interaction should be evaluated with respect to anticoagulant-associated hemorrhage using a well-organized epidemiologic studies and/or the FAERS database.

After the approval of dabigatran, the FDA received numerous reports of severe dabigatran-related bleeding events [7]. Safety advisories have been issued by the FDA, the European Medicine Agency, and the Australian Therapeutic Goods Authority [17-19]. The reports of increased bleeding with dabigatran differed from those in the RE-LY trial, and were likely the result of passive reporting in the FAERS database, which can lead to reporting bias. Currently, the increase in severe bleeding events associated with dabigatran in the FAERS database is regarded as the result of reporting bias [7]. Thus, regulating authorities have not altered the safety profile of dabigatran, based on its overall benefit-risk profile [17, 20].

Several post-marketing studies provide more data on the bleeding risks among patients with atrial fibrillation [15, 16]. A large post-marketing study of dabigatran evaluating 134,414 elderly patients showed the comparative safety of dabigatran versus warfarin in general practice settings between October 2010 and December 2012 [16]. This analysis confirmed a reduced risk of major bleeding and intracranial hemorrhage with dabigatran. In a press release, Boehringer Ingelheim, Inc. pointed out that the FDA analysis supported the positive safety and efficacy profile of dabigatran in the RE-LY trial [21].

Another retrospective post-marketing study evaluating 9,404 Medicare patients over a 6-month follow-up period reported that dabigatran was associated with a higher incidence of major bleeding relative to warfarin, a higher risk of gastrointestinal bleeding, but a lower risk of intracranial hemorrhage [15]. Their results differed from the RE-LY trial, which showed no difference in the rates of major bleeding with dabigatran and warfarin. The risk of major bleeding among dabigatran users was especially high for African Americans and patients with chronic kidney disease. These results should be interpreted with caution, due to the relatively small size of the study.

Disproportionality analysis has several limitations that are inherent to the nature of the data and require consideration prior to drawing conclusions. In general, ROR cannot be used to infer the comparative strength of causality [22, 23]. Rather, it offers a rough indication of the signal strength, used to generate hypotheses to search for unknown potential adverse reactions [24]. It is impossible to evaluate the “true” risk of hemorrhage without information concerning the total number of patients administered dabigatran. Since dabigatran and warfarin users are very different in several factors that directly affect the risk of bleeding, failing to adjust would bias the results, as our unadjusted estimates indicate. While hemorrhagic events have been document in the FAERS database, careful attention must be paid to the interpretation of the results.

Recently, the use of quantitative measures, in addition to qualitative analysis, has become increasingly important in signal detection for pharmacovigilance [22]. Several researchers have demonstrated that disproportionality measures can provide new, causal insights. These studies each use an approach that might circumvent biases, such as selection and reporting biases. Mitigating the effect of confounding factors by such approaches enhances the robustness of results. For example, van Puijenbroek et al. evaluated the association between two drugs and a single event (drug-drug interactions) using a statistical interaction term in a logistic model to calculate the adjusted RORs [11].

To our knowledge, reports on safety signal detection using logistic regression analyses focusing on age stratification are scarce. This study was the first to evaluate the association between aging and dabigatran and warfarin in GI and nervous system bleeding using the FAERS database by logistic regression. We adjusted the crude ROR by coding the terms of gender, reporting year, drug, and stratified age groups in the logistic model. In our logistic regression analysis, the adjusted RORs after adding adjusting terms were different than the crude RORs (Table 6). Thus, the adjustment of variables might influence the reporting ratio of adverse events. We demonstrated that the effect of age on the association between dabigatran and GI bleeding cannot be ignored in spontaneous adverse reporting. We consider our results valid, due to the appropriate analysis methods and the special attention paid to potential bias.

Until more evidence is available, prescribers should carefully monitor bleeding complications in elderly patients with renal impairment, a group that is known to have an increased risk of bleeding. This information could potentially be useful for improved management of GI bleeding during dabigatran treatment, and may be particularly beneficial to prescribers.

We sought to evaluate, using a real-world setting, any differences in bleeding between dabigatran users after adjusting for patient differences using appropriate analysis methods. Our study indicates the importance of comparing the safety profiles of newer and traditional drugs using post-marketing real-world data. After considering the causality in the current analysis, further epidemiological studies are recommended in elderly patients.

Acknowledgements

This research was partially supported by JSPS KAKENHI Grant Number, 24390126.

Conflict of Interest

The authors declare no conflict of interest.

References

1. Pokorney SD, Sherwood MW, Becker RC. Clinical strategies for selecting oral anticoagulants in patients with atrial fibrillation. J thromb thrombolysis. 2013;36:163-74

2. Connolly SJ, Ezekowitz MD, Yusuf S. et al. Dabigatran versus warfarin in patients with atrial fibrillation. N Engl J Med. 2009;361:1139-51

3. Boehringer Ingelheim Pharmaceuticals. Pradaxa (Dabigatran Etexilate Mesylate) Product Information. Ridgefield, DC, USA: Boehringer Ingelheim Pharmaceuticals. 2011

4. Stangier J, Rathgen K, Stähle H. et al. Influence of renal impairment on the pharmacokinetics and pharmacodynamics of oral dabigatran etexilate: an open-label, parallel-group, single-centre study. Clin Pharmacokinet. 2010;49:259-68

5. Eikelboom JW, Wallentin L, Connolly SJ. et al. Risk of bleeding with 2 doses of dabigatran compared with warfarin in older and younger patients with atrial fibrillation: an analysis of the randomized evaluation of long-term anticoagulant therapy (RE-LY) trial. Circulation. 2011;123:2363-72

6. Beasley BN, Unger EF, Temple R. Anticoagulant options--why the FDA approved a higher but not a lower dose of dabigatran. N Engl J Med. 2011;364:1788-90

7. Southworth MR, Reichman ME, Unger EF. Dabigatran and postmarketing reports of bleeding. N Engl J Med. 2013;368:1272-4

8. Sakaeda T, Tamon A, Kadoyama K. et al. Data mining of the public version of the FDA Adverse Event Reporting System. Int J Med Sci. 2013;10:796-803

9. Bate A, Evans SJ. Quantitative signal detection using spontaneous ADR reporting. Pharmacoepidemiol Drug Saf. 2009;18:427-36

10. Van Puijenbroek EP, Bate A, Leufkens HG. et al. A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol Drug Saf. 2002;11:3-10

11. Van Puijenbroek EP, Egberts AC, Heerdink ER. et al. Detecting drug-drug interactions using a database for spontaneous adverse drug reactions: an example with diuretics and non-steroidal anti-inflammatory drugs. Eur J Clin Pharmacol. 2000;56:733-38

12. MedDRA. www.meddra.org

13. DrugBank. www.drugbank.ca

14. Takahashi H, Echizen H. Pharmacogenetics of warfarin elimination and its clinical implications. Clin Pharmacokinet. 2001;40:587-603

15. Hernandez I, Baik SH, Pinera A, Zhang Y. Risk of Bleeding With Dabigatran in Atrial Fibrillation. JAMA Intern Med. 2015;175:18-24

16. Graham DJ, Reichman ME, Wernecke M. et al. Cardiovascular, Bleeding, and Mortality Risks in Elderly Medicare Patients Treated with Dabigatran or Warfarin for Non-Valvular Atrial Fibrillation. Circulation. 2015;131:157-64

17. Drug safety communication—safety review of post-market reports of serious bleeding events. Food and Drug Administration. http://www.fda.gov/drugs/drugsafety/ucm282724.htm

18. European Medicines Agency updates on safety of Pradaxa. European Medicines Agency. http://www.ema.europa.eu/docs/en_GB/document_library/Press_release/2011/11/WC500117818.pdf

19. Dabigatran (Pradaxa): risk of bleeding relating to use. Australian Therapeutic Goods Authority (TGA). http://www.tga.gov.au/safety/alerts-medicine-dabigatran-111005.htm

20. Questions and answers on the review of bleeding risk with Pradaxa (dabigatran etexilate). European Medicines Agency. http://www.ema.europa.eu/docs/en_GB/document_library/Medicine_QA/2012/05/WC500127768.pdf

21. FDA Study of Medicare Patients Reaffirms Safety and Efficacy Profile of Pradaxa® (dabigatran etexilate mesylate) for NVAF. Boehringer Ingelheim, Inc. http://us.boehringer-ingelheim.com/news_events/press_releases/press_release_archive/2014/11-03-14-fda-study-medicare-patients-reaffirms-safety-efficacy-profile-pradaxa-dabigatran-etexilate-mesylate-nvaf.html

22. Egberts AC, Meyboom RH, van Puijenbroek EP. Use of measures of disproportionality in pharmacovigilance: three Dutch examples. Drug Saf. 2002;25:453-8

23. Montastruc JL, Sommet A, Bagheri H. et al. Benefits and strengths of the disproportionality analysis for identification of adverse drug reactions in a pharmacovigilance database. Br J Clin Pharmacol. 2011;72:905-8

24. Pariente A, Gregoire F, Fourrier-Reglat A. et al. Impact of safety alerts on measures of disproportionality in spontaneous reporting databases: the notoriety bias. Drug Saf. 2007;30:891-8

Author contact

Corresponding author: Mitsuhiro Nakamura, Laboratory of Drug Informatics, Gifu Pharmaceutical University, 1-25-4, Daigaku-Nishi, Gifu, 501-1196, JAPAN, Tel: +81-58-230-8100, Fax: +81-58-230-8105, E-mail: mnakamuraac.jp

Received 2014-10-2
Accepted 2015-2-25
Published 2015-3-28

Citation styles

©2024 Ivyspring International Publisher. Terms of use