https://orcid.org/0000-0002-6244-2168
Figures
Abstract
Background
Glycemic control remains suboptimal in developing countries due to critical system deficiencies. An innovative mobile health (mHealth)-enabled hierarchical diabetes management intervention was introduced and evaluated in China with the purpose of achieving better control of type 2 diabetes in primary care.
Methods and findings
A community-based cluster randomized controlled trial was conducted among registered patients with type 2 diabetes in primary care from June 2017 to July 2019. A total of 19,601 participants were recruited from 864 communities (clusters) across 25 provinces in China, and 19,546 completed baseline assessment. Moreover, 576 communities (13,037 participants) were centrally randomized to the intervention and 288 communities (6,509 participants) to usual care. The intervention was centered on a tiered care team–delivered mHealth-mediated service package, initiated by monthly blood glucose monitoring at each structured clinic visit. Capacity building and quarterly performance review strategies upheld the quality of delivered primary care. The primary outcome was control of glycated hemoglobin (HbA1c; <7.0%), assessed at baseline and 12 months. The secondary outcomes include the individual/combined control rates of blood glucose, blood pressure (BP), and low-density lipoprotein cholesterol (LDL-C); changes in levels of HbA1c, BP, LDL-C, fasting blood glucose (FBG), and body weight; and episodes of hypoglycemia. Data were analyzed using intention-to-treat (ITT) generalized estimating equation (GEE) models, accounting for clustering and baseline values of the analyzed outcomes. After 1-year follow-up, 17,554 participants (89.8%) completed the end-of-study (EOS) assessment, with 45.1% of them from economically developed areas, 49.9% from urban areas, 60.5 (standard deviation [SD] 8.4) years of age, 41.2% male, 6.0 years of median diabetes duration, HbA1c level of 7.87% (SD 1.92%), and 37.3% with HbA1c <7.0% at baseline. Compared with usual care, the intervention led to an absolute improvement in the HbA1c control rate of 7.0% (95% confidence interval [CI] 4.0% to 10.0%) and a relative improvement of 18.6% (relative risk [RR] 1.186, 95% CI 1.105 to 1.267) and an absolute improvement in the composite ABC control (HbA1c <7.0%, BP <140/80 mm Hg, and LDL-C <2.6 mmol/L) rate of 1.9% (95% CI 0.5 to 3.5) and a relative improvement of 21.8% (RR 1.218, 95% CI 1.062 to 1.395). No difference was found on hypoglycemia episode and weight gain between groups. Study limitations include noncentralized laboratory tests except for HbA1c, and caution should be exercised when extrapolating the findings to patients not registered in primary care system.
Author summary
Why was this study done?
- In many countries, health outcomes are impacted by the consequences of poor glycemic control among people living with diabetes owing to the low quality of primary care.
- Digital technologies have been increasingly adopted as an innovative solution, especially during the epidemic of communicable diseases like Coronavirus Disease 2019 (COVID-19).
- However, very limited literature reports its clinical effectiveness in primary care settings, and the majority of favorable results are from studies with small sample size.
What did the researchers do and find?
- A mobile health (mHealth)-based digital platform named Road to Hierarchical Diabetes Management at Primary Care Settings in China (ROADMAP) was designed and tested through a community-based cluster randomized controlled trial that covered 19,546 participants from 864 communities across China.
- Compared with usual care, the intervention lowered the glycated hemoglobin (HbA1c) level by 0.3% and improved the HbA1c control rate by 7.0% absolutely and 18.6% relatively. No additional hypoglycemia episodes and weight gain were found relevant to the intervention.
What do these findings mean?
- To our knowledge, this study is the largest in testing the effectiveness and safety of an mHealth-based diabetes management with a diverse coverage of primary care settings.
- The findings indicate that integrated mHealth solution could be effective and safe in improving diabetes control in real life and could be more feasible and acceptable during the pandemic of COVID-19.
- Detailed findings in implementation and evaluation of the intervention could inform the promotion of effective and rational use of limited healthcare resource in China and other low- and middle-income countries (LMICs).
Citation: Jia W, Zhang P, Zhu D, Duolikun N, Li H, Bao Y, et al. (2021) Evaluation of an mHealth-enabled hierarchical diabetes management intervention in primary care in China (ROADMAP): A cluster randomized trial. PLoS Med 18(9): e1003754. https://doi.org/10.1371/journal.pmed.1003754
Academic Editor: Sanjay Basu, Harvard Medical School, UNITED STATES
Received: January 18, 2021; Accepted: August 4, 2021; Published: September 21, 2021
Copyright: © 2021 Jia et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data Availability Statement: All data can be viewed in NODE (http://www.biosino.org/node) by pasting the accession OEP000201 into the text search box or through the URL: http://www.biosino.org/node/project/detail/OEP000201. Researchers can apply for access to anonymized study data and associated documents through NODE. Any cross-border transfer of individual dada must obtain official approval following legal procedures. The export of human-related data is governed by the Ministry of Science and Technology of China (MOST) in accordance with the Regulations of the People’s Republic of China on Administration of Human Genetic Resources (State Council No.717).
Funding: Funding: This study was funded by the Bethune Charitable Foundation (X-J-2017-014, http://bqejjh.org.cn/) through one donation from Tonghua Dongbao Pharmaceutical Co., Ltd (the ROADMAP study group, PI: WJ). Wuxi BioHermes Bio&Medical Technology Co., Ltd provided point-of-care HbA1c analyzers (A1c EZ 2.0) for universal testing of HbA1c (the ROADMAP study group, PI: WJ). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: Competing interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: WJ is a member of the Editorial Board of PLOS Medicine. The authors declare that no other competing interests exist.
Abbreviations: BP, blood pressure; BPHS, basic public health services; CI, confidence interval; CONSORT, Consolidated Standards of Reporting Trials; COVID-19, Coronavirus Disease 2019; DBP, diastolic blood pressure; EOS, end of study; FBG, fasting blood glucose; GEE, generalized estimating equation; HbA1c, glycated hemoglobin; ICC, intraclass correlation coefficient; ITT, intention-to-treat; LDL-C, low-density lipoprotein cholesterol; LMIC, low- and middle-income country; mHealth, mobile health; MI, multiple imputation; OR, odds ratio; ROADMAP, Road to Hierarchical Diabetes Management at Primary Care Settings in China; RD, risk difference; RR, relative risk; SBP, systolic blood pressure; SD, standard deviation
Introduction
While facing the rising prevalence and growing burden of type 2 diabetes, health systems worldwide, particularly in low- and middle-income countries (LMICs), are also threatened by the consequence of poor glycemic control among people living with diabetes [1–4]. In China, the prevalence of diabetes has reached 11.2%, with an estimated population of 129.8 million [5]. To ensure access to primary care, which is essential for people affected by diabetes and other chronic conditions, a national basic public health services (BPHS) program was introduced in 2009 [6]. The program offers people with diabetes with a minimum of 4 blood glucose tests per year, as well as blood pressure (BP) measurement and lifestyle consultations and medication instructions at each clinic visit, and necessary referral. Diabetes management attained remarkable progress since then, in increased self-awareness (30.1% in 2010, 36.5% in 2013, and 43.3% in 2017) and antihyperglycemic treatment coverage (25.8%, 32.2%, and 49.0%, respectively) [5]. However, the control of glycemic level remains suboptimal with little improvement (49.2% in 2013 and 49.4% in 2017) [5], having only 5.6% of people with type 2 diabetes achieved the composite diabetes control (glycated hemoglobin [HbA1c] <7.0%, BP <130/80 mm Hg, and low-density lipoprotein cholesterol [LDL-C] <2.6 mmol/L) [7]. These indicate strong needs for more effective solution to address poor glycemic control with limited healthcare resource.
The amount and the quality of care are factors affecting the attainment of glycemic control. The BPHS assured that quarterly blood glucose tests was highly unlikely to be sufficient, given the guideline recommended frequency was 2 to 4 times per week for fasting blood glucose (FBG) and postprandial blood glucose testings for patients taking oral antihyperglycemic medication [8]. Critical system deficiencies, including limited and burned-out workforces, unavailable essential medicines and basic technologies, fragmented delivery of care, interoperable information systems, and financial restraints have contributed to the low quality of primary care in preventing and controlling major chronic diseases in developing contexts [4,9–12]. Similar challenges exist in China [13,14]. Recent studies on China’s primary care call for quality improving efforts on (1) enhancement of the quality of training for primary healthcare physicians; (2) establishment of performance accountability to incentivize high-quality and high-value care; (3) integration of clinical care with the BPHS; (4) strengthening of the coordination between primary healthcare institutions and hospitals; and (5) building primary healthcare system on digital and innovative technologies [13,15,16].
With the global internet penetration, digital technologies have been increasingly adopted in studies that aim to improve glycemic control [17,18]. However, text messaging is a dominant tool, limited literature reports clinical effectiveness in primary care settings, and the majority of favorable results are from studies with small sample size [18–20]. In addition, while a multitude of mobile applications for type 2 diabetes are available on the market, most focus on self-management or hospital appointments, few attempts to improve the quality of diabetes care in primary care settings by combining enhanced blood glucose monitoring and performance monitoring in a hieratical digital system [21–26].
Therefore, a mobile health (mHealth)-based digital platform named Road to Hierarchical Diabetes Management at Primary Care Settings in China (ROADMAP) was designed and had its effectiveness and feasibility on diabetes control tested through a trial during 2017 to 2019 in diverse primary care settings in China. The hypothesis was that contracted continuing service based on increased monitoring for blood glucose and BP for patients with type 2 diabetes, capacity building, and regular performance evaluation for service providers, facilitated by an mHealth platform, would lead to improved diabetes control. This paper summarizes the main results of the trial.
Methods
This study is reported in accordance with the Consolidated Standards of Reporting Trials (CONSORT) guidelines for cluster randomized controlled trials (S1 CONSORT Checklist) [27]. Detailed study protocol has been described elsewhere [28]. In brief, we conducted a community-based cluster randomized controlled trial for 12 months to compare the effectiveness of the ROADMAP intervention with usual care.
Setting and participants
The trial involved a total of 864 communities in 144 counties across 25 eligible provinces in mainland China, covering developed/less developed and urban/rural areas. Within each province, an average of 6 counties and 36 subordinate communities (6 communities from each county) participated in the trial. In each community (cluster), an average of 22 participants were selected at random from a full list of type 2 diabetes patients from local BPHS system. Eligible participants were individuals with diagnosed type 2 diabetes who had registered in the BPHS system; aged 18 to 75 years; residing in the community for the last 6 months with no plan of relocating; and voluntarily participating in the study with informed consent. Patients were excluded if suffering severe physical or psychological problems; unable to attend the site visit; unable to consciously answer questions; women in the process of, or planning for, pregnancy or breastfeeding; or participating in any other clinical trials within the last 6 months.
Randomization and masking
Randomization was performed centrally upon the completion of baseline data collection. Every 6 communities (i.e., clusters) enrolled by county hospital investigators within same county were randomized into either intervention or usual care group in a 2:1 ratio (i.e., 4 clusters to intervention and 2 to usual care). All participating counties were stratified by the economic development level of their affiliating provinces and urban/rural locality with the purpose of enrolling equal number of counties for each of the 4 strata. Given the nature of cluster study design, the intervention was implemented as open label after randomization. Assessments, at baseline and end of study (EOS), were carried out by trained local investigators who had no information of group allocation. Before the database was locked, statisticians stayed blinded throughout the study until the statistical analysis plan was finalized without any knowledge of allocation [29].
Interventions
The multicomponent intervention contained a contracted service package, service providers–targeted capacity building, and performance review for service delivery, which was facilitated through a provider-facing smartphone application Graded ROADMAP and a website. The application connected doctors from primary care clinics within the communities and county hospitals and constituted the hierarchy of care team in the region.
Contracted service package was a set of structured diabetes management services, initiated by blood glucose monitoring and BP measurement at each monthly clinic visit. Fingertip blood glucose monitoring was performed with a Graded ROADMAP–bundled unified meter. A minimum of one fasting and one postprandial blood glucose (on the same day); one BP measurement; one diabetic peripheral neuropathy screening; diet and physical exercise consultation and medication instruction were suggested at each face-to-face visit. The primary care doctors kept track of the results of delivered services with Graded ROADMAP app and took on a role alike gatekeeper by proactively providing patients with routine contacts, monitoring and evaluation, and lifestyle instructions. The app would remind the primary care doctor to request a referral when it captured an indication in patient’s record. The referral request then reaches the doctors in the county hospitals with patient’s profile and health record for further treatment. County hospital investigators routinely oversaw the implementation of the contracted service following a designated operation manual.
Capacity building consisted of 2 half-day compulsory structured training sessions held at the provincial and county level, using a train-the-trainer approach. The county doctors were trained at provincial level and became the trainer of the county level session. The training materials were developed by the Chinese Diabetes Society based on practicing diabetes guidelines, with the purpose of upskilling service providers by addressing theoretical and operational barriers to type 2 diabetes management and treatment in primary care settings. The structural trainings at provincial and county level were delivered one by one by national and provincial working groups, respectively.
Performance review was calculated in real time based on routinely collected data through Graded ROADMAP, and the results were published on the designated website and quarterly circulated through WeChat (the most popular app for communication, social media, and mobile wallet in China) among the hospitals and communities. The performance assessment was conducted monthly by the national working group. It covered both process and outcome indicators including the frequency of delivered blood glucose and BP measurements, the number of diabetes complication screening, and the number of referrals, as well as the changes in blood glucose/BP levels. Detailed contents and implementation of the intervention was described previously in the published protocol [28].
Within the intervention group, participants were further divided into 2 subgroups based on the utilization of an optional patient-targeting smartphone application (Your Doctor), which supported health education and communication with their contracted doctors. The Your Doctor app was provided to all the participants by their community doctors, but the app use was based on their willingness and capability in using a smartphone. Participants logged in to this app for no less than 4 times throughout the intervention were defined as active Your Doctor users, while those with less than 4 logins were defined as inactive users. Considering the high penetration of smartphone and mobile internet in China [30], we expected that around half of the participants in the intervention group would be Your Doctor active users.
Outcomes
The primary outcome was HbA1c control rate (<7.0%; target A) at EOS. The secondary outcomes include the percentage of patients achieving both systolic blood pressure (SBP) <140 mm Hg and diastolic blood pressure (DBP) <80 mm Hg (target B) per Chinese diabetes guideline during the study [31]; percentage of patients achieving LDL-C <2.6 mmol/L (target C); optimal control of composite ABC targets (HbA1c <7.0%, BP <140/80 mm Hg, and LDL-C <2.6 mmol/L; 130/80 mm Hg was also used as alternative BP target to facilitate comparability with other studies); percentage of patients achieving FBG <7.0 mmol/L; changes in levels of HbA1c, BP, LDL-C, FBG, and body weight; and episodes of hypoglycemia (blood glucose <3.9 mmol/L).
HbA1c was obtained from a centrally distributed point-of-care HbA1c analyzer (A1c EZ 2.0) [32]. Standard laboratory tests of blood and urine samples, including fasting glucose, lipid profile, creatinine, and kidney function, were performed by qualified laboratories at county hospitals. Data collection, storage, and reporting followed the “Mobile Application Information Service Regulation” issued by the Cyberspace Administration of China in 2016. Individual participants were encrypted and de-identified. Feedback and queries raised on data integrity, authenticity, and accuracy, as well as schedule management, were achieved using unique identifier reporting.
Statistical analysis
Detailed sample size calculation methods have been described elsewhere [29]. Briefly, 19,008 eligible patients from 864 communities (22 patients each) would be recruited at baseline, and 16,416 participants (14% drop rate) complete the EOS, which could provide an 89% power to detect a 5% absolute increase in the primary outcome for the intervention group. The sample size calculation assumed that 40% of participants would have well-controlled HbA1c (<7%) at EOS in the control group, with an intraclass correlation coefficient (ICC) of 0.15. All analyses were conducted at the individual level following the intention-to-treat (ITT) principle.
The primary analysis of the intervention effect for primary outcome was conducted using a logistic regression model with generalized estimating equations (GEEs) accounting for clustering within communities and with adjustment of baseline continuous HbA1c. This was the prespecified alternating model in case of nonconvergence of the log binomial regression [29]. The odds ratio (OR) along with the indirectly derived relative risk (RR) and risk difference (RD) were reported. The RR was indirectly derived from the OR using the following formula: RR = OR ÷ [1 − unadjusted p0 in control × (1 − OR)] [33]. Sensitivity analyses included (1) using SBP <130 mm Hg and DBP <80 mm Hg as target to evaluate BP control and optimal control of composite ABC targets for diabetes patients; (2) further covariate-adjusted analyses; (3) imputed analysis: the multiple imputation (MI) technique using a fully conditional specification was adopted for the imputation. A total of 10 sets of imputed data were created and analyzed using the primary model and then pooled to estimate the treatment effect. When performing MI, the missing continuous values of HbA1c, FPG, BP, and LDL-C were imputed using linear regression, then dichotomized to get imputed binary outcomes. The variables used in the imputation model included patient characteristics including age, sex, education, economic development level, locality (urban/rural), duration of diabetes, key medical history, and comorbidities; level of HbA1c, FPG, BP, and LDL-C at baseline; level of HbA1c, FPG, BP, and LDL-C at EOS; cluster indicator; and treatment (intervention/control); and (4) subgroup analysis.
Further comparison between active and inactive users of Your Doctor within the intervention group was conducted using an inverse propensity score weighted method based on the primary model. A similar analysis strategy was used for secondary outcomes but replaced with linear regression models with GEE by specifying an identity link for continuous outcomes. The comparison of the number of hypoglycemia episodes was tested using Poisson regression analysis with GEE and with adjustment of baseline episodes of each hypoglycemia. All statistical analyses were done with SAS (SAS Institute, version 9.4).
Patient and public involvement
Regional investigators and doctors from primary care facilities had input into the study design, the development of the intervention tools, pilot testing phase, and implementation of the intervention through roundtable and periodical national/regional review meetings. Participating doctors were informed of the study progress through monthly newsletters and progress reports. Patients with type 2 diabetes in different areas were interviewed for needs analysis at the preparatory and pilot phases and interviewed again about implementation barriers and facilitators and the burden of their participation as part of a process evaluation at EOS. To encourage active engagement, participants received their results from baseline and EOS assessments. The main results of the study will be disseminated to doctors and participants to boost community involvement in type 2 diabetes management beyond the study.
Ethics and dissemination
The present study had obtained ethics approval (No: 2016–149) from the Institutional Review Board at Shanghai Sixth People’s Hospital, where the leading principal investigator is affiliated with, before the study commenced. Written approval from each participating site was granted by the local hospital research ethics committee and other relevant regional regulatory bodies. All trial participating doctors and patients had provided signed informed consent prior to participant recruitment. Findings from this study will be widely disseminated to participants, academia, and public through peer-reviewed journals, conference presentations, social media, and other applicable mechanisms.
Results
Participant allocation and baseline characteristics
Between June 2, 2017 and July 26, 2018, 19,601 eligible patients were recruited from 864 communities. A total of 19,546 completed the baseline assessment and underwent randomization, of whom 6,509 from 288 communities were assigned to usual care and 13,037 from 576 communities to the ROADMAP intervention. Moreover, 17,554 patients (89.8%), 5,794 (89.0%) in control and 11,760 (90.2%) in intervention group, completed the EOS assessment at 1 year of enrollment (Fig 1).
HbA1c, glycated hemoglobin; ROADMAP, Road to Hierarchical Diabetes Management at Primary Care Settings in China.
Baseline characteristics were well balanced between 2 groups (Table 1). In general, 45.1% of patients were recruited from economically developed areas, with 49.9% from urban areas, a mean (standard deviation, SD) age of 60.5 (8.4) years, 41.2% male, 6.0 years of median diabetes duration, mean (SD) HbA1c level of 7.87% (1.92%), 37.3% with HbA1c <7.0%, mean (SD) FBG level of 8.67 (3.46) mmol/L, and 37.1% with FBG<7.0 mmol/L at baseline. The self-reported prevalence for hypertension was 51.8%, dyslipidemia 33.4%, macrovascular disease 18.2%, diabetic retinopathy 11.5%, diabetic nephropathy 7.1%, and diabetic peripheral neuropathy 7.4%. The coefficient of intracluster correlation for the primary outcome (HbA1c level <7.0%) was 0.074.
Primary and secondary outcomes
Univariate analysis showed that HbA1c control (<7.0%) at EOS was significantly higher in the intervention group (43.6%) compared with the control group (37.5%), an absolute increase of 6.1%. After adjusting for baseline HbA1c and clustering, the primary model showed that the 1-year intervention improved HbA1c control by 18.6% (RR 1.186, 95% confidence interval [CI] 1.105, 1.267) relative to the control group and an absolute increase of 7.0% (95% CI 4.0%, 10.0%). The composite ABC control increased by 21.8% (RR 1.218, 95% CI 1.062, 1.395) relative to the control group and an absolute increase of 1.9% in the primary analysis. Slightly higher control rates were also found for FBG and BP in the intervention group compared with the control group, but the absolute difference or the RRs were not statistically significant. No increase was found for LDL-C (Table 2).
The secondary outcomes measured as continuous variables were presented in Table 3. The multivariable analysis controlling for baseline value of the analyzed outcome and clustering showed that the intervention had an increased effectiveness in lowering the level of HbA1c by 0.30% (95% CI 0.21%, 0.38%), FBG by 0.47 (95% CI 0.32, 0.62) mmol/L, and DBP by 0.6 (95% CI 0.1, 1.1) mm Hg. No significant change was found for SBP and LDL-C.
Sensitivity analysis using BP <130/80 mm Hg as target, covariate-adjusted analysis, and imputed analysis regarding the primary and secondary outcomes did not show significant difference when compared with the primary models (S1 Table for binary outcomes and S2 Table for continuous outcomes).
The comparison of the binary outcomes between active and inactive users of Your Doctor among the intervention group is presented in S3 Table. The primary multivariable regression model suggested no significant difference for blood glucose, BP, and composite ABC control between the 2 subpopulations, apart from higher LDL-C control rate in the active users.
Hypoglycemia and weight gain
With significant improvement in blood glucose control, the intervention did not cause an increase in any type of hypoglycemia (S4 Table) and body weight (Table 3).
Subgroup analysis
Subgroup analysis illustrated in S1 Fig shows that the difference for HbA1c control between the studied subgroups was not statistically significant, although better HbA1c control was found among participants with age<60 years than age ≥60 years, duration of diabetes ≥6 years than <6 years, from less developed region than developed region, and rural than urban locality.
Intervention implementation
S5 Table shows that the yearly frequencies of delivered FBG test, postprandial blood glucose test, and BP measurement were 8 to 11 times in the intervention group, which did not achieve the protocol prescribed “monthly” target. A positive association between FBG test frequency and FBG control was detected (S2A Fig), but the trend was not found between BP measurement and BP lowering (S2B Fig). During the intervention period, less than 1 among every 5 patients (0.19/person/year) was successfully referred to upstream doctors, and only 4% of participants in the intervention group were Your Doctor active users (S5 Table).
Diabetes management activities between the 2 groups were compared using patient self-reported data. Participants in the intervention group reported slightly more examinations for blood glucose, BP, HbA1c, and blood lipid than those in the control group. Greater differences were found in mean frequencies of examination for diabetes foot and neurological complication. The improvement in these diabetes management activities in intervention group was also reflexed in the significant increase of total score of SDSCA by 1.2 (95% CI 0.3, 2.0) when compared with the control. The changes in medication were weak in both groups, and a slightly higher use of oral antihyperglycemic drugs was found in the intervention group compared with control (S5 Table).
The performance evaluation report was calculated automatically using the routinely collected data through the ROADMAP platform for intervention group. The algorithms of ranking were based on the monitoring frequency and control rate of the most recent blood glucose and BP as well as patient referral. The results could be assessed anytime by authorized doctors and responsible investigators, and a hard copy will be shared monthly. S6 Table exhibits a real example of the monthly performance report for 24 counties in September 2018.
Discussion
To our knowledge, ROADMAP is the largest randomized controlled trial in testing the effectiveness and safety of an mHealth-based diabetes management with a diverse coverage of primary care settings. After 1-year follow-up, we found that the intervention lowered the HbA1c level by 0.3% when compared with usual care, which led to an absolute improvement in the HbA1c control rate of 7.0% (95% CI 4.0% to 10.0%) and a relative improvement of 18.6% (RR 1.186, 95% CI 1.105 to 1.267). We also observed an absolute improvement in the intervention group in the composite cardiometabolic ABC control rate of 1.9% (95% CI 0.5 to 3.5) and a relative improvement of 21.8% (RR 1.218, 95% CI 1.062 to 1.395). The increased control of FBG and BP and the reduced levels of FBG and DBP favored the intervention as well. No increasing episodes of hypoglycemia and weight gain were found to be associated with the intervention. The findings indicate that integrated mHealth solution, which facilitates blood glucose monitoring and performance evaluation, could be effective and safe in improving diabetes control in primary care settings.
Regular blood glucose monitoring is the basis for optimized diabetes management. The minimum requirement of 4 blood glucose tests for each patient per year by the BPHS in China has been an improvement in universal coverage of diabetes management, but it is far from enough. Our study showed that a slight increase in the frequency of blood glucose monitoring was accompanied by other behavior change and better glycemic control. Studies from Europe also demonstrated that app-supported blood glucose monitoring could improve HbA1c and had the potential to reduce the cost [34,35]. A fixed 2 tests of blood glucose per month was required in this study, but a varied frequency of monitoring based on individualized goals and willingness has been suggested by other study [36].
Persistent uncontrolled glycemic level is one of the referral indications as per the Chinese diabetes guideline. However, the average referral frequency found in the intervention group of the ROADMAP study was only 0.19 time per patient per year. This is disproportionately low considering nearly 60% participants did not achieve the HbA1c target (<7%). In addition to emphasizing the importance of intensive treatment toward the target, appropriate patient referral to/from upstream hospitals should be strengthened. These results suggested that purely technological support was insufficient, and other measures including strict pathway or regulation with clear responsibilities and result-oriented budgeting should be considered.
Previous studies have yielded the promising potential of applying digital health tools in diabetes management. A meta-analysis examined 181 trials assessing diabetes management interventions, which involved multiple players including health system, healthcare providers, and patients [37]. The pooled result showed a mean HbA1c reduction by 0.28% (95% CI: 0.21, 0.35), ranging from −1.80% [38] to 0.70% [39]. Among them, interventions based on digital health such as electronic patient registry [40], electronic decision-making [41], and information relay from patients to clinicians demonstrated on average a 0.28% to 0.36% reduction in HbA1c level [42], which is consistent with the findings from our study. The trials that reported an improvement of over absolute 1% HbA1c were mostly those adopting active pharmacist engagement [43,44] or interactive workshops for patient education [38,45], indicating that strengthened functionalities on medication and patient education should be included in the ROADMAP platform in the future.
Despite the variability in intervention strategies, the majority of studies reporting positive effect were dominated by trials with small to medium sample sizes, while the trials with larger populations tended to be negative [18], particularly for those implemented among patients broadly recruited from communities [46]. Acknowledging the difficulties in implementing complex intervention in diverse context, we adopted several strategies to increase intervention fidelity, including clarifying intervention flowchart using table stickers, providing remote trainings and reminders through WeChat, and real-time access to performance report and ranking through a designated website portal.
Even with above strategies, the intervention delivery remained inadequate. Based on the app records, the frequency of blood glucose test was 18.7 times/year, less than the required 24 times/year; the frequency of BP measurement was 8.6 times/year, less than the required 12 times/year; the referral (0.19 times/year) was also significantly underused as explained above; and only 4% of participants in intervention group positively used Your Doctor to communicate with and receive education from their doctors, although we expected the number was around 50%. Possible reasons reported by other studies for low fidelity in implementing a complex intervention include the need of longer learning time [47–49], competing workload [50], and not user-friendly for elderly users [23]. Drawn from our process evaluation, however, fidelity issues were largely driven by low expectation of treat to target for blood glucose/BP by both patients and service providers, which resulted in negligence of alerts to abnormal blood glucose/BP measurements and referral indications by service providers and low patient compliance to instructions given by the providers.
The ROADMAP study presents several strengths. First, this study has a diverse coverage of 864 communities across 25 Chinese provinces. It provides an opportunity to explore context impact and make context-fitting adaptation for future scale-up. Secondly, the participants in the ROADMAP study were randomly selected from all patients registered in the studied communities. The representative sampling enhances the generalizability of the findings. Thirdly, a designated mobile phone–based electronic data collection application base on our past trial experiences [51] was adopted to support the project management and data collection, which largely increased the quality and efficiency of project implementation and quality. Furthermore, the centrally distributed HbA1c analyzers for point-of-care testing guarantees the accuracy and consistency of the primary outcome throughout such a large study.
There are several limitations in the ROADMAP study. The participants were recruited from registered patients with diabetes; caution is suggested when extrapolating the findings to a general population with type 2 diabetes. Apart from HbA1c, other physical and biochemical examinations were conducted using various local resources, although quality requirements had been met during site selection. Third, we underestimated the frequency of blood glucose monitoring in usual care, which resulted in a relatively small improvement in frequency of blood glucose monitoring in our study. A larger effectiveness on diabetes control might be achieved if more frequent individualized blood glucose and BP tests were adopted. Finally, only 4% of participants in the intervention group were Your Doctor active users. Potential bias and low statistical power made it unreliable to detect the effectiveness of Your Doctor.
In conclusion, the primary care system in China, as in other LMICs, is struggling to meet the gaps in chronic disease management, including diabetes, and innovative solutions are needed. The ROADMAP study offers a supportive and collaborative solution to tackle the suboptimal diabetes control in primary care via an integrated mHealth-based platform. More interactive health education to raise the awareness of the importance of treat to target for both patients and community doctors, more close engagement of upstream hospital doctors, leveraging home-tested glucose data to support diabetes management, and adding individualized medication guide to the mHealth platform may further improve the effectiveness of such an mHealth platform. We expect that the findings from the ROADMAP study could inform the improvement of the BPHS programs and promote effective and rational use of limited healthcare resource in China and other LMICs.
Supporting information
S1 CONSORT Checklist. CONSORT 2010 checklist of information to include when reporting a cluster randomized trial.
CONSORT, Consolidated Standards of Reporting Trials.
https://doi.org/10.1371/journal.pmed.1003754.s001
(DOCX)
S1 Fig. Forest of subgroup analysis for primary outcome—HbA1c <7% at EOS.
EOS, end of study; HbA1c, glycated hemoglobin.
https://doi.org/10.1371/journal.pmed.1003754.s002
(PDF)
S2 Fig. The relationship between frequency of BP/FBG monitoring and BP/FBG control across 12 months in intervention group of ROADMAP study.
BP, blood pressure; FBG, fasting blood glucose; ROADMAP, Road to Hierarchical Diabetes Management at Primary Care Settings in China.
https://doi.org/10.1371/journal.pmed.1003754.s003
(PDF)
S1 Table. Sensitivity analysis for estimated effect (binary outcomes) of intervention compared with control.
https://doi.org/10.1371/journal.pmed.1003754.s004
(DOCX)
S2 Table. Sensitivity analysis for estimated effect (continuous outcomes) of intervention compared with control.
https://doi.org/10.1371/journal.pmed.1003754.s005
(DOCX)
S3 Table. Comparison of primary and secondary binary outcomes between Your Doctor inactive and active users within intervention group.
https://doi.org/10.1371/journal.pmed.1003754.s006
(DOCX)
S4 Table. Number of patients and episodes of hypoglycemia within 1 month before EOS.
EOS, end of study.
https://doi.org/10.1371/journal.pmed.1003754.s007
(DOCX)
S5 Table. Comparison of key diabetes management activities between 2 groups during 1-year follow-up in the ROADMAP study.
ROADMAP, Road to Hierarchical Diabetes Management at Primary Care Settings in China.
https://doi.org/10.1371/journal.pmed.1003754.s008
(DOCX)
S6 Table. Monthly performance of intervention implementation in the ROADMAP study—results of 24 out of 144 counties in September 2018 as an example.
ROADMAP, Road to Hierarchical Diabetes Management at Primary Care Settings in China.
https://doi.org/10.1371/journal.pmed.1003754.s009
(DOCX)
Acknowledgments
The authors acknowledge personnel who engaged in helping us accomplish all trial procedures. Appreciation for the support from trial steering committee and efforts made by the ROADMAP Study Group, all participating clinical research associates, investigators, doctors, and patients, as well as those who helped to facilitate trial preparation, execution, and evaluation. The ROADMAP Study Group was composed by Dalong Zhu, Dongmei Li, Guangyao Song, Haoming Tian, Hong Li (Zhejiang Province), Hong Li (Yunnan Province), Hongyu Kuang, Huili Zhang, Jianying Liu, Jie Liu, Jing Liu, Li Chen, Li Yuan, Liming Chen, Lixin Shi, Minxiang Lei, Ping Liu, Puhong Zhang, Qifu Li, Qiu Zhang, Qiuhe Ji, Yadong Sun, Yanbing Li, Yuqian Bao, Yuzhen Liang, Weiping Jia, Zhigang Zhao, Zhongyan Shan, and Zilin Sun, alphabetically ordered by given name. We thank Professor Craig Anderson for providing constructive guidance in writing this manuscript.
References
- 1. G. B. D. Diseases, Injuries, Collaborators. Global burden of 369 diseases and injuries in 204 countries and territories, 1990–2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet. 2020;396(10258):1204–22. Epub 2020/10/19. pmid:33069326.
- 2. Khan MAB, Hashim MJ, King JK, Govender RD, Mustafa H, Al Kaabi J. Epidemiology of Type 2 Diabetes—Global Burden of Disease and Forecasted Trends. J Epidemiol Glob Health. 2020;10(1):107–11. Epub 2020/03/17. pmid:32175717; PubMed Central PMCID: PMC7310804.
- 3. Manne-Goehler J, Geldsetzer P, Agoudavi K, Andall-Brereton G, Aryal KK, Bicaba BW, et al. Health system performance for people with diabetes in 28 low- and middle-income countries: A cross-sectional study of nationally representative surveys. PLoS Med. 2019;16(3):e1002751. Epub 2019/03/02. pmid:30822339; PubMed Central PMCID: PMC6396901.
- 4. Aschner P, Gagliardino JJ, Ilkova H, Lavalle F, Ramachandran A, Mbanya JC, et al. Persistent poor glycaemic control in individuals with type 2 diabetes in developing countries: 12 years of real-world evidence of the International Diabetes Management Practices Study (IDMPS). Diabetologia. 2020;63(4):711–21. Epub 2020/01/07. pmid:31901950; PubMed Central PMCID: PMC7054372.
- 5. Li Y, Teng D, Shi X, Qin G, Qin Y, Quan H, et al. Prevalence of diabetes recorded in mainland China using 2018 diagnostic criteria from the American Diabetes Association: national cross sectional study. BMJ. 2020;369:m997. Epub 2020/04/30. pmid:32345662; PubMed Central PMCID: PMC7186854.
- 6.
National Health and Family Planning Commission P. Notice of the National Health and Family Planning Commission on printing and distributing the national basic public health service standard (Third Edition) China: National Health and Family Planning Commission, PRC.; 2017 [cited 2020 April 23]. Available from: http://www.nhc.gov.cn/jws/s3578/201703/d20c37e23e1f4c7db7b8e25f34473e1b.shtml.
- 7. Ji L, Hu D, Pan C, Weng J, Huo Y, Ma C, et al. Primacy of the 3B approach to control risk factors for cardiovascular disease in type 2 diabetes patients. Am J Med. 2013;126(10):925 e11–22. Epub 2013/07/03. pmid:23810406.
- 8. Jia W, Weng J, Zhu D, Ji L, Lu J, Zhou Z, et al. Standards of medical care for type 2 diabetes in China 2019. Diabetes Metab Res Rev 2019;35(6):e3158. Epub 2019/03/26. pmid:30908791.
- 9. Mendis S, Al Bashir I, Dissanayake L, Varghese C, Fadhil I, Marhe E, et al. Gaps in capacity in primary care in low-resource settings for implementation of essential noncommunicable disease interventions. Int J Hypertens. 2012;2012:584041. Epub 2012/12/20. pmid:23251789; PubMed Central PMCID: PMC3517842.
- 10. Dugani S, Afari H, Hirschhorn LR, Ratcliffe H, Veillard J, Martin G, et al. Prevalence and factors associated with burnout among frontline primary health care providers in low- and middle-income countries: A systematic review. Gates Open Res. 2018;2:4. Epub 2018/06/11. pmid:29984356; PubMed Central PMCID: PMC6030396.
- 11. Bitton A, Ratcliffe HL, Veillard JH, Kress DH, Barkley S, Kimball M, et al. Primary Health Care as a Foundation for Strengthening Health Systems in Low- and Middle-Income Countries. J Gen Intern Med. 2017;32(5):566–71. Epub 2016/12/13. pmid:27943038; PubMed Central PMCID: PMC5400754.
- 12. Blonde L, Aschner P, Bailey C, Ji L, Leiter LA, Matthaei S, et al. Gaps and barriers in the control of blood glucose in people with type 2 diabetes. Diab Vasc Dis Res. 2017;14(3):172–83. Epub 2017/05/04. pmid:28467203; PubMed Central PMCID: PMC5418936.
- 13. Meng Q, Mills A, Wang L, Han Q. What can we learn from China’s health system reform? BMJ. 2019;365:l2349. Epub 2019/06/21. pmid:31217222; PubMed Central PMCID: PMC6598719.
- 14. Li H, Yuan B, Meng Q, Kawachi I. Contextual Factors Associated with Burnout among Chinese Primary Care Providers: A Multilevel Analysis. Int J Environ Res Public Health. 2019;16(19). Epub 2019/09/25. pmid:31547533; PubMed Central PMCID: PMC6801650.
- 15. Li X, Krumholz HM, Yip W, Cheng KK, De Maeseneer J, Meng Q, et al. Quality of primary health care in China: challenges and recommendations. Lancet. 2020;395(10239):1802–12. Epub 2020/06/09. pmid:32505251; PubMed Central PMCID: PMC7272159.
- 16. Li X, Lu J, Hu S, Cheng KK, De Maeseneer J, Meng Q, et al. The primary health-care system in China. Lancet 2017;390(10112):2584–94. Epub 2017/12/13. pmid:29231837.
- 17. Fleming GA, Petrie JR, Bergenstal RM, Holl RW, Peters AL, Heinemann L. Diabetes Digital App Technology: Benefits, Challenges, and Recommendations. A Consensus Report by the European Association for the Study of Diabetes (EASD) and the American Diabetes Association (ADA) Diabetes Technology Working Group. Diabetes Care. 2020;43(1):250–60. Epub 2019/12/07. pmid:31806649.
- 18. Peiris D, Praveen D, Johnson C, Mogulluru K. Use of mHealth systems and tools for non-communicable diseases in low- and middle-income countries: a systematic review. J Cardiovasc Transl Res. 2014;7(8):677–91. Epub 2014/09/12. pmid:25209729.
- 19. Wang Y, Xue H, Huang Y, Huang L, Zhang D. A Systematic Review of Application and Effectiveness of mHealth Interventions for Obesity and Diabetes Treatment and Self-Management. Adv Nutr. 2017;8(3):449–62. Epub 2017/05/17. pmid:28507010; PubMed Central PMCID: PMC5421120.
- 20. Haider R, Sudini L, Chow CK, Cheung NW. Mobile phone text messaging in improving glycaemic control for patients with type 2 diabetes mellitus: A systematic review and meta-analysis. Diabetes Res Clin Pract. 2019;150:27–37. Epub 2019/03/02. pmid:30822496.
- 21. Huang Z, Lum E, Jimenez G, Semwal M, Sloot P, Car J. Medication management support in diabetes: a systematic assessment of diabetes self-management apps. BMC Med. 2019;17(1):127. Epub 2019/07/18. pmid:31311573; PubMed Central PMCID: PMC6636047.
- 22. Veazie S, Winchell K, Gilbert J, Paynter R, Ivlev I, Eden KB, et al. Rapid Evidence Review of Mobile Applications for Self-management of Diabetes. J Gen Intern Med. 2018;33(7):1167–76. Epub 2018/05/10. pmid:29740786; PubMed Central PMCID: PMC6025680.
- 23. Gao C, Zhou L, Liu Z, Wang H, Bowers B. Mobile application for diabetes self-management in China: Do they fit for older adults? Int J Med Inform. 2017;101:68–74. Epub 2017/03/30. pmid:28347449; PubMed Central PMCID: PMC5418079.
- 24. Hou C, Carter B, Hewitt J, Francisa T, Mayor S. Do Mobile Phone Applications Improve Glycemic Control (HbA1c) in the Self-management of Diabetes? A Systematic Review, Meta-analysis, and GRADE of 14 Randomized Trials. Diabetes Care. 2016;39(11):2089–95. Epub 2016/12/08. pmid:27926892.
- 25. Murphy ME, Byrne M, Galvin R, Boland F, Fahey T, Smith SM. Improving risk factor management for patients with poorly controlled type 2 diabetes: a systematic review of healthcare interventions in primary care and community settings. BMJ Open. 2017;7(8):e015135. Epub 2017/08/07. pmid:28780542; PubMed Central PMCID: PMC5724222.
- 26. van Olmen J, Erwin E, Garcia-Ulloa AC, Meessen B, Miranda JJ, Bobrow K, et al. Implementation barriers for mHealth for non-communicable diseases management in low and middle income countries: a scoping review and field-based views from implementers. Wellcome Open Res. 2020;5:7. Epub 2020/05/16. pmid:32399497; PubMed Central PMCID: PMC7194478.
- 27. Campbell MK PG, Elbourne DR, Altman DG; for the CONSORT Group. Consort 2010 statement: extension to cluster randomised trials. BMJ. 2012;345:e5661. PubMed Central PMCID: PMC22951546. pmid:22951546
- 28. Jia W, Zhang P, Duolikun N, Zhu D, Li H, Bao Y, et al. Study protocol for the road to hierarchical diabetes management at primary care (ROADMAP) study in China: a cluster randomised controlled trial. BMJ Open. 2020;10(1):e032734. Epub 2020/01/09. pmid:31911516; PubMed Central PMCID: PMC6955560.
- 29. Li XDN, Cheng F, Billot L, Zhang P, Jia W. Road to Hierarchical Diabetes Management at Primary Care (ROADMAP) Study in China: Protocol for the Statistical Analysis of a Cluster Randomized Controlled Trial. JMIR Res Protoc. 2020;9(4):e18333. pmid:32343257
- 30.
China Internet Network Information Center. The 40th statistical report on the development of China’s Internet released by China Internet Network Information Center [cited 2020 April 23]. Available from: http://www.cac.gov.cn/2017-08/04/c_1121427672.htm.
- 31. Chinese Diabetes Society. Chinese guidelines for the prevention and treatment of type 2 diabetes (2017 Edition). Chin J Diabetes. 2018;10(1):4–67.
- 32. Zhou R, Wang W, Song ZX, Tong Q, Wang QT. Evaluation of a new hemoglobin A1c analyzer for point-of-care testing. J Clin Lab Anal. 2018;32(1). Epub 2017/02/22. pmid:28220976; PubMed Central PMCID: PMC6817028.
- 33. Zhang J, Yu KF. What’s the relative risk? A method of correcting the odds ratio in cohort studies of common outcomes. JAMA. 1998;280(19):1690–1. Epub 1998/12/01. pmid:9832001.
- 34. Grady M, Cameron H, Levy BL, Katz LB. Remote Health Consultations Supported by a Diabetes Management Web Application With a New Glucose Meter Demonstrates Improved Glycemic Control. J Diabetes Sci Technol. 2016;10(3):737–43. Epub 2015/12/22. pmid:26685995; PubMed Central PMCID: PMC5038536.
- 35. Fritzen K, Basinska K, Rubio-Almanza M, Nicolucci A, Kennon B, Verges B, et al. Pan-European Economic Analysis to Identify Cost Savings for the Health Care Systems as a Result of Integrating Glucose Monitoring Based Telemedical Approaches Into Diabetes Management. J Diabetes Sci Technol. 2019;13(6):1112–22. Epub 2019/03/29. pmid:30917691; PubMed Central PMCID: PMC6835181.
- 36. Bridges JF, Brignac D, Thomas S, Kent C, Ogbuli M, Draper J, et al. Optimizing HgA1C and glucose monitoring frequency in patients with Type 2 diabetes. Med Sci Monit. 2012;18(12):CR693–7. Epub 2012/12/01. pmid:23197229; PubMed Central PMCID: PMC3560795.
- 37. Lim LL, Lau ESH, Kong APS, Davies MJ, Levitt NS, Eliasson B, et al. Aspects of Multicomponent Integrated Care Promote Sustained Improvement in Surrogate Clinical Outcomes: A Systematic Review and Meta-analysis. Diabetes Care. 2018;41(6):1312–20. Epub 2018/05/23. pmid:29784698; PubMed Central PMCID: PMC5961399.
- 38. Trento M, Passera P, Borgo E, Tomalino M, Bajardi M, Cavallo F, et al. A 5-year randomized controlled study of learning, problem solving ability, and quality of life modifications in people with type 2 diabetes managed by group care. Diabetes Care. 2004;27(3):670–5. Epub 2004/02/28. pmid:14988283.
- 39. Keyserling TC, Samuel-Hodge CD, Ammerman AS, Ainsworth BE, Henriquez-Roldan CF, Elasy TA, et al. A randomized trial of an intervention to improve self-care behaviors of African-American women with type 2 diabetes: impact on physical activity. Diabetes Care. 2002;25(9):1576–83. Epub 2002/08/28. pmid:12196430.
- 40. Tutino GE, Yang WY, Li X, Li WH, Zhang YY, Guo XH, et al. A multicentre demonstration project to evaluate the effectiveness and acceptability of the web-based Joint Asia Diabetes Evaluation (JADE) programme with or without nurse support in Chinese patients with Type 2 diabetes. Diabet Med. 2017;34(3):440–50. Epub 2016/06/10. pmid:27278933; PubMed Central PMCID: PMC5324581.
- 41. Rothschild SK, Martin MA, Swider SM, Tumialan Lynas CM, Janssen I, Avery EF, et al. Mexican American trial of community health workers: a randomized controlled trial of a community health worker intervention for Mexican Americans with type 2 diabetes mellitus. Am J Public Health. 2014;104(8):1540–8. Epub 2013/08/21. pmid:23947316; PubMed Central PMCID: PMC4103223.
- 42. Nicolucci A, Cercone S, Chiriatti A, Muscas F, Gensini G. A Randomized Trial on Home Telemonitoring for the Management of Metabolic and Cardiovascular Risk in Patients with Type 2 Diabetes. Diabetes Technol Ther. 2015;17(8):563–70. Epub 2015/07/15. pmid:26154338.
- 43. Al Mazroui NR, Kamal MM, Ghabash NM, Yacout TA, Kole PL, McElnay JC. Influence of pharmaceutical care on health outcomes in patients with Type 2 diabetes mellitus. Br J Clin Pharmacol. 2009;67(5):547–57. Epub 2009/06/26. pmid:19552750; PubMed Central PMCID: PMC2686072.
- 44. Fornos JA, Andres NF, Andres JC, Guerra MM, Egea B. A pharmacotherapy follow-up program in patients with type-2 diabetes in community pharmacies in Spain. Pharm World Sci. 2006;28(2):65–72. Epub 2006/06/23. pmid:16791717.
- 45. Trento M, Gamba S, Gentile L, Grassi G, Miselli V, Morone G, et al. Rethink Organization to iMprove Education and Outcomes (ROMEO): a multicenter randomized trial of lifestyle intervention by group care to manage type 2 diabetes. Diabetes Care. 2010;33(4):745–7. Epub 2010/01/28. pmid:20103547; PubMed Central PMCID: PMC2845019.
- 46. Chan JC. What can we learn from the recent blood glucose lowering megatrials? J Diabetes Investig. 2011;2(1):1–5. Epub 2011/01/24. pmid:24843455; PubMed Central PMCID: PMC4008009.
- 47. Heselmans A, Delvaux N, Laenen A, Van de Velde S, Ramaekers D, Kunnamo I, et al. Computerized clinical decision support system for diabetes in primary care does not improve quality of care: a cluster-randomized controlled trial. Implement Sci. 2020;15(1):5. Epub 2020/01/09. pmid:31910877; PubMed Central PMCID: PMC6947861.
- 48. McCulloch P, Taylor I, Sasako M, Lovett B, Griffin D. Randomised trials in surgery: problems and possible solutions. BMJ. 2002;324(7351):1448–51. Epub 2002/06/18. pmid:12065273; PubMed Central PMCID: PMC1123389.
- 49. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. Int J Nurs Stud. 2013;50(5):587–92. Epub 2012/11/20. pmid:23159157.
- 50. Wang Q, Kong Y, Sun J, Zhang Y, Yuan L, Wang J. What Are the Challenges Faced by Village Doctors in Provision of Basic Public Health Services in Shandong, China? A Qualitative Study. Int J Environ Res Public Health. 2019;16(14). Epub 2019/07/18. pmid:31311072; PubMed Central PMCID: PMC6679320.
- 51. Zhang J, Sun L, Liu Y, Wang H, Sun N, Zhang P. Mobile Device-Based Electronic Data Capture System Used in a Clinical Randomized Controlled Trial: Advantages and Challenges. J Med Internet Res. 2017;19(3):e66. Epub 2017/03/10. pmid:28274907; PubMed Central PMCID: PMC5362692.