Sources

The following databases were searched: MEDLINE,^® EMBASE,^® CINAHL^® (Cumulated Index to Nursing and Allied Health Literature), Cochrane Database of Systematic Reviews, International Pharmaceutical Abstracts,^© Compendex,^© INSPEC^© (which includes IEEE^®), Library and Information Science Abstracts,^® E-Prints in Library and Information Science,^® PsycINFO,^® Sociological Abstracts,^© and Business Source^® Complete. The search terms used are presented in Appendix A.

Supplemental searches targeting grey literature sources were conducted and included New York Academy of Medicine, SIGLE, U.S. HHS Health Information Technology, Health Technology Assessment reports from the U.K. Centre for Reviews and Dissemination, ProQuest Dissertations, National Library for Health United Kingdom (includes Bandolier), ProceedingsFirst, PapersFirst, National Technical Information Service, and Google. As part of the grey literature search, AHRQ made all references in their e-Prescribing, bar coding, and CPOE knowledge libraries available.

Search Terms and Strategies

Terms related to specific MMIT applications and in combination with both medication management terms and more general computer and technology terms, were prepared. The MEDLINE^® search formed the basis for all other databases, but searches were edited as needed depending on the features of the database being used. When possible, letters, editorials or commentaries, and animal studies were excluded electronically. No limits were placed on language or time to capture the global literature and early studies.

Organization and Tracking of the Literature Search

Searching was done in the fall of 2009 and updated in early summer 2010. The results of the searches were downloaded into Reference Manager^® version 10 (ISI ResearchSoft) and uploaded into our customized systematic review management system (Health Information Research Unit, McMaster University). The system is Web -based. It allows management of the systematic review process with improved auditing and control capabilities including automatic production of tables and tabulations. The system stores the full text of articles in portable document format (PDF) and tracks duplicates, results of title and abstract review, which articles were included or excluded with reasons, and data abstraction levels.

Title and Abstract Review

The study team reviewed titles and abstracts of all articles retrieved using prepared data abstraction forms (Appendix B, Sample Screening and Data Abstraction Forms). Two blinded, independent reviewers from a team of reviewers conducted title and abstract reviews in parallel. Both reviewers had to indicate that the article was to be excluded for it to be removed. Both reviewers also had to agree on inclusion for the article to be promoted to the next level. In the case of disagreements, a third reviewer determined if the article was to be promoted to the next level of screening.

This first review level was designed to detect all articles that reported on medication management with health IT assisting in the medication management process. Reviewers were instructed to consider applications as health IT if they were integrated with other information systems (rather than stand-alone applications or devices), with the systems being more than passive vehicles for data transfer. We defined health IT as electronic systems that collect, process, or exchange health information about patients and formal caregivers. We included articles only if the MMIT was integrated with at least one health IT system, such as EHR or EMR systems, and that it processed patient-specific information and provided advice or suggestions to either the health care provider or the patients and their families on issues related to health or wellness care. We excluded stand-alone devices (no integration) with the exception of personal digital assistants (PDAs) or handheld devices into which clinicians or patients entered patient-specific information to assist in medication management. PDAs are an important focus for AHRQ. All articles about transmission or order communication between pharmacist and clinical prescriber were also included and tagged as Electronic Data Interchange (EDI).

Review articles were passed through to the second level of screening. Once identified, the bibliographies of the reviews were screened for articles with potential for inclusion and their citations were put through the screening process starting at the title and abstract level if they had not already been captured by the original search. The systematic reviews were also included in the answers to the seven key questions where appropriate.

Defining Medication Management Health IT

To be clear on what kinds of applications were included in MMIT, the following outline for MMIT applications was devised and used by screeners.

MMIT systems or programs were included if:

• The computer or technology processed patient-specific information,
• The information provided by the system was relevant to one of the five phases of medication management or two ancillary aspects (education and reconciliation):
  • Prescribing or ordering medications,
  • Order communication (transmission, clarification, verification),
  • Dispensing,
  • Administering (by health care provider, patient, or caregiver),
  • Monitoring (signs, symptoms, or laboratory data to ascertain patient adherence, adverse events, or the need for medication adjustment),
  • Education (of patients or care providers, but not preprofessional education),
  • Reconciliation of medication lists,
• Someone (e.g., patient, caregiver, family, health care professional) received information in return that was, or could be, linked to patient-specific information used in decisionmaking,
• The technology was part of, or linked to, another electronic information system,
• The article contained outcome data related to one of the areas of interest set out in the key questions.

Articles were to be excluded if they were health IT systems or programs and:

• The IT component was only Web or local browsing of general health information databases or information resources (e.g., online textbooks),
• The system acted as a conduit of information only (except order communication of prescriptions between health care providers and pharmacists),
• Systems where no feedback was provided for patient care (e.g., surveys),
• The system did not help with medication management decision making or provide information about any of the medication management phases (prescribing, order communication, dispensing, administering, and monitoring), or education and reconciliation,
• Systems that made measurements but did not process the information,
• Stand-alone devices that do not integrate with information systems (except PDAs using patient specific information),
• The health IT application was used only to extract data (e.g., pill bottles that track opening and closing, smart infusion pumps not tied to other systems, studies using EMRs for data collection if the data were for quality improvement or other related tasks but not direct patient care).

Assessment of Study Quality

The included studies were assessed on the basis of the quality of their reporting of relevant data. Quantitative studies were assessed using the same criteria employed by Jimison et al.,³ in a previous AHRQ report. RCT scoring was based on Delphi consensus work by Verhagan and colleagues,¹⁰ and is referred to in this report as the ‘Verhagen/AHRQ RCT quality scale.’ Quality assessments of applicable articles were performed by more experienced reviewers to maintain consistency and accuracy. Studies with before-after, time series, surveys, and qualitative methods were not assessed for quality because few well-validated instruments exist and the study design itself is considered lower on the hierarchy of evidence.

Method assessments used for articles of the relevant design:

Verhagen/AHRQ RCT quality scale (scored out of nine)

1. Was the assignment to the treatment groups really random?
2. Was the treatment allocation concealed?
3. Were the groups similar at baseline in terms of prognostic factors?
4. Were the eligibility criteria specified?
5. Were outcome assessors blinded to the treatment allocation?
6. Was the care provider blinded?
7. Was the patient blinded?
8. Were the point estimates and measure of variability presented for the main endpoint measure?
9. Did the analyses include an intention to treat analysis

Cohort studies (scored out of ten)

1. Was there sufficient description of the groups and the distribution of prognostic factors?
2. Are the groups assembled at a similar point in their disease progression?
3. Is the intervention/treatment reliably ascertained?
4. Were the groups comparable on all important confounding factors?
5. Was there adequate adjustment for the effects of these confounding variables?
6. Was a dose response relationship between intervention and outcome demonstrated?
7. Was outcome assessment blind to exposure status?
8. Was followup long enough for the outcomes to occur?
9. What proportion of the cohort was followed-up?
10. Were drop out rates and reasons for drop out similar across intervention and unexposed groups?

Case-control studies (scored out of nine)

1. Is the case definition explicit?
2. Has the disease state of the cases been reliably assessed and validated?
3. Were the controls randomly selected from the source of population of the cases?
4. How comparable are the cases and controls with respect to potential confounding factors?
5. Were interventions and other exposures assessed in the same way for cases and controls?
6. How was the response rate defined?
7. Were the nonresponse rates and reasons for nonresponse the same in both groups?
8. Is it possible that over-matching has occurred in that cases and controls were matched on factors related to exposure?
9. Was an appropriate statistical analysis used (matched or unmatched)?

Case series (scored out of six)

1. Is the study based on a representative sample selected from a relevant population?
2. Are the criteria for inclusion explicit?
3. Did all individuals enter the survey at a similar point in their disease progression?
4. Was followup long enough for important events to occur?
5. Were outcomes assessed using objective criteria or was blinding used?
6. If comparisons of subseries are being made, was there sufficient description of the series and the distribution of prognostic factors?

Data Synthesis

Evidence tables with article details were created and ordered by key question, subquestion, and medication management phase as applicable (Appendix C). This offered another opportunity to check abstracted elements with the original articles; any errors were brought to the attention of the abstractors of the specific section for correction. Meta-analyses were not performed on any data because of the heterogeneity of the studies, as well as the nature of the observational studies in most sections.

Data Entry and Quality Control

General study data for each article was abstracted by one staff member and entered into the online data abstraction forms (Appendix B). Second reviewers were generally more experienced members of the research team, and one of their main priorities was to check the quality and consistency of the first reviewers’ answers and to perform the quality assessment where required.

Grading the Evidence

Because so much of the material was derived from observational studies, we did not provide grades for the evidence beyond quality scoring of the RCTs, cohort, case-control, and case series studies.

Peer Review

Throughout the project, the core team sought feedback from internal advisors and technical experts. These technical experts were members of the TEP and other content and methodology experts as needed. The report was reviewed in several stages, comments considered and incorporated into this final report. Members of the TEP and the peer reviewers are listed in Appendix E. Many of the TEP members also reviewed the initial version of the document. Both the members of the TEP and the review panel provided valuable comments and have made the final document stronger.