It’s hard to believe but this year marks the fifth anniversary of the National Academy of Sciences’ (NAS) 2011 report on the Environmental Protection Agency’s (EPA) Integrated Risk Information System (IRIS) program. The report identified systemic problems and offered sweeping recommendations to overhaul the program. So what has happened in the five years since the NAS report was issued? Some incremental changes have been made by EPA, but the critical improvements recommended by the NAS have not yet materialized and thus have not found their way into the latest IRIS assessments.
EPA’s Chemical Assessment Advisory Committee (CAAC) was formed following the NAS report to provide expert advice to the IRIS program on their assessments. ACC has long supported the creation of an independent panel to help EPA with its work. After its formation, the NAS recognized the CAAC’s value as a resource EPA can tap to help resolve the problems with IRIS and to help develop standardized methods. More recently, the lack of progress on the IRIS program has not escaped the attention of the CAAC.
In reviewing recent IRIS assessments, the CAAC notes that some improvements have been made. However, the CAAC also notes that there are important areas where EPA is still falling short when it comes to handling the scientific evidence underpinning its determinations. Here are two important areas raised by the CAAC that are worth highlighting:
Selecting Studies
In 2011, the NAS recommended the establishment of standard methods for evidence identification and clear guidelines for study selection. The NAS specifically recommended that the exclusion and inclusion of criteria by which studies are selected be clearly articulated in all assessments. After reviewing EPA’s most recent draft assessments, the CAAC found that the Agency is still missing the mark when it comes to identifying relevant studies.
This shortfall is illustrated in the CAAC’s final report on the IRIS draft ammonia assessment, which stated “EPA should clarify the criteria by which it determines the significance of specific limitations in studies. A clarification (or citations to relevant guidelines) as to how EPA judges a potential limitation to be a major one or not should be added.”
Regarding the draft assessment of trimethylbenzenes (TMBs), the Committee found “… that while the search strategy and rationale to select studies was clearly articulated, the exclusion criteria and implementation of those criteria was not as transparent.” The report went on to say that, “[w]hile it was clear which papers were used in the draft assessment, there were no means of determining which papers were excluded from the assessment. Thus the review does not provide sufficient documentation to determine if important papers may have been overlooked or considered and then omitted from consideration based on EPA’s criteria.”
Assessing Study Quality
The NAS also recommended that EPA develop a formal process for evaluating studies and integrating them into an assessment. The CAAC has concluded that EPA has yet to adopt a consistent systematic review process when conducting chemical assessments.
For example, in the TMB assessment the CAAC found the “… process of systematic review still needs development. Documentation of the process of identifying literature has progressed, but further development is needed in establishing standard practices for abstracting relevant data, for evaluating study quality, strengths and shortcomings, and for integration of evidence across studies.”
ACC and other stakeholders have provided similar feedback to EPA on these issues in the past, and we are encouraged to see the CAAC echo the same concerns. It is unfortunate that after five years, IRIS is still producing deficient assessments that do not reflect the 2011 NAS recommendations.
One way EPA could accelerate its efforts to improve the IRIS program would be to develop its long-awaited draft handbook for conducting assessments. We strongly urge EPA to facilitate a broad stakeholder discussion and seek public comment on this document before sending it to the CAAC for review. The handbook, when finalized, should provide a more methodical and transparent approach to evaluating chemicals.
