One way in which the National Institute of Environmental Health Sciences (NIEHS) National Toxicology Program (NTP) could demonstrate its commitment to transparency is by seeking public comment on its guidance documents prior to finalizing and implementing them. Not only would this help NTP to develop a robust approach, but it would also allow outside experts, all with diverse backgrounds and perspectives, to contribute what they have learned over the course of decades of experience.
When the National Research Council (NRC) conducted their review of the Integrated Risk Information System (IRIS) in 2014, they stressed the importance of ensuring broad and diverse stakeholder engagement in the IRIS process. There is no reason for the NTP program to be held to different standards, especially when developing new approaches that are relevant to multiple agencies and stakeholders.
With the publication earlier this summer of its new “Handbook for Preparing Report on Carcinogens Monographs,” NTP missed a valuable opportunity to increase confidence in the program’s accountability to the public and ensure its approaches and conclusions are based on the best available methodologies and information.
What the Handbook is and why it matters
The Handbook provides guidance and instructions for developing the Report on Carcinogens (RoC)—specifically, preparing monographs (focused evaluations) for substances the NTP considers for listing in the RoC. The monographs are the product of a literature-based review and evaluation. The Handbook is designed to provide a clear and formal process for NTP to follow in creating the RoC.
The Handbook is important because many stakeholders see the RoC as an authoritative body on the state of the science for a particular chemical or substance. RoC listings inform decisions by outside groups, including states and other federal agencies, to help manage the safety of certain chemicals and whether they should be present in products.
While NTP has always described its process for conducting literature reviews, the Handbook is essentially the first of its kind. It describes the actual approach for conducting a scientific evaluation for the RoC and includes sections on identifying and selecting studies, extracting data from studies, and evaluating the quality and utility of individual studies. It essentially spells out a process for NTP to follow that will determine the fate of a specific chemistry.
Although ACC views the current version of the Handbook as a needed addition to the NTP chemical review process, the guidance is not as strong as it should be—or as it could have been, if NIEHS had allowed other experts to offer their feedback first.
Below, there are three areas where NTP delivered improvements with its Handbook and seven significant and impactful areas where it could have delivered more improvements had it taken full advantage of the vast expertise outside of NTP.
What the Handbook delivers
- Standardized Approach: One important benefit of the new Handbook is that it creates a standardized approach for preparing RoC monographs. In theory, this approach would allow any third party to follow the guidance provided by the Handbook and conduct an independent review. It also enables interested parties to evaluate the monographs against the Handbook’s standards. If the standardized approach falls short, however, the scientific evaluations will suffer.
- Quality and Utility Discussions: Another benefit the Handbook delivers is its emphasis on the importance of clearly articulating the standards for judging quality and utility of research before beginning an assessment. When conversations about quality and utility take place prior to an assessment, scientists can ensure everyone is on the same page before beginning the hard work of conducting reviews, rather than afterwards.
- Sound Decisions (potentially): Most importantly, if the guidance provided is sound, the Handbook can help ensure the decisions the NTP makes about RoC candidate chemicals are well-informed, based on a robust evaluation of the scientific evidence, consistent with one another, and relatively simple to evaluate and understand.
What the Handbook missed
- Transparency: As mentioned earlier, the NTP could have demonstrated its commitment to transparency and accountability to the public by inviting outside comment on the Handbook prior to publishing it. Although the Handbook was subject to review by five individuals, two of them are former NTP board members, and, it appears, none of them specialize in the systematic review approaches NTP is trying to implement. In addition, the five reviewers do not represent a diverse set of stakeholders and scientific perspectives.
- Weight-of-Evidence: NTP doesn’t explain how negative evidence or conflicting evidence will be considered during the data integration step, nor does NTP emphasize that the highest quality/utility studies should be given the greatest weight. For instance, in NTP’s example of presenting tumor data from animal studies, the figure does not take study quality into account. Thus readers of the figure can come away with a misleading interpretation of the data. Similarly, the evidence-based tables should include the overall quality determination for the study.
- Mechanistic Data: For mechanistic data (critical information that may explain how chemicals act in animals and in humans), NTP seems to only rely on authoritative and secondary reviews. It is not clear why NTP would discount primary literature that would be extremely helpful. NTP also discusses relying on characteristics of carcinogenesis; however, the publication NTP relies on here is not publicly available, another example of NTP falling short when it comes to transparency. In addition, mechanistic data is useful for more than helping to interpret other studies, yet NTP does not seem to take full advantage of the utility of mechanistic data.
- Strength of Association: It is disappointing that NTP does not consider the strength of an association to be critically important. The Handbook suggests that for human studies, there could still be causal associations even though the strength of evidence (e.g., relative risk ratio) is weak. It’s not clear how NTP can infer causality if an association is weak.
- Listing Criteria: The NTP missed an opportunity to update the listing criteria for known and reasonably anticipated carcinogens. The current standards used by NTP do not consider the weight of all the available evidence but follow an incomplete “strength of some evidence” approach. If it is not clear why this is radically incomplete, imagine if a trial judge instructed the jury to follow the same protocol: “Ignore this piece of conflicting evidence and focus instead on whether these bits seem convincing enough—by themselves”!
- Confounding: When evaluating human studies, it is normal to consider factors that may confound a study result; NTP does not consider confounding as part of the study quality/utility evaluation but treats it separately. As the ability to understand and dissect confounding influences on a study relate directly to the value and quality of a study, these findings should be incorporated into the overall study quality/utility rating.
- Animal Studies: NTP focuses predominantly on the evaluation of human studies and spends less time discussing animal and mechanistic studies, which are often more widely available than data from human studies. Unfortunately, NTP does not consider these information streams as sufficiently informative or important to help inform human health hazards. The current NTP listing criteria can be fulfilled with two positive animal studies, which are assumed to be relevant to humans unless there is compelling evidence otherwise. However, NTP does not explain its standards for what constitutes compelling evidence, and this gap decreases the transparency of their decisions.
There is no doubt that NTP’s previous “black box” approach for developing the RoC needed to be changed, and the creation of its Handbook is a step in the right direction. However, it would be premature to view it as complete. It should be treated as a work in progress that could greatly benefit by incorporating review and comments from additional experts.
There is broad recognition in the NRC comments and elsewhere of the value of public input and dialogue, and so we strongly encourage NIEHS and other federal agencies that are developing and incorporating systematic review practices into their chemical specific evaluations to ensure that “handbooks,” “guidance documents,” and other important tools benefit from stakeholder input and dialogue.
