The Portal was founded as an independent organization in February of 2016. Nearly a year and a half later, I offer some reflections on the initiative: I present the rationale for why we insist on data transparency, why we did not adopt strict probe validation criteria, and why I believe the Portal can help improve the quality of the scientific literature even if our experts (Scientific Advisory Board, SAB) never agree on what makes a compound a chemical probe.

When the Structural Genomics Consortium (SGC) approached me about helping build the Portal, I was immediately intrigued by the opportunity. As an editor for Nature Chemical Biology, I handled large numbers of manuscript submissions reporting new chemical probes or applying chemical probes to discover new biology. In that role, I became keenly aware of the problems that are rampant in the chemical probe literature, and I longed for a resource like the Portal. It could have saved me hours of the time I spent looking for evidence that a probe reported in a paper under consideration was fit for the purpose chosen by the authors.

Red flags

During those countless hours of digging, I learned many things that have been invaluable in building the Portal. In short order, I developed a sensitivity for certain red flags that appear in chemical probe papers:

We performed a chemical screen, and our z-score was less than 0, but we proceeded to call our lead, hit compound a chemical probe anyway;
We report a chemical probe that inhibits Protein X, and we prove that the probe acts specifically on Protein X by showing that it kills several cancer cell lines;
We report the first specific and selective inhibitor of Protein X (IC50=300 uM); or
We apply Compound Y as a chemical probe that is a specific inhibitor of Protein X (validation data not show, or to be published elsewhere but not included for review) to reveal an unknown role for Protein X in {insert interesting biology here}.

While decisions on papers containing these red flags were relatively easy, I was always surprised by the sheer volume of papers the journal received that contained these types of red flags. I was doubly surprised when these papers were published elsewhere with little to no modification. I continue to be surprised that now, years later, this trend persists. To ensure the Portal is positioned to counter this trend, we require that data supporting the use of a chemical probe against a specific target is available, and we critically evaluate those data before we ask the Portal’s experts (the SAB) to rate and comment on a chemical probe.

There is no substitute for transparency when it comes to validation data. As I will discuss below, experts can and do disagree about what level of validation is required for a compound to be considered a chemical probe, but providing access to these data should be a non-negotiable aspect of any probe-evaluation process. When data are available, it is readily apparent that some compounds advertised as chemical probes are not. We triage these compounds; we either set them aside (perhaps more complete validation data will be forthcoming someday) or we post them as Historic compounds. When data are made available, high-quality chemical probes readily emerge; we endorse these compounds and see very consistent ratings from our experts. In some cases, however, inspection of validation data does not yield an obvious answer; we consider these gray-area compounds. If we post them for expert review, they typically earn between two and three stars, indicating that some SAB members want to see more data while others think they are good enough if they are applied with the proper care. As I discuss below, whether the Portal endorses these compounds as chemical probes or not seems less important than whether Portal users heed the nuanced advice from our experts about how to use these compounds to yield meaningful data.

Guidelines not check lists

To my knowledge, the first authoritative article to appear in the literature that aimed to provide guidelines for chemical probes was aptly titled, The Art of the Chemical Probe. Others soon followed, but with few exceptions, these articles emphasize a common theme: there is no one-size-fits-all definition for a chemical probe.

Chemical probes are designed to interrogate a specific protein target or group of targets. Thus, the characteristics a compound must exhibit to be considered a chemical probe depend critically upon that target in the context of other chemical tools that are available to interrogate that same target. For example, when the first well-validated inhibitor of Protein X is reported, the standards any new compound targeting Protein X will have to meet to be considered a chemical probe change immediately. The existence of chemical probe raises the bar for all subsequent compounds.

In recognition of this dynamic and context-dependent definition for a chemical probe, the Portal has not elected hard and fast criteria to define a chemical probe. Rather, we have enlisted experts from across multiple disciplines, professional backgrounds and geographic locations to help us evaluate compounds. By opting for a context-dependent evaluation process, one that depends on the opinions of experts rather than a checklist, we recognize that we have chosen a more difficult path, one that invites our experts to disagree with us and with each other. With each passing day, I grow more certain that this process is the only one that is fit for our purpose: to help scientists identify, source and apply chemical probes in their research.

When experts disagree

When I signed on to the Portal initiative, I was excited for the opportunity to contribute to a resource that I considered long overdue, that could help save millions (if not billions) of wasted research dollars and untold wasted hours. Despite my enthusiasm, I also had some concerns. My biggest concern was that the Portal would become the place where all compounds went to die. As a member of the Portal’s Board of Directors once said, “All compounds are on probation.” While I agree with this sentiment, I was afraid that by building the Portal as a resource designed to critically evaluate chemical probes, we risked emphasizing the weaknesses of compounds at the expense of recognizing their utility.

While Portal experts do provide critical reviews, I have been delighted to see that their criticism is both nuanced and constructive. If a chemical probe has activity against an unintended target, the SAB does not necessarily discount it as a chemical probe. They are more likely to acknowledge the weakness and then recommend experiments to help Portal users attribute any interesting biology to the correct protein.

Importantly, members of the SAB do not always agree with me or with each other. While resolving these disagreements can be time consuming, the existence of these disagreements highlights the need for a resource like the Portal, and the process of resolving them increases the value of the Portal.

The fact that experts disagree about whether a compound has been sufficiently validated to merit the designation chemical probe reflects just how difficult it can be to make a binary call about a compound. Although experts readily identify the low-hanging fruit (i.e., the obvious non-chemical probe) and the high quality chemical probe, there are many instances where compounds fall into a gray area – they have some characteristics of a chemical probe but important questions remain about its activity. Disagreements tend to arise around these compounds.

The dialogue around these compounds shows that experts have different priorities when considering chemical probes. Whether this dialogue happens spontaneously or whether it emerges through a process that I moderate behind the scenes, it ultimately appears in the Comments section of a probe page. By reading these exchanges, Portal users can learn not only about how and why priorities can differ, but also that experts who disagree about whether a compound is a probe or not tend to agree about the weaknesses of a compound and how those weaknesses should impact the way experiments are conducted and interpreted. For these gray-area compounds, I am beginning to believe that the Portal’s rating of the compound (i.e., 2 or 3 stars) matters far less than whether Portal users heed the advice of our experts and change the way they conduct and interpret experiments performed with these compounds.

It takes a village