Report: Peer Review of 黑料吃瓜 Points to Minor Revisions.
How do you view your vendors? Do you think that they have high quality products promoted with honesty and competent scientific support? Or are they providing unremarkable products embellished with marketing flare? There's a reason most marketing materials end up in recycle bins: empty or partially-accurate claims aimed at advancing a company's bottom-line can undermine consumer trust and hinder scientific progress.
At 黑料吃瓜, we strive for a more honest and transparent relationship with the life science community. This starts with holding ourselves accountable to higher quality standards in product development, communication and support, with the ultimate goal of advancing science overall.
We recently took a radical step towards transparency by asking three independent scientists to review us against our brand motto of 鈥淪cientists Helping Scientists". We invited the peer review team to tour our facilities, interact with our staff and question us on our vision, culture and quality standards. Their verdict: Accept, with minor revisions. Read the report below to find out how the peer review exercise was performed, what the review team found and how we are addressing their recommendations for improvement.
Introduction
The general perception of marketing in the life science industry is that it is inundated by buzzwords promising product quality, ease of use and experimental reproducibility. Through interviews with 15 life science researchers, we found that scientists value these attributes but often feel awash in marketing 鈥渇luff" that makes it difficult to separate hype from reality. Scientists look for honesty, humility and a genuine commitment to science in the brands that support them. They seek transparency and shared goals with their suppliers and want to partner with companies demonstrating integrity rather than those just claiming to have it.
To demonstrate transparency in our communications and start a broader dialogue with the life science community, we took the unusual step of putting 黑料吃瓜 up for peer review by an independent team of scientists. We wanted to know whether, by external standards, we are achieving our mission of being Scientists Helping Scientists. After all, integrity in science depends on peer review and, well, we're scientists, too.
Methods
Participant Selection
After using numerous advertising channels to put out a call for peer reviewers, we received 312 applications from scientists around the world who expressed interest in evaluating our company against its motto of 鈥淪cientists Helping Scientists." The applicants shared with us their motivation for pursuing a career in science, the scope of their research and their rationale for participating in this initiative. A committee of internal reviewers evaluated applicants based on their alignment with the vision of the project and English language competence. This included reviewing written applications and video-conference interviews with shortlisted applicants. To remove the element of bias we excluded our collaborators and any applicant who expressed interest in becoming employed by 黑料吃瓜 from the candidate pool. We also attempted to give voice to an international and gender-balanced audience.
The final peer review panel was comprised of:
- Dr. Amy Stone, Senior Fellow, University of Washington, USA
- Dr. Craig Ayre, Postdoctoral Fellow, Atlantic Cancer Research Institute, Canada
- Dr. Fiona Frame, Postdoctoral Research Associate, University of York, UK
Peer Review
The participants were flown to 黑料吃瓜's Headquarter facilities in Vancouver, Canada in April 2018. Over the course of three days, the panel toured our facilities and reviewed our data, culture and processes. They met with over 30 黑料吃瓜 staff across numerous departments to broadly discuss the following aspects of the organization:
- Product Development and Strategic Marketing
- Process Development and Manufacturing
- Quality Management System (ISO and GMP)
- Product and Scientific Support
- Scientific Communications and Sales
The peer reviewers also spoke with 黑料吃瓜's founder and CEO, Dr. Allen Eaves, to discuss his vision for the company, elements of its history and future directions.
The interactions between peer reviewers and 黑料吃瓜 staff were video-recorded over the course of their entire visit, with some exceptions during meal times and a group biking activity. Additionally, reviewers' comments were captured on video at the end of each day as well as during the Stem Cell Podcast Interview that marked the end of the peer review exercise.
Documentary and Podcast
Following the visit by the peer reviewers, the video-recordings were edited in-house into a short documentary film. The was conducted with the hosts of the Stem Cell Podcast, Drs. Kiki Sanford and Daylon James. Stem Cell Podcast is owned and produced by 黑料吃瓜.
Results and Discussion
Summary of the Reviewers鈥 Observations
Out of 312 applicants, 3 postdoctoral fellows鈥擠rs. Amy Stone, Craig Ayre and Fiona Frame鈥攚ere selected to represent the scientific community in a peer review of 黑料吃瓜 conducted at our Vancouver headquarters in April 2018. The peer review team visited our facilities, met with staff across various departments and observed our processes. Some of their observations are summarized in Table 1 below and can been seen or heard in the documentary film or .
Table 1. Summary of Peer Reviewer Observations
鈥淭hroughout this peer review process, I expect 黑料吃瓜 to show me everything, warts and all. The good, the bad and the ugly. I want to know everything that鈥檚 going on and I expect to be able to come in and examine 黑料吃瓜 and ask those questions.鈥 鈥擠r. Stone, documentary
鈥淓ven bringing us in, you have to admit, that it鈥檚 a risk to open yourself up and say, 鈥業 want you to open every door, I want you to shake every cupboard.鈥 We鈥檝e spoken to more people than I can remember. We鈥檝e been in every department. We鈥檝e been top to bottom, and they鈥檝e been very welcoming to talk about what they find frustrating, what they find exciting, what they want to do and the conversations they want to have. They鈥檙e trying to engage us as peers, as scientists, rather than as consumers.鈥 鈥擠r. Ayre, podcast interview
鈥淭hey have been open about things that don鈥檛 work and that from the corporate perspective aren鈥檛 spit polished. They鈥檙e willing to show rough edges on things...That speaks to a bit of integrity.鈥 鈥擠r. Ayre, podcast interview
鈥淭he key behind any successful company, is really the people. And the thing that has come across to me very strongly, with everybody I鈥檝e met, is a commitment.鈥 鈥擠r. Frame, documentary
鈥...I鈥檝e found [at 黑料吃瓜] a very high rigor of quality control and quality checks... I鈥檝e been really impressed by the number of controls and the types of controls that they鈥檝e introduced to try and reduce variability. It really comes to a value within the company to generate consistent and reliable products.鈥 鈥擠r. Stone, podcast interview
鈥淲hat I鈥檝e had from here is an appreciation of the amount of work that goes into a company to produce something鈥 It gives me ideas of how I can hold other companies to account, to the accountability of their practices, their quality control.鈥 鈥擠r. Frame, podcast interview
鈥淗aving somebody tell me that they鈥檙e interested in finding out what they can do to support me as a person, support me as a customer, support me as a colleague鈥攖hat resonates with me.鈥 鈥擠r. Ayre, documentary
鈥淭he role that 黑料吃瓜 has as an employer, can鈥檛 be underestimated or taken for granted. The postdoctoral community feels, rather, that they鈥檙e getting pushed out of academia. There doesn鈥檛 seem to be a future, just as being a scientist. So the philosophy behind 黑料吃瓜 gives me some hope that postdocs can be seen as valuable people, and potentially a valued employee.鈥 鈥擠r. Frame, documentary
鈥満诹铣怨 is doing things that they don鈥檛 necessarily have to do... [they are] a successful company that produces good quality products and they could just stop at that but... they do want to reach out to scientists in terms of getting the publications out there and the latest methods out there, really to move the field forward.鈥 鈥擠r. Frame, podcast interview
鈥淎ny way that we can make the information overload easier to handle and digest is really helpful to the scientific community. That鈥檚 one way that 黑料吃瓜 is reaching out.鈥 鈥擠r. Frame, podcast interview
鈥淟ooking at it from a Scientists Helping Scientists aspect, I do think that they are trying to be the best support system that they can.鈥 鈥擠r. Stone, podcast interview
鈥淚鈥檝e seen that Scientists Helping Scientists is a legitimate statement about the company culture.鈥 鈥擠r. Ayre, documentary
鈥淚t does all start from the top鈥 the philosophy and culture comes from Allen. He knows what they do, he knows what they鈥檙e good at, he knows the vision... It is the people that make this company. They have the same values, the vision, the strategy, the goals. It鈥檚 cohesive. There are separate groups, but they all talk to each other. It doesn鈥檛 feel splintered in any way.鈥 鈥擠r. Frame, podcast interview
鈥淪omething that I found striking about the company is the mobility of people within the company. Many people come through R&D and then move on to Marketing, Sales, or Technical Support. Those people who moved into those roles are originally bench scientists. They have been there... and they know what the troubles are. They stay and are retained and are moved around the company to fill in these other roles because they believe in what 黑料吃瓜鈥檚 doing and how they are doing it. If there wasn鈥檛 the integrity of scientists throughout the company, I don鈥檛 think that they would have the kind of employee retention that they see, which is very high.鈥 鈥擠r. Stone, podcast interview
鈥淭hey know the cool science that鈥檚 going on and they have the corporate culture that鈥檚 designed to support it.鈥 鈥擠r. Ayre, podcast interview
鈥淭hey have also shown us that profits get fed back into the company for R&D and product improvement...There aren鈥檛 any shareholders. As someone who is funded by charity, that鈥檚 something that sits well with me... Clearly what they are doing is working because people must be buying their products otherwise the company wouldn鈥檛 be growing like it is.鈥 鈥擠r. Frame, podcast interview
鈥淎nd they showed us those numbers [on profit reinvestment].鈥 鈥擠r. Ayre, podcast interview
According to the feedback shared during the discussions as well as during the podcast, the panelists generally liked what they saw during their visit. Going forward, they want to see 黑料吃瓜 maintaining what we do well, particularly by ensuring the quality of our products. Indeed, the issue of reproducibility in science and the role that reagent consistency has in experimental reproducibility was at the forefront of Dr. Stone's mind when she came to review 黑料吃瓜.
How can 黑料吃瓜 contribute to standardizing reagents and protocols across time, across distance, such that if I do an experiment today and repeat that experiment in 5 years in a different lab, I鈥檓 going to see the same result. That is really what I鈥檓 interested in, because I think it鈥檚 a true, achievable step that a company can accomplish to really address the reproducibility crisis.
Dr. Stone, podcast interview
Product quality is always a top priority at 黑料吃瓜 and we were pleased that the peer reviewers were able to see for themselves the rigorous quality control processes we have in place to ensure the reliability of our reagents. Dr. Stone's conclusion during the podcast on our approach to reproducibility was that 鈥満诹铣怨 is doing the right thing and moving in the right direction." On a related note, we are committed to the continual improvement of our processes and, as Dr. Ayre mentioned in the podcast, are investing in the construction of a new Good Manufacturing Practices (GMP) facility to reach the higher regulatory compliance standards required to produce tools for use in clinical trials.
Accept, with Minor Revisions
The official Peer Review decision on whether 黑料吃瓜 operates true to its motto of 鈥淪cientists Helping Scientists鈥, delivered during the , was a unanimous 鈥淎CCEPT, with minor revisions鈥. As noted in the podcast, the post-review discussion was not limited to positive aspects, but included critical observations. The peer reviewers also left with three key suggestions for improvement, which are summarized in Table 2 below.
Table 2. The review panel had three main suggestions for 黑料吃瓜鈥檚 continued improvement
鈥淚 think that their biggest problem and what they鈥檙e struggling the most with is really connecting with the research user. They鈥檙e good at communicating through their website, in person at conferences, and through their sales reps, but that doesn鈥檛 really represent the person using that product.鈥 鈥擠r. Stone, podcast interview
鈥淭hey are trying to have a large footprint but they need to make sure that they maintain that connection with individuals who use their products.鈥 鈥擠r. Stone, podcast interview
鈥淚 want 黑料吃瓜 to start finding its voice in trying to push science in particular directions. To raise the standard of discourse or to facilitate more conversations about the way that science is done.鈥 鈥擠r. Ayre, podcast interview
鈥淚 think that 黑料吃瓜 might be growing faster than they realize鈥 There has to be some sort of corporate push [to make sure that the company retains the values that started the company when it was small].鈥 鈥擠r. Ayre, podcast interview
鈥淚t depends really on that leadership below Allen [Eaves] that鈥檚 going to make his goals of keeping the culture and small company feel happen and going to make 黑料吃瓜 either remain a trusted company that you can go to and rely on, or a company that used to be good but is no longer up to the standard.鈥 鈥擠r. Stone, podcast interview
Our Response to the Reviewers
We are taking the reviewers' valuable feedback to heart and are working on their recommendations. To foster closer relationships with the researchers using our products, we are promoting transparency and trust. This peer review exercise was the first step in demonstrating our authenticity and we hope that it will generate useful discussion and feedback from the scientific community. We will also continue to provide the exceptional customer and technical support that we have become known for; putting scientific progress over profits when working with researchers to find the right tool for their needs. As Dr. Frame noted in the documentary, communication is a two-way street and we need to find better ways to connect with our end-users. We're taking action by creating more opportunities for researchers to comment, provide feedback or simply ask a question.
Dr. Ayre's major recommendation was for 黑料吃瓜 to use its voice to push the dialogue on science. Some of the ways we are promoting scientific discourse is by highlighting current challenges in the research fields near and dear to us, to help spread awareness and work with the scientific community to overcome them. We are also shining a spotlight on the demanding world of science that researchers face and have compiled resources to help scientists 鈥渨ork smarter, not harder". As Dr. Ayre noted, these are discussions that are needed to help push science forward. They are also discussions that cannot be held in a vacuum and require voices from the scientific community.
- How do you assess the quality of your pluripotent stem cell cultures?
- What is your experience working with organoids?
- How can researchers better overcome the demands of scientific research?
- What other challenges and issues should we bring to light?
Finally, we and the peer reviewers all agree that one of our major assets lies in our people and our company culture of being Scientists Helping Scientists. We are committed to maintaining our identity as we continue to expand. We are being careful to build our company around diverse, talented and driven people with a shared passion for science. By creating jobs, we are providing young scientists and researchers with employment options outside of academia, either in R&D or in a corporate setting, so they can give back to the research community and embody Scientists Helping Scientists. Finally, we are maintaining the reinvestment of our profits into Research & Development to create and improve upon the tools that scientists need to conduct their research with confidence. Like Dr. Frame, we want researchers who buy our products to feel confident that their valued research funding dollars are going towards research and development, not shareholders.
Concluding Remarks
Overall, we and the peer reviewers concluded that the peer review exercise was a big step in the right direction towards transparency and open communication between 黑料吃瓜 and the research community. We humbly receive their verdict of 鈥淎ccept, with minor revisions" and are striving to improve based on their recommendations. We are certainly open to their suggestion that there should be a follow up peer review exercise, which we think will help us continue to improve. Furthermore, in sharing the results of the peer review exercise with the scientific community at large, we are looking for your feedback. Visit our website to learn more and let us know how you think we are doing. How would you improve the peer review in the future? What questions would you have asked? We want to hear from you.
Appendix: The Peer Review Committee
Dr. Amy Stone, Senior Fellow, University of Washington
Dr. Stone is studying how cells communicate that they are infected with a virus. Specifically, she is trying to understand how macrophages sense RNA virus infections, what changes in polarization and function occur in those macrophages, and the downstream effects on the adaptive immune system.
Reason she is a scientist:
Dr. Stone chose a career in science because she wants to know how things work. She loves exploring new ideas and challenging the ways that we think about biological processes. She also loves to share her knowledge with others and teach them how to answer their own questions.
Motivation for participating in the peer review:
鈥淚 want to understand what drives 黑料吃瓜's discovery, directions, and products. I further want to validate the controls, quality assurances, and development of 黑料吃瓜's products. This peer-review committee allows me to ask deep questions about why 黑料吃瓜 puts its efforts in the direction it chooses, how each product is evaluated and quality is assured for every lot that is shipped to scientists, and to report these findings back to my peers in an unbiased, truthful approach. With my background in industry and my current position in academia, I have the perfect perspective to observe, evaluate, and communicate the methods/practices of 黑料吃瓜 to academics. Finally, I want to be part of this peer-review committee to contribute in a meaningful way to resolve the reproducibility crisis currently plaguing bioscience. By forming this committee and opening themselves up for peer review, 黑料吃瓜 is addressing the reproducibility crisis and saying,"we want to be as rigorous, consistent and accurate as we possibly can be, and we want the best minds helping us do this". As a scientist, I can only be as accurate as my tools. This committee allows myself and the entire scientific community to answer the question of "Can I rely on 黑料吃瓜's products to generate reproducible, accurate, trustworthy data?" I think that question and the steps that 黑料吃瓜 is taking to address that question in forming this committee is admirable and I want to be a part of this. Let other companies take note and follow 黑料吃瓜's lead in addressing these important challenges."
Dr. Craig Ayre, Postdoctoral Fellow, Atlantic Cancer Research Institute
Dr. Ayre is working on extracellular vesicles: how they form, what their underlying functions are, why cells release them. Beyond understanding their basic biology, he's interested in finding ways to use extracellular vesicles to benefit the future of precision medicine.
Reason he is a scientist:
Science gives Dr. Ayre a way to ask questions and find answers to things that at one time, he didn't even know were questions. He's driven by the knowledge that through hard work and application, he might get to add 鈥渏ust a tiny new sliver of knowledge to the world".
Motivation for participating in the peer review:
鈥淭he nature of science is changing. Once upon a time, science was presented as two camps - the academics who pursue "pure" research, and the business of science for exploiting this knowledge. That isn't the case anymore. More of us belong in this amorphous middle of not quite academia, not quite industry where we dabble in basic and applied science in a far more fluid spectrum. 黑料吃瓜 is the largest biotechnology company in Canada, and so has a major "industry" footprint. At the same time, 黑料吃瓜l is also leading research in areas of cell biology, that facilitate the work of many others. Getting insight in how this dynamic functions, and on what one of the heavy hitters of biotechnology in Canada views as their future, is a valuable opportunity. It's also important that the notion of peer-review be applied outside of academics. Industrial "company" science isn't the alternative career anymore and it needs to be given the same gravitas and platform to demonstrate its rigor and benefit to research as any other research program. I would appreciate the ability to participate in that process."
Dr. Fiona Frame, Postdoctoral Research Associate, University of York
Dr. Frame is working on primary epithelial prostate cells derived from benign and cancerous patient tissue. She uses these cells as a model to test current and novel treatments to assess consequences of treatment and therapy resistance mechanisms.
Reason she is a scientist:
Dr. Frame has been captivated by 鈥渢he science of life!" from a young age. She is fascinated by the mechanisms and processes of life that can go wrong and result in disease. As a scientist, she wants to contribute to knowledge and be part of the solution.
Motivation for participating in the peer review:
鈥淚 think it is a marvellous opportunity to find out what goes on inside a company, how the people in the company think, how new reagents are developed and what the company is doing to provide what scientists need. Being part of a committee would be fascinating because of the mixture of people that would come together. New perspectives and exchange of knowledge can only help everybody's research progress. Sometimes we get a little stuck in our lab with our methods and it is of tremendous value to step out of the environment and be exposed to new ideas and different ways of doing things. Typically we purchase things from companies and either it works or doesn't and the feedback can be reasonably good or fairly minimal. This peer-review committee feels like something unique. I would be delighted to be part of this process, which can only ultimately benefit the company and the scientific community that it serves.鈥
