Being both Correct and Non-Consensus
One of my favorite podcasts, Acquired, often likes to point out that in the investment world, you have to be BOTH correct and non-consensus to have outsized return on investment. If you’re correct, but everyone agrees (there is consensus), the fair market value is likely already baked into the price and you won’t see as big of a return on investment. If you have a non-consensus opinion but incorrect, you’re just going to lose money. You need to be both correct and non-consensus to leverage insights into opportunities and then opportunities into returns.
And this is hard. After all, the consensus opinion is often right, and even if its wrong, you’ll be with the crowd and not be embarrassed by disproportionately bad outcomes compared to others. Being non-consensus can be a question of timing (you want to be early to a trend), but equally there are many stories of people being much too early such that a market or the technology isn’t there. What does it mean to be correct and non-consensus in medical AI? In making choices on which collaborations and projects to work on, how I can chose high impact projects, but also high impact projects that would defy expectations or have outsized impact?
Fortunately, I think this is easier to be correct and non-consensus in medical AI, as the medical field is less clear what are the capabilities and opportunities of new technologies. Healthcare is slow to adopt those trends and limited by the ability to distinguish between hype and real innovations. Some of my assumptions and axioms might seem obvious when written out as below, but the healthcare system and researchers aren’t acting as if it’s true. If we can take learnings from general AI, it can be the foundation for a sizable moat in healthcare AI.
Axioms
AI is a transformative and generational technology. I think this is the most obvious and potentially most consensus conclusion right now, with computer vision in self driving cars, tremendous dictation and voice recognition capabilities in audio models, and LLMs being used for everything under the sun. But the corollary is:
AI is deep learning. The reason there has been so much excitement in the last decade in general CS has been the amazing abilities of deep learning and the use of GPUs for training that was previously not feasible. But still, healthcare systems are being pitched RF, SVM, or other non-deep learning technologies as AI solutions. They might be great products and I can’t paint with a broad swath that they all don’t work, but the expectations that they are “AI” is nonsensical.
Technology accrues advantages to the incumbents. There is tremendous value in being the last mile and already being in the ecosystem. By virtue of having a pre-existing relationship with customers and patients, the cost of deployment is lower and the value of implementation is higher (more likely you can capture the value of improved performance).
AI is data and data scale dependent. Understanding how to extract, clean, and organize data is important, but it’s much easier to get value out of historical data than to prospectively get more data. At the same time, in most tasks we’ve evaluated, there is no asymptote in the relationship between data scale and model performance, as more data consistently results in better models.
Domain Knowledge will always be necessary. In many projects, the AI model architecture is a hyperparameter, has only incremental impact on overall importance, but a lot of the hard work is recognizing the use-case, the pitfalls in the data, and understanding how to integrate is very use dependent. [2]
Human first. In all organizations, empowering the right people can make significant impact. Finding the right people to work with, motivated for the right reasons, and building consensus with the end users is critically important.
Implement first, if your AI model provides value, the money and business models will follow. In a fast moving field, it’s too late to wait for the grant to do the research project. Similarly, in businesses and start-ups, it’s less the initial idea and more the implementation and effort to build it out. Just go, and you’ll encounter the problems you haven’t even thought about at the beginning, but you’ll also be the first to build the muscle and knowledge in how to handle such challenges.
Even if all these things seem obvious, most healthcare organizations do not act in a rational way when it comes to these trends. Software or ‘dry lab’ is inherently cheap compared to basic science or many other institutional efforts, but many hospitals are not willing to invest in these trends. I have some ideas on what this means when it comes to behaviors healthcare organizations should have:
Behaviors
Invest in infrastructure. Not just AI infrastructure, but software infrastructure. Being on prem actually forces you to have the talent to manage such systems. It’s a headfake to assume you can outsource everything to the cloud. If you can’t hire the right talent to sysadmin your internal infrastructure, do you think you can choose and pay the right amount for even more expensive ML talent? A healthcare system seeking to cut IT costs by moving to the cloud is likely not the right IT org to build and maintain AI.
Invest in human capital. The right people are much rarer than the
data or infrastructure. Many people are interested in work in healthcare if the institutional inertia and bureaucracy can be overcome. No matter the organization, what makes people enjoy thier work is heavily influenced by the few people day-to-day they work with. Make it worthwhile for people. Hire people you want work with.
Build domain knowledge. “[T]he vast majority of early ML researchers were self-taught crossovers from other fields. This created the conditions for excellent interdisciplinary work to happen. This transitional anomaly is unfortunately mistaken by most people to be an inherent property of machine learning to upturn existing fields. It is not.” Brilliant article that I’ll just link to https://www.moderndescartes.com/essays/why_brain/
Corollary: Follow the data. Being stubborn is neither helpful in research or implementation. By building your own models, you’ll inevitably build intuition on which models are ready for "primetime” and which models are fun research projects but not worth effort to deploy. There are many clinical use cases that would great to have, but the data doesn’t show the performance you need. Drop it.
[1] I recognize this is a strong opinion, but if I had it my way, NEJM AI would only accept works that is either deep learning or a prospective RCT (ideally both!).
[2] This is not to minimize importance of model architecture design, but even NAS is very far from state of the art in most common modalities. If your unique advantage isn’t in designing AI models (for even in CS, not everyone is well suited to design architectures), its not the area that best maximizes your impact.