August AI outperforms OpenAI’s GPT-4, Google’s MedPaLM

Recently, ChatGPT helped diagnose a child’s disease after a string of doctors failed to identify the condition. When use cases for large language models in healthcare are slowly finding its way, an Indian health-tech startup has created a model that has not only outperformed OpenAI’s GPT-4 and Google’s MedPaLM on USMLE (US Medical Licensing Examination), but is available for free and has already helped over 2000 people since its launch.

Indian ‘August AI’
August AI, a large language model created by Bangalore-based health AI startup Beyond, aims to democratise access to high-quality health information. “August has been developed by a team of engineers, data scientists, and doctors, and aims to bridge the gap between doctors and patients,” said Samarth Sharma, part of the founding team at Beyond, in an interview with AIM. “Most importantly, August has been trained on proprietary health data that Beyond has generated over the years.” The model scored 94.8% on USMLE, whereas, MedPaLM scored 85%, and GPT-4 87.8%.

August is multimodal, that accepts input in various formats such as audio, text and PDF Lab, and delivers the output in text format. It currently aids people in handling physical and mental health concerns and simplifying complex health procedures. The platform is currently free for use on Whatsapp.

“Whatsapp is already on 2 billion phones around the world and people are comfortable using it, so it made sense to bring August to users through this platform,” said Sharma. “We want August to be accessible to everyone, regardless of socioeconomic status and technical sophistication.” Though the company is working on creating an app for August, they believe that Whatsapp is a great way to access the platform, and their key focus is to enable natural conversation on Whatsapp.

Fusion of LLMs
August is built on a combination of LLMs from multiple providers. “Over the past year, we have experimented with, disassembled, and fine-tuned various large language models to arrive at the core engine behind August’s health AI,” said Sharma. “We’re currently using a combination of BERT, LlaMA, GPT-4, to name a few, and most recently have experimented with LLama-2.

August has completed 1500 health consultations over the last year to understand the best ways to guide a person, and even fine-tuned the model around those conversations. “Utilising a proprietary, highly-tuned version of ensemble refinement, August provides high-quality answers to health-related questions,” said Sharma, who has optimised this technique for improving token efficiency and minimising hallucinations.

Manoeuvring AI healthcare
It has been noted in the past where GPT-4 has inclined towards providing societal biases in clinical decisions. With the medical field being a critical use case, any misinformation can have grave consequences, and August AI is critically working towards tackling it.

“Understanding the health of people from different ethnicities is still something that the overall healthcare ecosystem is not good at, and that bias translates to LLMs as well,” said Sharma. The company has been focusing on India and has been collecting India-specific data that has been integrated to August AI.

“August does not do any diagnosis. We don’t think health AIs are there yet,” said Sharma. “August is highly tuned to answer only health-information questions. The ensemble refinement deals with hallucinations and we’re using retrieval augmented generation to ensure it is drawing information from reliable sources when someone asks for specific information like exercise or products”

A long path ahead
August AI has an elaborate roadmap where they wish to scale the product through direct outreach to users and partnerships with distribution platforms that can benefit from having August as part of their ecosystem. The company even wishes to integrate booking of doctor appointments in the future.

Sharma believes that August AI is much more than just medical competency which is Google MedPaLM’s focus. “We’re doing much better than Google MedPaLM on the critical elements of empathy, holding a conversation and proactively checking in on people. While Google is building for the US, August’s focus on India and its empathetic conversation will be key differentiators for us,” said Sharma. Analytics India