Why we need a global standard for self-assessment tools

18 August 2020 in Tech

By Matteo Berlucchi, CEO, Your.MD

Sometimes companies in the same space as you miss the point.

The latest foray from a company in the consumer digital healtech space is a peer-reviewed article in a respected scientific journal. Nothing wrong with that.

Except it was spun into a media story suggesting the company was the first to use a particular technique (not true) and as a result, can perform better than a doctor at diagnosing online users

That is the spring from which my irritation flows — the pushing of a trope that instead of improving, ends up damaging the sector. You see, self-assessment tools can’t diagnose an online user and are not nor likely to in my lifetime.

Ever since I co-founded Your.MD with Henrik Pettersen, our chairman, I’ve been obsessed with one question: “What would make the public trust an online personal health assistant?”

You trust your doctor because they are the gold standard of care.

It wasn’t always that way. When doctors bled people, tried to cure kings and queens with mercury and generally did more harm than good, they were called quacks for a reason.

So what’s the difference? How did western medicine attain such a trusted position from such an unpromising platform?

In part, western physicians just got better after hundreds of years of trial and error. That’s not a luxury AI self-assessment tools are going to be afforded. The biggest rise in trust came with the concepts of evidence-based medicine, independent audits, inspection and ever-more predictable outcomes.

All companies involved in AI self-assessment services are driven by a desire to provide people with better support. AI can transform the world of health in many ways. However, we must stop trying to prove AI is better than a doctor at diagnosing. This is dangerous.

Diagnosing a patient in primary care is a non-binary exercise and skill that requires extensive trial and error strategies, supported by examination and tests.

This is a critical safety issue. If you’re seeing blood when you cough, the role of a self-assessment app is to tell you to seek treatment immediately, not to try and guess the cause of it! If you’re a male over 50 with a history of smoking, and you’ve been coughing for more than a few weeks, you should get checked for lung cancer, even if the AI calculates the chances of this outcome as low.

Triage is the sweet spot for AI symptom checkers, in other words, rating how serious a symptom looks for that person in the light of other, aggregated factors. This is different from diagnosis, and Healthily, the app developed by Your.MD, believes trying to prove, or giving the public the impression, that AI is better than a doctor at diagnosis is misplaced, counterproductive and potentially unsafe.

Having an AI-driven self-assessment tool in your pocket is incredibly powerful. However, the power of this technology to do good, and even save lives, is undermined if we make unfounded claims. Every one of us has the right to trust and understand our healthcare, whether it’s delivered by a human or technology. When we look at where health has often been in the past — rarified and hard for non-professionals to understand — sometimes transparency is the real breakthrough.

New regulatory standards in the UK and USA will lead to Healthily and other companies in this space being subject to audit but that still leaves the industry to catch up in the fields of evidence base and inspection.

Healthily has limited resources but is heavily investing in supporting:

  1. Inspection — The independent Professional Records Standards Body to create an inspection framework.
  2. Evidence — Imperial College’s Self-Care Academic Research Unit and the Royal College of GPs to create a Gold Standard for vignette testing.
  3. Standard — The World Health Organization and the International Telecommunication Union to create a global standard for benchmarking self-assessment tools.
  4. Strategy— An invitation to the world’s leading universities to design a real-world evidence study methodology.

Point 4 is the final part of the ‘trust” jigsaw — an evidence base that can’t be disputed and gives people confidence that the words in your palm can be relied upon as much as the words spoken in the consultation room.

Healthily was the first to enter the self-assessment tool market in 2015. We’ve conducted vignette tests like our competitors but never previously published them because we understood the inherent flaws and didn’t want to make false claims.

From our latest work using vignettes with Imperial College (which will be published later in the year), we know 90% of the time we give safe advice on an individual’s best next step to take.

That’s useful for an internal benchmark of progress, but because we’ve been conducting vignette testing since 2015 we also know they don’t represent what happens when real people use these innovative tools.

The answer to my original question: “What would make someone trust us?”, is unimpeachable evidence to back up grounded claims.

Important: Our website provides useful information but is not a substitute for medical advice. You should always seek the advice of your doctor when making decisions about your health.

The self-care app
Check your health, manage 100s of conditions, track your progress and much more. Download today.