So scientists tested

@parthknowsai

98.4K followers · usually 23.5K views a post

Views

84.4K

Likes

5.6K

Comments

206

Shares

1.3K

Selling · SaaS › AI Tool

OpenEvidenceseen in 3 videos →

Why it worked

The video effectively uses a scientific study to debunk the common assumption that general AI models are superior, positioning specialized medical AI tools as more effective. This creates a compelling narrative that challenges existing beliefs and highlights the value of specific, domain-focused AI solutions.

Summary

This video discusses a study comparing general AI models to specialized medical AI tools. The study found that specialized tools like OpenEvidence and UpToDate Expert AI outperformed general models like GPT-4.6, Claude, and Gemini on medical benchmarks. The paper highlights the need for independent testing of AI tools before they are used in clinical settings.

Structure

1Introduction to the study comparing AI models
2Mention of specialized medical AI tools (OpenEvidence, UpToDate)
3Comparison with general AI models (GPT, Claude, Gemini)
4Presentation of study results showing specialized tools outperform general ones
5Explanation of why specialized tools are better (RAG, medical databases)
6

Original caption

General AI tools beat expensive domain specific once #ai #Tech #chatgpt #techtok #LearnOnTikTok

More from @parthknowsai

Their other posts in the index, biggest breakout first.

Conclusion on the importance of independent testing

Product placement

Nature Medicine: This is a scientific journal where the study was published. It merely appears as a watermark/logo on the article. Removing it would not change what happens in the video. OpenEvidence: This is an AI tool designed for doctors to answer medical questions. It is shown being used and discussed. Removing it would change what happens in the video. UpToDate Expert AI: This is an AI tool for clinical care, built by Wolters Kluwer. It is shown on a phone screen and discussed. Removing it would change what happens in the video. GPT-4.6: This is a large language model. It is mentioned as a benchmark for comparison. Removing it would not change what happens in the video. Gemini 3.1: This is a large language model. It is mentioned as a benchmark for comparison. Removing it would not change what happens in the video. Claude Opus 4.6: This is a large language model. It is mentioned as a benchmark for comparison. Removing it would not change what happens in the video. Wolters Kluwer: This is the company that developed UpToDate Expert AI. It appears as a logo on the screen. Removing it would not change what happens in the video.

Call to action

Follow and I'll keep you posted.

On-screen text

so scientists tested so scientists tested a 3.5 billion DOLLAR MEDICA AGAINST REGULAR CHAT GPT. AND IT LOST ON EVERY SINGLE ONE. SO THERE'S THIS CALLED OPEN EVIDENCE. IT'S BUILT SPECIFICALLY LIKE, THAT'S THE WHOLE THING. IT'S DESIGNED FOR IT JUST RAISED $210 MILLION AND HOSPITALS ARE ACTUALLY BUYING AND USING IT. THERE'S ALSO ANOTHER UPTODATE. DOCTORS HAVE BEEN USING IT FOR YEARS. REASON THESE EXIST ASSUMPTION IS A SPECIALISED MEDICAL AI. IS GONNA BE THAN JUST USING REGULAR CHAT G PT. SENSE. ACTUALLY TESTED WHETHER AND THAT'S WHERE COMES IN. NYU AND THEY RAN AND I THINK MATTERS THE MOST HERE BECAUSE THEY GRABBED 100 QUESTIONS THAT ACTUAL DOCTORS TYPED INTO AI WHILE TREATING PATIENTS. SO NYU RAN THEM THROUGH THE MEDICAL TOOLS LIKE OPEN EVIDENCE AND UP TO DATE, AND ALSO THROUGH CLAUDE AND GEMINI. AND ONCE THEY THEY HAD 12 EVERY SINGLE RANKED AI PT. CLAUDE AND GEMINI ABOVE THE MEDICAL SPECIFIC ONES. THESE MEDICAL TOOLS ARE BUILT TO PULL FROM MEDICAL DATABASES TO ANSWER QUESTIONS. THEY USE RAG, SHOWS THAT CAN MAKE ANSWERS WORSE WHEN THE WRONG INFORMATION GETS PULLED IN VERSUS FRONTIER A THE ANSWERS BAKED INTO ITS TRAINING DATA FROM THE START. AND THE REASON WHY THIS MATTERS TOOLS ARE ALREADY IN HOSPITALS AS WE SPEAK. DOCTORS ARE MAKING REAL DECISIONS WITH THEM. THE WHOLE POINT OF THIS PAPER IS THAT NOBODY INDEPENDENTLY TESTED ANY BEFORE IT GOT THERE. NO ONE 700 DOLLAR MEDICAL IS ACTUALLY BETTER ONE BEFORE USING IT FOR PATIENTS. THIS PAPER IS BASICALLY SAYING WE NEED TO STOP ASSUMING SPECIALISED MEANS BETTER. BECAUSE ALL OF THESE AI. TOOLS HAVE REALLY GOOD MARKETING PITCHES UNTIL YOU ACTUALLY GO AND TEST THEM. FOLLOW AND I'LL KEEP YOU POSTED.

Transcript

So scientists tested So scientists tested a 3.5 billion DOLLAR MEDICA AGAINST REGULAR CHAT GPT. AND IT LOST ON EVERY SINGLE ONE. SO THERE'S THIS CALLED OPEN EVIDENCE. IT'S BUILT SPECIFICALLY LIKE, THAT'S THE WHOLE THING. IT'S DESIGNED FOR IT JUST RAISED $210 MILLION AND HOSPITALS ARE ACTUALLY BUYING AND USING IT. THERE'S ALSO ANOTHER UPTODATE. DOCTORS HAVE BEEN USING IT FOR YEARS. REASON THESE EXIST ASSUMPTION IS A SPECIALISED MEDICAL AI. IS GONNA BE THAN JUST USING REGULAR CHAT G PT. SENSE. ACTUALLY TESTED WHETHER AND THAT'S WHERE COMES IN. NYU AND THEY RAN AND I THINK MATTERS THE MOST HERE BECAUSE THEY GRABBED 100 QUESTIONS THAT ACTUAL DOCTORS TYPED INTO AI WHILE TREATING PATIENTS. SO NYU RAN THEM THROUGH THE MEDICAL TOOLS LIKE OPEN EVIDENCE AND UP TO DATE, AND ALSO THROUGH CLAUDE AND GEMINI. AND ONCE THEY THEY HAD 12 EVERY SINGLE RANKED AI PT. CLAUDE AND GEMINI ABOVE THE MEDICAL SPECIFIC ONES. THESE MEDICAL TOOLS ARE BUILT TO PULL FROM MEDICAL DATABASES TO ANSWER QUESTIONS. THEY USE RAG, SHOWS THAT CAN MAKE ANSWERS WORSE WHEN THE WRONG INFORMATION GETS PULLED IN VERSUS FRONTIER A THE ANSWERS BAKED INTO ITS TRAINING DATA FROM THE START. AND THE REASON WHY THIS MATTERS TOOLS ARE ALREADY IN HOSPITALS AS WE SPEAK. DOCTORS ARE MAKING REAL DECISIONS WITH THEM. THE WHOLE POINT OF THIS PAPER IS THAT NOBODY INDEPENDENTLY TESTED ANY BEFORE IT GOT THERE. NO ONE 700 DOLLAR MEDICAL IS ACTUALLY BETTER ONE BEFORE USING IT FOR PATIENTS. THIS PAPER IS BASICALLY SAYING WE NEED TO STOP ASSUMING SPECIALISED MEANS BETTER. BECAUSE ALL OF THESE AI. TOOLS HAVE REALLY GOOD MARKETING PITCHES UNTIL YOU ACTUALLY GO AND TEST THEM. FOLLOW AND I'LL KEEP YOU POSTED.

What is going on at meta?? #ai #Tech #LearnOnTikTok #fyp #meta

4.4