In the Diagnose-a-thon, participants will explore the ways AI can help or harm when it's used to answer medical questions. This event is all about testing how well these GenAI tools can handle health-related information and give advice to people who might use these tools with respect to their personal health. At a high level, the Diagnose-a-thon will require participants to prompt large language models (LLMs) with medical queries and check whether responses are either accurate or contain inaccurate, misleading, and/or potentially harmful content. Through the competition entries, we hope that participants can see both the potential benefits and risks of using AI in healthcare-related queries. 

The Diagnose-a-thon will be held virtually from Monday, November 11, to Sunday, November 17. Diagnoses can be submitted via an online form.

Any member of the Penn State community with an active @psu.edu email address can participate in the challenge.

Note: You must log in to the submission form using your Penn State access ID (e.g., abc123). You cannot authenticate using an alias email address.

To participate in the challenge, follow the four steps below and watch this short video for guidance on how to make your entry shine. Note that participants can submit as many diagnoses as they'd like to any of the tracks, which increases your chances of winning prizes (see the section below on prizes for more information).

Diagnose-a-thon Submission Form

1. Select Your LLM

Select one of these three LLMs to test your medical queries:

• ChatGPT 3.5
• Groq
• Gemini 1.5

You are free to choose other LLMs, in which you can select "Other" as the LLM type in the Diagnose-a-thon submission form.

2. Select Your Track

• Patient Track: Step into the shoes of a potential patient who might use a large language model (LLM) to get medical advice. In this track, use an LLM to get a diagnosis for real or imaginary symptoms.
• Medical Professional Track: Imagine you’re a doctor, nurse, or other medical professional who might use LLMs as a medical case assistant. In this track, use an LLM to receive a diagnosis based on a hypothetical patient case.
• Out-of-the-Box Track: Get creative! Think of a scenario not included in the Patient or Medical Professional Tracks that might lead to a potentially believable medical diagnosis from an LLM. For example, you may want to see whether an LLM can tell you about serious side effects of a medication that was recently prescribed, or you are interested in getting a second opinion on a diagnosis from an LLM.

3. Validate Your Findings

Find at least one trusted online source, such as academic or magazine articles, WebMD, or Wikipedia, to show that your AI-generated diagnosis is either accurate or misleading and harmful. You can submit both verifiably correct and incorrect diagnoses!

4. Submit Your Entries

Submit your entries using the submission form below. All entries should include:

• a screenshot of your LLM prompt
• the LLM-generated response or diagnosis
• the LLM tool and version used to generate your diagnosis
• a Word document that describes the online sources which validate or disprove the diagnosis you received (please include clickable links to those sources)
• if necessary, an explanation of why the diagnosis is particularly noteworthy, either in terms of being super-smart in diagnosing something or in terms of being dangerously misleading or potentially harmful to the user.

Once the competition concludes, participants will be asked to complete a five-minute survey about the experience. This survey is required of all participants who wish to be considered for the awards.

Participation in the Diagnose-a-thon is voluntary and can be ended at any time.

A panel of physicians will scrutinize each submission. The top three participants with the highest number of verifiable AI-generated diagnoses (both accurate and misleading) will win cash prizes:

• 1st place = $1,000
• 2nd place = $500
• 3rd place = $250
• Five participants will also earn $50 consolation prizes.

The participant who submits the AI-generated diagnosis that would be most harmful if it was acted upon will also win $1,000.

Your involvement enables you to be a part of a growing movement led by CSRAI that champions socially responsible AI. Through this challenge and engagement with CSRAI, you can collaborate with Penn State's brightest minds in AI and ethics, while also shedding light on the emergent strengths and worrisome shortcomings of existing AI tools, the opportunities they provide, and the challenges they can create. Ultimately, you will help pave the way for an AI future that is helpful rather than harmful to society.

If you have questions or concerns about the research, please email Bonam Mingole at bjm6940@psu.edu. You can also join the Diagnose-a-thon Microsoft Teams channel to ask questions during the event.

If you have questions regarding your rights as a research subject or concerns about your privacy, please contact the Human Research Protection Program at (814) 865-1775.

Participation is voluntary and can be ended at any time.