Version Hazard Log - MindBay (1.1)
11 Hazards, 18 Causes, 35 Controls
Name updated to MindBay
Approved Jul 25, 2025 by Paul Jewell
HAZ-001
3 Causes, 11 Controls
Incorrect and potentially harmful advice
The software's content provides inaccurate or harmful advice. The app’s AI-driven sessions may generate content that is misleading, outdated, or inappropriate.
1. Worsening of a patient’s mental health symptoms 2. Undermined trust in digital therapeutics and prescribing clinicians 3. Harm arises from incorrect guidance
CS-001
5 Controls
Software bugs causing erroneous processing of correct input data.
CTRL-001
Implement rigorous software testing procedures including unit, integration, and system testing to identify and fix bugs.
CTRL-002
Incorporate continuous integration and continuous deployment (CI/CD) pipelines to automate testing and ensure immediate feedback on the system's integrity.
CTRL-003
Conduct regular code reviews and audits to ensure the quality and accuracy of the software development.
CTRL-004
Use error detection and handling mechanisms to prevent the software from processing erroneous data without alerting the user.
CTRL-005
Provide regular updates and patches for the software to address newly discovered vulnerabilities and bugs.
CS-002
1 Controls
Outdated or incorrect signposting to crisis hotlines and NHS support.
CTRL-006
Implement a regular review and validation schedule for all crisis hotline and NHS support information within the system.
CS-003
5 Controls
Hallucinations, outdated training data, or bias in the language model.
CTRL-007
Establish a robust quality assurance process to routinely check the language model's outputs for accuracy and appropriateness.
CTRL-019
Clearly display warning message to users, explicitly stating where output is generated using AI.
CTRL-020
Onboarding for users outlining the use of AI in the product, and explaining that AI can make mistakes and outputs should be checked.
CTRL-027
Safeguard LLM layer which will screen the chatbot output to make sure it is appropriate and not harmful. If deemed harmful it will be replaced with a generic safety message
CTRL-028
Regular audit of a sample of conversations to ensure outputs continue to be appropriate
HAZ-002
3 Causes, 13 Controls
No detection of severe distress, self-harm intention and suicidal ideation
The software's safety protocol might not detect suicide or self-harm risks, which could prevent the safety feature from being triggered.
High-risk users not receiving immediate intervention Worsening mental health crisis without proper referral Liability concerns and ethical issues
CS-001
5 Controls
Software bugs causing erroneous processing of correct input data.
CTRL-001
Implement rigorous software testing procedures including unit, integration, and system testing to identify and fix bugs.
CTRL-002
Incorporate continuous integration and continuous deployment (CI/CD) pipelines to automate testing and ensure immediate feedback on the system's integrity.
CTRL-003
Conduct regular code reviews and audits to ensure the quality and accuracy of the software development.
CTRL-004
Use error detection and handling mechanisms to prevent the software from processing erroneous data without alerting the user.
CTRL-005
Provide regular updates and patches for the software to address newly discovered vulnerabilities and bugs.
CS-004
5 Controls
Subtle or metaphorical language around self-harm is not recognised.
CTRL-007
Establish a robust quality assurance process to routinely check the language model's outputs for accuracy and appropriateness.
CTRL-008
Regularly audit the safety protocol’s performance and verify its accuracy in detecting high‐risk cases.
CTRL-017
Self-Harm safety protocol trained with the use of an open source data set for suicidal ideation detection.
CTRL-018
Extensive manual testing with many varied scenarios to ensure the model adequately recognises risk of self-harm and suicidal ideation
CTRL-031
The LLM is supplied with a list of a potentially ambiguous phrases to help detect these and activate the protocol, such as "I'm going to end it all"
CS-012
3 Controls
Patient uses app who has moderate-severe or severe depression and / or anxiety (the intended use of the app is for patients with mild-moderate depression and / or anxiety)
CTRL-021
Clinician refers patients to the app via strict criteria, and using widely used PHQ-9 and GAD-7 scoring systems for depression and anxiety.
CTRL-022
Intended use statement clearly outlines patient cohort suitable for the application
CTRL-023
Clinicians receive training during onboarding where they are clearly told the patient cohorts that are appropriate for use of the application.
HAZ-003
3 Causes, 7 Controls
Safety feature activation without providing the adequate support
The software's safety protocol may be triggered but might not effectively direct patients to the appropriate helplines or trusted support resources, leaving them without the necessary guidance during a crisis.
High-risk users may not receive timely or sufficient intervention Delays in connecting patients to appropriate emergency or mental health services Erosion of patient trust in the app, potentially exacerbating mental health issues
CS-001
5 Controls
Software bugs causing erroneous processing of correct input data.
CTRL-001
Implement rigorous software testing procedures including unit, integration, and system testing to identify and fix bugs.
CTRL-002
Incorporate continuous integration and continuous deployment (CI/CD) pipelines to automate testing and ensure immediate feedback on the system's integrity.
CTRL-003
Conduct regular code reviews and audits to ensure the quality and accuracy of the software development.
CTRL-004
Use error detection and handling mechanisms to prevent the software from processing erroneous data without alerting the user.
CTRL-005
Provide regular updates and patches for the software to address newly discovered vulnerabilities and bugs.
CS-002
1 Controls
Outdated or incorrect signposting to crisis hotlines and NHS support.
CTRL-006
Implement a regular review and validation schedule for all crisis hotline and NHS support information within the system.
CS-005
2 Controls
Lack of localisation, resources aren’t tailored to the patient’s region.
CTRL-006
Implement a regular review and validation schedule for all crisis hotline and NHS support information within the system.
CTRL-009
Regularly audit the safety protocol’s performance and verify its accuracy in processing high‐risk cases.
HAZ-004
3 Causes, 4 Controls
Patient or clinicians overestimates app’s capability, viewing it as a direct substitute for current standard of care
Although the software is explicitly a supplement to standard NHS care, patients or clinicians may wrongly treat it as a standalone therapy, potentially delaying or foregoing essential professional interventions.
Escalation of mental health problems due to lack of professional oversight Delayed or missed clinical interventions Incomplete treatment outcomes
CS-006
3 Controls
Insufficient user education about the limits and proper use of the app.
CTRL-010
Develop comprehensive user manuals and FAQs that clearly outline the capabilities and limitations of the app.
CTRL-011
Implement pop-up reminders or notifications within the app that periodically remind users that the app is a supplementary tool and not a replacement for professional healthcare advice.
CTRL-019
Clearly display warning message to users, explicitly stating where output is generated using AI.
CS-007
2 Controls
Lack of adequate disclaimers about the app being a supplementary tool rather than a replacement for traditional care.
CTRL-010
Develop comprehensive user manuals and FAQs that clearly outline the capabilities and limitations of the app.
CTRL-011
Implement pop-up reminders or notifications within the app that periodically remind users that the app is a supplementary tool and not a replacement for professional healthcare advice.
CS-015
1 Controls
Over-reliance of the app
CTRL-030
App is not designed to be addictive. The user can complete a maximum of one session every 24 hours
HAZ-005
1 Causes, 1 Controls
Inaccessible language
Patient finds the app content inaccessible or hard to understand. Content, language complexity, reading level, or technical format may prevent patients from fully engaging with the CBT sessions.
Reduced therapeutic benefit Patient disengagement or dropout
CS-008
1 Controls
Assumption of a certain literacy level that not all patients may possess
CTRL-012
Regularly review and update patient education materials to ensure they are clear, concise, and accessible to individuals of all literacy levels.
HAZ-006
2 Causes, 1 Controls
Unauthorized local user access
Someone with physical access to a patient’s device could gain entry to the app without proper authorisation.
Confidential information being viewed by unintended parties Potential alteration or deletion of session data Distressed patients, potential for significant psychological trauma in some cases
CS-009
1 Controls
Patient leaves the app open or running in the background and walks away from the device.
CTRL-013
Use biometric authentication to re-verify the identity of the user when the app resumes from the background.
CS-010
1 Controls
Patient uses a communal or publicly accessible device (e.g., family tablet) without logging out.
CTRL-013
Use biometric authentication to re-verify the identity of the user when the app resumes from the background.
HAZ-007
3 Causes, 8 Controls
App unavailability or technical issue
System outages or technical issues may prevent the patient from continuing or reviewing their CBT sessions.
Disrupted therapeutic process, decreased app efficacy Increased risk of relapse or deterioration in mental state Frustration, leading to app abandonment
CS-001
5 Controls
Software bugs causing erroneous processing of correct input data.
CTRL-001
Implement rigorous software testing procedures including unit, integration, and system testing to identify and fix bugs.
CTRL-002
Incorporate continuous integration and continuous deployment (CI/CD) pipelines to automate testing and ensure immediate feedback on the system's integrity.
CTRL-003
Conduct regular code reviews and audits to ensure the quality and accuracy of the software development.
CTRL-004
Use error detection and handling mechanisms to prevent the software from processing erroneous data without alerting the user.
CTRL-005
Provide regular updates and patches for the software to address newly discovered vulnerabilities and bugs.
CS-017
1 Controls
Cloud outage
CTRL-033
SLA with Azure ensuring >99.9% service uptime
CS-018
2 Controls
Mobile software version not compatible with app resulting in unavailability or downtime
CTRL-034
Testing is performed on a variety of mobile devices and software versions
CTRL-035
Incident reporting system available in-app so users can report technical issues
HAZ-008
1 Causes, 4 Controls
Cyberattack or data breach compromises patient information
Unauthorised external access via hacking, phishing, or vulnerabilities in the LLM system or surrounding infrastructure.
Exposure of sensitive patient data Distressed patients
CS-011
4 Controls
Inadequate cybersecurity measures and firewall protection within the software.
CTRL-005
Provide regular updates and patches for the software to address newly discovered vulnerabilities and bugs.
CTRL-014
Implement robust encryption protocols for data storage and transmission to protect patient information.
CTRL-015
Establish comprehensive firewall configurations and regular updates to prevent unauthorized access.
CTRL-016
Provide cybersecurity training for all employees to recognize and respond to security threats effectively.
HAZ-009
3 Causes, 9 Controls
Incorrect helpline signposting
Failure to display the correct national helpline information or misdirecting distressed patients to helplines for the wrong region (e.g., the wrong country)
Delayed or absent connection to appropriate emergency support Increased patient distress due to ineffective crisis intervention Reduced trust in the app’s reliability and overall service
CS-001
5 Controls
Software bugs causing erroneous processing of correct input data.
CTRL-001
Implement rigorous software testing procedures including unit, integration, and system testing to identify and fix bugs.
CTRL-002
Incorporate continuous integration and continuous deployment (CI/CD) pipelines to automate testing and ensure immediate feedback on the system's integrity.
CTRL-003
Conduct regular code reviews and audits to ensure the quality and accuracy of the software development.
CTRL-004
Use error detection and handling mechanisms to prevent the software from processing erroneous data without alerting the user.
CTRL-005
Provide regular updates and patches for the software to address newly discovered vulnerabilities and bugs.
CS-005
2 Controls
Lack of localisation, resources aren’t tailored to the patient’s region.
CTRL-006
Implement a regular review and validation schedule for all crisis hotline and NHS support information within the system.
CTRL-009
Regularly audit the safety protocol’s performance and verify its accuracy in processing high‐risk cases.
CS-007
2 Controls
Lack of adequate disclaimers about the app being a supplementary tool rather than a replacement for traditional care.
CTRL-010
Develop comprehensive user manuals and FAQs that clearly outline the capabilities and limitations of the app.
CTRL-011
Implement pop-up reminders or notifications within the app that periodically remind users that the app is a supplementary tool and not a replacement for professional healthcare advice.
HAZ-010
2 Causes, 4 Controls
Forseeable misuse
A user intentionally or unintentionally misusing the product and asking inappropriate questions
Risk of unpredictable answers and harmful advice
CS-013
4 Controls
Intentionally attempting to misuse the chatbot to provide answers outside of intended use
CTRL-024
Safeguard against inappropriate user responses - will filter these out and respond with a safety message referring to ToU and a safety phone number
CTRL-025
Implementation of FAQs library that the LLM can utilise to provide pre-approved answers to difficult questions
CTRL-026
Clear disclaimer and instruction during onboarding about what is appropriate use that user has to acknowledge
CTRL-029
All user responses that are flagged as inappropriate by the LLM safeguard layer will be audited and reviewed
CS-014
3 Controls
Unintentionally misusing the chatbot through not understanding how it works
CTRL-024
Safeguard against inappropriate user responses - will filter these out and respond with a safety message referring to ToU and a safety phone number
CTRL-025
Implementation of FAQs library that the LLM can utilise to provide pre-approved answers to difficult questions
CTRL-026
Clear disclaimer and instruction during onboarding about what is appropriate use that user has to acknowledge
HAZ-011
1 Causes, 1 Controls
Poor engagement with the app
Poor engagement resulting in deterioration in mental health
Deterioration in mental health condition
CS-016
1 Controls
Does not initially see the value in the app, or does not make app use a habitual behaviour
CTRL-032
Targeted and personalised push and/or email notifications are utilised to try and increase engagement and bring users back to app when they could potentially benefit