Building an AI-Powered Medical Record Summarization Platform With 88% Accuracy

At the crossroads of legal and medical documentation, manual organization and extraction of information from vast medical records stood as a common barrier hindering efficiency and productivity.

Traditional approaches, like outsourcing or relying on an in-house team, proved time-consuming, expensive, and prone to inaccuracies. This process often stretched over days and incurred substantial costs, particularly for more extensive cases.

The client, with a background as an expert injury witness across various domains such as sports, workplace, motor vehicle, and aviation accidents, recognized the difficulty faced by medical and legal professionals when reviewing large volumes of medical records.

This necessitated a transition from the traditional and laborious method to a far more scalable approach, achieved by harnessing AI for the comprehensive analysis of extensive medical records.

The client initially collaborated with a Ukraine-based software company to bring their vision to life. However, after a dedicated three-year endeavor, they encountered challenges getting the product to market. Notably, limitations in the technology stack, such as the initial use of an older version of Kotlin for the backend, necessitated a refactoring into .NET to improve efficiency and scalability.

Identifying these gaps, the client sought a proficient software development company specializing in artificial intelligence with substantial experience in the medico-legal domain to launch their product successfully.

Delving deep into the needs of the industry, we understood that our goal was to develop a pioneering AI-driven medical record summarization tool that could organize, categorize, and summarize thousands of diverse medical documents with high accuracy and speed.

Furthermore, considering that the client had already made advancements in the product development process, our role involved thoroughly analyzing the existing codebase. Our task was to identify gaps or shortcomings and strategically modify the system to launch a scalable and feature-rich Minimum Viable Product (MVP) as soon as possible.

Here’s a detailed roadmap of how our Maruti Techlabs team executed this project:

Phase 0: Three-week Audit Workshop

Collaborating closely with the client's core team, we conducted a detailed three-week audit. This workshop involved key stakeholders, including our Chief Technical Architect, VP of Data Science and Technology, Head of DevOps, and Product Development Leads.

The audit thoroughly examined the project's frontend, backend, and data service aspects, involving a detailed analysis of source code, architecture, cloud infrastructure, documentation, and processes. The four core standards that were a key part of this code audit were - maintainability, reliability, efficiency, and portability.

The primary goal was to deliver a comprehensive review, precisely defining our work scope. We outlined preferred methodologies and anticipated collaborative outcomes, ensuring transparency and alignment of expectations.

Here’s a high-level brief outlining what the audit encompassed -

• Examination of the current technology stack 
• Software architecture assessment 
• Code quality check 
• Review of potential maintenance issues 
• Technical documentation assessment 
• Recommendations

Key insights from our Audit underscored several critical aspects:

• Data Science Model Performance: The performance and accuracy levels of data science models were concerning. The system took about 46 minutes to process 100 pages. 
• Adherence to Software Development Practices: Consistency in following standard software development practices was lacking. 
• Outdated Technology Stack: Performance and scalability issues were traced back to an outdated technology stack.
• Documentation: There was a lack of comprehensive documentation across the entire product (frontend, backend, Data Science). API versioning was also missing, which could lead to integration challenges in the future.
• Unit Test Cases: The absence of unit test cases highlighted the scope of improvement in early issue detection, code quality elevation, and streamlined QA efforts.
• Code Clean up: The need for clean-up of redundant, dead, or unused code surfaced, aiming to enhance code maintainability and overall system efficiency.

These insights were a foundation for our strategic approach to building the tool.

Phase 1 - Developing the AI Models

Adhering to the client's demand for a versatile medical record summarization tool, we structured the project into a comprehensive suite of multiple AI models. We designed the AI models based on 4 stages of the workflow - 

Stage 1 - Data Collection and Annotation

Stage 2 - Document Classification

Stage 3 - Data Extraction

Stage 4 - Summarization

Here's an in-depth overview of the various stages of the workflow embedded in our AI medical record summarization tool:

• Stage 1 - Data Collection & Annotation

Document collection lies at the heart of a medical summarization tool. The diverse documents users submit are tagged, labeled, and annotated at this stage. This helps transform intricate medical records into well-organized sets, facilitating enhanced data accessibility.

We collaborated with the subject matter experts from the client’s team to define a tailored annotation protocol for the documents ranging from admission sheets and discharge summaries to physical therapy reports. The protocol plays a crucial role in labeling and organizing the data within the documents.

If the user submits a new document type not part of this predefined list, a new annotation protocol is tailored for that type. This adaptability ensures the tool can effectively handle various medical document types.

Several medical documents were annotated in the thousands and formed the basis of training data for the tool. To meet the client's scalability and performance requirements, our team adopted node optimization, incorporating dynamic configuration decisions based on document size to enhance resource allocation and system efficiency.

The implementation of automatic image pulling on each node significantly reduced the overall wait time for document analysis and processing from 20 minutes to a mere 6 minutes.

• Stage 2 - Document Classification

Within a medical summarization tool, data classification plays a critical role in processing. It helps systematically categorize medical documents into specific types, such as admission sheets, laboratory reports, or discharge summaries.

In this model, the following categorizations occur –

• Document Type

We incorporated AWS Textract, XGBoost, and Bio-clinical Bert algorithms for document text embedding generation and classification. Bio-clinical Bert is a specialized language representation model for biomedical and clinical text. Pre-trained on extensive biomedical and clinical datasets, this model is further coupled with XGBoost to identify and process texts associated with medical documents.

The models extract the relevant elements and labels of the annotated documents and classify them as medical or non-medical. If the document type is non-medical, it is not processed. If the document type is marked as medical, it is tallied against the original list of document types and further processed.

• Handwritten Text

Handwritten clinical notes are vital for a patient's medical history and treatment journey.

We tailored the Yolov5 algorithm to decipher the contents of each page. If more than 40% of the content on the page is handwritten, then the page is not considered for processing.

However, if the handwritten content is <40%, the algorithm extracts relevant elements from the page. It was further configured to ensure that signatures and overlay comments are not considered.

• Checkbox Mark Detection

The checkbox mark detection model is crucial in scanning medical forms to identify critical information such as the patient's obesity status, diabetic condition, pregnancy status, and pre-existing diseases.

The Yolov5 algorithm was further leveraged here to automatically recognize and interpret the marked checkboxes on forms. This model helped rapidly and accurately retrieve essential information, providing a more comprehensive understanding of the patient's health profile.

• Stage 3 - Data Extraction

Considering the end goal of generating medical summaries and outlining events chronologically, extracting critical medical information from patients' healthcare records is fundamental for generating insightful medical summaries. These summaries must encompass comprehensive details related to diagnoses, prognoses, treatments, medical history, and more.

Leveraging Textract’s OCR capabilities facilitated the extraction of information, while Bio-Clinical Bert played a crucial role in highlighting critical insights within the extracted texts. We further utilized DistillBERT and LightGBM with TinyBert embedding for named entity recognition (patient name, doctor name, date, etc) and data classification.

In response to the client's emphasis on cost optimization and scalability, our data scientists implemented a caching mechanism in Textract. Furthermore, to enhance efficiency, the Textract Async API was employed to handle multiple requests concurrently, thereby reducing the API hit rate of paid services. This strategy met the client's financial objectives and streamlined the process for diverse data extraction needs.

We leveraged a combination of AWS Comprehend Medical and ChatGPT for entity recognition and mapping connections between symptoms, diagnoses, treatments, and outcomes. This enabled a thorough extraction and relation mapping between different entities.

AWS Comprehend Medical is a natural language processing (NLP) service that Amazon Web Services (AWS) provides. It is designed to analyze and understand natural language text using machine learning algorithms.

AWS Comprehend Medical is primarily responsible for understanding medical entities like symptoms, diagnoses, treatments, and outcomes. The output from Comprehend is then passed onto ChatGPT for further refinement and extraction of ICD (International Classification of Diseases) / CPT (Current Procedural Terminology) codes. ICD codes are used for diagnosing and classifying health conditions, while CPT codes are used for describing medical procedures and services.

• Stage 4 - Chronological Order & Summarization

The final output is the chronological indexing with a structured timeline, facilitating efficient treatment progress tracking, pattern identification, and trend recognition.

The portal presents the summary through four distinct tabs, each encapsulating a unique medical record summary.

1. Chronology and Timeline Summary
   The final summary provides a detailed timeline of a patient's medical history. This comprehensive overview helps us understand the progression of illnesses, treatments, and significant medical events over time. 
2. Treatment Summaries
   Treatment summaries offer a condensed overview of the therapeutic interventions a patient has undergone. It highlights critical medical procedures, medications, and their outcomes.
3. Diagnostic Summaries
   Diagnostic summaries briefly summarize a patient's diagnostic procedures, test results, and medical assessments. It offers a comprehensive snapshot of a patient’s health status.
4. Prognosis Summaries
   Prognosis summaries encapsulate a patient's anticipated health outcome based on their medical condition, treatment responses, and potential developments.

Each summary includes hyperlinks to the source documents, simplifying the retrieval of specific details and enabling the cross-checking of information. These links establish connections between related pieces of information, providing users with a comprehensive and holistic view of the patient's health.

Through these models, the medical record summarization tool successfully condenses extensive medical records into concise and informative summaries while retaining critical details.

Phase 2: Custom Software Development

With precise client requirements and a detailed code audit, we established the architectural design and technology stack. We opted to break the architecture into microservices, not just for scalability and performance but the monolithic architecture was difficult to maintain, and it was challenging to add new features or fix the bugs. Any code change can have ripple effects. Moreover, the code is not validated by unit tests which makes it worse and more vulnerable to performing any changes. This was complemented by AWS hosting architecture featuring Kubernetes for robust infrastructure and streamlined deployment.

• Backend

Recognizing that the existing backend, initially built in Kotlin, was not leveraging the latest technology stack, we initiated a strategic refactoring process, transitioning to .NET Core. This shift provided an avenue to integrate modern technologies, frameworks, and tools, ensuring alignment with contemporary development standards. Parallely, the Data Science stack included Python, AsyncIO, PyTorch, XGBoost, YOLOv5, TinyBERT, Bio_ClinicalBERT, Pandas, Numpy and Boto3.

• Frontend

For frontend development, we selected React along with a blend of TypeScript and Flow, leveraging additional technologies such as Redux, Redux-Form, CSS-in-JS, Webpack, Yup, Vite.js, and SCSS. Recognizing the data-intensive nature of the project, we opted for Redis, ElasticSearch, and PostgreSQL as the foundation for our database.

• Team Structure 

With these technical requisites at the forefront, we meticulously assembled a skilled team to spearhead the project to success. Our process adhered to three core principles:

• Past Collaboration: Developers who have worked together in the past foster a smoother team dynamic.
• Skills Match: Developers with precise skills matching the technical prerequisites needed to accomplish the project.
• Diverse Expertise: Team with varying experience levels, offering a well-rounded blend of senior and junior talent for a comprehensive perspective.

For the smooth execution of this project, we assembled a team of backend, frontend, QA, DevOps, BI, and Data Scientists. This team operated under the guidance of a technical project manager and technical lead, ensuring a well-coordinated and efficient workflow across the project lifecycle.

Phase 3: Portal Development

Leveraging the design wireframes as a foundation, our frontend team meticulously crafted a sophisticated portal. The portal enabled end-users to effortlessly -

• Upload extensive medical records,
• Streamline payment processing,
• Access patient information, and
• Receive intricately extracted medical summaries.

We introduced multiple purchase packages tailored to the document's page count to enhance user flexibility. This feature was seamlessly integrated with Stripe, ensuring a smooth subscription process.

Following successful payment transactions, the portal efficiently guides end-users through uploading medical records in PDF format. These documents are then processed by a stream of AI models to extract, categorize, organize, and summarize medical records.

The resulting summary is thoughtfully organized into sections—meticulously detailing medical history, diagnosis, prognosis, treatments, and handwritten notes chronologically. This structuring ensures a clear and organized presentation of the patient's journey. Users are offered the convenience of downloading the summaries in any predefined format, thus providing a user-centric experience.

Phase 4:  Product Support

With the product being launched to a broader audience, it is critical to ensure that the right support mechanisms are in place.

We put together a product support team that works in a close capacity with the account manager on the client side. This strategic alignment ensures a tailored and continuous support experience, enhancing overall end-user satisfaction, gathering necessary feedback, and optimizing the product's functionality.

This encompassed troubleshooting, guidance on tool usage, and prompt issue resolution, all delivered through an iterative approach that accommodated the evolving needs of end-users.

Understanding the Mechanism of Product Support

We used Jira to manage product support, with any issues logged via a customer-facing portal. Customers, as well as the client, can log a support ticket requesting bug fixes or improvements in the portal.

Here is a detailed overview of the product support mechanism -

1. Support Ticket
   The customer(s) or the account manager on the client side can log a support ticket reporting an issue. Our product support team takes on the tickets based on the severity level and pre-defined SLAs.
2. Diagnosis of the Issue
   Our Product Support Agent works on diagnosing the issue and referring to solutions from the existing knowledge base. The agent parallely communicates with the customer or account manager for more information to better understand the challenges and associated root cause(s).
   After this diagnosis, the agent precisely classifies the support ticket into an improvement or a bug.
3. Addressing an Improvement
   If the support ticket diagnosis identifies an improvement, then the product support agent will pass on the relevant details to the Product Owner and the Technical Project Manager for further review. They would plan a user story for the same and incorporate it within the product roadmap.
4. Addressing a Bug
   If the support ticket diagnosis identifies a bug, the product support agent would pass on the relevant details to the QA team, who would, in turn, file a bug, followed by the Product Owner prioritizing it within the sprint backlog.

Once the bug/story is completed, the ticket is closed, and the customer(s) or account manager on the client side will be notified of the same.

Thus, with a strategic focus on product support, Maruti Techlabs actively nurtured a positive user experience and played a pivotal role in sustaining the project's success.

Phase 5 - HIPAA Compliance

To instill trust and foster widespread adoption, a product must rigorously adhere to the stringent regulations established by the Health Insurance Portability and Accountability Act (HIPAA).

Our team collaborated with an external audit agency to meticulously formulate a concise HIPAA compliance implementation plan. This strategic initiative aimed to navigate the intricate landscape of healthcare regulations. The external agency played a crucial role in identifying potential system vulnerabilities and areas for enhancement.

Subsequently, a comprehensive checklist was developed, encompassing tasks, policies, and procedures. Our team implemented integrated security measures, including data encryption, backup protocols, and role-based access control. Additionally, efforts were directed toward educating employees and team members on HIPAA, underscoring privacy and security as core priorities.

To fortify our commitment to compliance, Business Associate Agreements (BAAs) were executed with all third-party vendors and partners. Detailed records covered compliance efforts, training initiatives, and assessments, ensuring a comprehensive approach.

Upon completing the Minimum Viable Product (MVP) for the Beta launch, the platform underwent a rigorous audit and continuous monitoring by an external agency. This proactive approach to control monitoring ensures robust compliance, providing full visibility into the compliance status at all times.