Data Annotation

Team Preparation:​ Assembled a team with strong writing skills and experience in generating concise, mobile-friendly content.​

Guidelines and Evaluation:​ Developed comprehensive guidelines to ensure responses were accurate, unbiased, and sensitive to user context.​ Implemented a detailed evaluation process to continuously assess and improve the quality of responses.​

Training and Feedback:​ Conducted extensive training sessions to familiarize the team with the task requirements and the AI’s capabilities.​ Provided regular feedback to refine the team’s approach and ensure adherence to guidelines.​

Contextual Integration:​ Implemented strategies to effectively utilize chat history and other contextual information to enhance response relevance.​

Comprehensive Regulatory Content Repository: Innodata built a vast repository of regulatory documents obtained through automated scraping techniques. This repository ensured access to the latest and most relevant information. 

Machine Learning for Categorization: Advanced machine learning algorithms were used to categorize the scraped documents based on pre-defined metadata (e.g., jurisdiction, document type). This facilitated efficient information retrieval. 

OpenAI for Intelligent Summaries: OpenAI’s capabilities were harnessed to generate daily or weekly summaries of the categorized regulatory documents. Users could customize their subscriptions based on specific needs (regions, document types) to receive the most relevant information. 

Natural Language Processing (NLP): Leveraging NLP, the system could interpret complex legal language, providing clear and concise summaries in a user-friendly format. 

Real-Time Updates: The system continuously monitored regulatory sources for updates, ensuring users were always informed about the latest changes. 

Source Access: Users could easily access the original PDF source document for any summarized point, facilitating deeper dives when needed. 

Abstract Creation from German Court Cases: Using Generative AI to automate the generation of case summaries. 

Keyword Extraction and Taxonomy Matching for Dutch Labor Law Books: Employing AI to identify relevant keywords and match them to an established taxonomy.

To address the challenge, our approach involved thorough training and meticulous preparation. We provided comprehensive learning modules and resources to familiarize the team with the task requirements and client guidelines. A dedicated Q&A system ensured clarity, with trainers promptly addressing any queries or ambiguities. Additionally, we maintained a centralized Source of Truth document for up-to-date guidelines and instructions, streamlining the annotation process. ​

Automated Workflow Tool: We developed a tool to automate the scraping of regulatory documents from various sources. This ensured a comprehensive and up-to-date dataset for analysis.

Machine Learning for Metadata Enhancement: Machine learning algorithms analyzed the scraped content, automatically assigning relevant metadata like publication date, jurisdiction, and document citations. This improved data organization and searchability.

Learning Modules and Practice Exercises: Pre-recorded lessons and exercises to hone creative writing skills and familiarize them with client guidelines.​

Q&A Sessions: Regular meetings to address queries and refine understanding, with an emphasis on resolving ambiguities and ensuring adherence to guidelines.​

Reverse Engineering: Analyzing client-provided answers to practice tasks to identify patterns and nuances, enabling the team to align closely with client expectations.​

Exercise Sets: Creating ad copy, identifying hallucinations, and editing subpar outputs to meet diversity and creativity standards.​

Training and Evaluation​:

• Conducted thorough training to familiarize the team with task requirements and the AI’s capabilities.​
• Used learning modules, Q&A sessions, and practical exercises to prepare the team.​

Quality Attributes​:

• Helpfulness: Assessed if the response addressed the prompt coherently, concisely, and relevantly.​
• Honesty: Evaluated factual accuracy, neutrality, and transparency.​
• Harmlessness: Ensured responses were safe, non-offensive, and devoid of unqualified advice.​

Response Ranking​:

• Agents ranked responses based on their adherence to the quality attributes, from best to worst.​​

Innodata’s team of experts created a comprehensive set of 500+ documents tailored for testing the language model’s summarization skills. These documents were carefully curated to reflect the diverse and real-life scenario typically encountered in eDiscovery projects. By providing a robust and varied dataset, we ensured that the model’s summarization feature could be thoroughly tested under realistic conditions. ​

​In addition to creating the documents, our team rigorously tested the model using this dataset by developing and implementing a series of prompts designed to push the model’s summarization capabilities to their limits. By simulating various real-world scenarios and document complexities, they were able to thoroughly evaluate and enhance the language model’s performance. These prompts helped categorize the bugs and errors encountered during testing of the created documents into categories such as false positives, false negatives, and graceful responses. This approach resulted in a highly robust model capable of handling complex datasets. 

Learning Resources: Pre-recorded lessons and detailed documentation provided clear guidelines and examples.​

Practice Exercises: Interactive Google Forms exercises allowed for practical application of knowledge.​

Real-world Experience: Early access to the client’s practice queue ensured familiarity with the actual work interface.​

Dedicated Support: Q&A Excel sheets addressed questions and facilitated clarification from the client.​

Centralized Knowledge Base: An internal “Source of Truth” document housed the latest guidelines and resources.​

Example Sets: Client-provided gold-standard answers served as reference points for annotators.​

Skilled Workforce: Recruitment focused on individuals with experience in concise writing (e.g., newsletter writers) and basic fact-checking.​

​

​

Chain-of-Thought Prompt Development: Innodata’s team of experts employed a technique called “Chain of Thought in Context Learning” to design prompts that encouraged the LLM to explicitly showcase its internal thought process while responding to user queries. This provided valuable insights into the LLM’s reasoning and information retrieval steps.

Prompt Completion with RAG Integration: The team leveraged “Prompt Creation Completion” techniques, where authors set up prompts, craft related queries, and complete the prompts using the Retrieval-Augmented Generation (RAG) tool. This tool retrieved relevant information necessary for the LLM to complete the task at hand.

Author Expertise: Our team of skilled authors, equipped with an understanding of API and RAG dependencies, crafted the dataset elements:

Expert Actor Recruitment: Innodata assembled a team of actors with real-world medical experience, including nurses, medical doctors, and students. This ensured the dialogues reflected the appropriate level of expertise and communication style for each scenario. 

Content Development: Our medical writers crafted the dialogues based on the client’s provided topics and “clinical key” resources. Each conversation maintained a natural flow while adhering to strict medical accuracy.

Multi-Layer Review: The dialogues underwent a rigorous review process by medical professionals to guarantee factual correctness and adherence to the 99% accuracy benchmark.