Measuring AI Impact in Legal Practice

BY ROBBIE PHILBRICK, RACHEL WILKA & MICHAEL CALLIER

One of the biggest challenges facing organizations today is understanding how and when to use artificial intelligence (AI). Implementing AI can be a resource-intensive endeavor, requiring project management, the adoption of new technology, and often the need to develop customized plans for implementation to fit specific needs. Incorporating AI can be particularly challenging in the legal space, where the implications of incorporating AI include ethical, legal, practical, and professional considerations. These challenges are compounded by the rapidly evolving nature of AI products, which makes it difficult to keep up with emerging risks and opportunities.

As a result, prioritization is essential. Not all AI use cases offer the same value and evaluating them requires a structured approach. In this article, we’ll explore the following three-part framework for assessing and prioritizing AI opportunities:

1. Identifying potential use cases.
2. Evaluating identified use cases.
3. Creating an overall scoring framework.

Identifying Use Cases

Identifying where AI might be a useful tool within an organization begins with understanding organizational needs and pain points. Start by identifying the different user roles and teams within your organization, along with their specific goals and responsibilities. The organization’s broader priorities should also be considered. From there, organizations can examine the key challenges each group faces—especially those that make it harder to achieve their goals. These challenges often show up as inefficiencies, reduced client satisfaction, or increased risk.

Organizations may want to consider the following:

• What tasks/goals cannot currently be met due to limited organizational resources or the absence of supporting technology? These may be tasks that lend themselves well to automation, either due to their complexity, the manual effort they require, or a combination of the two.
• Are any tasks universally disliked within your organization? If there’s work that everyone dreads, AI could alleviate those burdens. For instance, automating first-draft contract generation might save hours of monotonous labor.
• Are response times too long? If the legal department struggles to meet the needs of client teams promptly, AI might streamline workflows and shorten turnaround times.
• What work is repetitive and routine? Repeatable and predictable tasks, such as compliance reviews or document comparisons, often lend themselves well to automation.
• What risks are consistently overlooked? Are there areas where risks are routinely undetected or where errors go unnoticed? AI could help improve accuracy and oversight in these areas.
• Are there inconsistencies across teams? When different attorneys or teams handle similar tasks in varying ways, standardizing processes through AI could enhance both quality and consistency.

Evaluating Potential Use Cases

After identifying potential use cases, the next step is to assess their viability by weighing the benefits, costs, and risks—while also identifying any potential obstacles. Each of these factors can provide critical insight into whether a project is worth pursuing.

Benefits

AI can deliver value across a range of categories. One obvious benefit of AI usage is saving time—research shows that in just five years, generative AI could save users up to 12 hours of work per week.¹1  www.thomsonreuters.com/en/insights/articles/save-time-and-achieve-more-with-ai#:~:text=Professionals%20believe%20that%20the%20amount,person%20freed%20up%20in%202025. However, there are other less obvious benefits, such as improvements in employee job satisfaction and financial gains. Measuring these benefits requires understanding both the qualitative and quantitative impacts that an AI application might have. Here is a list of performance indicators to measure the potential impact of use cases and help you select the right one:

• Time savings: As one example, auto-generating initial drafts of Statements of Work (SoWs) could save hundreds of hours annually. To quantify the value of time savings, first determine how much time personnel spend on SoWs per week, then determine the hourly rate of that personnel. A full-time employee working a standard 40 hours weekly totals 2,080 working hours annually. If their total annual compensation is $100,000, their hourly rate would be approximately $50. Ten hours of time savings per week would result in $500 of weekly monetary impact.
• Cost reduction: AI could be used to assess and summarize contracts for regulatory responses, minimizing the need for outside counsel to manually review all but the most critical agreements. Organizations can start their cost reduction estimates by calculating the amount of outside counsel spend for similar events.
• Responsiveness: Faster initial response and turnaround times (TATs) improve the client experience. TAT analysis should account for the entire process being targeted for improvement—from intake to resolution.
• Risk reduction: AI minimizes errors in contract review and due diligence processes by finding needles in document haystacks, such as flagging unreported open-source software in technology transaction contracts or missing asset disclosures in bankruptcy proceedings. To quantify potential impact, calculate current error rates and related costs on a granular level when comparing different use cases.
• Resource optimization: Automating reviews of contracts below a certain risk or complexity threshold could free attorneys for higher-impact tasks. Measure the opportunity cost of low optimization by quantifying the proportion of time attorneys allocate to low-risk, low-complexity work and the annual cost for those attorneys.
• Cross-departmental impact: AI solutions that also benefit other teams—like sales or procurement—can strengthen the business case for investment. Identify and quantify additional efficiencies or savings realized by other departments.

By applying these measurements, organizations can prioritize projects that offer the highest return on investment, not just in terms of cost but also in time and strategic benefits.

Costs and Detractors

Even high benefit use cases may not be feasible due to significant costs and detractors. Some to consider include:

• Resources required for either purchasing AI or developing AI: Developing AI and integrating it into your systems always involves costs, whether direct purchasing costs, developer time building a model, and/or implementation costs.
• Development time: Off-the-shelf solutions may have quick implementation timelines, but custom models could take months or even years to build.
• Data quality: AI performs optimally with structured, accurate data. Poor data quality, including missing or inaccurate information, can significantly increase costs or reduce the feasibility of achieving meaningful results.
• Transition challenges: Introducing AI often involves creating new processes and retraining staff, which can be complex and time-consuming.
• Time to value: Justifying even a high-benefit use case can be difficult if gains may not be realized for months or years, especially in organizations with frequent strategic pivots.
• Maintenance: Ongoing maintenance can add to the total cost of ownership and offset benefits.
• Switching costs: Replacing existing solutions with AI may incur sunk costs or other trade-offs. For instance, transitioning from a vendor offering may mean writing off committed funds, and moving away from bundled software could disrupt workflows that rely on those bundles, even if they are not directly related to the AI implementation.

Creating an Overall Scoring Framework

Creating a scoring model helps standardize the evaluation of AI use cases by balancing benefits against costs and risks. While some cases might be straightforward—e.g., comparing outside counsel costs to AI development costs—others require a more nuanced analysis. It’s important to understand what requirements are “must-haves” versus “nice-to-haves” for an implementation to be successful. A successful scoring model should consider both quantitative and qualitative factors as discussed above, as well as other criteria tailored to organizational priorities and requirements, and ultimately provide a scaled/numeric value of each:

• Quantifying benefits: Use metrics like hours saved multiplied by the hourly value of attorney time to estimate the monetary impact of efficiencies.
• Assessing costs: Account for development, data standardization, and training expenses, as well as ongoing maintenance.
• Long-term strategic goals: Evaluate whether a project lays the groundwork for future efficiencies or additional use cases, prioritizing foundational projects higher.
• Downstream impact: Analyze whether the project generates downstream efficiencies for other teams or aligns with broader organizational priorities.
• Qualitative inputs: Include employee/user feedback on impact, priority, and alignment with strategic goals to ensure a holistic assessment.

By applying these principles, organizations can balance benefits and costs effectively, ensuring alignment with both immediate needs and long-term objectives. For instance, if standardizing data for AI use would consume most of the projected time savings, the project might only be worth pursuing as part of a larger, strategic effort to build scalable infrastructure for future AI initiatives.

Risk Overrides

In some cases, risks may outweigh potential benefits, regardless of cost-benefit analysis. High-risk areas might include:

• Inaccuracy: Any use of AI includes a risk of inaccuracy, which may be more or less detrimental depending on the area of application. For example, when AI drafts a severance agreement incorrectly, evaluate what errors are most likely to occur, the rate of such errors, and what their impacts would be.
• Data sensitivity and privacy concerns: AI solutions often process sensitive data, raising concerns about privacy and security, especially if AI output could potentially be shared with external parties. Data exposure risks might disqualify a project outright unless mitigated.
• Regulatory constraints: Certain uses of AI, such as in hiring or credit scoring, are subject to strict regulations and oversight that will either explicitly prohibit using AI or create additional regulatory requirements if AI is used.
• Consumer impact: Projects that increase the likelihood of discriminatory or otherwise harmful impacts on consumers are often untenable without significant safeguards.
• Lack of ownership: Without a clear owner to champion the implementation, even high-benefit projects might fail.

In many cases, these risks can be mitigated by introducing human oversight (i.e., Human in the Loop: having a person review AI-generated outputs) or running outputs through other human-controlled processes before proceeding.

Bringing It All Together

Once you’ve developed a standardized approach to evaluate use cases, you can step back and assess your overall AI strategy:

• Are all teams benefiting? Ensure the AI roadmap distributes value in a way that considers the needs of all departments and sub-teams fairly.
• Are there foundational building blocks? Shared infrastructure can support multiple use cases and maximize return on investment. For example, there may be an AI tool that could help both the sales and legal departments.
• Is there clear ownership? Every project needs an accountable leader, both for adoption and for ongoing maintenance/program management.
• Are resources aligned? Ensure adequate funding and support for approved use cases.
• Are risks manageable? Monitor the overall risk profile and adapt as needed.
• Are the intended users bought in? The best AI model in the world means nothing if it never gets used after implementation.

Conclusion

Measuring AI’s impact in legal practice is both an art and a science. It is crucial to recognize that AI’s impact is often gradual and iterative, not immediate or absolute, which should temper expectations and guide decision-making. Continuous oversight, feedback loops, and governance are essential to refine models, address new challenges, and adapt to evolving organizational needs. AI projects succeed when treated as dynamic investments, not static solutions, and user investment is the number one predictor of success, so securing buy-in is critical. Ultimately, adopting AI is a learning curve—one that evolves with the organization and its priorities. While the ways we use AI and its potential impact will change over time, having a framework and process for evaluating different use cases can remain consistent.

---

MORE ONLINE > Previous parts of this series can be found at www.wabarnews.org:

> Implementing Generative AI Effectively in Legal Teams

> How to Prepare Information Architecture to Leverage AI Effectively

NOTE

1. www.thomsonreuters.com/en/insights/articles/save-time-and-achieve-more-with-ai#:~:text=Professionals%20believe%20that%20the%20amount,person%20freed%20up%20in%202025.

---