Scaling gen AI in US state and local governments: Opportunities, challenges, and the path to achieving large-scale value
Governments can scale gen AI by enhancing technological infrastructure, engaging the workforce, and building effective governance structures
Naman Chaurasia
John O'Leary
Joe Mariani
Sushumna Agarwal
Generative AI is transforming industries, economies, and societies in ways we could only imagine a few years ago—and it is also making inroads in government. In its 2024 State CIO survey, the National Association of State Chief Information Officers asked state CIOs if employees in their organization were using generative AI tools in their daily work—and more than half answered “Yes.”1
In light of a potentially constrained budget environment, US state and local governments may be looking at tools that can help them deliver on their mission more efficiently. In fact, gen AI is already generating impact.
Internal back-office operations: Indiana is leveraging gen AI to streamline access to state archives. Staff can quickly search millions of documents spanning a century, from historic business records to recent financial filings. The AI understands plain-language queries, making searches fast and intuitive. This technology helps Indiana improve transparency and efficiency, turning vast historical data into accessible, useful information.2
Internal service operations: The ability for gen AI to analyze text and generate natural, easy-to-understand phrasing in different languages is helping governments communicate more effectively. The New Jersey Office of Innovation is leveraging gen AI to rewrite emails sent to customers in plain language. As a result of this change, the office has observed a 35% faster response rate from customers.3
Public-facing uses: State and local governments are implementing chatbots that answer constituent questions, both in individual departments and across multiple government services. New York City, for example, launched one of the first public-facing gen AI-powered chatbots in October 2023, built on top of MyCity Business and NYC.gov. Citizens can ask straightforward questions such as “How can I apply for a driver’s license?” or “What services do I qualify for?” While still evolving, the chatbot could help New Yorkers better navigate the complexity of government services.4
While gen AI pilots and experimentation continue, a Deloitte survey suggests that a large majority of organizations, both public and private sector, have deployed less than one-third of these experimental pilots into production,5 and even fewer have reached the scale necessary to meet mission objectives.6
Multiple factors can contribute to organizations getting stuck in the pilot and experimentation phase, including insufficient preparation for scaling up, concerns over potential risks and bias, insufficient technical expertise, and lack of dedicated funding streams or incentives to adopt AI in government.
State and local governments are caught between the desire to harness value from this powerful new tool, and their rightful concern about unleashing an untested technology within critical public services.7 The big challenge? How can state and local governments move forward to capture the value of gen AI at scale while controlling risks?
How organizations worldwide are harnessing generative AI to revolutionize IT, operations, and communications
Generative AI offers a wide array of applications. According to Deloitte’s State of Generative AI in the Enterprise quarter four report,8 which surveys global leaders in both the public and private sectors, gen AI is having the deepest impact on information technology, operations, and marketing for respondents (figure 1).
There are numerous examples of gen AI applications across similar functions in government agencies. As these agencies increasingly adopt these tools, they are witnessing significant improvements in efficiency and productivity.
Information Technology
The Utah state government has adopted gen AI to enhance cybersecurity. Handling two terabytes of data daily, the tool has improved alert quality and actionability, enabling proactive risk mitigation.9
Operations
Over the past year, the US Department of State has integrated various gen AI tools to enhance efficiency and productivity. These tools assist with drafting emails, translating documents, and brainstorming policy. They have collectively saved employees tens of thousands of hours, allowing them to focus on more strategic tasks.10
Pennsylvania and Colorado state governments have made certain AI tools available to a section of employees across various agencies. In Pennsylvania, this initiative has resulted in average savings of more than an hour per day among users. In Colorado, more than one-third of users have saved at least six hours each week.11
Marketing and communications
The Swindon Borough Council in United Kingdom uses gen AI to translate and simplify government documents for its residents, who collectively speak more than 30 languages. The tool has reduced translation costs by 99.96% and cut translation time from 16 days to 14 minutes.12
Customer service
The Colorado Department of Local Affairs used gen AI to significantly reduce the time required to identify process improvements in its housing voucher program. This advancement allowed the team to spend less time on discovery and more time implementing solutions. Previously, any process improvement exercise required detailed information gathering from individuals involved in the process and lengthy discussions to build process maps. However, the team transitioned to using a computer consultant named “Coco.” Coco conducted one-on-one interviews with dozens of stakeholders, asked questions in natural language, and synthesized all the information to create process maps. Staff members are now utilizing these process maps to redesign the processes effectively.13
The potential benefits of gen AI at scale
Generative AI is a powerful tool, but it needs to be adopted at scale to realize its transformative benefits. Scaling entails more than just engaging a large number of users; it also means embedding gen AI into an organization’s core processes.
AI at scale can be a powerful tool for government. In 2024, the US Department of Treasury started using AI tools to detect fraud in government payments, preventing or recovering $4 billion in improper payments—an improvement of over $3.3 billion from the previous year. Like many high-impact applications of AI, the Treasury’s “Do Not Pay” service crosses organizational boundaries, connecting state unemployment agencies with the Social Security Administration’s Death Master File, a database that holds information of deceased individuals, to reduce improper payments.14
The potential benefits of AI at-scale are accelerating. The emergence of agentic AI—that is, more autonomous, goal-directed AI-based tools that can coordinate other automation tools—is opening new areas of advanced automation (figure 2). AI agents are like conductors of orchestras being able to coordinate the action of other automation tools like gen AI, robotic process automation, and even human workers to create more efficient workflows. For example, an AI agent could streamline vehicle registration renewals by orchestrating the actions of several tools: using machine vision to extract information from forms and supporting documents, employing rules engines to determine eligibility for renewal, escalating cases to human workers if issues arise or if the application is not immediately eligible, and finally, using gen AI to issue renewal notices.
As noted earlier, gen AI will need scale to generate true transformative value for governments. However, a survey of local government practitioners in the United States revealed that despite the growing relevance of AI across various sectors, fewer than 6% of respondents “have prioritized AI as a significant focus in their service delivery strategies.” Nor are local governments getting the capabilities that would put them on the path to scale. Only 10% of the communities surveyed have an individual in charge of overseeing AI efforts, and just 9% have organization-wide policies that govern AI usage.15
So how can state and local governments move beyond small pilot experiments with gen AI to implementing it at scale?
Government’s unique path to scaling gen AI
Scaling gen AI in government can be challenging because it cannot simply imitate the commercial sector, which is often more advanced in technology adoption. In the commercial world, clear metrics such as sales and revenue provide straightforward ways to measure the return on investment from any technology, including gen AI. Business executives, driven by financial incentives, typically pursue a top-down, centralized approach to technology adoption.
In contrast, governments can sometimes struggle to measure the impact of gen AI on mission effectiveness. Deloitte’s State of Generative AI in the Enterprise survey revealed that government employees and tech professionals are more eager than their private sector counterparts to experiment with gen AI. However, this enthusiasm is not as prevalent among surveyed public sector leaders (figure 3).
Another finding of the survey was that government and public service providers, unlike their private sector counterparts, believe that maximizing access to gen AI for the workforce is more likely to drive the most value from these initiatives. This suggests that the path to scaling gen AI in government likely lies in providing wider access to gen AI tools across the workforce (figure 4).
One strategy for enabling wide access to gen AI among government workers is the creation of an AI marketplace—a controlled technology environment that allows users to create and deploy solutions to their problems. If the solutions work, the marketplace has governance mechanisms in place to scale them across the organization.
The marketplace provides a platform with multiple building blocks, enabling users to quickly build their own gen AI use cases. Depending on the sensitivity of the data, it can offer options to choose between multiple open-source large language models (LLMs) and on-premise LLMs. On-premise LLMs, hosted on the organization’s cloud infrastructure, protect data from being sent to open-source LLMs. The platform enables users to connect their data and workflows to AI tools through a user-friendly interface, allowing employees to utilize gen AI capabilities without needing to understand complex coding. For instance, a caseworker could easily upload relevant policies and create a gen AI-powered chatbot, making the technology accessible to nontechnical staff.
Additionally, marketplaces help agencies support data security and compliance. They enable necessary guardrails to reduce risks when the solutions are scaled across the organization. These guardrails include confirming that data is not being used to train external LLMs, managing role-based access to data and tools, and blocking personally identifiable information from being sent to public LLMs. Usage data can also be monitored to support compliance and security, providing a safe and effective environment for government employees to harness the potential of gen AI. The Department of Defense Chief Digital and Artificial Intelligence Office has already implemented an AI marketplace,16 and the Department of State may soon follow suit.17
What does at-scale AI look like, and where is the North Star?
There are four dimensions to AI readiness: strategy, governance, people and technology. The good news is that many organizations likely already have some form of these capabilities in place.
The challenge is that there is a significant gap between pilot and at-scale capabilities, and organizations accustomed to AI only in pilot form can find it difficult to gain the capabilities needed to scale (figure 5). In other words, while every organization may have some form of AI strategy, governance, people, and technology, the capabilities in each of those buckets are very different for pilots versus at-scale adoption. While pilot-level capabilities might suffice for initial experimentation, scaling AI solutions requires a more comprehensive and integrated approach across all dimensions.
Challenges to building gen AI scaling capabilities for state and local governments
Understanding the components of a successful gen AI effort is one thing; implementing the necessary structures and culture is the real challenge. The unique circumstances of government can create equally unique barriers that can make it difficult to acquire at-scale capabilities. Some of the most common barriers that have emerged through our interviews and surveys include risk wariness, constrained budgets, and limited technical expertise.
Barrier: Risk wariness
With sensitive data and the public trust at stake, some government leaders are wary of the risks associated with gen AI. However, failure to adopt poses risks as well. The challenge lies in finding ways to reap the benefits of gen AI while minimizing any undue risk.
Actions that can help:
- Gain executive support: In government organizations, rank-and-file public workers are often more eager to employ gen AI than more senior leaders (figure 3). In fact, we found that government employees reported being more eager to test and experiment with gen AI. However, this enthusiasm is not as prevalent among surveyed government leaders, who are more cautious. Some actions for fostering support may include quantifying the benefits of AI for the government mission; creating AI fluency within the executive group that is closely tied to everyday service delivery; and creating an innovative space and a culture where adopting a cutting-edge technology is celebrated, even if the initial efforts sometimes produce sub-optimal results.
- Adopt tailored or tiered governance: Gen AI is a tool that can be used in different ways. Small pilot applications typically shouldn’t be subject to the same level of governance as a large use case that directly impacts thousands of constituents. California has outlined its oversight considerations for gen AI applications, categorizing them into three risk levels: low-risk use cases merit “standard monitoring and lightweight evaluations;” moderate use cases “warrant more involved oversight;” and high-risk systems “require intensive evaluations, qualitative assessments, and risk mitigation measures.”18
- Build true governance, not extra paperwork: There is a conundrum associated with today’s more capable gen AI and agentic AI use cases. On one hand, how do you know something is low-risk or high-risk if you don’t perform a risk assessment? On the other hand, you don’t want to make tedious risk assessments a barrier to delivery. One way to resolve this conundrum is to integrate risk controls directly into the platforms and tools where AI solutions are developed. Having control over features such as role-based access to data, automated detection of security vulnerabilities and configurations, visibility, and control over the type of LLM being used for various solutions, and the ability to monitor red flags such as toxicity, bias, and inappropriate responses provides actionable governance that can be both understood and tracked.
- Improve data availability: For generative AI to deliver accurate and tailored information, it must have access to comprehensive data from multiple agencies. The richer the data drawn from these sources, the clearer the view of an individual’s unique circumstances, which in turn enables more complete responses. Confirming that gen AI can access data drawn from across organizational boundaries is key.
Barrier: Constrained budgets
Gen AI has the potential to save government organizations millions of hours and perhaps dollars, too. But adoption often involves upfront costs for investments in technology infrastructure or subscriptions. In times of budget constraints, making these investments can be difficult, especially when they are competing with other more long-standing budget items.
Actions that can help:
- Decide how to measure success: A key to competing in tight budgets is demonstrating a clear return on investment. This can be difficult with AI, as measuring mission benefits in government is notoriously challenging. How do you measure a constituent’s satisfaction level or the impact of faster service? Identifying ways to measure the mission outcomes improved by gen AI is the first step toward justifying its cost.
- Get the funding right: Even when dollars are available, public funding can often be inflexible and siloed. For instance, if you can’t spend on acquisition and operations at the same time, that can be a problem. If funding is only available for program A, it becomes challenging to support gen AI, which aims to enhance the missions of programs A, B, and C. Therefore, identifying the right funding mechanisms to support long-term gen AI procurement and operations is important.
Barrier: Limited technical expertise
Making informed decisions about the costs and risks associated with gen AI implementation can be difficult without a foundational understanding of the technology, how it works, and its various options. Our surveys indicate that government leaders are experiencing this expertise gap, with only 17% of respondents reporting high or very high levels of gen AI expertise in their organization, compared with an average of 40% across other industries.19 So, how can state and local leaders quickly gain the expertise they need?
Actions that can help:
- Offer broader access to the technology: One of the main barriers to developing AI expertise in government is simply a lack of access to the technology. It is difficult to become proficient with a tool that is not readily available for use. Only 1% of government respondents in a recent survey said that 60% of their workforce had access to gen AI tools, the lowest rate of access of any industry.20 Providing the workforce wider access—with effective guardrails—will help build AI fluency across the organization.
- Embrace modularity: Building AI solutions in a modular manner can help solutions adapt as new technologies emerge and avoid vendor lock-in. With a modular architecture, organizations can swap out different LLMs, change hosting venues, or even add new tools as conditions warrant.
Getting started on the path to scale
There are many ways gen AI adoption can go wrong. You can fail to make progress by making one big bet or by launching 100 little proofs of concept. But the good news is that there are numerous ways for the 50 states to get it right. The building blocks are the same, but the exact path to scale will be different for every state. However, a few signposts can help keep you on the right path.
Build tech foundations: Currently, many states have strategy plans but may not have the tools to make that strategy a reality. If you get the tech platforms right, it helps bake strategic values into the very processes and infrastructure that employees will use to make new AI tools. This foundation of a strong tech platform can naturally support governance, training, and other necessary capabilities, ultimately allowing wider access to AI without incurring additional risk.
Engage the workforce early: Employees will find high-impact use cases with gen AI only if they are actively engaged. Lack of engagement can hinder successful implementation. Organizations should empower workers to build tools and foster a strong culture so that guardrails and proper outcomes are internalized at every step of the process.
Transparency: Being transparent with the public and workforce can help build trust in the public sector.
Generative AI has the potential to be a key tool throughout the public sector. By prioritizing robust infrastructure, workforce engagement, and building strong governance, state and local governments can effectively pave the way to implement gen AI at scale to better drive effective mission results.
Continue the conversation
Meet the industry leaders
William D. Eggers
Sean Conlin
by
Naman Chaurasia
John O'Leary
Joe Mariani
Sushumna Agarwal
The authors would like to thank William D. Eggers for his thoughtful feedback; and Chris Stehno and Hari Murthy for their inputs in developing the draft.
Cover image by: Sonya Vasilieff
Visit the Deloitte Center for Government Insights
Access more insights for the defense, security and justice, government health care, state and local government, transportation and infrastructure, human services, and higher education sectors.