In fiscal year 2023, the US federal government spent approximately $190 billion on R&D. This was a roughly 13 percent increase over fiscal year 2022, making it one of the largest R&D spending jumps in recent history.1 Amid geopolitical shifts and rising competition abroad, legislation such as the CHIPS and Science Act and the National Quantum Initiative Act reflects a growing priority on securing the country’s place as a leader of global R&D well into the future.
To deliver on the full potential of US R&D expenditures, many in the federal government are looking to the private sector. Over the past several decades, the private sector has replaced the government as the driving force behind R&D, accounting for 73 percent of overall R&D spending in the country in 2020. By comparison, the US government accounted for nearly 70 percent of global R&D spending when organizations such as the National Laboratories and the Defense Advanced Research Projects Agency were founded in the 1950s and 1960s.2 Indeed, even in the US Department of Defense’s (DOD’s) 14 Critical Technology Areas, the DOD recognizes that the private sector is leading innovation in 11 of them.3 For example, venture capital investment in 2023 was equivalent to about 450 percent of DOD investment in advanced computing and software and about 800 percent of the department’s investment in biotechnology (exhibit). Considering the private sector more broadly, there are similar trends in additional areas, such as quantum science, in which industry has invested more than $5.5 billion (nearly 250 percent of DOD investment) since December 2022.4
Today, companies have developed sophisticated approaches to maximize the impact of their R&D activities. Already, Silicon Valley concepts such as “disruption” and “a start-up mindset” have entered the government lexicon, but they have yet to be truly embraced by public sector R&D organizations. While public and private sector R&D often differ in their nature and objectives, they share a common need to accelerate the time to impact and generate tangible outcomes for stakeholders in both the short and long term.
During times of crisis, US governmental R&D organizations have repeatedly risen to the challenge, delivering breakthrough technologies on extraordinary timelines. Now, the question is how to maintain that dynamism during a steady state. Drawing on lessons from leading R&D organizations in the pharmaceutical, aerospace, and venture capital (VC) sectors, we have distilled six insights that could potentially be adapted to the public sector context. These best practices could help US federal government agencies—and potentially governments in other countries—make the most of their R&D expenditures, accelerate technological innovation, and effectively deliver on their missions for years to come.
Go broad on ideas and deep on execution
Successful R&D organizations cast a wide net to source ideas and then make large, concentrated investments in a few select candidates with the most-promising risk/reward propositions.
In VC, for example, one of the world’s preeminent fund managers has developed a multipronged approach to investing: seed a broad range of early-stage companies, identify the high-potential candidates that emerge, and then back this smaller group of companies with significant capital. The firm raises two separate investment funds: one for seed- or early-stage ventures and another for later-stage (Series A+) investments. This strategy ensures that considerable capital is on hand to invest in high-potential candidates and accelerate their commercialization and deployment.
This “fast follow-on” approach has become a hallmark of VC investing, in large part because it reflects a fundamental truth about innovation: it is difficult to determine which early-stage technologies will yield the most value, and placing large bets during the initial stages of development is often inefficient. However, once the front-runners emerge, an infusion of capital can significantly shorten the time to launch—a critical factor for success in both the private sector and public sector.
US government leaders have repeatedly emphasized the need to accelerate the pace of innovation and shorten the cycle time in transitioning cutting-edge technologies from the lab to the field. Developing and maintaining faster technology adoption and acquisition routes will play a crucial role in realizing that goal. Successful acceleration will also be a matter of knowing which projects to prioritize and back with additional resources; this is where strategic clarity, effective governance, and appropriate reprogramming authorities will be essential. Federal organizations may need to maintain competencies across all topics, but they will likely want to target a handful of large investments for the greatest impact.
Focus on distinctive capabilities
Top performers recognize the risks of being a jack of all trades and master of none; they take a critical eye to their organizational strengths and weaknesses and deliberately orient their R&D activities around existing competitive advantages.
In the late 2000s, a large pharmaceutical company was falling behind its competitors. Its largest sources of revenue were under threat from generic competition, it had suffered several large and expensive clinical drug failures, and its share price had plummeted to its lowest point in a decade. Fast forward to today, and the company is among the most valuable pharmaceutical companies in the world. What led to this remarkable turnaround?
The solution was a return to basics. After conducting an in-depth review of its R&D failures, the company found that it had been playing to its weaknesses (commercial sales estimates) instead of its strengths (scientific expertise). In response, leadership decided to double down on just three therapeutic areas in which the company had distinctive capabilities. This discipline allowed the company to be more targeted in its R&D resource allocation, make strategic moves (such as acquisitions) to strengthen this expertise, and attract top talent in these areas—ultimately creating a virtuous cycle that deepened expertise and furthered the company’s competitive advantage.
Hundreds, if not thousands, of US governmental organizations are undertaking or investing in R&D activities, and those activities involve hundreds of thousands of partners. It is therefore unsurprising that a lack of alignment can pervade these efforts. This can result in not just duplicated efforts but also missed opportunities for collaboration. Creating transparency into the R&D portfolio, taking a critical lens to each R&D organization’s differentiated capabilities, focusing efforts on those areas, and offering incentives for collaboration are four ways to potentially increase R&D efficiency and effectiveness.
Not all differentiated capabilities represent persistent and defensible competitive advantages in the long term. Changes in technology are increasingly challenging many traditionally secure competitive advantages, as the recent exponential growth in the use of large language models and generative AI shows. Organizations should be mindful of technology trends that may undermine their own competitive advantages. It is more important than ever to have the infrastructure and flexibility to rapidly establish new sources of competitive advantage while retaining a strong core business. Thus, technologies that accelerate R&D itself could potentially jump to the front of the queue. Development trends in full-stack software, digital engineering, and generative AI are massively accelerating R&D velocity. Falling behind on process R&D can therefore be even costlier than falling behind on a given product. Nevertheless, the US government’s historical emphasis on product R&D has limited the amount of resources flowing to process-changing technologies.
Clearly define success metrics—and keep users at the center
R&D and innovation are not ends in themselves; they are means to an end. It is therefore critical to define success up front (for example, by evaluating R&D activities in the context of the organization’s mission statement and strategic objectives) and establish rigorous, data-driven performance-tracking mechanisms to evaluate progress.
R&D often requires large investments, long timelines to impact, and countless possible pathways. As a result, the potential costs of failure and misalignment can be significant. For example, another global pharmaceutical company incurred a multibillion-dollar loss when a therapy billed as having “blockbuster” potential was pulled from the market after less than two years because of negligible patient uptake. The main reason: the product’s value proposition centered on solving a problem that most users did not perceive to be significant (the method of drug delivery). In solving that problem, the company created another problem that many users cared more about (the size and portability of the delivery device). This failure to fully understand user preferences pulled billions of dollars of resources away from other potential R&D activities that could have seeded the company’s next generation of drug therapies.
As this example illustrates, some of the most important criteria for success revolve around the end user—customers, citizens, or military service members—while also considering other stakeholders (such as acquisition officers) who may be involved in delivering solutions. Thus, when evaluating R&D projects, US governmental organizations should consider whether the solution meets end users’ needs (product–market fit) and has a clear avenue to deliver the solution into end users’ hands (path to market). In doing so, the organization can involve prospective end users early and often in all R&D activities, even (and perhaps especially) when that end user is not well defined. This approach also bridges the physical and organizational gap between researchers and end users—a gap that is particularly acute in the context of the US government.
Staying focused on user needs may also mean adapting requirements to seize available opportunities and enable partners to add their unique contributions. Private sector R&D rides on the waves of others, not against them. Recent US government efforts to integrate commercial capabilities—the cloud, zero-trust IT, and as-a-service offerings, to name a few—suggest that the process of defining requirements may likewise need to evolve alongside cutting-edge technologies (that is, shift to a more agile and iterative process that integrates evolving user needs and the latest technological advances).
US governmental organizations can also consider go-to-market details (that is, the time between initial ideas and consumer launch) by sharpening their focus on elements such as producibility, maintainability, and user experience. Such factors can be considered not only up front but also regularly throughout the R&D life cycle to ensure that underlying assumptions still hold.
Evaluate success at the portfolio level, not the project level
Best-in-class organizations recognize that R&D is a team sport. They use overall portfolio performance rather than individual project performance to manage resources and directly reward R&D personnel.
To understand this principle, consider again the example of VC. One leading corporate VC firm has made hundreds of investments over the past 15 years or so, but more than 80 percent of its portfolio has been written off or is yet to return value. Another 15 percent generated only modest returns. And less than 2 percent of the firm’s portfolio has been true home runs (defined as returning more than two times the invested capital). At the individual project level, this may look like a losing strategy. But at the portfolio level—that is, looking at the firm’s overall returns—the VC firm has far surpassed expectations because the returns on that 2 percent have more than made up for the cost of the other, less successful investments.
Similarly, top-performing R&D organizations understand that most efforts fail but that the few successes can make it all worthwhile. Many private sector organizations even structure their R&D teams’ incentive packages to be based not on individual project performance but on performance of the overall R&D portfolio. A number of leading biotechnology companies provide further individual incentives to R&D project leaders to end their projects when appropriate, helping ensure that the team’s resources are dedicated to the most promising projects.
For US governmental organizations, this approach could include integrating cross-team collaboration and efficient resource-allocation elements into performance reviews, introducing other nonmonetary forms of recognition for terminating underperforming projects, and adapting organizational structures and processes to break down silos between individual R&D project teams. These would be significant changes to current US government processes and structures, but they could help to catalyze the next wave of US government R&D.
Accept failure as part of the process—and learn from it
Because R&D projects inherently have a low probability of success, R&D organizations must accept failure as part of the process. The most effective R&D organizations aggressively terminate projects that lag behind expectations and rapidly reallocate resources to the projects with the most potential. They also have robust mechanisms in place to diagnose the root causes of failures, synthesize key lessons learned from those experiences, and integrate those lessons into their ongoing operations. A leading aerospace company has made iterative, full-scale testing of its vehicles a hallmark of its R&D work, the speed of which has significantly outpaced that of many competitors in the industry.
As the pace of innovation increases, the government will also need to shift from a legacy “big bang” approach (taking three to five years to build a single rocket, for example) to an iterative development model that can adapt to changing circumstances and accelerate time to (sometimes literal) launch. That shift will likely involve more-frequent failures in developmental and operational tests, which can be embraced so long as the test is generating sufficient data and insights to justify the expense.
A more iterative approach also necessitates changes in how US governmental R&D organizations allocate funding, empower their leaders, and delegate decision making. These leaders need the flexibility and the decision-making authority to rapidly reallocate funding and resources, including personnel, across projects in a relatively short period of time (for example, every three to six months). And the right governance mechanisms must be in place to ensure that those decisions are aligned with the mission of the organization.
In practice, US governmental organizations will need to strike the right balance between research scientists and “decision scientists” (those who are making the call to cut projects and reallocate resources). Executives at several corporate VC firms approach this by bringing together the full range of stakeholders to form a comprehensive portfolio review board that is responsible for objectively vetting R&D decisions through a holistic lens. Collectively, the board has not only sufficient technical knowledge to understand the potential of each R&D project but also sufficient distance to make an impartial decision rooted in strategic alignment. Criteria for go/no-go calls at each stage gate might therefore include scientific potential, a clear path to market, and strong alignment with strategic objectives and goals.
Construct a balanced, tailored R&D portfolio
R&D organizations in both the private and public sector are expected to simultaneously advance near-term priorities and long-term objectives. This requires balancing the R&D portfolio to not only align with the organization’s target risk profile but also enable the full range of R&D investments—from high-risk, high-reward “moon shots” to feasible, near-term enhancements.
In the pharmaceutical industry, for example, investors want to see both strong revenues for drugs currently on the market and a robust R&D pipeline of drugs that will generate future earnings. Shareholders reward companies that demonstrate both short-term and long-term value creation and that pursue a diversity of opportunities while still leveraging their competitive advantages. Indeed, many of the largest acquisitions in the industry (running into the tens of billions of dollars) were undertaken to rebalance short- and long-term potential.
The principle also applies to public sector R&D organizations. After realizing that most of its R&D resources were allocated to near-term priorities, one of the US uniformed services decided to set aside 20 percent of its science and technology (S&T) budget for the development of transformational capabilities—long-term, low-probability projects that could significantly advance its strategic objectives. This decision ensured that significant S&T resources would be working toward long-term mission success, regardless of the near-term priorities that might arise. Taking a broader, portfolio-level perspective can enable R&D leaders to allocate risk across projects and meet the full range of their mission objectives.
Part of this approach is also about understanding which technology areas require large-scale US government support to achieve mission objectives as well as which technologies already benefit from significant backing in the private sector. This could include faster pathways for industry to commercialize research led by the government and faster pathways for the US government to partner with commercially led technology. In the case of the latter, this can consider more than just funding: earlier access to testing, certification, or risk-tolerant US government missions may be more valuable to R&D partners.
Historically, R&D funded by the US government has yielded remarkable technological breakthroughs, including the internet, GPS, the epinephrine autoinjector, and virtual reality. Now, the current increase in R&D spending and US government leaders’ heightened focus on innovation create an opportunity to achieve similar feats. By applying key lessons from the private sector, the US government can embark on a new era of innovation and once again become a driving force in global R&D.
