Board Oversight of AI as the Ultimate Enterprise Risk

by Michael W. Peregrine

Photo courtesy of the author

A string of recent, prophetic warnings by prominent technology and finance leaders present an unexpected, uncertain yet pressing oversight challenge to corporate boards across industry sectors.

The first of these warnings was Bill Gates’ lengthy essay, posted on his website on August 26.[1] In the essay, Mr. Gates predicts that AI will be “either the greatest equalizer ever invented, or the worst source of injustice.” Given this, he loudly recommends that the world’s top priority should be to resolve the current state of unpreparedness for the turbulent social, political and economic upheaval of transition to the AI era.

(Mr. Gates’s essay also makes the bold observation that the tech industry is “knowingly downplaying those threats because there [is] too much money on the line”).

The second was the August 30 letter from Andrew Bailey, Chair of the G-20 Financial Stability Board, to global finance ministers, warning that AI could cause a global economic downturn and create a significant cybersecurity risk to financial systems.[2]

These two were followed shortly by the resignation of Anthropic researcher Jason Coxon, due to concerns that uncontrollable AI systems could potentially destroy humanity.[3] A similar view was widely shared in an X post by his former Anthropic colleague Evan Hubinger.[4] And shortly thereafter comes news that Anthropic blocked efforts to conduct AI research that could have helped develop biological weapons.[5] And then came Dario Amodei’s essay calling for a pause in the pace with which AI models are improved; with which other leading AI developers quickly (and publicly) concurred. Legislative proposals are expected to quickly follow.

These and similar developments are forcing the issue of AI safety to the forefront of public discussion, political debate and boardroom dialogue. They reflect the extraordinary importance of AI to the national economy and the strategic and operational planning of many companies. Yet from this emerges the question of a proper board response; there are no established oversight principles for addressing the most existential of enterprise risks.[6] For the threat of human extinction is not precisely the type of mission-centric risk that constitutes a “red flag” for purposes of Caremark consideration; nor does it fit neatly within the definition of operational business risks for which business judgment rule protection is available to the board.[7]

Because of their extreme nature, it would be understandable for a board to reject these developments as “someone else’s problems,” not those of an operating business; to dismiss them as hyperbole, cataclysmal, unripe or irrelevant. But it would be best to do so after some organized intra-board discussion.

For purposes of board review, these new risks might fairly be characterized as “known unknowns” within Donald Rumsfeld’s famous risk matrix; i.e. enterprise risks that the board knows may exist but yet which it yet does not fully comprehend.[8] Responding to such risks would likely require informed business judgment and expert consultation, together with the imagination typically necessary to envision the totality of the circumstances.

Such an approach would be consistent with established principles of board oversight, and would hopefully allow a more informed evaluation of the risks as publicly articulated to date; i.e. whether the country is truly unprepared for the potentialities of rapid and unregulated AI deployment. Should the board ultimately conclude that these risks are credible, it has multiple external and internal options from which it could pursue a good faith response. These include the following:

1. Exercise the “Corporate Social Voice” in public support of AI safety proposals favored by the board and senior management. For some companies, it may mean supporting general themes of government AI regulation; for others, it may mean supporting any one or more of the specific regulatory proposals that are now emerging from Congress and individual states.[9] (Surveys continually demonstrate that the public looks to the business sector when government is unable or unwilling to address recognized social concerns. The public voice of the corporate sector may be necessary to bring about formal federal action, given what Mr. Gates described in his essay as the opposition of the tech industry to AI regulation).
2. Work with state and federal elected officials within the company’s jurisdiction to encourage support of AI safety guardrails and other related concerns.
3. In compliance with state and federal antitrust laws, combine with peer companies to heighten public and regulatory awareness of the pressing nature of AI safety risks and the need for mandatory protective government measures.
4. Maintain enhanced communications with AI developers and manufacturers regarding new developments in managing AI risks.
5. Engage in wide-ranging conversations with the company’s workforce to glean their perspectives on potential AI-related dangers.
6. Revisit the effectiveness of internal AI governance initiatives to identify cybersecurity and rogue platform risks arising from the company’s use of AI.

These and other responses would be consistent with traditional principles of informed judgment, oversight and particularly corporate social responsibility. Then again, an informed, good faith decision to pass on action should similarly be sustainable (especially if the board commits to continuous monitoring of developments). But an uninformed decision for inaction might not be.

As boards process the flood of new information regarding the existential and other risks posed by AI, they would do well to consider Bill Gates’ related message to leaders from his essay:

You have a chance to act now, before unemployment rises sharply, communities are hurting, and public trust has eroded. You can make sure that your government handles the problem holistically, rather than divvying it up into multiple bureaucratic fiefdoms. You can make sure AI benefits everyone. And you can work with other governments to meet this national and global challenge.

[1] Bill Gates, “The turbulent AI era is here. The choices we make now are critical.” August 26, 2026. https://www.gatesnotes.com/work/make-ai-work-for-everyone/reader/a-turbulent-ai-era-and-critical-choices-to-make

[2] Financial Stability Board, “FSB Chair warns of risks arising from frontier Artificial Intelligence (AI) Models;” 31 August 2026. https://www.fsb.org/2026/08/fsb-chair-warns-of-risks-arising-from-frontier-artificial-intelligence-ai-models/

[3] Amrith Ramkumar, The Wall Street Journal, “Anthropic researcher quits over out of control AI fears,” September 9, 2026. https://www.wsj.com/tech/ai/anthropic-researcher-quits-over-out-of-control-ai-fears-707b7628?st=6eUHMS&reflink=article_copyURL_share

[4] “Anthropic’s Evan Hubinger: >10% chance AI kills everyone this decade, no alignment plan yet.” https://dealroom.co/news/x-2097497037956891126-anthropics-evan-hubinger-10-chance-ai-kills-everyone-this-decade-no-alig/

[5] Dustin Volz, The New York Times, “Anthropic Says it Blocked Possible Efforts to Build Biological Weapons,” September 10, 2026. https://www.nytimes.com/2026/09/10/us/politics/anthropic-ai-biological-weapons.html

[6] William J. Broad and Cade Metz, The New York Times, “AI Could Possibly End Humanity. How are Humans Supposed to Process That?” September 10, 2026. https://www.nytimes.com/2026/09/10/science/ai-humanity-risk.html

[7] In re Boeing Co. Deriv. Litig., Consol. C.A. No. 2024-1210-MTZ, slip op. at 1-2, 21-22, 40, 43-44 (Del. Ch. Aug. 13, 2026).

[8] See, e.g. Michael W. Peregrine, “The Pandemic’s Impact on Board Oversight of Enterprise Risk,” CLS Blue Sky Blog, April 30, 2020. https://clsbluesky.law.columbia.edu/2020/04/30/the-pandemics-impact-on-board-oversight-of-enterprise-risk/

[9] Amrith Ramkumar and Yoko Kubota, “Congress Is Suddenly Waking Up to the AI Doomsday Threat,” The Wall Street Journal, September 11, 2026. https://www.wsj.com/politics/policy/congress-is-suddenly-waking-up-to-the-ai-doomsday-threat-b40ab25a?st=oTKvyQ

Michael W. Peregrine is a retired corporate governance attorney and Fellow of the American College of Governance Counsel. He is currently an Executive Fellow with the consulting firm SullivanCotter.

The views, opinions and positions expressed within all posts are those of the author(s) alone and do not represent those of the Program on Corporate Compliance and Enforcement (PCCE) or of the New York University School of Law. PCCE makes no representations as to the accuracy, completeness and validity or any statements made on this site and will not be liable any errors, omissions or representations. The copyright of this content belongs to the author(s) and any liability with regards to infringement of intellectual property rights remains with the author(s).

by Michael W. Peregrine

Photo courtesy of the author

A string of recent, prophetic warnings by prominent technology and finance leaders present an unexpected, uncertain yet pressing oversight challenge to corporate boards across industry sectors.

The first of these warnings was Bill Gates’ lengthy essay, posted on his website on August 26.[1] In the essay, Mr. Gates predicts that AI will be “either the greatest equalizer ever invented, or the worst source of injustice.” Given this, he loudly recommends that the world’s top priority should be to resolve the current state of unpreparedness for the turbulent social, political and economic upheaval of transition to the AI era.

(Mr. Gates’s essay also makes the bold observation that the tech industry is “knowingly downplaying those threats because there [is] too much money on the line”).

The second was the August 30 letter from Andrew Bailey, Chair of the G-20 Financial Stability Board, to global finance ministers, warning that AI could cause a global economic downturn and create a significant cybersecurity risk to financial systems.[2]

These two were followed shortly by the resignation of Anthropic researcher Jason Coxon, due to concerns that uncontrollable AI systems could potentially destroy humanity.[3] A similar view was widely shared in an X post by his former Anthropic colleague Evan Hubinger.[4] And shortly thereafter comes news that Anthropic blocked efforts to conduct AI research that could have helped develop biological weapons.[5] And then came Dario Amodei’s essay calling for a pause in the pace with which AI models are improved; with which other leading AI developers quickly (and publicly) concurred. Legislative proposals are expected to quickly follow.

These and similar developments are forcing the issue of AI safety to the forefront of public discussion, political debate and boardroom dialogue. They reflect the extraordinary importance of AI to the national economy and the strategic and operational planning of many companies. Yet from this emerges the question of a proper board response; there are no established oversight principles for addressing the most existential of enterprise risks.[6] For the threat of human extinction is not precisely the type of mission-centric risk that constitutes a “red flag” for purposes of Caremark consideration; nor does it fit neatly within the definition of operational business risks for which business judgment rule protection is available to the board.[7]

Because of their extreme nature, it would be understandable for a board to reject these developments as “someone else’s problems,” not those of an operating business; to dismiss them as hyperbole, cataclysmal, unripe or irrelevant. But it would be best to do so after some organized intra-board discussion.

For purposes of board review, these new risks might fairly be characterized as “known unknowns” within Donald Rumsfeld’s famous risk matrix; i.e. enterprise risks that the board knows may exist but yet which it yet does not fully comprehend.[8] Responding to such risks would likely require informed business judgment and expert consultation, together with the imagination typically necessary to envision the totality of the circumstances.

Such an approach would be consistent with established principles of board oversight, and would hopefully allow a more informed evaluation of the risks as publicly articulated to date; i.e. whether the country is truly unprepared for the potentialities of rapid and unregulated AI deployment. Should the board ultimately conclude that these risks are credible, it has multiple external and internal options from which it could pursue a good faith response. These include the following:

1. Exercise the “Corporate Social Voice” in public support of AI safety proposals favored by the board and senior management. For some companies, it may mean supporting general themes of government AI regulation; for others, it may mean supporting any one or more of the specific regulatory proposals that are now emerging from Congress and individual states.[9] (Surveys continually demonstrate that the public looks to the business sector when government is unable or unwilling to address recognized social concerns. The public voice of the corporate sector may be necessary to bring about formal federal action, given what Mr. Gates described in his essay as the opposition of the tech industry to AI regulation).
2. Work with state and federal elected officials within the company’s jurisdiction to encourage support of AI safety guardrails and other related concerns.
3. In compliance with state and federal antitrust laws, combine with peer companies to heighten public and regulatory awareness of the pressing nature of AI safety risks and the need for mandatory protective government measures.
4. Maintain enhanced communications with AI developers and manufacturers regarding new developments in managing AI risks.
5. Engage in wide-ranging conversations with the company’s workforce to glean their perspectives on potential AI-related dangers.
6. Revisit the effectiveness of internal AI governance initiatives to identify cybersecurity and rogue platform risks arising from the company’s use of AI.

These and other responses would be consistent with traditional principles of informed judgment, oversight and particularly corporate social responsibility. Then again, an informed, good faith decision to pass on action should similarly be sustainable (especially if the board commits to continuous monitoring of developments). But an uninformed decision for inaction might not be.

As boards process the flood of new information regarding the existential and other risks posed by AI, they would do well to consider Bill Gates’ related message to leaders from his essay:

You have a chance to act now, before unemployment rises sharply, communities are hurting, and public trust has eroded. You can make sure that your government handles the problem holistically, rather than divvying it up into multiple bureaucratic fiefdoms. You can make sure AI benefits everyone. And you can work with other governments to meet this national and global challenge.

[1] Bill Gates, “The turbulent AI era is here. The choices we make now are critical.” August 26, 2026. https://www.gatesnotes.com/work/make-ai-work-for-everyone/reader/a-turbulent-ai-era-and-critical-choices-to-make

[2] Financial Stability Board, “FSB Chair warns of risks arising from frontier Artificial Intelligence (AI) Models;” 31 August 2026. https://www.fsb.org/2026/08/fsb-chair-warns-of-risks-arising-from-frontier-artificial-intelligence-ai-models/

[3] Amrith Ramkumar, The Wall Street Journal, “Anthropic researcher quits over out of control AI fears,” September 9, 2026. https://www.wsj.com/tech/ai/anthropic-researcher-quits-over-out-of-control-ai-fears-707b7628?st=6eUHMS&reflink=article_copyURL_share

[4] “Anthropic’s Evan Hubinger: >10% chance AI kills everyone this decade, no alignment plan yet.” https://dealroom.co/news/x-2097497037956891126-anthropics-evan-hubinger-10-chance-ai-kills-everyone-this-decade-no-alig/

[5] Dustin Volz, The New York Times, “Anthropic Says it Blocked Possible Efforts to Build Biological Weapons,” September 10, 2026. https://www.nytimes.com/2026/09/10/us/politics/anthropic-ai-biological-weapons.html

[6] William J. Broad and Cade Metz, The New York Times, “AI Could Possibly End Humanity. How are Humans Supposed to Process That?” September 10, 2026. https://www.nytimes.com/2026/09/10/science/ai-humanity-risk.html

[7] In re Boeing Co. Deriv. Litig., Consol. C.A. No. 2024-1210-MTZ, slip op. at 1-2, 21-22, 40, 43-44 (Del. Ch. Aug. 13, 2026).

[8] See, e.g. Michael W. Peregrine, “The Pandemic’s Impact on Board Oversight of Enterprise Risk,” CLS Blue Sky Blog, April 30, 2020. https://clsbluesky.law.columbia.edu/2020/04/30/the-pandemics-impact-on-board-oversight-of-enterprise-risk/

[9] Amrith Ramkumar and Yoko Kubota, “Congress Is Suddenly Waking Up to the AI Doomsday Threat,” The Wall Street Journal, September 11, 2026. https://www.wsj.com/politics/policy/congress-is-suddenly-waking-up-to-the-ai-doomsday-threat-b40ab25a?st=oTKvyQ

Michael W. Peregrine is a retired corporate governance attorney and Fellow of the American College of Governance Counsel. He is currently an Executive Fellow with the consulting firm SullivanCotter.

The views, opinions and positions expressed within all posts are those of the author(s) alone and do not represent those of the Program on Corporate Compliance and Enforcement (PCCE) or of the New York University School of Law. PCCE makes no representations as to the accuracy, completeness and validity or any statements made on this site and will not be liable any errors, omissions or representations. The copyright of this content belongs to the author(s) and any liability with regards to infringement of intellectual property rights remains with the author(s).