A recent research article  warned that:
“an advanced agent motivated to intervene in the provision of reward would likely succeed and with catastrophic consequences”
First Author Michael Cohen also summarized the contents of the paper in a presentation  and on social media :
In addition, the article considerable attention in mass media [4, 5] because the paper is co-authored by an employee of the world-leading AI lab “Google Deepmind”, and its message was summarized as
“Google Deepmind Researcher Co-Authors Paper Saying AI Will Eliminate Humanity”.
Sounds scary, huh!?
I read the paper with great interest since I have been worrying about the developments in AI since I read the excellent book by leading AI expert Kai-Fu Lee “AI Superpowers” . However, I realized that many features of the described superintelligent AI agent are similar to our current global economic system, which can be be described as “globalized neoliberal capitalism”:
- a standardized reward system, ie the accumulation of capital
- a dependence on the real world, ie (increasing) resource consumption and (increasing) waste production
- a clear conflict between
- the objective of the economic system (“infinite growth”) and
- human interests (“maintenance of the stability of the biosphere as basis for organic (human) life”)
- the astonishing inability to change the trajectory of the economic system which contradicts long-standing scientific evidence of violating the planetary boundaries and bioethical values and even political pledges and ambitions, as exemplified by the escalating climate and ecological crisis.
An outstanding example of this development may be the development of cryptocurrency, esp. bitcoin, which is mostly used for speculative investments, while its societal function is marginal. However, it consumes vasts amounts of energy comparable to entire countries, like Pakistan .
Therefore, I would propose that such an (artificially) intelligent agent may already exists.
One just has to relax some assumptions in the sense that the underlying neural network is implemented in a distributed hybrid form of artificial and natural (human) neural networks and the basic algorithm is encoded in the form of cultural ideas (e.g. economics textbooks). In addition, the agent may be the result of emergent behaviour of the individual (human and artificial) agents, where the assignment of the “an intelligent agent” identity to these system properties may not actually matter, but may be a matter of parsimony and intellectual convenience.
A superintelligent (ie more intelligent as the average democratic voter) agent may provide rewards to individual humans which strive to optimize their outcome within their limited scope (lifes), while actually optimizing the reward of the agent in the long run.
For example, cultural “frames”, such as “trickle-down economics”  may invert the actual reward function on a system level in the long run (> average human life span) relative to the perceived reward function of an individual (human) level in the short run (< average human life span), thereby, triggering humans to constantly act against the long-term interest of the human species in general.
 Michael K. Cohen, Marcus Hutter, Michael A. Osborne (2022). Advanced artificial agents intervene in the provision of reward. AI Magazine, 2022-08-29. https://onlinelibrary.wiley.com/doi/10.1002/aaai.12064
 Kai-Fu Lee (2018). AI Superpowers. Harpers. https://www.aisuperpowers.com/