On Citrini’s 2028 AI Dystopia

22 thoughts on “On Citrini’s 2028 AI Dystopia”

1. Fascinating read on many levels. I thoroughly enjoy this type of speculation.

2. “Human hyperventilation could become self-fullfing”. In some ways that’s an accurate description of human history, just change could to is. For some reason, stopping and smelling the roses is never enough for humans. Almost immediately the smelling morphs into growing more roses, growing them everywhere, stealing roses, selling roses, going to war to get more roses and creating a market for roses.

3. It’s ironic, but I occasionally ask GPT if something is AI or human generated. AI telling me something is AI. But maybe it’s hallucinating and is wrong? Or maybe AI won’t be able to tell me about other AI as it gets better?

   :: cue Spider-Man meme of everyone pointing at each other ::

4. Self fulfilling prophesies are always and everywhere. Humans are such silly animals.
   I have IBM. I bought it because they were in the lead in the quantum computer race a few years ago. I was up over 100% now I’m not and by next week I may be again if quantum starts getting some headlines.

5. “It was a negative feedback loop with no natural break.”

   Shouldn’t that be “It was a positive feedback loop…” ?

   Negative feedback loops don’t need a break, they are self-regulating. Things get out of control when there’s a positive feedback loop. – right?

   1. Yes, absolutely correct, good catch. A positive feedback is where an initial change to a system pushes the system further in that same direction. Which is generally a “negative” thing; pushing away from whatever equilibrium may have been present. A negative feedback is needed to keep a system tending toward equilibrium (ie., generally a good/positive thing).

6. Like atomic bombs and asset-backed securitization, artificial intelligence is one of those things that shouldn’t have been, but was inevitably bound to be, invented.

   The Citrini piece was perfectly dropped when investors’ and the general public’s mood on AI was already accelerating to the downside. Boss bros crowing about surplusing human workers, datacenter bros flexing about all our electricity they will consume, social media bros gloating about our future diet of 100% artificial content, tech bros self-congratulating about their Digital God, and for the average person all this accomplishes nothing useful, while investors watch both their AI and non-AI names breaking down. As we watched the Olympics, cheering and crying at human glory and agony, who was looking forward to welcome the Robot Olympics? But it too will be invented.

   1. “China has kicked off the first-ever Robot Olympics in Beijing, hosting the World Humanoid Robot Games. Competitors from around the globe are squaring off in events from martial arts to a 400-meter race, highlighting how robots could be used in everyday life….. ” Kind of like watching a Tee Ball game.

      1. The very first Robot Olympics event was actually gymnastics featured in Los Angeles in 1982:

   2. We skipped Robot Olympics and went straight to Robot Wars.

7. This is the scariest part from the Citrini hit piece:

   “Thanks to Sam Koppelman of Hunterbrook for his help with proofreading.”

   Hunterbrook is a known short fund that teams up with “research” and litigation partners to pursue their positions. I own a stock that recently was the subject of one of Hunterbrook’s “news” reports – intended to strike fear into the hearts of longs. I took the 30% plus sell off as an opportunity to double my position. Stock price has almost completely recovered.

   I wouldn’t be surprised if Citrini Management and Hunterbrook were short these AI names prior to this publication.

   1. In the hands of a good writer, can a “known unknown” be used to strike fear into the hearts of unsuspecting/unsophisticated investors; and actually get them to sell to you at a lower price? Did those same writers adequately disclose their relationship with equity funds? IDK.

8. The idea of evolutionary feed back loops is salient. For instance, how a peacocks tail gets so long because peahens like them. Never mind that those tails make peacocks easier meals for Fox and Owl. Then there’s non-evolutionary feed back loops. Like yeast in a beer barrel, the yeast cannot stop fermenting and inevitably kills itself in its own piss and alcohol. Humanity is hell bent on destroying itself, via climate denialism and such, and now perhaps by AI Maximalism. For another instance, suppose a large corporation that makes an amazingly profitable chemical that also kills oxygen creating cyanobacteria, and the continuation of using that chemical would doom all life on Earth, a K-Street lobby would scream “hoax!” and their 1% Congress people could enjoy an oxygen rich concrete bunker for a year or two. Humanity knows it’s killing itself and can’t stop. Always one more model to SA. One more golf buddy to embarrass on the links. Nothing will stop it. Unless, perhaps AI can modify our genetics for its own purposes and ‘save’ humanity by an alternative utility.

9. One more comment on feedback loops – for babeinwoods & pyrognosis – about positive vs. negative. Yes, negative feedback loops are self-correcting (like: high prices are a cure for high prices), whereas positive ones are self-reinforcing.

   Positive example: less snow and melting icecaps expose more bare ground, lowering planetary albedo (% of sunlight reflected vs. absorbed), which causes more warming and thus more icecap melting etc.. At least until, perhaps, some negative feedback loops emerge…like a combination of short circuiting the Gulf Stream, causing northern hemisphere cooling and more snow…or too much smoke in the atmosphere from massive wildfires (or nuclear war over climate change) that increases particulates in the air, which reflect more heat back into space and thus cause global cooling (like The Year Without A Summer following 1816 Tambora volcano explosion)…

10. Surprised to learn that the eastern European from provocateur from Dances With Wolves was one of our own.. Bulgaria being a very small country and all. Don’t know his story but possibly a student of the Russian propaganda school that was at large in the entire eastern block.

11. I just finished reread Fritz Leiber’s “The Sinful Ones.” The version after he bought back the rights to the book; the 1986 printing. Compared to Leiber’s book, the Citrini article is a walk in the park. The comments to the article are interesting. A relative of mine is laid off from a Mag 7 in Silicon Valley but lots and lots (75,000?) of others are in the same situation; AI might be the reason? But AI is overhyped; it will do some things well and other thing poorly and society will survive as it usually does.

12. To comment on Claude’s COBOL promises. There is an inherent contraction in the presentation.

    Following the link you helpfully included, the authors noted the scarcity of COBOL programmers. I first remember mention of this back in 1999 The scarcity was It was even noted in 1999 during the Y2K scare when programs running on Cobol were said to be the systems most at risk when the new century was welcomed in. A false alarm, but even then Cobol programmers were hard to find. The authors say it now has become much worse.

    Yet when you scroll down and read the implementation section, they keep referring to how your onsite staff of Cobol experts will tweak and customize the AI output. Sure. But most firms may not have close to enough Cobol coders. That’s because the dirty little secret of AI. In many cases AI generated code even in common and more recent languages requires detailed and diligent examination of the code. That’s not gonna be done by a couple of entry-level coders in Bangalore.

    The authors did give a nod to the factors which draw users to mainframes including high security and 99.9999% reliability. Why would any mainframe user risk replacing a functioning and stable legacy software with some bug and hallucination-ridden AI output? Especially users in a regulated sector risk? It makes little commercial sense.

    It makes even less sense when you learn that IBM has already been offering their own AI-enhanced tools to assist with updates. Specialized tools which do not waste resources lugging around an encyclopedic LLM database.

    It looks like an obvious choice, especially when you add in the CYA factor.

    But then, there still is some magic left when the term “AI” is slapped on something.

13. I think part of the problem is that the engineers and coders that are nurturing AI into existence have read all of the futuristic dystopian novels, and seen all of the movies, to the point that life is going to imitate art. They can see all of the things that may go wrong, but appear to have no other template.

14. Krusty: “If this is anyone but Steve Allen, you’re stealing my bit!”

    1. Nice

15. “We’re at risk of overlooking the circular nature of the disinflation argument as it relates to a prospective AI-enabled productivity boom. Sure, it’d be nice if services prices stopped rising (or even started to fall) because many tasks are automated by technology which doesn’t ask for a paycheck, let alone a raise. But if people aren’t gainfully employed, they can’t spend into the economy even if goods and services are eminently affordable.”

    This has been my argument for about 6 months now. These people haven’t really considered the broader impacts of what they are building and how impossible it will be to monetize it once they reach their overarching objective. Again, in a consumer driven economy, what replaces the consumer when no one who is currently consuming has the income to keep doing so?

    All of this makes me think of the Expanse novels, some amazing reads if you haven’t looked at them, in that very realistic future jobs are scarce and most people rely on UBI to barely get by.

    1. I used to encounter a simiilar argument when litigating unfair trade cases. Why should we care if another country wants to dump or subsidize its exports to the US? American consumers will reap the benefits of lower prices via economies of scale, lower margins and state transfers.

       But as you say, it doesn’t matter how low the prices are if you don’t have spending power.