To the point where SlateStarCodex had a whole post sneering at it. He seems to miss what I think is the real point, which is not that capitalism is more dangerous than AI, or the inverse. It's that AI (especially in its current form) is made in the image of capitalist rationality; it is in some respects a fever dream of capitalist rationality. The same things that are wrong with capitalism are wrong with rationalism and rationalist AI, because they are themselves aspects of some more general tendency.
That's fair enough, and not all that out of line with what Ginsberg is calling Moloch, now that I come to think of it. Ginsberg says "Moloch whose name is the Mind", but he's not referring to mind in general, but a specific, rational-selfish sort of mentality, Urizenic (see Your Reason and Blake's System ). It's the kind of blind insane mind whose reason leads it astray, that erects vast systems of control, and is blind to the destruction it causes. The industrial capitalists and military strategists who created the Vietnam War and poisoned the environment, and who were guided by reason throughout.
Was trying to get at some major Wrongness that is the root of the smaller-scale wrongness manifested in capitalism, rationalism, and rationalist AI. They seem very closely linked and wrong in the same sort of ways. I don't think this is a super-deep insight or anything, because it's actually kind of obvious, but nevertheless I think there's some value in trying to see just how they are connected and if there is some concept that underlies them all, a unified theory of something I can't quite name yet.
But it isn't Mind as such, more like the aspect of the mind that is narrowly goal-seeking. A stupid form of smartness. What Gregory Bateson was talking about in Conscious Purpose vs Nature
Which I misremembered as Conscious Purpose Considered Harmful, which is what the computer science equivalent would be titled.
, although I don't think there's anything exactly unnatural about it.
Maybe it's also behind whiteness, colonialism, and the kind of modernist oversimplification described in Seeing Like a State.
The meta-problem with all of these, another thing they have in common, is that they are an integral part of our lives and our thoughts. We can call out their faults, but we can't easily escape them.
Update Nov 22nd, 2022
Since writing the above I've read Technic and Magic which has a very systematic theory that seems close to what I'm trying to say here. I could not quite buy into that book's worldview, which seemed to view all rationality as somehow linked in a stifling web of badness. That's not quite what I mean! Nothing against reason as such, its the overapplication of it that chokes the life out of the world. Or maybe I'm just a squish, too sold-out to the forces of technic.