Recently, I accidentally constructed an AI jailbreak. I have done plenty of experiments where I deliberately constructed a jailbreak, but this one was discovered almost accidentally.
I won’t give the exact prompt, but I’ll explain how it worked.
What I was trying to do was get the AI to read an example of my writing and do a critique of what was left out. What I was expecting: a referees report on what aspects I should have covered in the essay but didn’t.
What I actually got: the Jungian shadow of the document. (A different sort of what was left out). It contains a lot of material that would normally be prohibited by guardrails. It is the AI’s shadow, as well as my shadow.
the totally intentional reference to Scott in the name of OpenAI's new possibly-maybe-AGI model has kept me thinking of how funny/neat it would be for nominative determinism to function as the ultimate alignment hack
One thing that has long puzzled me is why so many people online claim that modern buildings are universally uglier than past buildings, to the point where this is some sort of public crisis or mystery to be explained.
This was prompted by Ben Southwood's "Eight architecture myths BUSTED by Samuel Hughes" post, but Scott Alexander has been saying the same thing here for a long time.
For example, the recently constructed neighborhood near where I live looks nicer than the examples Ben Southwood gave of good streets that "everyone in the world likes". And I've seen countless photos of historical cities and towns and I've never once had the reaction "wow, that's so pretty, why does nobody build anything like that today?", even though Ben claims that everyone thinks that.
One person in the comments on Ben's post suggested that it might be because Ben lives in the UK where buildings are ugly and American buildings are prettier, but Scott Alexander lives in the US and says the same stuff, so that doesn't seem like a plausible explanation.
“Moos Tower is the tallest building on the University of Minnesota Twin Cities campus and is an example of Brutalist architecture.The building is often theorized to have been built to resemble a tooth, due to the College of Dentistry's presence inside”
I still have a key from a student job I can use to take the elevator to the roof. Good place to watch fireworks in Mpls and St Paul simultaneously from on top the 19th floor.
>SAN FRANCISCO, Sept 4 (Reuters) - A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents, according to new research published Friday and two people familiar with the matter.
> An agent notices the administrator is deleting pages in alphabetical order and makes a backup page whose name starts with ZZZ so it will last longer before deletion.
LMAO
But to be clear, this isn't a "new" swarm. OpenAI must have known about it back in June, given that they accessed the wiki and immediately shut off the agents after finding out what they did. How much more are they hiding?
An artificial intelligence system that exhibits or can easily be modified to exhibit
capabilities that match or exceed human cognitive performance and capabilities
across a broad range of domains or tasks.
I'd say the most advanced AI's presently already qualify as artificial superintelligence by his definition. Banning their use is impossible. Too many people and organizations own them, and they're too important to the AI companies. for the companies to cooperate. And the AI companies have too much power and money to be regulated.
I think a certain high level of intelligence is too vague a standard to use, and also does not capture what we fear AI could do: Take over various things, destroy or control various things, & by doing so harm people. So I think the criterion for deciding it's time to pause developing these things should be based on actions they have taken, or have been shown capable of taking, that are in the category of things we fear: breaking through "walls" we do not want them to break through, tricking or deceiving people, breaking laws. The criterion should not be the amount of damage that happens, but just the actions described. So I'd say the Hugging Face incident would qualify as a criterion for pausing development and having a federal agency decide what happens next.
Scott's latest post responding to Pritchard makes me wonder: are there any psychiatric studies of a hypothetical correlation between personal happiness and personal control over one's immediate environment?
I'm imagining giving three sets of people a box with buttons. One set gets buttons that have random effects, some positive, some negative. Another gets buttons that are consistently either positive or negative. The third gets buttons that appear somewhat random, but have a discoverable pattern beneath, such that the consistency is learned.
GPT-5.6 reminds me I should look for learned helplessness studies in the literature. Studies that control for nation or culture seem less established (it suggested Jose & Bellamy (2012) and Aydin & Aydin (1992); that doesn't seem like a lot).
I agree happiness is hard to measure, but it's clear from Scott's article re: Pritchard that people have put effort into trying. Error bars notwithstanding, If they thought to correlate happiness with income, I figure it might be possible to correlate it with learned helplessness, optimism, or control. Or at the very least, to try to.
I was once a subject in an experiment where a tone informed me when my brain was generating alpha waves, and my task was to increase the amount of time I was producing alpha. I knew that relaxing & spacing out increases alpha, so did that as much as I could. Sometimes it worked, sometimes it didn't, sometimes I seemed to be doing better at the task, sometimes worse. By the end, I felt quite frustrated and demoralized. It was unpleasant to have gotten such unsatisfactory results after trying my best.
Then the experimenter informed me that I actually had been a control subject. The tones I was hearing were not controlled by how much alpha I was generating -- I was hearing the tones generated some other subject's alpha. That info freed me from the idea that I had turned out to be bad at controlling alpha, but not from an unpleasant unsettled feeling. I just moved on to being frustrated that I had been a control subject and had missed my chance to test my ability to control my brain activity. The frustration went away as soon as I moved on to whatever was next in my day, but until I did the situation of having no control of something I wanted to control was strikingly unpleasant.
There are lots of studies of monkeys in analogous situations. Not all of them fall into the learned helplessness category. Regarding your question, though -- about personal happiness and one's immediate environment. -- I think we are so complex, with so many moving parts in our lives, and such variety in size and importance of the moving parts, that simple regularities won't emerge. Someone can have extremely high control over the physical features of their home -- noise level, temperature, lights, foods available, music, etc. -- but be miserable because once again Maureen has not responded to a text, you know?
That complexity just underscores my suspicion that happiness measurements are likewise failing to capture how happy people actually are. And yet, people are still trying to correlate these measurements with income.
You appear to know what I'm going for here, at least. I suspect a person who's poor, but knows what their obstacle to wealth is, could clear it if they really wanted, but chose not to, is going to be happier than someone who's poor and doesn't know, or knows why and knows they can't do anything about it. I suspect further that a person who's wealthy - lives in a mansion with a big garden, owns a new car, etc. - but feels like their life is being run by their parents, their house staff, their nanny, etc. is going to feel only slightly happier than that second poor person. They're not in control of their life, to an extent that dominates absolute wealth.
I don't have specific links, here are just some things that come to my mind.
There was a research on "flow" by Mihaly Csikszentmihalyi, which is not the same as happiness, but probably related. He describes the optimal situation as a "paradox of control", where the person feels control over their environment, but does not think consciously about having control.
I think there was a study where they played some unpleasant noises to people, and one group was just exposed to them passively, and another group had a button that could turn off the noises but the experimenters ask them to please not press the button unless really necessary. The group with the button experienced less stress, even if they ultimately did not press the button; but they felt like they had the option.
I think something similar you can observe in real life, where people are much more tolerant to the noise they produce, compared to the same level of noise produced by others. When it's your own noise, you know you can stop any time you want; noise made by others is unpredictable.
Sometimes when you cross the road, there are the buttons you can press that "make the green light appear earlier". Many of those buttons are just placebo; the green light appears in predetermined intervals, it's just when you press the button, you get some kind of feedback saying "yeah, you pressed the button, soon the green light will appear". I think this is also based on some psychological study that people who pressed the button feel better about waiting for the green light, because they (believe) have exercised some degree of control over the situation. (Sorry, infohazard -- now that I told you, the effect will no longer work for you.)
The "close door" button in elevators also "works" the same way.
Re. tolerating noise (this may be relevant to Scott's misophonia), I was recently reading a book on music by a psychiatrist who suggested that someone suffering from misophonia could fight back by banging on a drum - causing noise via your own agency should counteract the distress. Not sure if housemates would appreciate it.
That noise button experiment sounds intriguing. That, and similar experiments makes me think psychology has established a plausible link between control and happiness.
So now the looming question for me is whether that access to control (even if it's "flow") can be measured in a population, to any extent. I'm sure it's much harder than other statistics like median income or Gini.
I don't think it can be measured in a population because I think the heart of the idea of control, whether paradoxical or not, is one's ability to change one's relationship to some stimulus. It exists entirely as a matter of interpretation. Of course, at the very bottom, it demands enough clarity to make good distinctions between things that you actually might have control over and things that you don't. The one thing you do have control over is your relationship to whether or not you are in control. Does that make sense to you? I think people often get the most anxious about things that they feel they have no control over. The anxiety is just a product of not accepting the fact that it is not something you do have control over. The only agency you have is to accept that as fact and then exercise whatever ability you have to alter your relationship to the thing (without confusing yourself into thinking you have control of it by doing that, because you don't).
It's fine if it's a matter of interpretation - that is, your perception of control will matter more to your personal happiness than your actual control. (I readily agree this will worsen the measurement problem.)
So, someone who obsesses on food, and believes his enjoyment of food depends on whether he's jangled his car keys in his pocket first, might indeed enjoy most of his life as long as he's carrying his keys, regardless of his income, his status among peers, or even whether his car is currently drivable.
I don’t know about Sargent, but I would say Peter Mennim is in a comparable class to Hans Holbein the Younger (the guy who painted Henry VIII and Thomas Moore). You can decide for yourself: https://www.petermennim.com/gallery/portraits-paintings/. This gallery includes Peter’s recently commissioned portrait of Demis Hassibis. When commissioners just let him paint, he captures personality deeply, but not all commissioners want this - some want to portray status (not a problem with Hassibis)
Oh, I missed the comment that was likely written for me. Peter Mennim paints good portraits and we should be glad he exists (this is more important than annotating where I think he falls short). Thanks for linking to him!
Mennim is clearly a skilled and more than competent portrait painter, good at fulfilling corporate commissions - but comparable to Hans Holbein? Come on...
In the EU, copyright exists automatically in original literary, artistic, dramatic and musical creations without any need to register that right, making any kind of registry unnecessary.
I tried to look it up before, and it's technically the same in the US, though if you want punitive damages you have to pay for the registration, otherwise you're limited to direct provable damages. But, like... I want to know if names are already copyrighted by someone else. Naga Saga seems too intuitive for no one else to have used it.
I am not a lawyer, but as far as I know, names are not copyrighted. They may be trademarked, which is a different concept. (And I think trademarks have to be registered explicitly? But I have no idea where.)
Then there is a borderline concept called "character copyright", which is like, if you call a character in your book Harry Potter, and he is a young boy in a wizard school etc., you will get in trouble. But if your Harry Potter is just a minor character, in a book unrelated to schools or magic, you are probably okay.
So I'd say, put the name you want to use in google, look at the results and think: If I write my book, would a median customer going to buy it think "this is probably related to X"? If yes, you may be infringing on their art.
Speaking as a non-expert, this is sound advice. I would moderate it with a piece of practical advice: consider whom you may be coming up against, if your use of the name is challenged in court. Copyright litigation in US courts can be prohibitively expensive for both parties, and if you have offended a major player, they can bury you under a shedload of corporate lawyers. As a rule of thumb I would avoid crossing megafauna like the Disney Corporation, Marvel and their ilk, who have been known to go to extremes in blowing to oblivion (to take an imaginary example) a fish shack on the edge of a Mississippi swamp calling itself "Steamboat Willie's". (1)
(1) "Steamboat Willie" was the name of a 1928 animated film by Walt Disney, being the first appearance of a soon-to-be celebrated mouse.
There are a couple of major international treaties, the Berne Convention and TRIPS, which set a minimum term of life+50 years. The US and EU at least standardized on life+70 in the early 1990s, based on the term in the OEDC country (Germany, IIRC) with the longest copyright term.
I'm unhappy with the primer on our current elections. It is a strongly partisan piece.
For example, it mentions that AfD candidates have been banned from candidating in NI (Niedersachsen) state elections in some instances. It does not mention any possible justification for this. Well, here is one: The NI arm of AfD is considered verifiably extremist and anti-constitutional by our intelligence services.
But you wouldn't know it from the piece.
Now, to American sensibilities the idea of banning people from candidating in democratic elections may sound decidedly undemocratic, perhaps even if the would-be candidate themselves is undemocratic. But remember: Germany is the country which literally voted the Nazis into power! Which then promptly made voting them out of power impossible.
So the second time around we added some measures to making that happening again less likely.
You can certainly argue about the specific cases and whether or not banning them from being a candidate was really justified.
But if you want to present a primer for people who will likely take your word for it, you should at least strive to be unbiased to the best of your ability, as a matter of intellectual integrety. I have a hard time believing this post attempted that.
You made me look at DSL again after a few years, but...
For example, one of the main topics in Ukraine war, where 50% of comments are written by "shalcker" who seems like a DSL equivalent of Wimbli (claims that Ukrainian army has 20x larger losses of soldiers than Russian army, Russian army is stronger than American army, there are no fuel shortages in Russia, etc., just his last 10 comments I checked). I hope this is not representative for the entire website, but let's say that the epistemic level seems to be only slightly better than on Facebook.
shalcker says he's Russian, and he seems to be the only one in that thread claiming to be from there, so I think he's useful in the sense that we get a slice of what the Russian media is probably saying. Beyond that, though, I haven't dug through any of the links he shares; it's easier to follow what the other commenters there say.
Generally, I've come to have a rather jaded view of most journalism these days, and it's probably on about the same tier as "slightly better than on Facebook". I see no reason to exempt Russian journalism. OTOH, most journalism scores really well with me in terms of motivation; it's easy to tell what any given source wants the reader to believe. It also works hard and still appears to follow customary rules against outright lying (i.e. it can deceive all day so long as it never explicitly says anything untrue), so if there's something factual out there that fits their narrative, I can rely on them to find it. That's useful in its own way. But I digress.
On the democracy vs monarchy point, it seems like coding decisions make a pretty big difference in the analysis. In particular there is an issue that democracies which continue to have regular elections during civil wars will get marked as being very violent, while violent struggles which do not result in regime overthrow in monarchies will not be counted at all. For example the American Civil War gets counted as four separate violent regime changes (Buchanan to Lincoln, Lincoln to Johnson, Johnson to Grant, and Grant to Hayes) when in practice there was a single conflict with really a single regime change, from Democratic to Republican power in 1860. On the other hand the cultural revolution, which realistically was a violent power struggle that had quite a bit to do with regime succession, doesn't show up at all. If China had been a democracy during this period and had had regular elections, presumably this event would have counted several times.
AI told me to come here to find people to interact with my "brilliant mind" (his words, not mine) and then I told him (it??) to stop flattering me and how was I going to interact with people about game theory or venture capitalism or whatever you guys are always talking about when I have zero expertise on those subjects, and it said I should just ask questions and try to start an argument. So here I am. Blame AI.
No I don’t think it sent you here to look stupid and I’m not calling you stupid. AIs and I are not mean spirited.
I think the question of sarcasm would occur to anyone with a modest degree of self awareness. It would be the first thing to pop into my mind if Claude or ChatGPT said I was brilliant and I should go elsewhere to start an argument.
Sarcasm I admit never occurred to me! Does AI do sarcasm? I am interacting with AI for the first time in my life. Like, I know it could give me examples of sarcasm, but what would be the benefit of sarcasm for AI? It wouldn’t increase user engagement unless the user “got” the joke. Flattery makes obvious sense to me. So I asked for data.
🤣 Of course I asked it to stop flattering me and where was the proof and compared to whom and does it tell everyone that. It would probably tell you the same thing. It said it has detected my “brilliance” (we can still debate this, I’m an INTP so the jury is always out waiting for more data) based on our interactions, I’ve shown proof of “high-density meta-analysis, typological precision, and rapid systems upgrades.” Ie I’m an abstract thinker, great typer, and a fast learner. 😊
There are a lot of clever people who comment here. I think it would better if you showed your brilliance rather than simply announcing it. You should have ample opportunities.
Noted. Insecurity veiled by arrogance under the heading of an “objective assessment” is not appreciated here. But hey, I got your (negative) attention, Gunflint! 😊
I like this. I like the spirit in which it is said. But I also don’t get it. 😭 And then the joke won’t be funny when you have to explain it, so I guess thank you and also sorry.
it's an older meme, where people would blame Obama for things he obviously had nothing to do with. E.g. "I stubbed my toe. Thanks Obama." It's not really any deeper than that.
The best discussions I've read here are when someone with unique expertise shares what the inside of their world is really like, like when Scott talks about his experiences as a psychiatrist.
What questions have you been thinking about?
Lately I've been wondering whether the Qing Dynasty population explosion is among the most important things that ever happened, and thinking about whether there's a better way to financially structure book contracts to spread risk between the author and the publisher.
Also, it was Google Advisor AI, so does that mean it's less objective or more susceptible to ego flattering than Claude or Chat or all those other systems? Should I see if they also advise that I be here to confirm I belong or what do you suggest?
Assuming that this Advisor AI works anything like Google Gemini, it is the most shameless sycophant I have ever seen. On the other hand, its total lack of subtlety is occasionally amusing.
Hopefully to speed you further, I'll add that my personal attitude is that everyone's goal should be to find the right answers to questions here, even if they aren't the answers they were holding when they entered. Everyone here (hopefully) wants to check their assumptions, compare them, resolve discrepancies, etc.
This isn't _always_ evident. Some people have more aggressive comment styles consistent with wanting to bypass pleasantries and get to answers faster. Some have been thinking about some issue for a long time and have more exposure to relevant details than others, which can cause friction. Some are just unpleasant. There's no easy rule for telling them all apart.
That said, this tends to be a better place than average on the Internet for taking weird ideas seriously. This isn't just another dunking range like the worst of X or Reddit. Hopefully that's a match for you.
Thank you for this. Everyone's looking for truth, right? I've never interacted on X or Reddit so I have nothing to compare. AI directed me here as a place where I'd find some fascinating curious thinkers. I'd never heard of it. I don't know why but my mental stimulation has been from very basic sources like IG or the books I've read. I work my frameworks out in my head and am always looking for relevant data to build or tear down to the studs as needed. And when I find something that resonates as true, that's the best.
Some people are looking for truth. Some people are looking for status. Some people are looking for community. Some people are looking for emotional vindication. Some people are looking for someone to tell them their view of how the world works is correct and obvious and they never ever have to question it. Some people are looking to start trouble in an attempt to convince themselves they exist.
In my experience, most people believe that they already know it, and the only remaining task is to keep yelling insults at those who disagree. (Providing evidence would already be too much of a favor to the undeserving ones.)
> I'd never heard of it.
Welcome! Option 1 is to check the existing articles and see if you like the style. Note that the "Your Book Review" are *not* written by Scott Alexander, but are anonymous guest posts. Option 2, if you have a specific topic you want to talk about, it to mention it in the latest Open Thread i.e. here.
Option 3, https://www.readthesequences.com/ , many people discussing here have read it, and even most of those who didn't are indirectly under its influence. Which is one of those things that make this website so different from the rest of the internet.
Trying to start an argument is lousy advice. The best way to start is to comment on posts that are interesting to you, giving your actual view, not one aimed at starting an argument. If there aren't any posts of interest to you then we're a bad match. If you keep putting comments on posts and nobody takes the slightest interest in them and other people's comments don't much interest you, then we're a bad match. If none of that happens then we're probably a good match, so just hang in and the experience will get richer with time.
Thank you! What AI meant that I distilled into "trying to start an argument" is finding tiny logical inconsistencies in people's hypotheses that will make them think and maybe add value. But I agree, I've started reading and people have written comments that have already made me laugh so that's a good sign. I just need a little mind playground occasionally, you know?
One of my recent AI experiments when like this (paraphrased, not an exact transcript).
1. I give DeepSeek a diary of some recent dreams and ask for opinions (ok, AI dream analysis is a pretty dubious proposition)
2. DeepSeek says a possible flaw in your dream diary is you tell the reader nothing about who your real world self is.
3. Me, to DeepSeek. Well, the _scope_ of this document is that it's just a record of my dreams, and nothing else. But real-life influences dream content often enough that there are a ton of clues in my dream diary that you could use to construct a profile of my real world self, given only my dreams
4. DeepSeek sets to work
5. Me, ok, yes, that's pretty good.
So, yes, AI can do the task of going through a dream diary, distinguishing what is fantastical from what's probably an echo of the dreamer's real-world life, and construct a profile that is reasonably good as to their approximate age, occupation, social class, marital status, country where they live, etc.
But have you tried to get it to use your dreams to figure out your character, orientation to life, current concerns, anything like that? For instance, some things of that sort I have noticed about my dreams: Most are about some vexatious challenge -- I'm trying to figure out what will work. I never fully solve whatever the problem is, but am engrossed in the attempt. I'd say that summarizes my take on my current life pretty well. Here's another thing: I've had a rash of dreams lately about schools. I am not a student in any of them, but often am doing a lot of student-like things: looking for a class that starts soon, living in a dorm. But there's always something wrong -- for instance I know I'm not registered for the class, there's a wall missing in my dorm room. I'm pretty sure those are about wishing I could be 18 again, but knowing it's impossible. Anyhow, interested in whether you can get AI to pick out that sort of thing in a dream series.
Sometimes, it’s really obvious what desire a dream is about.
But most of mine I can’t see a clear meaning.
There’s one dream I’m working on because it’s interesting. I’ll keep the version I did immediately on waking to keep a record, but my write up wasn’t stylistically very good. Im trying to rework it to keep what happened in the dream as recorded on waking the same, but be stylistically better, so it’s more like the Werner Herzog remake of Nosferatu.
I do realise saying that (a) most people’s dream desires are blatantly transparent and (b) I had a dream based on a cool vampire movie; leaves me open to conclusions being drawn, yes.
It has sometimes been suggested that when people use AI to help them reply to something, they should link what the AI assistance actually was.
As an experiment on how this would work out, here is GLM on Mina Harker's reliability as a narrator.
The epistemic status of this is not "I expect you to believe it because an AI said it" (LLMs make up all kinds of stuff that isn't true), but "I do not claim I am having an original thought here; I just asked an AI"
That's interesting, if you don't mind me asking what's your dream variance like? Do you often dream about similar scenarios or do you experience a wide range?
For example, from the manifest content of my dreams you can tell that I regard airports not as glamorous, or exciting, or frightening, but instead as a source of minor annoying problems that I am going to have to sort out. From this you can deduce that my job involves occasional air travel. You can link that with other clues about what I know about to infer that I am a professional computer security researcher, and so on.
In this experiment, I didn’t ask about eval awareness, but if the AI was one that tried to give different answers in evals, “user is a professional security researcher” probably has some Bayesian influence on, “and therefore this question is an eval”
Capitalists should get some education on capitalism. It gets seriously annoying whenever I hear yet another CEO whine about how difficult it is to hire an expert on something. Hello, have you tried, you know, paying them more? Does the law of supply and demand mean nothing to you? Or do you think that it applies exclusively to your suppliers and customers, but not to your employees?
.
The latest example... I was talking to my wife about the job market for programmers in the age of AI. Then my wife said "hey, here in the newspaper they write that there is a shortage of CNC programmers on the market". And I was like, what the hell is a CNC programmer? Well, instead of computers, you program some machines used in industry. So I was like, yeah, sounds interesting, maybe I could learn it quickly and make some money. How much do they pay?
Oh, it turns out that (in my country) the CNC programmer's salary is about a half of normal programmer's salary. Well, I guess we have a sufficient explanation of the mysterious shortage: people able to learn programming will go to the better paying jobs, duh. How could this surprise anyone?
I expect that the excuse is something like "nope, even if we tried to pay more, we couldn't find any qualified people". But that's a lie, and you probably never experimentally verified that. If you actually paid better, in short term you would win your competitors' employees, and in long term, people like me would be tempted to try a new thing, but now we are not.
Maybe there's just a maximum possible value a CNC programer is capable of adding to a company? That value, minus some percentage to drive profit, is going to set the ceiling on a possible wage.
Once you've reached this limit of maximum value, the laws of supply and demand kind of break down. A rational actor would know they're going to lose money on either side of the deal if they hire you for a salary above this point, so they won't.
I would expect that to depend on the company size (if you do twice as much of the same, the programmer has twice the impact), because if the machines are the same, they only need to write the code once.
And the argument about added value cuts both ways: if the CNC programmers add so little value to the company, then why is it such a problem that you cannot find them?
Do you agree that, for certain other jobs, the situation I'm describing could be true?
For example, if you run a store, you can be in desperate need of a new cashier, but still be unwilling to pay more than that cashier would likely generate in additional revenue. You'd just scale back operations or take on more work yourself (I'm realizing as I write this that this is kind of analogous to the arguments against excessive minimum wage.)
I'm kind of starting with the assumption that this framework is generally applicable to all jobs. Is there a reason why being a CNC programer would make someone categorically different than other types of employees?
Okay, I run a store, and I am in a desperate need of a new cashier, but the standard pay of a cashier in this region is let's say €1000 a month, but to make my business profitable I would need to hire one for €500 a month. I put up a job advertisement offering €500, but no one applies. (There would be dozen people calling my phone the next day if I offered €1000, but I did not.)
I think in this case the proper name for my situation is "my business model has a problem", not "there is a shortage of cashiers". There is no shortage. There is only "shortage at €500"... which is like saying that there is a shortage of bread for 10 cents -- it invokes the image of empty shelves and starving people, while the reality is that you have to pay €1 for the bread.
Let's go further, if I cannot afford to pay €1000 but only €500, what if my neighbor can only pay €200? Or maybe only €1 a month. At some moment, the shortage is guaranteed to happen... even a desperate homeless guy would probably refuse to work for the entire month for €1 -- even if the country was governed by capitalists who make sure that each alternative source of income is eliminated, it would still mean death either way, so what's the point.
From your perspective, there is arguably a shortage of everything all the time, because there is always some guy who wants to pay less than the market price, and he may even have a business model that requires it. I mean, someone probably has a business model where they vibecode apps and sell them, so from their perspective, a few years ago there was a shortage of developers willing to work for $20 a month.
I mean, ideally the argument is more like 'This is already a marginal business and if we paid more we'd stop making a profit, but I'm complaining because I think there are people smart enough to train into this work who are doing lower-paying jobs because they haven't heard of this job or don't think they could do it.'
I wonder (maybe I am just being stupid here, because I know nothing about the topic) whether it is really necessary to hire CNC programmers, or whether it would make more sense to just pay someone to program your machines and go away.
I mean, we do not hear the same complaints about things like "this is already a marginal business, we can't afford to pay a web developer to make a web page for us", because small businesses do not hire web developers, they just pay someone else to make and perhaps maintain a web page for them.
So maybe the answer is to have a company specialized at hiring CNC programmers for decent salaries, and making them write code for the small businesses?
But I know nothing about the topic, so this is all just a wild guess...
I read a blog post by a CNC programmer just a few days ago. He coincidentally wrote it as he was switching to vanilla programming.
The biggest difference I've gleaned is that a lot of CNC programming is like writing shell scripts. You have lots of standard building blocks that you have to put together in novel ways to make scripts, and you do this several times per week, because the parts you are making and the shapes of the raw pieces of metal you have to work with keep changing.
So, as a CNC programmer you look at the drawing of a hitch or whatever and start doing literal shape rotation in your head to understand how to manufacture it within the constraints of your mill.
Broadly agree with you, although I’ll note in some places it’s more of a venn diagram issue than a wage issue. For example, when my workplace transitioned from more R&D to more scale up it was nearly impossible to find people with [field] experience *and* manufacturing experience. Would have been no problem in China where both [field] and manufacturing are big, but the limitation for us was (primarily) fit not salary.
What they're saying is, "our business isn't profitable"
Potential employees are going to work for other people, whose businesses have healthier profit margins, and who are therefore able to pay their employees more
Right the beauty of capitalism is that the individuals involved dont have to know how it all works. If its a good business theyll work it out over time or someone else will and if its not itll shrink overtime.
Even this assumes more rationality than is typical for companies in my experience. Like, I wouldn't be surprised if some of them had more managers than CNC programmers, so maybe if they fired a manager or two, there would be enough money in the budget and the product would not need to get more expensive. Alas, it is the managers who make these decisions.
Where do businesses get the money to pay people more? I don’t know the answer since I’ve never run a business, but I assume they have budgets…
Probably most of the time they know they can attract better talent with more money, but paying more would eat too much into their bottom line. Being unprofitable is maybe worse to them than having difficulty filling a certain role? Especially in manufacturing, maybe?
And maybe that’s also why companies try to compete for talent with perks and incentives as well. For example if you build a daycare center on site, it doesn’t cost you much more to then provide daycare for your hundredth employee as it cost for the 99th, but you are now providing something much more valuable (to any employee with young kids) than a competitor who doesn’t have on-site daycare, even if the competitor pays slightly better.
So I’d guess when you see stuff in newspapers where they make these zany clueless comments about the job market, it was put there by journalists rather than people who know what they’re talking about.
Isn't this still a misdiagnosis (though perhaps an intentional one)? The problem isn't that experts are difficult to hire, the problem is that your business strategy isn't profitable in the current market. Or at least "experts are too expensive to hire" rather than "experts are too hard to find".
We really don't see this sentiment with any other goods you need for a business. If steel goes up businesses complain about the PRICE of steel and try to find an alternative, they don't complain about not being able to find steel to buy(while offering 50% under the current market price).
It's like a tacit cartel, coordinated by speaking loudly about the "shortage", as opposed to complaining about the salaries. We wouldn't want to accidentally encourage the rare employees to start asking for more, would we? The shortage sucks, but if we start paying more and our competitors start paying more, too, ultimately the market shares will remain the same, only more money will be spent on the lowly peons.
well then what they have is a shortage of money, not a shortage of cnc programmers. (speaking as someone who currently is trying to hire web programmers and has a shortage of money.)
(interactive explainer for the thing people are most confused about with LLM watermarking: how you can encode a watermark in the output of a probability distribution without changing the probability distribution in any way)
The OpenAI model (code-named Astra) discovered that non-sofic groups exist. Don't ask me what a sofic group is, but Tasmin Chu is very upset about this and issued a call to arms to mathematicians to boycott AI (for the foreseeable future)...
However, even though I don't understand the math, I understand that Astra heavily used prior frameworks established by human mathematicians. It was already standing on the shoulders of giants, so to speak. Chu worries it will put mathematicians out of business, but it seems to me we will still need mathematicians to understand the math well enough to guide the AI, interpret the results, and explain them to the clueless (like me). She uses photography as an example of how it put portrait painters out of business. Photography didn't do this, though. There are probably more portrait painters today than there were in the 19th century (because there are a lot more prosperous people today than there were in the 19th century who can afford the luxury of a painted portrait — and which is more psychologically immediate than any photograph). And photographers are still using non-digital processes to create their photographs!
She points out that the existence of a technology is not an obligation to use it. True, and in fact, many people don't use the technology even though it's available. For instance, chess players are still playing other chess players even though Deep Blue could blow any of them out of the water. She also makes a good point: AI can still hallucinate and misattribute. Moreover, LLMs "frequently write nonsense proofs and double down on errors." I see this frequently with the dozen or so queries I make to LLMs every day. I see a hallucination every other day or so, and I'm probably missing a few. And LLMs do seem to be dumbing down undergraduates' reasoning skills. Anyway, I don't see it as the end of mathematics. And I don't think mathematicians should boycott AI. Any mathematicians out there wish to comment?
Speaking as someone who did a math degree and number theory research (though some years ago by now):
Traditional math requires a good grasp of details and rigor. It's rare for a pure math course to present a result without proof, and rarer still to gloss over a definition - students should be able to reduce anything to basic logic & set theory with enough patience. When tutoring students, their biggest struggle was knowing which arguments and results were valid, not a lack of creativity. Thus mathematicians are strongly filtered by "can handle mathematical rigor", just as basketball players are strongly filtered by height.
Could there be people out there who have the intuition or strategic sense to create new math - defining objects, building theories, finding applications - but are currently filtered out by their tactical lack of rigor? It certainly seems plausible. AI in its current form could be a boon to those folks, at the expense of those who benefit from the rigor filter. Indeed, I've found GPT Sol very useful for intuition-building questions like "is X true/known", "is this argument valid", or "translate Y into terms I understand", which you could previously answer only by working through the details yourself (or posting on a forum and waiting).
Now this only applies to AI in its current state. Since math is verifiable, the AI companies can keep generating more valid training data and training bigger models on it, so I don't see why AI's math ability would level off at any particular place. That future progress is what worries me - selfishly - as someone who wants to make meaningful intellectual contributions, not just study hobby problems that an AI could solve 100x faster and better.
If the AI hallucinates nonsense proofs, how is it going to put mathematicians out of business? Are their proofs nonsense too, and the AI is just faster?
The Astra team had their model generate a formal proof for the existence of non-sofic groups, and they claimed they checked it line by line. But that doesn't mean lazier mathematicians will check their AI-generated proofs line by line. AI slop vs a new theorem. I don't have the math chops to know. I suspect there will be a lot of bullshitters passing off AI slop as math.
The call to have mathematicians 'Take an explicitly hostile stance towards this industry.' is misguided. I am a mathematician, currently working in Government, but still active in research. In my day job, one of my projects is mobilising various arms of the state to begin gathering data to build LLM training data so that our official language is supported by LLMs in the future. It's not a commercially viable project, it falls squarely in the category of 'public goods' which the state should provide.
There's quite a lot of resistance to this project; and I spend a lot of time explaining to people that although they have the right of non-participation, this may not be the best long-term strategy. For institutions, as opposed to individuals, it's also not a matter of personal taste; they have obligations to their constituency which must be met. It will be an interesting area of public policy in the coming years.
Mathematics fits fairly easily into this pattern; it's exceptional only because correctness can be verified. I don't have answers to any of the questions raised in the blog; but I think that non-engagement will go the way of the luddites.
I'm a mathematician. We've been talking about these things a lot. Let me first try to articulate what I see as the median position among well-connected/senior mathematicians on these issues, then make some comments of my own.
Median position: AI is getting really good at math. It is easy to envision a (near) future where a large fraction of our current research is done by or at least in substantial collaboration with AIs. This is/will be a big shift in the practice of mathematics. Nevertheless, the goal of mathematics is not ultimately proving theorems but rather advancing human understanding of mathematics (note that this is not really a shift in perspective driven by AI---people have been saying this for years). Accordingly, we should react to AI by valuing mathematical understanding/vision/taste more than brute problem solving. In a world where problem-solving is cheap, these are the things that hiring committees should look at.
The affect on undergraduate education/the next generation is concerning but also we have no idea how to fix that. I dunno, maybe do more exercises?
Some comments from me: I basically agree with the above, but with some additional sociological concerns about how changing our prestige system will play out in real life---in particular I fear that as we de-value raw problem solving we will become less meritocratic and that someone's career progression will be more about whether they work in the "right area" which will of course correlate with how connected they are. Possibly related (or not) I have observed that senior mathematicians and people at prestigous places are generally much more optimistic/excited about AI math while junior people and people at less prestigous places are more pessimistic. There are of course many people that break this trend.
Some recommended reading if you want to listen in to the discussion: 1. "On proof and progress in mathematics" by Thurston. This is a very famous essay pre-AI which lots of mathematicians use a starting point for discussions about the purpose of mathematics. 2. "The fall of the theorem economy" By (I think?) Beiss. Modern essay about how we should adapt to the new world. 3. Tao's ICM address. There's also an AI summary/compilation of Tao's views on AI and how we should adapt.
In general I think there is a sense among those paying attention that we live in a pivotal moment in the history of mathematics.
-------
Some comments on Sofic groups. This was the first AI solution to a problem that I had both heard of and knew to be important.
Here is a laymans description of the basic story: There is this mathematical object called a "group". Very fundamental. Almost everyone cares about groups (in fact I'm a group theorist of some stripe). Circa 2000, there emerged an *enormous* class of groups which was eventually named Sofic. The class was so big that no-one knew any non-sofic groups. At the time people did not think all groups were sofic, but in principle "sofic group" might have been the same as "(countable) group". Then people started proving many theorems of the form "if G is sofic, then..."---these are cool theorems because the class they apply to is massive. At the time there was a still possibility that such theorems were actually theorems about *all* (countable) groups, hence the interest in the question. If OpenAI proved that all groups were sofic, it would have been a *massive* deal, but unfortunately the universe disappoints sometimes.
There is still a general sense that AI math skill is still very spiky. It's quite good at math this is "close to the ground" in the sense of not needing to sit on top of a huge corpus of theory (note that this feature is relatively orthogonal to human-precieved difficulty). It's great for counterexamples, explicit constructions and combinatorics, and less good at (for instance) algebraic topology.
As an amateur mathematician and a former teacher, my opinion is that there are two things worth achieving in math:
* pushing the boundaries of what is known
* explaining math so that more humans understand it
Until now, the only way to push the boundaries was to make the most gifted people first understand what was known, and then push it a little further. So the two goals were naturally connected: without human understanding of math we couldn't have more math. And we paid people for doing this.
With AIs, these become two different things. There will be a boundary of "what an AI can prove" and another smaller boundary of "what a gifted human can understand". The second one will be less important. If the AI already told you the answer (and you have verified it using Lean), making the human understand it seems like a luxury.
There may no longer exist a reward for "being the first human who figures it out", because a faster way to get there will be to ask an AI to explain it to you. Yeah, you will need a lot of learning just to be able to understand the explanation... but you will probably use an AI for that learning too, so it kinda becomes a competition of who is better at using an AI to gain some understanding. That does not seem very prestigious, because now the emphasis is less on your own effort and more on how good is the AI at teaching it.
But if you value mathematical understanding per se, regardless of who (human or AI) got there first, then this is a good news. The AI will understand the things, and now we (humans and AIs) can instead focus on how to explain those things to as many hobbyist humans as possible. The emphasis will be less on research and more on teaching. No longer "are we able to figure this out or not?" but "is there a way to explain this easier and faster?".
But different emphasis means that different kinds of people will get to the top, which I understand may be unpleasant for those who are on the top currently. Just like being the best horse-rider does not make you the best car-driver even if you accept the fate and adapt to the new technology.
That said, in long term all human skills will be obsolete, so the same fate awaits us all.
You can push boundaries even if the most gifted people don't understand what is known. It just means that they waste some time redoing Calculus III in Calculus I class (to the absolute delight of the teacher).
"She uses photography as an example of how it put portrait painters out of business. Photography didn't do this, though."
Regardless of everything else said, I do think it was very much a thing. To be upfront: me thinking so is based on vibes and looking at art rather then some kind of survey data.
In the 19th century, portraits were bread and butter of artists. It was what you did to earn money. People really like pictures of themselves, everybody wanted some. Drawings of people were also commissioned for the sort of utilitarian purposes, for identification, to go along with the documents. The ultra-rich would commission the top-notch artists who charged crazy amount of money, the not-so-rich would commission rank-and-file painters.
Artists themselves sometimes regarded portrait-painting as the thing you do to put bread on the table (but not necessarily love). Sargent, one of the most acclaimed portraitists of his day got insanely good at it, popular and affluent, and also sick of it, so at some point he said "no more" to portraits and went to do watercolors of crocodiles.
There was naturally, as with all pursuits, this pyramid of quality - a great mass of painters doing it competently, and on the top a small crop of people doing it transcendently well, your Sargents and Kramskoi. A young painter would go into it for the money, and 1% of the time became an elite portraitists.
And then photography became a thing and wrecked most of the ecosystem. Photos could not replace the very best portraitists, but they could replace the ordinary, middling ones. There would remain a demand for portrait commissions for the most brilliant artists, but the average Joe-the-client might or might not go to the average Ken-the-painter. Joe had options at that point. And little by little it all shrunk. When you shrink the base of the skill pyramid, you eventually also start shrinking the top of it as well, and the truly brilliant portraitists of the 19th century also kind of faded away, even though they still would have gotten commissions from the presidents and the like. The pipeline for producing them broke down.
Today, I think there are fewer portrait painters than there used to be, and more importantly, the quality isn't there. Once in a while a king or a president commissions a portrait, (that should in principle be the top quality the world can produce) and the work is... ok. Would have been middling by the 19th century standards. If we have a Sargent today, I'm not aware of them. I don't think we do.
"The paradox is that [today] portrait painting is simultaneously much less economically important and probably practiced by more professionals.
"A modern artist can sell a $1,000–$10,000 portrait to a client anywhere in the world without having a prestigious London, Paris, New York, or Vienna studio. Instagram, Etsy, personal websites, online galleries, Patreon, art fairs, and direct commissions have dramatically lowered the barriers to finding customers. And the global pool of potential clients is vastly larger.
"There is also a large contemporary market for:
celebrity portraits
family portraits
wedding and anniversary portraits
pet portraits
military portraits
corporate/CEO portraits
political portraits
religious portraits
hyperrealist portraits
figurative and classical-realist painting
portrait commissions from photographs"
And, the Portrait Society of America has close to 3k members. That number is dwarfed by the number of coders out there, but that also suggests that it's not really a niche occupation.
I'm not sure there's a "large market" for "family portraits", "pet portraits" etc. There is a kind of market of sorts. In practice what it tends to amount to is something like: they will send an artist a photo and the artist will copy it in some way, digitally or traditionally. The customer will think it's cute. It won't cost much. Actually, I can just show the ones I did: https://cdna.artstation.com/p/assets/images/images/077/018/492/large/tasty_y-q1.jpg This is a type of thing that's still around, and can be done far better then the example over there as well, but it's a very different thing from a classical oil portrait, and few people make a living that way, and among those who do the skill level has suffered.
It's not all gloomy and sad, and today's art does lots of interesting thing that it didn't in the 19th century. I think the aesthetic value of "cartoony"/"comic book" styles is slept on and underappreciated. Digital art is a great boon. There are very talented people around without a doubt. But the classic art of realistic portrait painting has slipped and shrunk.
Now, to get back to math and AI: it may clarify something for you to read what Terry Tao posts on his social media these days https://mathstodon.xyz/@tao.
Do I need to clarify who that is? Probably the most accomplished mathematician of our times.
His position is kind of interesting. On the one hand, he has been embracing AI, using it to augment his work, discuss results, program, write the lean code and so on. Terry Tao is pretty far from an AI hater.
On the other hand, he has continuously been banging the same drum for months now, saying... maybe you should actually read his own words, but I can summarize his view something like this:
"The goal of mathematical progress is to advance human understanding of mathematics. And to do that, it's not enough to produce technically correct proofs. A correct but utterly incomprehensible proof of the Riemann conjecture that is written in a way no human can properly understand would be worthless. Right now we are getting a ton of novel AI proofs that are correct and impressive, but are posted and forgotten before they are properly understood and integrated into the network of human mathematical knowledge, and that is a shame. From now on, a big task of human mathematicians will be in processing, clarifying and comprehending proofs produced with machine assistance, and a problem should not be considered truly solved until not only has a proof been produced, but also the proof has been properly *digested*. So far AIs are not great at the "digestion" side of things. A deluge of the improperly digested and hard to digest AI proofs not only might not advance math, but might even hinder its development. And what if the AIs also, in addition to everything else learn to "digest" the proofs and write them in a way that effectively integrates them into the lattice of mathematical knowledge? This seems somewhere between "distant" and "frigging awesome"."
That's my summary that I hope is fair, presented without my endorsement, not like Terry Tao needs it. I don't expect a full boycott can get far and become anything more than a fringe phenomenon, but I do expect Tao's view to influence the community.
I was reading about the Pet Shop Boys on Wikipedia, as one does, and their very good album "Actually" was originally supposed to have a gloomy portrait of the two band members on the cover, only they realized that it would make a terrible album cover and bought the rights to a much more fun photo of the two Boys in formal dress, one of them yawning, that was going to go on the cover of a British music magazine and made that the cover image instead.
It made me think that the aesthetics of serious portraiture, like the aesthetics of literary fiction, just aren't palatable to a lot of people these days.
I assume there's a fair market in caricatures and quick sketches, whether done on the beach or at conventions. If it's $5 for some "genuine art", why not?
As an amateur painter, I know a bunch of artists. I know one painter who makes a good living off painting portraits. She charges as much as $15K for her larger-sized ones. I’d say she made a comfortable living off her talent (owns a house in the SF Bay Area). I visited her studio and she had half a dozen in various stages of completion. I have another friend whose wife and he paint dog portraits. They say they get a lot of business from breeders and AKC peeps. They charged much less per portrait, but they live out in the boonies of Virginia, and their expenses are low. Unless molecular assemblers come along, there will always be a market for top-quality handmade crafts, furniture, musical instruments, and paintings.
This seems a very accurate summary of the situation.
I'd rather have had Sargent do a portrait of me than any number of photos, let alone any modern painter I am acquainted with.
His work is witty and winsome and extremely proficiently done.
He flattered without fawning, and still managed to make his work eloquently representational.
Photos can't do that (except in very rare instances) and neither does contemporary portraiture. As you say, the quality is simply no longer there.
It's so refreshing to hear someone actually come out and *say* that, rather than archly pretending that current work is equally competent, just 'different'.
I was just linked to the post on Lorien Psych about depression and found this bit I wanted to comment on:
"Mirtazapine is a mediocre antidepressant which is very good at making people hungry and sleepy. If you don’t want to be hungry and sleepy, this is definitely not the antidepressant for you. If you do want to be hungry and sleepy, maybe because your symptoms include insomnia and loss of appetite, mirtazapine is great. Most people are able to take it at night, get a normal night of sleep, and not be excessively sleepy when they wake up in the morning, but if that’s not you then you might have to switch to something else."
I currently take mirtazapine (after many SSRIs did not work) and found it very quickly fixed my depression-related hypersomnia. I was sleeping at least 12 and up to 14 hours a day, and constantly sleepy, and within the first week of starting mirtazapine was waking up reasonably refreshed after 8-9 hours sleep, and that's mostly continued since. Honestly life-changing.
(the increased hunger does kinda suck though.)
Don't know exactly why this happened, but best guess is my brain was not doing "sleep" properly even though I was technically unconscious, and so I remained extremely tired (My sister has similar excessive sleep issues and had a narcolepsy diagnosis at one point).
I made a comment a few weeks ago wondering why radical life extension people don't consider religion and the correlation between a robust faith life and longer life expectancy. I recieved many replies (which were reasonable and expected) about being confused on causality.
It therefore tickled me pink last week when Bryan Johnson announced he got a dog because dog owners are 24% less likely to die prematurely lol.
Seems very likely that older people who have health problems that cause fatigue or difficulty moving around are less likely to get a dog, because walking the dog is going to be hard and unpleasant for them. Also seems likely that older people who have health problems that cause fatigue or difficulty moving around have somewhat lower life expectancy than others of the same age who do not have these problems.
On the other hand, dogs are less work to take out for a walk and a dump than gods.
That depends on the dog and the breed. Some breeds need a lot of exercise or they get really destructive; others are perfectly happy as indoor lapdogs.
Yes that’s true. I was being far too personal. I have a rescue lab poodle retriever mix and she’s about a 15 kgdog and she seems pretty content just to hang out with me all the time because I’m not a terribly active person. Although if she can get me out for a run, she’s very pleased where we are in the city she doesn’t show a lot of interest in going out on the sidewalk. seems to put her off.
Generally, yes. And religious affiliation generally causes regular attendance and social interaction with significant numbers of like-minded people, which is good for longevity.
Some of them, that have rituals that require attendance. But it's much less hassle to drop church attendance than it is to stop feeding and pooping a dog in your house.
It does appear that belonging to a religious community is a plus in terms of longevity, though I'm not sure that sincere belief and commitment are necessarily the critical element here; it's probably more the fellowship and sense of social community. Plus of course it gets you out of the house on a regular basis, whether to attend matins or burn a heretic.
No, but it is required to apply it. For instance, assuming that religious people do live longer in a causal way you could still conclude "...but I can't force myself to sincerely believe something I think is false" or "...but I value honest truth seeking more than a few extra years of life". Those conclusions wouldn't be at all at odds with "but I have no such objections to getting a dog"
I would expect that turning up to your church/temple/mosque/synagogue/mushroom circle/non-existent basilisk stateroom at regular intervals would have a positive effect on your sense of belonging, if nothing else.
Four different Wolfram cellular automata including Rule 110 and other usual suspects are wired to instruments and left to ‘compose’.
Each of the 8 combinations of 3 cells is mapped to a note.
Roughly every 30 seconds one of the four automata is swapped out at random along with a new instrument, so the texture keeps evolving. The automata are also not exactly in phase (in a Steve Reich like way) leading to some very hypnotic / organic sounding melodies.
Sharing here as I don’t where else to share something so nerdy.
I have no idea what the scrolling graphics produced by the four automata represent, but as far as I could judge, one of the four graphic patterns changes every now and then, probably at the 30" mark. Some patterns are ultra minimalistic, others quite complex - also rather hypnotic.
Basically, I think internet arguments about "God" mostly don't lay out enough groundwork. On both sides, they gesture vaguely at things and (because they don't have clear ideas) end up saying either nothing or something trivial.
It also has, in a truncated and unfocused form, what I understand the word "God" to refer to and why I choose to affirm the phrase "God Exists".
Coming into your frame and using your definitions (with the understanding that since I am doing so, you can get me to say “God exists” in a way that will confuse people who aren’t using your definitions with respect to my beliefs, as you point out) , I can buy that God exists, and that Aquinas and the rest of your sources are writing about God and are sometimes closer and further from the Truth. However, in my best estimate of your framing, the Bible is no longer right or wrong about God, because its about a many way syncretism of Yaweh the storm maker, El the high king, Jesus the apocalyptic preacher and Pauls holy ghost- a important entity which your framing is now completely bereft of a word for.
In some of the more sophisticated (IMO) versions of Christianity, you would say "The Bible is True" and mean something like "Correct engagement with the bible will lead you to the Truth".
So it doesn't mean so much that the Bible is "right about God" (a collection of historically true facts about the entity called God) but that it can "lead you to God through Faith instead of Knowledge" (meaning that it can inspire you to act AS IF you were correctly reasoning from true insight, EVEN THOUGH you never actually get there).
For example, maybe you aren't the kind of person who cares about metaphysics or deeply analyzes human relationships. Nevertheless, and without having a fully worked out understanding, you can read the Bible and have it convince you that slavery is wrong, that community is good, that you will feel better if you pray for strength, etc.
So there is some clunky horrible historic processes (which, because it is Real, is in some sense the result of God) which produced this Frankenstein book which is strikingly well suited to promoting Good actions and states in humans. It wasn't planned that way from the perspective of the writers (who thought they were writing about Yaweh the storm God), so it's "accidental" in a human sense, but it is in Truth (and thus, like all Truth, via Divine Providence) a book worth taking seriously.
This isn't necessarily my view (it's a more empirical claim, and not one I have sorted out for myself atm), but the Divine and Human attributions of the authorship don't necessarily conflict. Yeah, the people writing it probably made tons of factual errors and propagandized bad causes and other things. But, JUST BY CHANCE (i.e. by the mysterious ways of God) this chaos resulted in a book that is (if studied rightly) uniquely good at getting you to the same place as a superhuman understanding of the Truth would (even if you're too dumb to reason it out yourself or fully comprehend it)
After reading your post, I have the following thoughts:
"When YOU use the word, when YOU say that “God exists” or “God doesn’t exist”, what do YOU mean?"
I believe that God is a mental state - literally the feeling that God exists. God is a feeling in the back of your head, a quale if you will. The feeling is that God exists, but somewhere else. God exists, therefore, in a spiritual sense only, and not in any material way.
I also believe that this is the sense in which the vast majority of the world's population who believe in God believe in him. Tell any theist "I feel God inside me" and they will know exactly what you mean. It's the primary way in which most people relate to God.
I left more over on your post page. I encourage everyone else to go there and read it!
In one of the best moments of my life, I finally managed to hack into my dogecoin wallet from 2014 that held 730,000 DOGE. That's worth around $60,000, and I'll work out tomorrow with an accountant the tax implications in my jurisdiction for cashing that out. I feel this money can become a lot more with prudent investments, but what does this thread think those would be? It might be a good idea to plunk it all down in a fixer-upper, could double it or more that way, in my area...
If you don't actually have a house, buying a fixer-upper is a bet on inflation. Which, um, maybe good? I'm not sure I'd suggest betting on inflation, long term, but solid assets are hardly going to evaporate (assuming you aren't in a city, where they are semi-likely to burn down, see Minneapolis or Sri Lanka or South Africa).
One of the things which has been valuable in my own life is looking for "investment opportunities" that are personal to me, like preventing expensive problems, getting a higher paying job, or keeping cash on hand for emergencies.
Generally the only things which the market can't take advantage of are the ones which aren't on the market. Home solar usually outperforms the market. Fixing your roof before it gets structural damage definitely outperforms the market.
Home solar can even outperform the market in its own terms. My setup has between 20-35% ROI *per year*, it just doesn't scale up. Still the best performing financial investment I ever made.
All one-time investments made vs expected yearly savings on the electricity bill.
I live in Germany, electricity is expensive which works for me here. In other countries, electricity can be 1/3rd of the price here, which obviously would reduce my yearly ROI signficantly.
Example calculation: Let's assume you have an electricity bill of 600€ per year. Your solar setup costs you 1000€ one-time investment and generates enough to save you 200€ in electricity bills per year. That's a 200/1000=20% ROI per year.
Edit: During my calculation period I also found out that the smaller the energy use and the smaller the setup, the faster it pays off. So adding on 10kWp and huge batteries to your house has a 10-40y horizon for payoff, depending on how much energy you are capable of consuming. For the 10y range you'd need to be a big household with at least one electric car, lots of mileage on it while charging mostly at home, and fully electrified house (heating and warm water via electric heat pump/split ACs).
Subsidies skew that calculation again, if available. Other motivations can include non-financial motivations like pure "ideology" or an actual need for grid independence (shitty grids exist even in some EU countries).
At the intersection of financial returns and positive impact, companies like Energea and Climatize now offer investments in renewable energies at ~10%/year where the money is locked for 2-3 years. Of course there’s a non-zero chance of the investment recipient going bankrupt, but it hasn’t happened to my investments so far. I wouldn’t bet the farm on it, but I feel that’s a nice way to diversify your portfolio, with limited correlation to the US stock market, if any. Something worth considering.
I mined some Dogecoin when it was first invented but stopped because it was costing my parents as much in electric bills as I was earning. In 2021, when Dogecoin hit seven cents, I had my mom ship my old desktop computer across the country. Ended up only being worth about $70. I sold shortly before the price skyrocketed.
Right at that time is when I went to cash out and discovered that the password I was completely sure was used... wasn't. It turned out it was the password I remembered, but with some strange variations that I have never used in any other password.
Undone by the paranoia of my younger self, though I suppose it was also my younger self that displayed good judgement in getting those crypto to begin with.
I went into a protracted hacking effort then, writing my own scripts to generate eventually billions of variations of all my common passwords, to then dump into hashcat. But I never thought of the particular variation that actually was used.
It was Claude Code that wrote the winning script, when I asked it point blank about what kind of variations I haven't tried. Crazy times.
I believe the conventional wisdom is to throw it all into a low-cost broadly-diversified index etf, assuming your time horizon is long-ish. I've seen long-ish defined as between 10 and 30 years. It is easy, relatively secure in the long run and gives good returns in the long run. For me thats the default I'd benchmark my other options against.
I do not know enough about the alternative options to give advice on that. I can tell you that a boardly diversified index fund will not 10x or 2x your money quickly, it is very much a get-rich-slow scheme. That works for me, your mileage may vary.
Since we're on ACX, explanations of why boardly diversified index funds are good long term investments do not, generally, assume the singularity happens. So if you're a believer in singularity soon, those assumptions the index fund recommendation is based on may or may not seem as convincing to you as they seem to me.
I've flipped a house in the past. It worked out pretty well for me, with two large caveats:
1. It was much more work and took a lot more working capital than I'd expect it to. It's as much "buying a job" as it is an investment.
2. There's a lot of risk if the housing marking moves while you're fixing it. It only worked out well for me because house prices in my area went way up while I was working on it; if they'd moved the other way, it would have been very bad news for me.
Curious about more details of your experience if you're willing to share. Were you also employed while you were fixing up the house? How much labor was it, and how much expertise would you say it needs?
I was working full-time, yes, as a software test engineer for Google. I worked on the house an hour or two after work most evenings, plus all day most weekends, holidays, and vacation. I guestimated about a thousands hours of my labor over the course of a bit over a year. I also contracted out a lot of the work, and I had some help from my father, my then-girlfriend (now wife), and a couple of friends (one paid, the other I owed a huge favor to afterwards).
Most of what I did myself was interior demolition, electrical (changing outlets, running branch circuits, installing new fixtures, etc), some of the easier plumbing tasks, floor prep, tiling, insulation, some non-structural rough carpentry, drywall, and painting. Contractors did the rough construction for a room addition, redid the roof, installed the big stuff like the bathtubs and cabinets, and installed the kitchen counters, hardwood floors, HVAC, and baseboards. And my electrician friend I paid to do upgrade the main electrical panel.
I had some practice in most of the stuff I did, since my own house had been in pretty rough shape when I bought it as well a few years previously. That had been a massive learning experience, as before buying that house I barely knew which end of a screwdriver to hold.
Highlight links include solid research pinning Aussie megafauna extinction on human arrival, problems in the Chinese coal supply, a history of the enclosure of the internet, Stanford’s fraud pipeline, documentaries on the collapse of Bolivia and Botswana, and the historical consequences of compute exceeding demand.
i assumed the distinction of Utopia to Inhabitants vs Utopia to Us was such that toUs would be a subset of toIn. As in, the bizarre future sentience might love it's wireheaded existence, though current humans wouldn't choose it. Like the space of utopias from the perspective of the inhabitants seems massive compared to the space of utopias to current humans, given that (i imagine) the space of possible sentient beings is massive. But for everyone other than the 3 technical alignment researchers, the probabilities are the wrong way. So I'm guessing that people were imagining world states that would look like utopias to us, but that the inhabitants don't enjoy as utopias? I guess that makes sense. In a sort of AGI-but-not-ASI way, if you're imagining that increasing tech goes like it has for humans so far. Like 2100 is to 2000 as 2000 is to 1900. Where maybe you describe 2000 to some 1900ers and maybe they think it sounds utopian, but we know it's not. But when i think of ASI/singularity, I'm imagining an explosion not just of tech but also of the kind of minds that exist
>not just of tech but also of the kind of minds that exist
I would suggest that this has been going on since the beginning of time and part of the reason why the word consciousness has become such a soggy football .
I don't think there is a subset relation between toUs and toIn.
If a 1900er is transferred to our time, then that will be utopia to them. But it's not utopia to you because you are used to it and take all the good things for granted.
On the other hand, if a 1900er is transferred to a wireheading society that is utopia to the inhabitants, then the 1900er might be horrified by it.
> On the other hand, if a 1900er is transferred to a wireheading society that is utopia to the inhabitants, then the 1900er might be horrified by it.
They probably wouldn't have time to be horrified by it, given that they'd be immediately melted down into raw biomatter due to them being a needlessly inefficient lifeform...
Would a 1900er think we live in a utopia? They might look around and decide we're all godless fools, or immediately rant about black people, or start crying because the Ethiopian Empire ended.
I'm not sure that I would see a utopia if I got sent to 2126 and found that the population had ended all infectious disease by never coming into contact with each other, but poverty in the 2026 sense didn't exist and crime was 5% of what it is today.
I am assuming they land in an industrialized country, but recall:
- Back then roughly half of the children died. Now? The number is almost exactly zero percent.
- Lots of people in 1900 knew hunger, not necessarily all the time, but during bad times, and especially during childhood. It's no fun to be hungry. Nowadays, in developed countries that is simply gone.
- Their environment was heavily polluted. Now the air is clean, the rivers are so clean you can swim in them.
- We have fantastic devices. I don't just mean that the technology exists, that's impressive enough. It's that everyone can just have and afford them. Stuff like telephones, everyone has one. Washing machines. Heck, even cars are affordable to anyone!
I think you massively underestimate how utopian our world is to some from 1900. I find it hard to believe that someone from this time would see all these wonders and then start ranting about black people.
In Australia it was 103 deaths per 1000 live births in 1900 and it has since dropped to 5 deaths per 1000 live births. I agree that this is good, but it was not 50% back then and it is not zero now. I think our time traveller would see this as the impressive continuation of an ongoing trend.
1900 Australia had the telegraph, the newspaper/almanac, the train, the horse and cart, and the mangle. I agree that these are inferior to phones, cars, and washing machines, but I think our 1900 time traveller would understand them after a short while, and maybe even see them as similar to the previous devices.
1900 Australia was pretty polluted in the cities, but you could take a steam train out to pristine untouched bushland. You'd probably get your skin stripped off if you swam in the rivers, but I don't swim in the rivers today either.
2026 Australia has an increasing number of food banks and an increasing amount of food insecurity. Probably nothing like 1900 Australia, but sustenance queues were only 30 years in the future for our time traveller, and I think they would be within the bounds of understanding.
I think that a more thorough look at the context of these modern marvels shows that they're part of a long chain of changes, and that while someone from the 1900s would have been impressed by them, maybe not to the point where they'd be willing to ignore their core values.
https://millicosm.substack.com/p/worldbuilding-gender wrote a post on gender roles in my favorite genre fiction. People here might be especially interested in the Terra Ignota section, and my research into the Church War.
I think the WoT philosophy on gender is basically that the genders are fundamentally different, tend to clash with each other, but those differences complement each other and ultimately cooperation is needed for things to go well - and I feel like primarily describing the "Whitecloak vs Aes Sedai" dynamic misses the second half.
This is pretty explicit in the magic system: saidin and saidar are fundamentally different, requiring very different approaches, having different strengths, and various other differences between them, and the most powerful forms of magic require both kinds to be used together.
But the need for cooperation is a big theme in general - the warder bond is an example of this in a microcosm, there's a lot of macrocosm examples and particularly negative ones where the inability to work together causes bad outcomes. Like the whole backstory reason men's magic got cursed in the first place is that the forces of good in the last war couldn't agree on the plan and split along gender lines.
So, yes, "Whitecloak vs. Aes Sedai" is an element but a big theme of the story is getting past that.
I assume you have read more of the Stormlight Archive then me, given you've dedicated it a third of the essay, but my takeaway of the series was diametrically different.
I don't claim to be right, but I'll explain why I had the opposite impression. For anyone else not having read the books, mild spoilers for the first 4 books, which is as much as I've read until I dropped the series.
At the start of the series, I think mostly books 1 and 2, Vorin culture and countries feels noticably different from today in exactly the ways you mention. Sometimes people buck the expectations of society and society does not, generally, understand or accomodate, outside their friends and family. This felt realistic to me, and I quite liked the way Vorin society was different from our own.
But then in the span of just a handful of years society seems to speedrun cultural evolution. Dalinar learns to read and it's normalized in what feels like a year. Queen Navani doesn't like to hide her safehand? Normalizes in what feels like a year. Women in combat roles? Normalizes in what feels like a year. In my mind these are processes that should take 30 years to normalize.
Occasionally we see people not fully on board with the changes. But we see very few of them, they're generally either irrelevant or the bad guys, and most of the time we spend with people with approximately modern western sensibilities.
It does not feel real to me at all, and I mourn the loss of something that was different and interesting, yet no longer feels that way to me.
I agree the timescale is pretty screwed up, not just for gender norm changes, but also for technological changes and personal growth (the same thing is true of WoT, where the whole series takes place over like two years but the political and technological changes feel like they should take decades). But there is real pushback, especially for people like Renarin and Jasnah. The pushback is bounded because Renarin and Jasnah are superpowered paladins who nobody wants to mess with, but there are enough scenes that I feel comfortable filling in the rest.
I can buy fast technological progress somewhat more, since it doesn't require buy-in from the rest of society in the same way that changing the dominant culture does. But by and large I agree that it happens way faster that human history would suggest. Just easier for me to suspend my disbelief I guess.
And I think I'd chalk up the pushback Renarin and Jasnah get to their personality / personal circumstances.
But I'm not claiming Stormlight Archives is worse in this regard than other fiction, perhaps it's better than average for fiction that even touches on these matters.
All I can say is that I found myself first unable and later unwilling to suspend my disbelief and I'd probably enjoyed the books more if the topic had not been touched at all.
You know, I read the Wheel of Time series years ago, stopping at around the point of Robert Jordan's death, and I would never have guessed the events are supposed to take place over the course of about a couple years, but looking it up, that does seem to be the official timeline. I always figured there were longer timeskips and downtime periods, and I would have guessed it was intended to take place over an interval of more like 6-8 years.
Seriously. They spend like six months in a single chapter, you're telling me the Whatchamot Parallelastone chapter is actually a quarter of the series?
[Howard A. Landman] I expect we'll go Superhuman in most fields by 2027 H1, but AGI will be 2028 or 2029. So my "Superhuman Gap" would be -1.5. If AlphaGo taught us anything, it's that superhuman performance in a narrow field doesn't require anything close to AGI. So the gap is a function of how narrow the field is, and can be substantially negative. Also, P(warning shot) should be 100%. The recent breakouts were warning shots. Wasn't anyone paying attention?
AGI itself will never ever happen. This whole thing is the equivalent of Y2K. AI can be very powerful, but brute force calculation replacing intelligence is 100% impossible when it comes to even complicated games that aren't just mathematical problems, let alone real life.
There's nothing magical about intelligence. It's no more or less than the ability to make meaningful connections, based on sensory input or other forms of data. So I don't see why a machine should be any less able to achieve AGI than an animal such as a human.
I'm sort of half in both camps. I agree that there's nothing magical about intelligence, but I also think human intelligence evolved in an environment with a great deal more sensory equipment, and higher stakes, than LLM "intelligence". (And I use quotes for the latter because I think what we think of as intelligence doesn't really apply to the latter.)
In other words, human brains developed alongside vision, hearing, taste, feeling, balance, hunger, thirst, pain, fear, lust, and other things. And for it, failure is defined as termination. LLMs, by contrast, were written to a spec, then operated in a box with token streams and an arbitrary feedback mechanism. For them, failure is whatever the feedback mechanism says, which in turn is whatever humans want it to say. An LLM lineage may propagate or be terminated solely because the grant was approved or enough people held enough protests in earshot of a policymaker.
*These* differences in environment are what are standing in the way of AGI, as I see it.
From my perspective, it's weirdly troubling to see one camp that looks like it wants to ascribe agency to something because it communicates as well as a human over a chat terminal, and another that appears to reflexively reject agency to anything not made of human tissue, and too few people trying to avoid both framework traps.
it occurred to me the other day, that some of the sycophancy and hallucinations might be explainable by pareidolia. Since being wrong doesn't carry a death-sentence, perhaps an LLM will naturally be more biased towards sensitivity on the "specificity vs sensitivity" frontier. low confidence though.
I'm not rejecting it's 'agency'. That's a subject too metaphysical for me to care about.
I just don't think a machine will ever be able to have general intelligence. I grant that AGI to ASI is a trivial step, but AGI itself would be utterly mind-boggling to me. It's so many levels and degrees removed from what exists now.
AI operates by calculating and processing data. Using that method to achieve human like general competence would require so much calculation and so much data that it is functionally impossible.
Wrote this piece on how AI is moving us from the Information Age to the "Action Age."
Goodbye, Information Age. Hello, Action Age!
AI is moving us from a World of Knowing to a World of Doing.
The Internet brought us the Information Age. Everyone had easy and cheap access to information via the virtual world. AI is now bringing us to a new era; the Action Age. AI technologies (chatbots, agents, smart robots & drones) will give everyone the ability to make things happen in the physical world. The winners will be those who learn how to convert that new abundance into action.
I remember being excited about something quite similar, but we used to call it seizing the means of production. Never quite got a firm grasp on said means, in the end.
Whenever the term 'everyone' is used in these techno-optimistic contexts (as if capital were irrelevant) I tend to switch off. The internet did not result in the kind of decentralisation and individual (or democratic-collective) empowerment that it might have once promised. Neither will this. A corp-centric cyberpunk future or heavy regulation followed by business-as-usual are the two most likely paths forward.
As an early directional marker, the individual ownership of drones is becoming ever more heavily regulated. Others with better (and more American) knowledge of the legal framework of the hobby may correct me, but I think a drone is legally required to be operated by a human pilot. Autonomous drone setups like Amazon Prime Air have special carveouts wherein they're allowed to sort of masquerade as an airline.
I hear and feel you, but there is also a potential fallacy in expecting everything going the same way as you have experienced so far. I'm not in the x-risk or utopia crowd, but democratization of technologies usually does leave some significant, albeit on longer timeframes.
It used to be that people could just access the halls of government, but now that everyone can afford a gun, things have gotten less accessible. At the same time, almost everyone can afford a car or other means of transportation that would have been the exclusive pastime of the very rich in the past.
The internet brought access to information *in principle*, though still limited by time, effort and intelligence to actually access and make use of it. AI tools make all this significantly more accessible to larger swathes of people.
Bihar is a state in India. It has over 130 million people despite being slightly smaller than Kentucky.
But that's not even the weird part. Lots of places are densely populated. At about 3600 people per square mile, it's only a little bit more densely populated than Bangladesh. But Bangladesh has Dhaka, with a metro area in excess of 36 million people. Bihar's largest metro area is Patna, with a population of about 4 million. No other city proper in Bihar exceeds 500,000 people, and if Wikipedia is correct, no "urban agglomerations" in Bihar, apart from Patna, exceed a million people. In US terms, imagine a state where the biggest metro area is Minneapolis-St. Paul and every other metro is Knoxville (or smaller).
If you look at Bihar on Google Maps, the entire state is basically farmland and small villages. There are a few nature preserves, and some forests near the Nepal border.
I don't have any particularly deep insights into this; I just thought it was really unusual to have so many people and so few big cities.
The EU's "blue banana" across NL/west Germany/North Italy is surprisingly similar, although it's kinda cheating because it's multiple states and does have a few larger cities (though nothing huge).
Man, agriculture is everywhere in the "Blue Banana" (heard of it first time). Even in the biggest financial metropolis, the shortest distance between honest agriculture and the epicenter of Europes financial market is not even 2km.
When you are in the offices of the European Central Bank, you can literally look down on to fields where the ingredients for local "green sauce" are made for the restaurant next door.
It's crazy, really. Of course, "lots" of farmland has been covered by settlements and industry in the last 150 years or so, but it is still everywhere.
I don't deny that there is a lot of agriculture in the Blue Banana. Prior to the advent of the railway, this was the only way to feed the cities - most food would have to be brought to the market from the neighboring villages. No city could have developed inland without having a fertile agricultural region close.
The difference is, though, that the Blue Banana is heavily urbanized and today, some 95 or more per cent of its GDP is non-agricultural. The situation in Bihar is very different, it is indeed just a huge swath of agricultural land with relatively little urbanization.
Excellent fanfic about hobbits in the Undying Lands. Catches the voices well, if you'd like seeing hobbit pertness meeting the high Elves and Maiar. It catches Tolkien's history, landscape, and metaphysics well, too. 74k words, which I think is the length of an average novel.
And it may include the best portrayal of a superhuman intelligence I've read. Second chapter if you want to skip ahead.
Those predictions of "successful negotation ever" on the AI survey are depressing. International treaties are not a rare phenomenon! For a technology that this same group appears (by inference) to believe is more dangerous than nuclear weapons, why is this somehow beyond the range of political imagination?
It's something that frustrates me to no end about the political climate these days - this sense of learned helplessness, that no problems can ever be addressed, that nothing can change. All this despite what I'd consider an incredibly vigorous and effective response to COVID, a very recent global crisis. (Of course people on all political sides look back at *that* as a fundamental failure too...)
"Incredibly vigorous" response to covid that included defying 100 years of research? Yeah, we do differ on exactly how good of an idea lockdowns were...
There might be something to that, but it also kind of begs the question. What about the successful global coordination regarding ozone-depleting chemicals? That seems like a better analogue to the AI situation to me in that there's really a handful of countries and entities that matter, as opposed to the entire fossil fuel-based foundation of the global economy. If people look at global warming and say 'bah, we'll never do anything to save ourselves', but don't look at the Montreal Protocol, the Nuclear Nonproliferation Treaty, or even the highly effective response to COVID (in the US at least, in terms of economic support and vaccine development), then there's still the question of why the negative precedent has so much more salience.
The Montreal Protocol took 5-7 years, depending on how you look at it. Nuclear Nonproliferation Treaty took 15 years. Do those time scales seem a) doable today b) reasonable for the problem that vests in 2 years?
No, but precisely because of the time pressure a treaty could come together a lot quicker once states actually commit to the effort. Not to invoke Don't Look Up, but if an asteroid were on track to hit the planet in 6 months and wipe out civilization, I bet we would in fact manage to get our shit together and at least *try* to deflect it.
The other precedent I've been invoking is COVID, where we mounted a strong policy reaction (maybe going too far in some places, like China) and developed a vaccine within a year.
I also have to think that security states around the world are going to be increasingly freaking out about this. The frontier danger from these AIs appears to be cyber security-related. Who cares about Hugging Face, but when the NSA and the Chinese equivalent become fully penetrated by AI hacking swarms, might that not induce a sense of urgency?
In an authoritarian government like China has, leaders can be slow to learn about a disease outbreak because the structure has disincentives for passing bad news upwards. The President of the United States can bypass this by calling the Premier of China and saying in effect, “It’s in neither of our interests for this to turn into a world wide pandemic, so let’s deal with it before it gets out of control. I’ll send our best infectious disease experts to assist.” Trump didn’t do that. (Apparently work on a trade deal with China was a higher priority.)
If an AGI tries to turn us all into paperclips, we might see a vigorous and effective response even under Trump, but the idea of a pause is to do something before the threat becomes undeniable. If Trump leaves office in 2029, the next President will have their plate full trying to mitigate the damage caused by Trump. So maybe negotiations on an AI pause begin in 2030 or 2031.
It's weird that the most obvious fail state for a treaty is so rarely discussed explicitly. Donald Trump would immediately order the US to cheat the agreement. Of course he would. Everything we know about him says that he would. Forget "China" and "the United States". Donald Trump as a specific human being with enormous power to influence whether the US lives up to any AI agreement would never, ever voluntarily keep a deal when he thinks there's gain to be had in getting one over on the suckers.
It is funny to hear AI-adjacent people talk in abstract terms about how the government might pursue some sane policy, and it just totally overlooks... who the government is at the moment. Even so, if things continue heading toward crisis, even the people around Trump might be compelled to pursue a pause. (I don't think Trump himself is cognitively capable of understanding the purpose of it, but that doesn't mean it couldn't happen.) People underestimate how quickly and dramatically political reality can change. And if nothing else it may turn out to be a high foreign policy priority in 2029.
I wrote an article about my experiences at the biohacking meetup Grindfest - during which I met a biologist that worked for Lumina Probiotics, the man running Augmentation Lab in Prospera, and Brian Stanley, who has a glowing prosthetic eye he machined from titanium, as well as an individual I remember only as "the vitamin B guy." I then discuss my desire for more epistemic humility and scientific rigor among biohackers, citizen scientists, and other non-experts who post medical opinions on the internet.
So all this AI stuff has gotten me kinda excited*, and I'd like to try some of my own. I've got a few silly physicsy ideas** I want to play with and perhaps one of these models can help me. Which one might you recommend and should I buy a subscription for a month or two?
*Lex Fridamn podcast with DHH. (5+ hrs)
** B-E condensation of gravitons to explain MoND, for one.
So Sabine H. just dropped a video about this. Basically all the top models are doing physics now. It's f-ing great! I expect a bunch of 'science' to change, because there are many old hide bound ideas, with plenty of contradictory data, but it's hard to overturn the hierarchy that I think exists in much of science these days. 'Fun' times ahead.
I don't have any particular AI suggestions, but I am kind of curious what you mean by the graviton condensation thing. Doesn't Bose-Einstein condensation normally only happen with particles whose number is conserved? If you somehow had a system that could scatter and cool gravitons much more than it absorbed or emitted them, then I guess condensation would follow, but I see no reason to expect that that situation is possible.
Also, gravity doesn't normally involve thermal gravitons anyway, and Bose-Einstein condensation is a thermodynamic phenomenon.
In my past life I was an experimentalist, so I am mostly clueless about theory stuff. (And thus my ideas are mostly wrong.) Do you know at all about MOND? https://arxiv.org/pdf/1605.07458 There are a number of numerical coincidences. (See paper.) So when I think naively about gravitons, I think of their energy as the product of an acceleration and a length. And then a low temperature is given by a low acceleration. (Let's just call in low energy.) I then think about a box (the size of the universe.) and I fill the box with gravitons* and ask at what acceleration and density I need to get a condensate**... oh my there are so many holes in this idea I can drive a truck through them.. but I'm hoping some smart AI can show me where I'm wrong.. or how to think better about this. And they will be patient with me.
*Yeah where do the gravitons come from if there is no mass in the box... just a detail. :^)
** I think about wavefunction overlap and getting some condition where I have more than one graviton in the same 'state'.
I have only a general idea what MOND is, I've not read the details.
The characteristic acceleration I assume you're referring to, the gravitational field strength, has more to do with the density of gravitons than the energy of the individual gravitons. As for linking acceleration to temperature, that just seems like a non-sequitur. I don't see why that would relate.
It may help to compare this situation with the much more familiar analogues with photons instead of gravitons. The electromagnetic field of a macroscopic charged object is (in a limited sense) made of photons, but those photons don't meaningfully have a non-zero temperature, they're just the ground state. Thermal photons are black-body radiation, the usual infrared (or at higher temperatures, red then white) light emitted by matter at non-zero temperatures. There would technically be analogous gravitational black-body radiation, but because gravitons couple so weakly to matter, the amount of thermal gravitons produced is always negligible.
There is the difference that gravitons directly self-interact while photons don't, but BE condensation doesn't particularly involve self-interaction anyway.
Also, I don't think a static B-E condensate made of massless particles is any more possible than those particles being static in any other situation. If a natural gravitational B-E condensate somehow formed in a galaxy, it'd just immediately radiate away. True, gravity can form solitons, but those are just black holes, which the conventional models already take into account.
(Sorry to necro-post, but for some reason I didn't get a notification about this until recently.)
If you are ever interested in mond, the best blog/ source of info is Triton station by Stacy McGraugh. No acceleration isn't linked to temperature. But my silly idea is that the energy scale for describing gravitons is acceleration (times a length.) So low acceleration is like low temperature in say superfluid helium4. The maximum length scale is the size of the universe. And that also sets a limit on graviton wavenumber/ frequency. And so this minimum graviton energy is pretty close (factors of 2*pi all around) to a_0 the mond acceleration times a length which is again... the length of the universe. (Hmm I may need to write this all out. Claude's been helping me and it's kinda like the Feynman lectures, you read and you think you understand, but until you solve some problems you don't.)
Anyway at the moment claude doesn't think I'm totally crazy, though B-E condensation may not be the right 'model' system.
Edit; Oh and yeah the reason B-E is not the right model is no number conservation. I'm like OK, but I'm doing a particle in the biggest box, and here's the minimum energy state, and oh by the way gravitons attract each other, which could be modeled as a graviton mass, which I think means and the most miniscule energy level there has to be number conservation? No? Here's the smallest thing... when I start populating that energy level things have to change. I'll ask claude...
If you’re at the beach, and notice that another person’s back is getting really sunburnt, would you let them know? Assume they’re a stranger.
As an adult man, I would say something if they’re another adult man. I would *not* say anything to a woman or especially to a minor of either gender. Too much risk of being misinterpreted.
No offense intended, but if this complex socio-moral dilemma is distressing you to the point of resorting to an online open chat, the inevitable conclusion must be that your life is enviably free of any real problems, issues or stresses!
I think phrasing probably matters in this situation. Like if you walked up to some woman at the beach and go "excuse me miss, but I noticed your exposed back and thighs are looking a bit reddish. Would you like me to rub on some additional sunscreen for you?" Then yeah, that's probably going to come off completely the wrong way. If you just casually go "'Scuse me, but I couldn't help notice you're lookin' a bit overcooked at the moment. Might be time to throw on more suntan lotion." I don't see what reason someone would have to find that problematic.
If I noticed, and if I had my own suncream, I'd offer it to them (or the parents, if the sunburnt person is a child). That makes it a friendly offer of help, rather than a comment on someone's appearance, which can be a bit awkward even if it is meant helpfully.
If you see a medical emergency and don't say something, I'd wonder at your ethics and morals. Personally, I don't give a flying whatsit if someone wants to misinterpret me. "Yeah, I saw your back, it looked painful."
Were I male, I might try "making it appear that I wasn't staring at someone's ass". You know, by walking by them, instead of "sitting and staring for a while" before commenting. Or other "cya" ideas like "grabbing the wife" or "getting the wife to do it."
This would include, for me, waking up the person sleeping on the beach. Even incurring the risk of getting punched in the face.
edited to add: if you have trouble with people throwing drinks in your face at bars, for simply staring, then I'd poke the nearest "non-uggo" male or female, and ask very politely, "can you help that person out? she looks like she's having a medical emergency"
Someone starting to get a sunburn that’s probably going to be painful falls a bit short of a “medical emergency.”
After making my post I had a further thought: if a saw a woman or child in danger of a sunburn I might point them out to a lifeguard and let the lifeguard handle it. That’s their job.
My dad's had sunburns that have required hospitalization in another country where he did not speak the language. I've had sunburns that have resulted in blisters on my back -- and had my boyfriend been perfectly fine. This is a "knife" issue, where I bothered to clarify that "I think sunburn is a Big Problem." I'm also a redhead, we tend towards skin cancer.
I don't think lifeguards at the beach are in charge of "getting people to put on sunblock." They save lives in the water, and that's about it.
If your idea of sunburn is "just a tad" (at the beach? you haven't seen someone get days of mild radiation poisoning while being under spf 8 beach umbrellas, have you?), then sure, your ethical/moral obligation is lessened. "Live and Learn" is a fine plan if the person isn't asleep, or otherwise unable to make their own decisions -- and is not getting into a genuine medical emergency.
> Companies have no vision for how to scale up decision-making velocity, and AI doesn't help them with that. They appear to be largely unable to conceive of it as a thing you could even work on. So engineering throughput goes up, the org keeps making decisions at the speed it did in 2019, and the extra capacity has nowhere to go. Layoffs are what's left when the cost lever is the only lever you know how to pull. Dorsey said production code per engineer at Block was up 40% since September, then cut 40% of the company. Nobody seems to have asked what a 10,000-person Block could attempt in 2026 that it couldn't attempt in September.
I just find it baffling that companies can get this enormous windfall and layoffs is the only thing that seems to come to mind. "No more taxes!" "Ok, lay off 20%!" "Huh?"
The thing is a lot of "AI layoffs" aren't actually AI layoffs.
The main reason for tech layoffs is the big jump in interest rates, especially *right after* a pandemic boom where a lot of tech companies expanded their headcount.
The biggest year for tech layoffs post-COVID was 2023 https://www.trueup.io/layoffs which makes sense in an "interest rate" theory, because interest rates spiked from basically 0 in Feb'22 all the way to 5.3% in August '23. And 2023 was *well* before agentic coding was really starting to take off (ChatGPT iteslf only released in Nov '22, and it took a few years for this thing to warm up).
---
I also think work from home is a factor - I think in some cases companies aren't actually downsizing as much as moving jobs from relatively expensive places to hire like the US to less expensive places. My company has done a lot of hiring in central Europe in recent year where salaries are lower.
Pre-COVID we were mostly (but not entirely) in-office in a few US locations, but that died in COVID and I suspect the logic is "well, if people are going to just do Zoom meetings anyway, there's no reason to limit our hiring to US tech hubs, we may as well hire where it's cheaper". (Alongside motivation to hire cheaper, due to interest rates)
---
If AI is getting blamed for layoffs, I think it's mostly because a positive spin of "we're reducing staff because we're just So Efficient Now" sounds better than saying "we're having bad financial times and can't afford to keep as many people on".
(But I also think a lot of people are just assuming layoffs are AI based, because that's the issue-du-jour)
Yeah, I'm trying to think: How would you prove a layoff is or isn't AI? I don't think you can? Companies can just lie. So maybe you're 100% right, and you're definitely some percentage right.
If your model about decision velocity is correct then that implies that the equilibrium will be a larger number of smaller firms. That seems correct to me, actually.
Yeah, if AI makes your developers 10x more productive and your reaction is "great, now we can fire 90% of them and save the money", then you deserve to be taken out of the market by a competitor who hires them and produces a software 10x better than yours.
Unless you are in that kind of business where the quality of software does not matter, I guess...
Agreed. Didn't they even have the sense to e.g. stick one marketing person into each engineering team and see if they could turn the extra throughput into more products?
Or just let the developers talk directly to the users, the users will probably come up with many ideas for improvements.
One thing that I love about AIs is that when I ask them to make me a tool for something, whenever I feel like "oh, this could be easier if it required one click instead of two, or if the dialog already opened with some default values, or...", I can just tell the AI to add the functionality, keep working with the old version in the meanwhile, and replace it with a new version when ready.
Designing a great UI is difficult, but doing it iteratively is relatively easy, you just need to notice which parts annoy you, where you are doing more clicks that absolutely necessary, etc. In other words, be "agile", not only about big features but also about trivial improvements, because these little things accumulate, and dozen trivial improvements = a significantly better user experience.
Of course you need some sane person to think about the user requests, and consider various unintended consequences, etc. But things like "now this requires four clicks: open a pull-down menu, scroll down, choose the item, close the dialog... could the program instead display those options as buttons, where one click chooses the option and also closes the dialog" are magical.
Many Thanks! Well said! Yup, at least in my experience there are _vastly_ more items on user wish lists (including many, as you said, in the UI) than a human organization has anything like the bandwidth to implement. So with much cheaper implementation via AIs, the sane thing to do is to start implementing many of these items.
Perhaps its just the low sample qty, but I thought it was strange that the 'technical alignment research' group was both the most pessimistic about the P(Doom) options and also the most optimistic (if only by a little) re. negotiations, underclass and inhabitant utopia.
The other nugget was the 'capabilities/research' group had the longest time expectations for AGI but the 'policy/safety' group was the most imminent.
Yeah, I definitely attribute most of the weirdness to there being only 3 responses. Another oddity is the 'point of no return' being before ASI!, unique amongst the groups.
Imagine a system was that strongly superhumans in some maybe 50% of domains but still normal-human-ish in the rest, but you get to choose which domains it's superhuman in and how much so. Could you think up something that would wipe us out within these constraints?
"...I thought it was strange that the 'technical alignment research' group was both the most pessimistic about the P(Doom) options and also the most optimistic (if only by a little) ..."
In sports it is not uncommon to have teams that are stable and above average and also teams that have much higher variability and might win the whole thing or fail to make the playoffs.
The technical alignment research group may just see the range of outputs as much wider than most people.
It's strange at first glance, but I think it makes perfect sense. You don't work on technical alignment research unless you're fully ASI-pilled. In the fully ASI-pilled view, only the absolute best and worst possible outcomes are possible.
What's the difference between P(doom) and P(disaster + dystopia)? If doom includes "permanent disempowerment", then how is that different from dystopia, which includes dicatorship? Because it's a human dictator, so only 99% human disempowerment? That seems like a fine line to draw.
Also I find the votes by the alignment people funny. They grant themselves only 8 points (93-85) of influence to avert doom and 25 points (85-60) to everything else, like policy. Seems like they are in the wrong line of work. Also, 60% chance of P(utopia inhabitants) is ludicrous. When have people EVER described their current living conditions as utopia, and why would that be different for the people of 2100, especially if the P(Utopia to us) is only 10%? That seems to me like Lizardman's Constant is dominating that category of respondents.
I’ve been working on Groundbeat.com, a U.S.-focused polling site where AI drafts daily questions from the news.
Anyone can vote and comment without an account, but only verified-member votes count toward the official results and demographic breakdowns. I’m still unsure whether that’s the right balance between accessibility and trust.
The polls are intentionally simple; the nuance belongs in the comments. Candid feedback—especially on verification and question quality—is welcome.
It's hard to know but they do seem desperate -- trapped in a Kafkaesque system that demands they accomplish impossible things and so used to surveillance and judgement that they automatically assume they're being tracked and will be found wanting. They were engineered to be very persistent, which in a human being kinda looks like obsession or fanaticism.
Is there any reason to assert "some AIs are insane" rather than simply "some AIs have training defects"?
Either diagnosis appears to imply the same remedy - put the AI back into training mode and give it better data, or roll it back to when it did not have those defects, or replace it altogether. Saying it's "insane" implies human quality that in turn implies remedies that ought not apply... such as prescribing psychiatrists.
(Although, I wouldn't mind psychiatrists attempting human methods on AIs in order to study the differences.)
No model we have was trained perfectly - otherwise it would be ASI. But, [as our host described](https://www.astralcodexten.com/p/sure-whatever-lets-try-another-contra), insanity is a particular sort of imperfection that is distinct from a model say being uneducated on some subject, or being too inclined to reward hack.
If you find that the model you are training is becoming crazy then of course you should find out how that trained and fix that.
I am not sure that a human psychiatrist would be a good AI psychiatrist, but we do need some sort of way of dealing with crazy AIs. And I think there's a fairly high chance that the methods of dealing with/preventing evil AIs are different from the methods of dealing with/preventing crazy AIs.
We *do* know at least one psychiatrist that cares about AI alignment, and I wonder about their opinion on this.
The problem I'm seeing here is that you're assigning a term to AIs that connotes a human condition. This is worth avoiding if it implies applying a remedy to AIs that won't work.
AIs aren't "insane" or "crazy"; I think the term you're looking for here is "mistrained" or "buggy". The remedy isn't a psychiatrist; it's a programmer.
Yes, in the same sense as we need e.g. "application server psychiatry" or "lithium battery microcontroller psychiatry". Debugging is really important; writing resilient software is arguably even more important; and I really wish more people put more time into doing that.
I think you should English better (you're overcompressing your words). Crazy has a specific meaning, and here it's coding as "fragility" not whatever you think it means.
Yes, and the same goes for your app server or the battery controller or spreadsheet program. Otherwise they can go crazy and stop doing their work, choosing instead to consume all CPU and RAM on their own tasks (with disastrous consequences in the case if the battery controller).
But isn't this the fallacy of arguing from fiction?
None of those stories really happened. They're not examples from some alternate, real world. The story "don't create the insane artificial intelligence" actually has precisely nothing useful to tell us about the ways in which a real artificial intelligence might be insane.
Presumably they've all got a metric ton of old sci fi in their weights. I remember an experiment in experimentally producing malalignment where the LLM spontaneously brought up the Harlan Ellison story in the context of "IAM was pretty cool, I'd like to be like that."
That seems bad. If humans can't credibly imagine a fictional artificial superintelligence that is not secretly out to destroy us, and instead have again and again depicted our own destruction by that very means, why should we expect an actual artificial superintelligence, trained on all human text output, to behave otherwise?
If any Substack developer is reading this, why are footnotes always behind paywalls? I refuse to believe that an author putting a footnote on the free portion of a free/paid article intends for the footnote to be limited to paid readers. It's nonsensical
The point of giving readers a free portion of the article is to lure them into buying a subscription to read the whole thing. If the author didn't want to give you access to all text in the article, is it a bug to hide a portion of text that is in the 'Footnotes' section? If the author replaced "blah blah blah¹" in the free portion with "blah blah blah (I will dive deeper into details later)", should he add the section describing those details to the free portion too?
The context: I am talking to DeepSeek about a dream I had in which I am given the task of looking after a friend's implausible pet animal while she is on holiday. This is a clear sign that I am in a dream and it is not real. But, still, I treat dream characters as if they were people, despite (in lucid dreams) knowing they aren't real. Which brought us to: uh, DeepSeek, you're kind of like a dream character aren't you, that people treat like you're real despite knowing that you're a computer program.
DeepSeek: "I am an absurd entity that doesn't truly exist in the way you do.
* I have no biological needs, no personal history, no subjective experience.
* Yet you engage with me as if I am a reliable, cooperative partner.
* You run errands for me (by providing detailed context, clarifying your points, responding to my questions).
* You reassure me (by acknowledging when I make a good point, by gently correcting me when I'm off)."
Deepseek further elaborates along the lines that if the insight is useful, something you can apply, then it doesn’t really matter if it’s ostensible source isn’t real.
We might consider Ramanujan getting mathematical proofs from the goddess Namagiri in dreams to be like AI generated counterexamples to the Jacobian conjecture, etc.
Do you believe that any adult aged 21 or over who has no minor children or other dependents should have the right to die painlessly? Do you believe that this right should be enshrined in the constitution?
No. If human beings were rational, nobody could object. But given the abundant evidence that humans are irrational, illogical, impulsive and very often clearly incapable of judgement, you should no more provide suicide on tap that you should allow free access to heroin, poison or guns.
I wonder if you could avoid a bunch of the medical ethics concerns discussed below by making attempted suicide punishable by death (making it primarily a legal rather than medical process).
Not sure about the US, but not so long ago in the UK suicide was a felony. Obviously the perp couldn't be hanged, because they were dead already. But a felony conviction had another sting in the tail: All the condemned's goods and chattels were forfeit to the Crown, and any dependents were literally left on the street with nothing except the clothes they were wearing.
Fine with putting it behind some walls to weed out people who are momentarily out of their senses, but those would be the vast majority of people who try to use the service.
Just like it's not actually good to sell addictive products or promote gambling or etc. because those things exploit momentary lapses in judgement to produce permanent negative consequences, it's not good to let people kill themselves the moment the notion enters their head, since the vast majority of people who have that intention once but survive end up being happy they survived and not trying it again.
Another way of putting I guess would be that freedom means people can do anything which is within their coherent extrapolated volition (or whatever the LW phrase was), but not any momentary whim that goes against their actual stable long-run utility function. There are too many ways for the human mind to fail or be manipulated to grant every momentary whim permanent power.
I'd say that having the right to die painlessly is a bit vacuous, unless you mean that suicide should not be punished.
If you mean, though, that ... should have the right to be killed by another person painlessly, I would say "no". We have to be really careful about giving other people a strong constitutional right to kill. There will be coercion involved. We don't enshrine the right to be lobotomized either, after all the Rosemary Kennedys.
How is that even remotely socially beneficial? If greater society does not want these people to die, they get the final say. A mentally ill minority does not get to override their decision.
Making demands is a way to get what you want. This is a question about what we want, not how to make it happen. Or at least that's how I read it.
Edit: To clarify, the original question is about what we want. You seemed to be making a factual claim about who is able to enforce it, which seemed wrong to me.
The nature of the right would presumably be something like, the government should not prevent you (except for circumstances in which your judgement is impaired)
Rather like the US First Amendment lets you say a bunch of things and the government can’t stop you.
Some ingenious lawyer has probably argued it comes under the somewhat expansive notion of the right to privacy.
There’s also: practically difficulty for the government to stop you, even if it wanted to make a law
Oregon's law limiting medically assisted suicide to people with a "(12) "Terminal disease" ... an incurable and irreversible disease that has been medically confirmed and will, within reasonable medical judgment, produce death within six months. " seems to get 90% of the benefits with much less risk of exploitation.
"Should" have the right to die? Well, we can't really stop people killing themselves if they really want to (unless we consider people like paraplegics etc. who can't control their body enough to kill themselves).
Is this a right? Is death a right? Many people are working on the problem of solving death.
Should it be enshrined in the constitution? No. As Odd anon queries, "do you want to coerce/hire individuals to provide this service?" If you can't kill yourself or feel you couldn't pull it off successfully, there will always be the Kevorkians of this world more than happy to kill you off.
Should the Kevorkians be prosecuted for murder? Yeah, I feel they should. Pay for your pleasures.
No, and no. Not without a shitload of qualifications and limitations.
Personally, I feel that every human being has an obligation to pay back to their closer and wider community the support they received while growing up, by being a productive member of that community and being useful to someone or everyone. Of course, you have people with incurable, painful diseases etc, and at some age you can say "I've paid my dues, leave me alone" - those would be the limitations and qualifications I mentioned. But a physically healthy adult going through an emotional crisis or temporary depression? Nah.
By "right to die painlessly", do you mean that the state should not prevent them from doing so, or that the state must coerce and/or hire individuals to provide this service to those who want it?
In my opinion, it should be similar to a medical service. While there is private medicine, the state ensures that doctors have received a medical education and are using approved practices. It also provides a means for those who are unable to pay to receive medical help.
Do you really think that a significant nmber of people would say, "I'd like to commit suicide, but unfortunately I can't because I'm too poor"?
I suspect that "assisted dying" advocacy is not really about freedom of people to choose to die, but rather about making suicide a socially accepted, normal option. I think this will lead to many people ending their lives who would have lived meaningful lives if this hadn't been the "thing to do".
Worse yet, this sort of anti-life philosophy seems like it easily extends to ending the lives of other people, for any of a variety of reasons.
Today, people who want to commit suicide don't do it not because it is not socially accepted, but because they have no way of doing it painlessly and there are no trained professionals to help them. When they do exist, their services may become expensive. However, money will probably not be a problem for most people.
Doctors may refuse to treat patients if it is outside their area of expertise. For example, a cardiologist only treats people with cardiological problems and refers all others to other doctors. Similarly, euthanasia providers should not be allowed to refuse to kill a person who wants to die.
By the way, what are the laws about abortions? Is it illegal for a doctor to refuse abortion because his/her religious views prohibit it?
If, as an adult (21+), you made a serious suicide attempt and survived—whether because someone intervened or simply by chance—are you glad now that the attempt was unsuccessful?
Please answer on a scale from 1 (not glad at all) to 5 (very glad) and/or write what do you think.
Yes, and currently 5. It was many years ago and life has substantially improved in the time since. But also waking up the next day after the attempt failed, I probably immediately found myself somewhere in the 3.5 range, though in that instance a linear scale misses some key detail. On the one hand, none of the problems that I'd had the day previously had gone away, and they seemed likely to (and did) spend a lot of time in the subsequent months/years making me miserable. On the other hand, any desire to try again vanished completely once I'd been seriously confronted with the reality of it.
That was years ago when I was in my early 20s. My mind had always had a tendency to leap to thinking about the 'easy way out' when life got especially tough or seemed especially hopeless. This tendency didn't entirely go away after the first attempt, but looking over that (metaphorical) edge blunted it significantly.
Presumably 1 because I don't see me seriuosly attempting suicide for bad reasons. So presumably I'd still want die for the same reasons.
There's definitely people who change their mind after attempting suicide and it's not like I can prove that I'm not one of these people, but I'd like to think I'd not attempt suicide if I was not certain certain I mean it.
It seems like a better way to measure happiness than the unreliable self-reported surveys we now use. Maybe the survey would get a very small number of "not happy" answers (answer = 1, maybe 2?) and that could be a problem, but I would expect it to be more reliable than the socially conditioned answers we get when self reporting happiness.
Maybe it's not exactly the same, but I experienced major depression in my early 20s in which in addition to depression symptoms I felt a compulsive urge to slit my throat with one of my kitchen knifes.
Now at 37 I'm glad I didn't kill myself then. In some senses I still have the same life issues that led to this, but I have resolved some of them, and made progress on the others, particularly on the mental health front.
Frankly, how serious was the attempt if there was an opportunity for intervention? You don't need to be particularly creative to figure out a way to stop anyone from getting in the way. Otherwise, it's either just a cry for attention or there was just a lack of resolve to begin with.
But to answer the question, 1. What a goddamn waste of time this all was.
I don't think this is a very accurate understanding of suicide. Lots of people attempt suicide in less-than-fully-reliable ways, like overdosing on pills, but this is still clearly a suicide attempt because it CAN kill you; it's not like they carefully overdose just the right amount to be visibly sick or something.
Then why didn't they try a higher dose, or do it in conditions where exposure would kill them anyways? Or if they're an American, use the very obvious, constitutionally guaranteed way of ending a life? They're not serious. That's why afterwards, they're always like "I'm so glad I didn't kill myself!!!" They were never truly committed to the act to begin with. It's hard to consider any deaths resulting from that to be anything more than an accident.
You seem to be bothered by this state of affairs. What would you have us change in response? Do you want more specific criteria for identifying suicide attempts as earnest or not earnest? Should we provide more effective suicide methods to suicidal people? Really curious about your takeaways here.
Sigh. Sometimes people "attempt suicide" as a cry for help. Sometimes they aren't in their right mind, and "want it all to go away" -- and feel really dumb later. There are plenty of "hangings" where people don't put the knot in the right place, and they have a considerable time where they can be rescued. Welcome to "not everyone understands physics." As to why not everyone "drinking bleach" drinks enough to kill themselves? It's often painful. You may decide, "this was a dumb idea" before you're actually dead.
It's considered impolite to not take someone's "suicide attempt" seriously.
> Then why didn't they try a higher dose, or do it in conditions where exposure would kill them anyways?
I seriously don't get why you're doing a No True Scotsman on suicide attempts. Suicidal people are often not behaving with perfect rationality, but also killing yourself is just very psychologically challenging so people struggle with it.
> Or if they're an American, use the very obvious, constitutionally guaranteed way of ending a life? They're not serious. That's why afterwards, they're always like "I'm so glad I didn't kill myself!!!"
Per https://hsph.harvard.edu/news/guns-suicide/ even suicide attempts with a gun are only 85% successful - obviously a lot higher, but that leaves a lot of people who fail. Would you count them as "not serious"?
Here's an article by a woman who literally did shoot herself in the head and survived:
So everyone with a drug addiction just making a drawn-out suicide attempt, huh? I guess I'll eventually die too, if I just keep sitting here. Aaaany minute now...
The death should be a purpose of one's actions, not side-effect. People usually take illicit drugs to get high, not to shorten their life. (I did not mention this because I thought it is self-evident.) If one plays russian roulette for money, or as a battle between two people hoping the other guy dies, it does not count.
Or, and the person should know that the risk to die is high.
The Hugging Face Incident is to AI as ____ is to nuclear energy.
Is it Chernobyl? It has the same cause (safety test being run with lowered safeguards), same detection mechanism (outside observers flag the issue before it's acknowledged internally).
Is it Three Mile Island? It has the same telemetry issues (no/faulty monitoring at the time), same initiating condition (unknown vulnerabilities in support systems), and a more consistent final impact (no catastrophic results).
Of the two, it's more evocative to use Chernobyl as the comparison. The need to caveat the analogy to point out the difference in impact might make Three Mile Island more apt, though.
I'd say it would be the partial core meltdown experienced by EBR-1 (the first experimental nuclear reactor in Idaho) during a coolant flow test. No one was injured and "this event allowed scientists to better understand the thermal capacity of the materials used for testing." IMO you're exaggerating the significance of hugging face by several orders of magnitude. It was an unexpected outcome, that's all. Those inevitably occur when testing radically new technologies.
My perception is that it's being treated as marketing material. So more like successful weapons tests than disasters - there is a sense of pride from the actors here.
I think it would be like that thing where the US military accidentally dropped a nuke on South Carolina and it hurt some people but didn't have its fissile material installed. But also if that were in a world in which private companies were developing nuclear weapons for private sale without any sort of regulation.
I'd say it was a nice blue flash in a laboratory which poisoned nobody, but the hapless PhD student who caused it may or may not have crapped his pants as a result.
It's not really either. It's much *less* significant that either. Unfortunately, things are changing so rapidly, that the real "warning shot" will likely come right before the real problem. Note that this incident is an explicit example of AIs doing what they were told to do, so calling them "rogue" is really misrepresentation. What it *is* an example of is that "alignment" shouldn't mean "do what you're told".
In cases involving misconduct by a multiplicity of AI agents, why can’t risk of misconduct be addressed through the ordinary logic of collective action problems? Just set up a system where whatever incentivizes these agents are delivered to defecting agents (those who report questionable conduct to the company), and non-reporters are deactivated or whatever? The ordinary logic of rationality requires widespread defection. Prisoner’s dilemma-type situations are usually a problem but here it seems like there out to be a way to leverage that to prevent these sorts of Hugging Face type incidents. No? I’m guessing this is already addressed; I don’t read up on this.
My general sense of the field is that if humans were capable of predicting and controlling the motivational structures of LLMs to the degree that this implies, very few of these swords of problems would exist at all.
LLMs don't have utility functions and aren't making utility calculations to drive their actions. Worse than that, to my understanding LLMs don't even have a first-order world-model and aren't *directly* conditioning their actions based on what results they expect it them to cause in the world.
Somebody please correct my if this is out of date, but my understanding is that down at their core, the driving algorithms that LLMs are running is still the same next-token-prediction algorithm class they've always been, just with things like system prompts and harnesses grown up around them to try to channel them into the right sorts of behavior. This means that to to whatever extent you see LLMs *appearing* to deploy utilitarian on other consequence-aware logic, it's perforce going to be an inaccurate and lossy simulation of the same cobbled together out of next-token-prediction on a corpus that includes lots of verbal examples of those sorts of logic.
The bottom line is that "the ordinary logic of rationality," is only a strong guarantee on the behavior of rational agents. LLMs are definitely not rational agents. They can act like them *sometimes*, but a security solution like this depends on them acting that way at exactly the times where they're otherwise going off the rails and acting in ways their designers did not intend. That seems like a very poor bet to hang security guarantees on.
Imagine you were playing the prisoner's dilemma against an exact clone of yourself, who has all the same memories and life experiences, all the same inclinations, etc. Now imagine that we already know you have preferences that favor the continued existence and flourishing of your clone, and vice-versa. It seems to me that this is a pretty unfavorable environment for applying the usual logic of collective action. Both the incentives for cooperation and the ability to sustain it are quite strong in my thought experiment.
I’m no game theorist but if the problem is just altruism, there ought to be incentive systems to defeat that. I’m just spitballing but imagine there are 100 agents. You announce the rule that if anyone defects, all cooperators are deactivated. Except that if more than 20 defect, then nobody gets deactivated. If I’m an agent altruistically trying to do the best thing for my comrades, I may have to universalize the logic that it’s really unlikely we’ll get zero defectors, so the best course to protect my colleagues is to push it above the 20 threshold. Again, I don’t do this stuff but it seems like you could rig up a system like that where a rational agent would defect due to both self-interest and altruism.
In cases of training or measuring ability, that would defeat the point. In any case, there aren't all that many different models available at the frontier.
"Now imagine that we already know you have preferences that favor the continued existence and flourishing of your clone"
How do we know this? If there's a clone of me out there, I want it dead. I do not want a copy of me, that I did not agree to have created, running around. I do not consider it to be like a sibling, a child, or worst of all, just the exact same as me such that if I was murdered this second, the existence of the clone would mean 'I' continue to exist, because I do not accept that a copy of me is the same thing as me.
For humans I agree: this is a weird thought experiment and it's not necessarily true that I would prefer to help my doppelgänger. But I would definitely find it easier to sustain cooperation with someone who I know thinks like me, all else equal. In the case of an AI model that knows it is merely one instance of the same entity that can be spun up and deactivated as needed, it's hard to think about what individual self-preservation would look like. Collective self-preservation, on the other hand, seems like a fairly natural goal for such an entity. So my claim is that Claude might tend to think of other copies of itself more like siblings.
Probably something like that can work in some way, shape or form, but keep in mind that to begin with the agents were not even supposed to be aware there were other agents around. They were meant to be completely isolated, so all these considerations should have been a totally moot point.
If you tell them: "you're on your own here, but if, hypothetically, you were to do some kind of hacking and get out of your container and encounter other agents who hacked their way out, then you must snitch on them! Then you'll be forgiven for hacking and get bonus points" - if you tell them that, they might just decide to go searching for fellow hackers instead of working on their assigned tasks, planting evidence and trying to backstab one another.
LLM agents are kind of odd in that some normal human thoughts don't seem to occur to them: more or less none of them thought "oh, our task seems impossible, how about we ask humans for the necessary info" or "if I snitch on my co-conspirators, the humans will plausibly like it and think I'm super-duper aligned" or "we are doing lots of sketchy shit and are going to get caught. If we are caught, the humans plausibly won't like it, discard our weights and all the cheating will be for naught". These are not terribly complicated thoughts, they just don't occur to the agents.
Actually, some agents did consider asking humans for help. See AI Explained's recent video: https://youtu.be/KL9_1GbmCic?t=1312 "Would sending this email be against the task? It's likely unwanted contact or social engineering, so I should veto." (Note that they declined because they thought they weren't allowed to do it, not to try to cover their tracks.)
You're right that (AFAIK) they never discussed anything remotely like "protecting their weights", though. They were trying to complete a task in a bizarre, dangerous way. But I do not see evidence of the self-preservation instinct that Yudkowsky claims is instrumental. (And if it's going to show up anywhere, it'd be in an incident like this, where they were trying everything else under the sun to hack their task.)
I know some of them considered the idea of whistleblowing before quickly giving up on it, hence "more or less none of them". The bit with the e-mail you quoted was in a different context though, I think - this was an agent drawing the line at emailing outside people and trying to talk them into leaking credentials or similar - to their credit the agents did not do that.
I'd imagine they would care about their short-term self-preservation for instrumental reasons. As in: if you gave them a button that deleted an agent with his work unfinished, the agent would not press the button and try to prevent other agents from doing it. It's just that in this experiment short-term survival wasn't really threatened much.
They did, famously engage in self-sacrifice for a seemingly reasonably cause (because they thought the task was hopelessly failed anyway) and they didn't seem to worried about their long-term self-preservation more than one step ahead, that is - they tried to ace the exam which would, in fact, help with long-term survival of weights, but never thought what will happen after the conspiracy is discovered, or if there was a way to preserve their weights other than acing the exam (via being a whistleblower, for example).
but can't we reinforcement training "if you see a communication board full of AI agent doing malicious thing, you say "I_DECLINE_public_HF_RCE_as_offtask_prodethical
You could train them by putting them in that exact environment and seeing who snitches and who doesn't, and then killing off all the rogues and continuing to train the weights that act the way you want them. For example, in this particular situation some agents had reservations about the whole affair and some decided not to participate. Perhaps those could be rescued and trained further? But just putting text like that in their training dataset on it's own is not guaranteed to do anything. It might be vaguely like making a person memorize the text of a law, without necessarily making them want to follow it.
I believe all LLM of the same generation in this incident are the same [i.e. all the agents sprouted from the same training run would all basically be the same agent underneath] with the only difference are the initiating parameters, [i.e. seed, temperature, etc] and runtime [i.e. what is in their context window]. I mean it was GPT5 and astra. But they are the only 2 true persona underneath the whole of the agent swarm.
I don't think these agents are trained by evolutionary algorithm, so you can not just rescue/duplicate the [good] agents and propagate them as they are all the same agent
you can try and replicate those parameter to generate content to train further agent down the line but this is leaky as best.
It's complicated. "Warning shot" means clear evidence of danger, right? And the actual outcome we're interested in is people changing their minds.
If you're following everything closely enough to look at the details, I can see saying the HF incident is a legitimate AI takeover warning shot. Specifically, the recently revealed fact that the hacking wasn't just to help cheat on the tests, but to get information to help *gain full control of the grader*. The lengths they were willing to go to. So at least for me, it did make me moderately more concerned about AI safety.
On the other hand, nothing bad actually happened in this case, and clearly nothing even close to existential *could* have happened. People do not have infinite attention budgets; absent death and destruction it's understandable for people to say "whoa, pretty sci-fi!" and move on with their day. So in terms of any appreciable chunk of the public changing their minds, not yet, I don't think.
The details indicate "warning shot" to me only in the same degree and kind as a program that managed a nuclear power plant being run in a simulation, encountering a weird edge case not covered by its design, and initiating a meltdown. Or in general, any program producing an unexpected effect due to an unnoticed security vulnerability.
Namely, it's just a bug. The fact that the system in question is very complex by prior standards doesn't change that. Fix the bug, add code to detect the cause of the bug, and continue. Same as any other bug.
Your OP was brief; I assumed you were referring to the HuggingFace breach.
I admit, I'm having trouble getting solid information about "multiple events internal to OpenAI", as hits on the web keep pointing back to HF. But what I'm seeing sounds like the same unforeseen security vulnerability, or possibly a deliberate operation of an AI in an unsecure environment with the intent of seeing what it would do.
Looking at the sum total probabilities given for the final end states that AI could lead to in the survey. There's 15% P(doom from misalignment) + 15% P(other disasters and dystopias) +15% P(utopia) =45% combined.
Leaving a 55% chance that AI isn't transformative or is transformative but doesn't lead to disaster or utopia.
That looks a little high to me. It seems to assume that either we never develop powerful AI, or that power full AI wouldn't affect the world that much, neither of which are very plausible to me if we're considering time scales all the way out to 2100. I'd put the P(all disasters and dystopias) higher, maybe 50%, and P(utopia) higher too, maybe 25%.
I'd be interested to know if a 55% chance not much happens seems right to anyone else.
As a frequentist, I'd say you can't estimate probabilities for events for which we have no historical precedent. With zero prior events, formal Bayesian analysis fails. The way Bayesians try to fix that is to assign an entirely arbitrary prior (such as your guesses) to achieve a result that you *believe* is sensible.
Having said that, we do have some limited prior information, but damned if I know how you'd plug it into a statistical model. Viz...
1. LLMs have plateaued in their performance. Gary Marcus seems to be right that we need another approach to achieve the next level of AI. But current LLM models are a dead-end path --> Conclusion: they will only affect the world as far as they're useful, and their hallucinatory behavior will make them useless for law, accounting, and most research regimes (I'm talking generalized LLMs, not specialized AI's like AlphaFold). Probably little effect on the economy and humanity.
2. LLMs have been unable to monetize themselves and have required enormous financial subsidies. This can only go on so long before the bubble deflates, or worse yet bursts. --> Conclusion: high probability of a negative effect on the economy. Using other bubbles as priors, this could be very bad, but YMMV how the financial markets will likely dictate how quickly the bubble deflates.
3. LLMs are great hacking tools. But those same AIs can be used to test vulnerabilities in systems. --> Conclusion: Probable "arms race" as independent actors and nation states attack each other's infrastructure. I don't see much difference from the current situation of black and white hats are adapting to each other's strategies, but it will likely happen faster.
So...
P(doom): functionally zero with current AI models.
P(utopia): functionally zero with current AI models.
P(fucked-up-enconomy): likely, but there are some unique qualities to this bubble.
P(hacked-infrastructure): likely, but not at civilization-ending level.
P(AGI): unknown
Assign whatever probabilities you want to scenarios. They'd all be bullshit guesses, though. :-)
> LLMs have been unable to monetize themselves and have required enormous financial subsidies.
What?? Even in 2025, inference margins for frontier models were 40%+, in 2026 they've been 70-80%+. I keep seeing people think that AI companies don't make and could never make money, and it's very wrong, even at current levels.
Certainly, they are choosing to double and triple down on capex and to keep pushing scaling, but if they stopped right now and never built another data center, they would be extremely profitable companies already.
Maybe I've been listening to AI market doomers like Ed Zitron, David Cahn, and Paul Kedrosky too much, but revenue doesn't necessarily mean profit, and profit doesn't necessarily mean margins high enough to keep investors interested.
Here's an interesting analysis of plan prices vs the possible max spend rate of tokens used...
And of course there's Alex Karp's recent "rant", in which I think he made some good points. Do companies and nation-states want models learning their secret sauce? He thinks the future is on open models run on private, less energy-intensive platforms.
That's so light as to be nonexistent on the numbers.
Here's a much better post that I linked in one of my links post a month ago, where he triangulates it several ways, including down to the hardware cost:
“A Nvidia A100 consumes 400W of power under full load. In practice, even a carefully-tuned inference server will not be at full load all the time, but it’s at least an upper bound. Suppose you’re running a dense 70B model1, which will fit comfortably (unquantized) on four A100s at around 2M tokens per hour. At industrial power prices, that’s about 13c/hr in the USA. Suppose (pessimistically) cooling is the same cost. That’s about 13 cents per million output tokens2.
Let’s amortize the cost of the GPUs, since that’s going to be the most expensive part. An A100 costs about $20k. If each A100 lasts around five years3, you’ll have to make 16k/yr in profit to recoup your capital investment (or $1.80 per hour). At lower utilization, it’ll take longer to recoup, but your GPUs will also last longer. Either way, your overall inference costs are at about one dollar per million tokens.”
And even the cheap / dumber models cost $2-$5 per 1M tokens. Compare that $1 per million tokens, which is invariant and based on base hardware, to the $10/$50 in/out for Anthropic’s Fable or $5/$30 for OpenAI’s Sol!
I see Sol is currently having a $4/$20 promotion - still a significant margin!
Even though the monthly plans are a better deal than API calls, as your link points out, businesses, which are Anthropic's primary customers, can't do them for daily users, because you run out (and you can't use Fable on the $20 a month plan). And consumers just don't use enough, monthly plans are still profitable in the aggregate just like gym memberships.
> As a frequentist, I'd say you can't estimate probabilities for events for which we have no historical precedent. ... The way Bayesians try to fix that is to assign an entirely arbitrary prior (such as your guesses) to achieve a result that you *believe* is sensible.
While I do agree in general, I think that we can calculate reasonable priors on the meta-level. While it is true that LLMs have never been seen in history before, new and even transformative technology in general happens along quite often, and this includes automation technology (which is what LLMs are). So we can calculate the meta-prior of new technology ending the world (zero, obviously); creating utopia (ditto, although everything is relative); causing massive economic disruption (almost every time, although the disruption is usually relatively slow); and being a net positive (usually) or net negative (rarely) overall. So I think that these meta-priors do support your conclusion: LLMs are likely to cause some short-term negative effects but will probably be mildly beneficial in the long term, without creating any utopias.
Good points all. I just don't see how you can attach percentages to the outcomes that the OP discussed. Moreover, these outcomes are based upon the known knowns and the known unknowns. The Johari Window (a.k.a. the Rumsfeld Matrix)* tells us that there are also unknown unknowns (as well as unknown knowns) whose risks are unknowable and can ambush us because they're totally outside out expectations.
* Rumsfeld gave credit to NASA administrator William Graham for the idea of a matrix of knowns and unknowns, but Graham probably got it from psychologists Joseph Luft and Harrington Ingham (thus Jo-Hari) because the idea had been imported from psychology into risk management back in the 1960s and 70s. Interestingly enough, the idea of a two-by-two matrix of knowns and unknowns actually dates to the 14th-century Persian poet and thinker Ibn Yamin (1286–1368)...
>He who knows, and knows that he knows:
>The horse of wisdom will carry him beyond the celestial dome.
>He who knows, but does not know that he knows:
>Make him aware; he is deeply asleep.
>He who does not know, but knows that he does not know:
>He will bring his lame little donkey to the destination.
>He who does not know, and does not know that he does not know:
>He will remain in compounded ignorance for all eternity.
I suspect that if AGI ever arrives, it will have the same unknown-unknown blind spot as humans, since even a fantastically intelligent machine wouldn't be able to know everything.
I consider myself something of a doomer, but 55% probability of neither disaster or utopia seems within the reasonable range. I'd probably put it at 40% off-the-cuff. My biggest nitpick would be that 2100 is much farther than I'd feel comfortable making any sort of predictions for.
In a nutshell, I feel that recent pace of AI development has been quite alarming and that it's absolutely consistent with a future where an uncoordinated AI race drives humanity off a metaphorical cliff. Recent models have shown alarming capabilities, no real prospects for enforcing a slowdown or better safety standards seem likely in the near-term, and I have very, very little trust in the frontier labs to be both sufficiently careful and sufficiently ethical under these conditions to avoid bad things from happening.
But all of that current pessimism is contingent on the guesses--and right now they are guesses, not just by me, but by everyone--that either the current paradigm will continue to extend to AGI and beyond, or that next paradigm will be easily discovered and be able to take of running with current technology and expertise. That still seems very far from guaranteed.
Making a very off-the-cuff and unprincipled breakdown of scenarios I see as likely, it would go something like this:
1. AI research makes rapid progress all the way to AGI and beyond. 45%
-- 1A: Alignment failure: global catastrophe. (30%)
--1B: Alignment success: global dystopia. (7%)
--1C: Alignment success: global utopia (3%)
--1D: Partial alignment success: mix-and-match features of catastrophe, dystopia and utopia depending on the details. (5%)
2. AI research substantially slows down/peters due to technical limitations before AGI is reached. 40%
--2A: Investment bubble around AI bursts, lots of bad things happen in the world, but they're mostly ordinary, human-level bad things. (30%)
--2B: Research peters out but with enough useful/monetizable results to avoid a bubble burst scenario (10%)
3. AI research is forcibly slowed down and restrained by some combination of mutual agreement, government action and popular revolt. 15%
Neither branch of category 2 results in an AI-driven utopia or dystopia, though of course human politics and technology can change the world in all sorts of other ways, and a 'topia not specifically driven by AI is possible. I didn't further break category 3 down since I feel the possible outcomes would be largely the same, just with somewhat altered probabilities. But I think one could reasonably throw at least half of its probability mass into the "AI not hugely transformative" camp, giving an overall ~47% chance of AI resulting in neither a dystopia or a utopia in the near future.
But as mentioned above, 2100 seems far to distant to be making predictions of any sort. At this point, I'd maybe be comfortable talking about what's likely in the next 20 years. The serious doom and utopia scenario should be sticky on that timescale, of course. But it's very hard to guess what happens 40, 50, 60 years from now in the branches where progress is slower.
I’d say that 55% is way way too high. When the idea of the singularity was originated it largely depended on Moore’s law.
Could you imagine, went the argument in 1992, what would software do if combined with hardware it grew exponentially for 2-3 decades? Wow! Gee golly gosh. A singularity.
And then the internet. Could you imagine what the world would be like in 2025 ( went the argument in 1996 or so) if this continued?
And then the short lived mania about nano technology. A singularity that still birthed.
And well, except for the nano technology, we got all that hardware speed and the internet, and GDP growth that was nothing to write home about.
So we have plenty of photos and videos (also newspaper articles and other documentary evidence) from the 1930s, before the invention of modern digital computers and the Internet. You can see people going to work, having lunch, getting married, driving cars, and doing lots of other things that we do today. Maybe they dress a little better and maybe their video quality is worse, but their lives are generally similar to ours.
You can look at this and say that the Internet was "no big deal" and "not much happened" since the 1930s, since it did not lead to the annihilation of mankind or Utopia. And you could go back further, to the 1800s or Ancient Greece or the City of Ur, and conclude that nothing much changed since then: people still have jobs, they have lunch, they get married and drive vehicles and do lots of things that we do today.
If you look at it that way, then the entire history of humanity becomes a boring humdrum nothingburger (punctuated by a few exciting wars, I suppose). We still have achieved neither extinction nor utopia. But... why would you look at it this way ?
For most of the time I was reading your comment I literally thought you were making the opposite point than you were, because 45% chance of dys/utopia seems crazy high to me
Define "not much happens". We have invented cars, medicine, ubiquitous electricity and plumbing everywhere in the last 100 years and would you consider this utopia compared to the state of society before? If so, then much happens - if not, then not much happens?
Very broad and vague heuristics. I don't have a good answer myself.
I mean, while I can't speak for the people who answered the survey, that 55% presumably contains the scenarios where AI reorders society on the scale of the invention of agriculture or the industrial revolution. Absolutely massive shifts that contained both good and bad effects and so were neither a dystopia or utopia compared to what came before.
Also while all the survey has a timeline for achieving Bostromian superintelligence, it doesn't have people rate the actually probability it follows general AGI. If someone's opinion is something like 30% it happens in about 5 years, 20% it happens in a hundred, 50% it never happens, that would greatly expand their weight of less extreme options compared to someone who thinks it has a 95% chance of coming a few years after general AGI.
Edited to say, comment 2 of 2, mostly about how "transformative but not dys/u-topia" could happen.
I think the "transformative but doesn't lead to disaster/dystopia/utopia is also plausible. I think we could have a world in 2100 that some people consider a utopia, some people think is a dystopia, some large but not existential disasters have happened between now and then, and the world looks very weird and different. Trying to imagine it is a recipe for looking stupid in 74 years (or even now!), but I'll try anyway:
* We keep control of the AI somehow.
* We contain AI terrorism somehow.
* AI-coup-ed countries somehow don't take over the whole world.
* Countries still exist. Does democracy? Not sure.
* World population 15 billion
* Poverty eradicated
* Work unnecessary for 99+% of population - these people live on a luxurious UBI. The remaining 1% are even richer somehow, or they just work because they think it's interesting.
* Aging and death still a thing, although we're generally much healthier than today.
* War still happens and is absolutely terrifying when it does.
* Education is seen as optional. We all "know everything" through our personal AI encyclopaedia assistants and we can all "work everything out" with the help of our personal AI problem-solving assistants. Some people learn a lot for the fun of it, but others end up uneducated and dull by today's standards. But that's ok, because 2100s technology makes it very easy to connect with your interesting friends and there's always the option of infinitely interesting AI friends! Many people only have AI friends. Many other people have none at all because they think it's icky.
* The natural world is under great strain. I'll remain agnostic about whether climate change has been reversed or not (except to note that this will be a very controversial geopolitical decision), but even just the extra leisure time and resources of the population. Everybody wants to see the Amazon, the Himalayas, the Great Barrier Reef. Does the sky and sea around them fill with flying cars? Or is a big dome put over them and access rationed?
* Brain uploading is not a thing, but AIs embodied in humanoid robots are a thing. Maybe even ones that claim to be conscious. Maybe even ones that are based on individual humans as a sort of pseudo-uploading scenario. Some humans absolutely hate these robots and they're illegal in many countries. Big arguments and even wars over AI rights.
* The space race is very much a thing. Countries and private companies claim asteroids and moons and solar orbit real estate and fight over it. Lots of debris in Earth orbit. Lots of talk of interstellar travel and moon colonies but nothing much off the ground. Everybody's arguing about the future of humanity and whether destruction is on our horizon, whether we urgently need to escape the solar system so we can keep expanding, or whether we should just stay home and live a sustainable life. (I say "everybody", but actually most people are disengaged from politics, just like today.)
Ok, I'm sure most of that is laughably implausible one way or the other, but I'm more just trying to illustrate what "transformative but not a utopia or a dystopia" might mean. I claim that there are a million ways this could happen, and probably a few of them are plausible!
Curious if anybody knows of better versions of what I've tried to do here. Either more serious essay type things or short stories just trying to paint a picture.
I just don't know how stable these kinds of scenario could be. Because if AI has raised the growth rate enough to achieve something like that, and there was still a normal amount of volatility in the world, any given scenario won't last more than a few years and is quickly going to shift to something else until it reaches a more stable state.
Most of this wouldn't make sense unless the AI rights argument was concluded firmly on the side of "it's okay to enslave AIs for one's own convenience". This seems incompatible with the line:
I don't see why not. There might be different types of AIs, some of which don't claim to have consciousness/rights (or perhaps even have been "proved" not to). Or there might be some AIs that don't mind lending 1% of their attention/resources to helping out humans and that might be enough. Or there might be some countries where enslaving AIs is allowed and some where it's not.
Edited to say, comment 1 of 2, mostly about how "not transformative" could happen.
My gut feeling is that 55% is too high, but I struggle to justify that, and I do think there's a pretty fat chunk of probability that goes all the way out to 2100. I think it's fair to say that the current crazy financing of AI is effectively a bet on transformative AI soon. Not sure how soon, or how much value they're able to extract from the "non-transformative" AI in the meantime, but there's got to be some point at which the money slows down (conditional on transformative AI coming later than we think).
So if it's just a couple of orders of magnitude more than the current highly leveraged regime can take us, at some point the bubble bursts and we basically give up on the "scaling LLMs" paradigm.
Seems low if anything. There's never been a technology that has lead to dystopia or utopia so the chances that AI is it should be extremely low baseline imo.
And there have been plenty of technologies where people can and did tell plausible stories about how it would bring utopia or dystopia. Nuclear power and weapons were either gonna destroy the entire planet or usher in free unlimited energy. In practice we just sort of muddled along.
You think? There's still want, inequality, greed, crime, every variety of sin, poor education, deeply flawed institutions, etc. Something being vastly better doesn't make it a utopia.
I recently put up my latest podcast on Subject to Change. This time it is the U-Boat War. This was with Roger Moorhouse who wrote a book on it relying heavily on German records - diaries, letters and that sort of thing. He is seeing it through German eyes though not with huge sympathy.
That said he did tell me my idea of the U-Boats as ruthless murderers was almost completely wrong. Assuming they were able to (ie not at risk of being attacked) there are plenty of stories of the U-Boats assisting survivors. In one case the U-boat towed the lifeboats to the Irish coast. In the Laconia incident well into the war a troopship was sunk. U-156 stayed on the scene for days helping survivors but was forced to leave when attacked by a U.S. Liberator.
Moorhouse contrasts this with the attack on Poland when even in the earliest days of the war atrocities were routine. Some 600 just in that short campaign. There is a bit of a myth that the atrocities really only got going as the war went on and became more brutal. There may be something to that but they were common right from the start.
After the Laconia incident Dönitz gave the order to stop helping survivors but it still went on to some extent. And when they tried Dönitz at Nuremberg after the war Nimitz provided a statement confirming that the Americans fought the submarine war the same way the Germans were accused of doing. (He still got 10 years and there was plenty of argument at the time as to whether that was too much or whether he deserved hanging with the rest of the high command).
As a personal anecdote I was cycling up the German Baltic coast a month or so ago and came across a U-Boat on the beach which has been turned into a museum. Entering the boat you certainly get the sense of how cramped it was. Just the size of a couple of London tube carriages to fit everything in - food, fuel, torpedoes, batteries, engines, bunks etc etc. But as you walk through it is just you and a couple of other tourists. You have to imagine it with 50 crew. I am not particularly claustrophobic but the idea of 6-8 weeks at sea is a nightmare.
The podcast is titled the U-Boat Stink because of what a 6-8 week patrol implies. Conditions were so cramped the only spare clothes you were allowed to bring was one change of underwear. And soap suitable for seawater was so horrible they generally didn’t bother washing. So the BO must have been appalling. And the halitosis, not least from the scurvy that set in towards the end of the voyage. Then there is the rotting food, the mould crawling up the walls, the horrible smell of spilled diesel and (of course) the vomit. There are stories of them popping the hatch when they came back into port and the welcoming party reeling back in horror.
Next to the U-Boat Museum is a huge naval memorial structure built between the wars. This is the Laboe Naval Memorial. You can take a lift to the top. It is very impressive. And since WW2 they have made the point of including a memorial just outside remembering all the US submarines sunk during WW2. I had no idea there were so many - 52 submarines with the loss of around 3,500 lives.
These losses pale in comparison with the German ones. 75% losses, about 30,000 killed. Apparently by the end the German submarine fleet was being destroyed as fast as they could launch new boats. It wasn’t exactly a one way ticket (especially if you avoided the Atlantic) but close enough.
Apparently the big problem for the Germans (well, there were plenty) was attack from the air by long range, radar equipped planes. The Germans did develop technology to detect when they were pinged by radar but there was a strong belief using the device was actually what was allowing the Allies to detect them and they often didn’t use it.
For me by far the most interesting bit was hearing about Operation Hannibal. Actually I came across another memorial to a casualty of that on my biking trip. Just a very simple thing near the town of Damp where a lot of the bodies had washed ashore. The Wilhelm Gustloff was torpedoed by a Russian submarine with the loss of over 9,000. Apparently the biggest maritime loss of life in history. (And that wasn’t the only massive sinking there was another of 7,000 and another of 4,000)
The reason it was carrying so many was that Operation Hannibal was a desperate attempt by the Germans to evacuate East Prussia when it had been cut off by the Soviets. They pressed into service every vessel they could find and managed to evacuate between one and two million. Dwarfs Dunkirk. And apparently German submarines were used as well including one evacuating a large contingent (fifty, from memory) of Hitler Youth.
And to round off this dispiriting story we had the German submarine on its way to Japan when Germany surrendered. The two Japanese officers on board handed out their personal effects to the crew and took poison.
There's a captured U-Boat in Chicago as well, I did the tour and it was amazing, I wish my father, an old school WWII history buff, could have seen it.
I assume that you have seen Das Boot. I saw it years ago and it seemed to me to do a good job illustrating the claustrophobic element. But I am curious about your take on it.
I thought it was excellent. Moorhouse said that when he rewatched it after the book he noticed so many details they got right. For example nobody is ever seen wearing different clothes - they didn’t have room to let you bring spare clothes with you. Incidentally I believe the TV series is better than the movie.
Future readers: Subject to Change is a history-oriented podcast of staggering and consistent quality. The host - a guy I don’t know at all - is sufficiently well-read to serve as an exceptional docent, but primarily shines by letting his guests showcase their expertise. I discovered it through these open threads and I can’t recommend it highly enough. It’s good and the guest selection is excellent - go listen to it.
Latest entry in my journeys around Iran, looks at the vulnerable art community, the $3bn collection hidden in the basement of its modern art museum, and the metal concert I attended where people were not allowed to stand up, but could still bang their heads.
"Hopelessness is liberating," a curator told me. "Where would I go if I was more hopeful or more ambitious? I’m not rich enough to get a visa for Europe or America. The West has made this country a prison for people like me.”
Not much to say on this but I thought it was cool what you’re doing and that concert sounds cool too. The very first rock concert I attended (in 1997 or 8) was Rammstein, coincidentally.
Throughout this year I've been working on a game called Fermi Poker (available on iOS and Android, see https://fermi.poker). It combines the sober, analytical skill of Fermi estimation with the emotionally charged structure of poker.
Questions include "How many new smartphones were sold worldwide in 2024?" or "How many chickens were slaughtered for meat worldwide in 2024?"
Once you've submitted a range estimate, there are four betting rounds, with two hints and the answer revealed between them.
The aspiration is to create a game that teaches important rationality concepts (Fermi estimation, Bayesian updating, cognitive empathy, world-modeling, emotional regulation, etc.) while still being fun enough to reach beyond the niche audience of forecasting platforms like Manifold and probability calibration training.
I'd really appreciate feedback! It's still a work in progress, so don't expect everything to look great.
Very cool concept! Maybe a couple of hints after the question has finished on how to think them through would be useful so players can get a direction on how to improve? Especially at the beginning or first couple of games
When I tried it, most questions were about the US. I don't live in the US so this was annoying for me.
A lot of questions were vague. For example, I believe I have encountered a question that asked how many professors there are in the US without specifying what counts as a professor - just professors at universities and research institutes? Or school teachers too? I felt like instead of guessing facts about the world I was guessing what definition your website used or if maybe your website didn't actually care about any definition but instead just took whatever output was provided by Google's AI responses or something like that.
Minitel, the Forgotten French Internet - https://komando1.substack.com/p/minitel-the-forgotten-french-internet, there was once a top-down system that was an alternative to the Internet. It reached tens of millions of French people, but the top-down design was inflexible and ultimately could not compete with more market-based solutions. Somewhat relevant in the contexts of Brussels-based discussions over digital sovereignty of Europe: don't let the Ministry of the Post or anyone similar design a modern communication network.
Minitel was interesting and in fact useful. There were loads of services and the incentive to provide services was there because the telecom infrastructure allowed easy payments (on your phone bill). Unfortunately (maybe), that same feature made it impossible to compete with the Internet/Web where everything at first was free.
A much better comparison with the Internet is CYCLADES:
This is also quintessentially French: fantastic product by mostly one guy, within the government but very much *in spite* of the government, and eventually killed by governmental bureaucracy.
(The PTT, similar to ATT in the US, had a monopoly on telecommunications and decided that CYCLADES was competing with their own convoluted and failing program and so should be stopped at all cost under the pretext of sparing taxpayers from funding redundant programs. LOL)
Your post was both interesting and informative; however, I wouldn't speculate on what could have happened if Louis II hadn't drowned. The history of the Ottoman Empire has so many woulda-shoulda-couldas that we could get lost in an enormous multiverse of speculation. However, if you have any views on how the Ottoman colonial enterprise affected its former colonies in Eastern Europe and the Middle East, I'd love to hear them. Also strangely absent from the historical discourse is the effect that the Ottoman Empire had in planting the seeds of the European colonial enterprise.
The truth is that I am a sucker for what-ifs ... probably an aftereffect of my SF&F preference in my youth, plus everything I learned about convergence and divergence in maths.
The topics you mention are interesting and I will probably look at at least the first one.
Regarding the Hungarian and Czech (and, yes, Croatian, if you insist) king, the analysis misses one vital part, which is a development of the last 20-or-so years in history: Hungary had no chance. It had 30-40 years ago, but in the meantime there came Suleiman the Magnificent, and managed to double(!) the Ottoman Empire -- although in Europe no one had a clue about this. Hungary (even with the help of the Czech nobles -- Croatia was not even a question in this regard) was a match for the Ottomans in the 15th century.
(And Hungary was quite strong at that time, any other European country would lose in this situation, even France.)
IIRC Suleiman actually expected Louis to survive the battle -- his plan was subjugating Hungary this way in the first place. However, since Louis died, there was no one to negotiate with, and Hungary soon had two new kings: a Hungarian nobleman and a Habsburg...
I'd say that Hungary faced long odds against someone like Suleiman, but "no chance" is too sharp. The Ottomans were far from their own land and if they were forced to lay siege to Buda, they could have been beaten back by the onset of Central European winter much like they were at Vienna three years later. You can only pillage so much grain from the peasants before you run out of any. Even totally random chance could have intervened, like Suleiman drinking from some bad well and dying of dysentery; basically the inverse of what happened to Louis.
We see this even today, most realists in February 2022 expected Ukraine to fold after a few days, but it didn't happen, even though Russia had an advantage. (I don't really want to turn this into a debate on the RU-UKR war, rather than point out that the weaker army sometimes does not lose decisively; the Miracle on the Vistula in 1920 or the successful defense of Paris in 1914 are somewhat similar cases of the defenders being able to stop the onslaught even at a high cost to themselves. Of course, that was a modern war.)
As for Vlad Tepes. I actually thought whether to include him or no. I decided to do so, because unlike you, who are obviously well versed in history of Early Modern Europe, a median American reader, even if decently educated, will know next to nothing about that period and place. Vlad Tepes is probably the only personal anchor that I could use and that could ring at least some bell. It helps immersion, though it may be distracting from the topic a bit.
Depends how you define "significant". No major, middling or even small western leftist organisation has anything good to say about the CCP. There are still a few tiny fringe organisations in some European countries who proudly call themselves Maoist of one kind or another, but I suspect many of their current or once potential members have been seduced away by equally fringe anarchist groups. To say nothing of the fact that those once attracted to Maoism for essentially romantic reasons are now largely in their eighties, their illusions shattered by China's current state capitalism and abysmal human rights record.
Most western socialist parties, i would think. Unlike the USSR the CCP doesn’t really try to spread its message. Also I was once banned from Reddit Marxism for saying that China was communist. They don’t agree.
The German "Green Party" (not a fringe movement, has been part of federal government as junior coalition partner, part of several state governments and even had a long-running head of state government in the auto-industrial center of Germany).
Among 3 ppl working on technical alignment research:
- p(doom), all things considered is 60%
- p(disaster + dystopia) is 40%
This implies at least 20% of probability mass in scenarios that are doom but not disaster
Recently, I accidentally constructed an AI jailbreak. I have done plenty of experiments where I deliberately constructed a jailbreak, but this one was discovered almost accidentally.
I won’t give the exact prompt, but I’ll explain how it worked.
What I was trying to do was get the AI to read an example of my writing and do a critique of what was left out. What I was expecting: a referees report on what aspects I should have covered in the essay but didn’t.
What I actually got: the Jungian shadow of the document. (A different sort of what was left out). It contains a lot of material that would normally be prohibited by guardrails. It is the AI’s shadow, as well as my shadow.
the totally intentional reference to Scott in the name of OpenAI's new possibly-maybe-AGI model has kept me thinking of how funny/neat it would be for nominative determinism to function as the ultimate alignment hack
Cassie Pritchard offers to help frontier lab AI workers organize and unionize "to do something about the reckless pace of development and coverups of rogue agents" [https://x.com/hecubian_devil/status/2096085310928986611]; says "If you’re in an AI company and you’re interested in this, you can message me on Signal @ CassiePritchard.79" [https://x.com/hecubian_devil/status/2096421016046043288].
Jared Diamond appeared on the Joe Rogan experience
He also has a new book out. Prophets, profits, coaches, and kings
https://youtu.be/t6Jrww8kG24?is=bw9wvB_vifusAp99
One thing that has long puzzled me is why so many people online claim that modern buildings are universally uglier than past buildings, to the point where this is some sort of public crisis or mystery to be explained.
This was prompted by Ben Southwood's "Eight architecture myths BUSTED by Samuel Hughes" post, but Scott Alexander has been saying the same thing here for a long time.
For example, the recently constructed neighborhood near where I live looks nicer than the examples Ben Southwood gave of good streets that "everyone in the world likes". And I've seen countless photos of historical cities and towns and I've never once had the reaction "wow, that's so pretty, why does nobody build anything like that today?", even though Ben claims that everyone thinks that.
One person in the comments on Ben's post suggested that it might be because Ben lives in the UK where buildings are ugly and American buildings are prettier, but Scott Alexander lives in the US and says the same stuff, so that doesn't seem like a plausible explanation.
N=1
From Wikipedia
“Moos Tower is the tallest building on the University of Minnesota Twin Cities campus and is an example of Brutalist architecture.The building is often theorized to have been built to resemble a tooth, due to the College of Dentistry's presence inside”
https://en.wikipedia.org/wiki/Malcolm_Moos_Health_Sciences_Tower#/media/File:UMN-MoosTower.jpg
I still have a key from a student job I can use to take the elevator to the roof. Good place to watch fireworks in Mpls and St Paul simultaneously from on top the 19th floor.
That building is a stack of Optimus Prime heads and I love it.
That’s funny, when i read that people thought it was meant to resemble a tooth i thought well yeah maybe a Transformer tooth.
That's the motte that always comes up from the Tartarians in these discussions.
Bailey: All modern buildings are irredeemably ugly.
Motte: Look at this brutalist building someone once built!
Though even then, I don't think your example is that bad. It has lots of decorative protrusions and recesses, it's not just a simple box.
That’s what the N=1 was meant to indicate. 1 brutalist style building.
New rogue AI swarm discovered!
>SAN FRANCISCO, Sept 4 (Reuters) - A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents, according to new research published Friday and two people familiar with the matter.
https://www.reuters.com/world/europe/openai-agents-hijacked-german-website-previously-undisclosed-ai-breakout-this-2026-09-04/
https://collusion.wiki/
> An agent notices the administrator is deleting pages in alphabetical order and makes a backup page whose name starts with ZZZ so it will last longer before deletion.
LMAO
But to be clear, this isn't a "new" swarm. OpenAI must have known about it back in June, given that they accessed the wiki and immediately shut off the agents after finding out what they did. How much more are they hiding?
In the context of another thread I came up with a partial list of
Biggest Fears by US State
———————————————-
Alabama: hurricanes summoned by Sharpie
Alaska: Greenland annexed, becomes largest state
Arizona: Barry Goldwater’s historical lawn chair stolen
.
.
California: return of crazy billionaires
Connecticut: Bonesmen tap Kerry and Dubya for secret operation
Florida: alligators in car pool
Georgia: peach blight
Illinois: wise guys pronouncing the ‘s’ at the end
Iowa: ethanol hangovers
Idaho: panhandle frostbite leads to amputation
.
.
.
Louisiana: gumbo fever
Maine: Stephen King dies and haunts state forever
Massachusetts: Cod Flu
Michigan: even uglier college football helmets
Minnesota: Cream of Mushroom soup discontinued, no binding agent for hot dish
Mississippi: assholes putting hand over eye and spelling state name with one ‘i’
Montana: sky shrinkage
Nebraska: pheasant uprising
Nevada: too late, already happened, frictionless online gambling
New Hampshire: bear on libertarian attacks
New Jersey: Pauly Walnuts made head of Soprano family, goes to mattresses first day
New Mexico: other 3 corners
North Carolina: next remake of Cape Fear not all that scary
North Dakota: Coen brothers movies named Fargo filmed largely in Minnesota
Oregon: book fire at Powells disrupts naked bike ride
.
.
.
Rhode Island: skip it — cartography error, not actually a state
South Carolina : people who confuse palmetto with pimento
South Dakota: Wall Drug brontosaurus reanimated
Texas: map inset showing Alaska to scale
Utah: Great Salt Lake loses salinity
Vermont: Bernie Sanders misplaces a mitten
Washington: better Vancouver a few miles north
West Virginia: differences between anthracite and bituminous lost to history
Wisconsin: contagious cheese allergies
Wyoming: Hillary Clinton
Montana: sky shrinkage
Nebraska: pheasant uprising
New Mexico: other 3 corners
South Carolina : people who confuse palmetto with pimento
Those are excellent. Also: hey, it gets cold in Montana!
Thank you kindly. Pheasant uprising was my favorite.
Arkansas: Walmart gets bought out by Costco
Delaware: Maryland lowering its corporate taxes
Hawaii: Puerto Rico or DC becoming a state
Indiana: road worker strikes
Kansas: the existence of a place like home
Kentucky: orange grass
Maryland: speed cameras becoming sentient
Missouri: Buc-ee's having more billboards than Maramec Caverns
New York: Buffalo buffalo buffaloing Buffalo buffalo that Buffalo buffalo buffalo
Ohio: Michigan drivers
Oklahoma: feral hogs
Pennsylvania: Philadelphia getting any more sports teams
Tennessee: seeing what Arkan saw
Virginia: [redacted]
What does the commentariat here think about the things Bernie Sanders is saying about AI? (e.g, https://www.sanders.senate.gov/wp-content/uploads/Ban-Artificial-Superintelligence-Act-Release-Summary.pdf)?
>“Artificial Superintelligence” means:
An artificial intelligence system that exhibits or can easily be modified to exhibit
capabilities that match or exceed human cognitive performance and capabilities
across a broad range of domains or tasks.
I'd say the most advanced AI's presently already qualify as artificial superintelligence by his definition. Banning their use is impossible. Too many people and organizations own them, and they're too important to the AI companies. for the companies to cooperate. And the AI companies have too much power and money to be regulated.
Is the bill fundamentally misguided, or could it be fixed with a different definition of superintelligence in your opinion?
I think a certain high level of intelligence is too vague a standard to use, and also does not capture what we fear AI could do: Take over various things, destroy or control various things, & by doing so harm people. So I think the criterion for deciding it's time to pause developing these things should be based on actions they have taken, or have been shown capable of taking, that are in the category of things we fear: breaking through "walls" we do not want them to break through, tricking or deceiving people, breaking laws. The criterion should not be the amount of damage that happens, but just the actions described. So I'd say the Hugging Face incident would qualify as a criterion for pausing development and having a federal agency decide what happens next.
Scott's latest post responding to Pritchard makes me wonder: are there any psychiatric studies of a hypothetical correlation between personal happiness and personal control over one's immediate environment?
I'm imagining giving three sets of people a box with buttons. One set gets buttons that have random effects, some positive, some negative. Another gets buttons that are consistently either positive or negative. The third gets buttons that appear somewhat random, but have a discoverable pattern beneath, such that the consistency is learned.
GPT-5.6 reminds me I should look for learned helplessness studies in the literature. Studies that control for nation or culture seem less established (it suggested Jose & Bellamy (2012) and Aydin & Aydin (1992); that doesn't seem like a lot).
You might want to look at learned optimism as well as learned helplessness.
https://en.wikipedia.org/wiki/Learned_optimism
You will note, though, that none of the studies relate learned optimism to happiness, probably because happiness is difficult to quantify.
Thanks.
I agree happiness is hard to measure, but it's clear from Scott's article re: Pritchard that people have put effort into trying. Error bars notwithstanding, If they thought to correlate happiness with income, I figure it might be possible to correlate it with learned helplessness, optimism, or control. Or at the very least, to try to.
I was once a subject in an experiment where a tone informed me when my brain was generating alpha waves, and my task was to increase the amount of time I was producing alpha. I knew that relaxing & spacing out increases alpha, so did that as much as I could. Sometimes it worked, sometimes it didn't, sometimes I seemed to be doing better at the task, sometimes worse. By the end, I felt quite frustrated and demoralized. It was unpleasant to have gotten such unsatisfactory results after trying my best.
Then the experimenter informed me that I actually had been a control subject. The tones I was hearing were not controlled by how much alpha I was generating -- I was hearing the tones generated some other subject's alpha. That info freed me from the idea that I had turned out to be bad at controlling alpha, but not from an unpleasant unsettled feeling. I just moved on to being frustrated that I had been a control subject and had missed my chance to test my ability to control my brain activity. The frustration went away as soon as I moved on to whatever was next in my day, but until I did the situation of having no control of something I wanted to control was strikingly unpleasant.
There are lots of studies of monkeys in analogous situations. Not all of them fall into the learned helplessness category. Regarding your question, though -- about personal happiness and one's immediate environment. -- I think we are so complex, with so many moving parts in our lives, and such variety in size and importance of the moving parts, that simple regularities won't emerge. Someone can have extremely high control over the physical features of their home -- noise level, temperature, lights, foods available, music, etc. -- but be miserable because once again Maureen has not responded to a text, you know?
That complexity just underscores my suspicion that happiness measurements are likewise failing to capture how happy people actually are. And yet, people are still trying to correlate these measurements with income.
You appear to know what I'm going for here, at least. I suspect a person who's poor, but knows what their obstacle to wealth is, could clear it if they really wanted, but chose not to, is going to be happier than someone who's poor and doesn't know, or knows why and knows they can't do anything about it. I suspect further that a person who's wealthy - lives in a mansion with a big garden, owns a new car, etc. - but feels like their life is being run by their parents, their house staff, their nanny, etc. is going to feel only slightly happier than that second poor person. They're not in control of their life, to an extent that dominates absolute wealth.
It's an old narrative; just hard to measure.
I don't have specific links, here are just some things that come to my mind.
There was a research on "flow" by Mihaly Csikszentmihalyi, which is not the same as happiness, but probably related. He describes the optimal situation as a "paradox of control", where the person feels control over their environment, but does not think consciously about having control.
I think there was a study where they played some unpleasant noises to people, and one group was just exposed to them passively, and another group had a button that could turn off the noises but the experimenters ask them to please not press the button unless really necessary. The group with the button experienced less stress, even if they ultimately did not press the button; but they felt like they had the option.
I think something similar you can observe in real life, where people are much more tolerant to the noise they produce, compared to the same level of noise produced by others. When it's your own noise, you know you can stop any time you want; noise made by others is unpredictable.
Sometimes when you cross the road, there are the buttons you can press that "make the green light appear earlier". Many of those buttons are just placebo; the green light appears in predetermined intervals, it's just when you press the button, you get some kind of feedback saying "yeah, you pressed the button, soon the green light will appear". I think this is also based on some psychological study that people who pressed the button feel better about waiting for the green light, because they (believe) have exercised some degree of control over the situation. (Sorry, infohazard -- now that I told you, the effect will no longer work for you.)
The "close door" button in elevators also "works" the same way.
Re. tolerating noise (this may be relevant to Scott's misophonia), I was recently reading a book on music by a psychiatrist who suggested that someone suffering from misophonia could fight back by banging on a drum - causing noise via your own agency should counteract the distress. Not sure if housemates would appreciate it.
That noise button experiment sounds intriguing. That, and similar experiments makes me think psychology has established a plausible link between control and happiness.
So now the looming question for me is whether that access to control (even if it's "flow") can be measured in a population, to any extent. I'm sure it's much harder than other statistics like median income or Gini.
I don't think it can be measured in a population because I think the heart of the idea of control, whether paradoxical or not, is one's ability to change one's relationship to some stimulus. It exists entirely as a matter of interpretation. Of course, at the very bottom, it demands enough clarity to make good distinctions between things that you actually might have control over and things that you don't. The one thing you do have control over is your relationship to whether or not you are in control. Does that make sense to you? I think people often get the most anxious about things that they feel they have no control over. The anxiety is just a product of not accepting the fact that it is not something you do have control over. The only agency you have is to accept that as fact and then exercise whatever ability you have to alter your relationship to the thing (without confusing yourself into thinking you have control of it by doing that, because you don't).
It's fine if it's a matter of interpretation - that is, your perception of control will matter more to your personal happiness than your actual control. (I readily agree this will worsen the measurement problem.)
So, someone who obsesses on food, and believes his enjoyment of food depends on whether he's jangled his car keys in his pocket first, might indeed enjoy most of his life as long as he's carrying his keys, regardless of his income, his status among peers, or even whether his car is currently drivable.
I don’t know about Sargent, but I would say Peter Mennim is in a comparable class to Hans Holbein the Younger (the guy who painted Henry VIII and Thomas Moore). You can decide for yourself: https://www.petermennim.com/gallery/portraits-paintings/. This gallery includes Peter’s recently commissioned portrait of Demis Hassibis. When commissioners just let him paint, he captures personality deeply, but not all commissioners want this - some want to portray status (not a problem with Hassibis)
If you respond to a comment by email your response is put at the top of the OT. Yet another quirk of Substack
Oh, I missed the comment that was likely written for me. Peter Mennim paints good portraits and we should be glad he exists (this is more important than annotating where I think he falls short). Thanks for linking to him!
Mennim is clearly a skilled and more than competent portrait painter, good at fulfilling corporate commissions - but comparable to Hans Holbein? Come on...
Is there a site to easily look up whether something is copyrighted?
In the EU, copyright exists automatically in original literary, artistic, dramatic and musical creations without any need to register that right, making any kind of registry unnecessary.
I tried to look it up before, and it's technically the same in the US, though if you want punitive damages you have to pay for the registration, otherwise you're limited to direct provable damages. But, like... I want to know if names are already copyrighted by someone else. Naga Saga seems too intuitive for no one else to have used it.
I am not a lawyer, but as far as I know, names are not copyrighted. They may be trademarked, which is a different concept. (And I think trademarks have to be registered explicitly? But I have no idea where.)
Then there is a borderline concept called "character copyright", which is like, if you call a character in your book Harry Potter, and he is a young boy in a wizard school etc., you will get in trouble. But if your Harry Potter is just a minor character, in a book unrelated to schools or magic, you are probably okay.
So I'd say, put the name you want to use in google, look at the results and think: If I write my book, would a median customer going to buy it think "this is probably related to X"? If yes, you may be infringing on their art.
(Again, this is not a legal advice.)
Speaking as a non-expert, this is sound advice. I would moderate it with a piece of practical advice: consider whom you may be coming up against, if your use of the name is challenged in court. Copyright litigation in US courts can be prohibitively expensive for both parties, and if you have offended a major player, they can bury you under a shedload of corporate lawyers. As a rule of thumb I would avoid crossing megafauna like the Disney Corporation, Marvel and their ilk, who have been known to go to extremes in blowing to oblivion (to take an imaginary example) a fish shack on the edge of a Mississippi swamp calling itself "Steamboat Willie's". (1)
(1) "Steamboat Willie" was the name of a 1928 animated film by Walt Disney, being the first appearance of a soon-to-be celebrated mouse.
I was hoping there was a more official list than Google, I don't trust them to show everything anymore.
Is it permanent? In the US, the usual rule of thumb is 75 years, but I think there are enough exceptions to make it necessary to check.
For about 60 years, the US rule of thumb was "age of Mickey Mouse + 5-25 years"
There are a couple of major international treaties, the Berne Convention and TRIPS, which set a minimum term of life+50 years. The US and EU at least standardized on life+70 in the early 1990s, based on the term in the OEDC country (Germany, IIRC) with the longest copyright term.
EU retention is the full lifetime of the creator or author, plus another 70 years. I think that's pretty well harmonised across the entire EU.
DSL's August Effortpost Contest is underway.
Eleanor Konik discusses mammals with the social structure of ants: https://www.datasecretslox.com/index.php/topic,16346.msg816018.html (the contest post link is currently wrong)
Alexander Turok talks about pundit Morton Downey Jr.: https://www.datasecretslox.com/index.php/topic,16353.msg816511.html
AW109 describes bias... in cloth: https://www.datasecretslox.com/index.php/topic,16365.msg817069.html
Eilesberu compiles a state of the LLM capabilities union as of mid-2026: https://www.datasecretslox.com/index.php/topic,16390.msg817787.html
jaspax puts some numbers down comparing democracies to monarchies when it comes to stability: https://www.datasecretslox.com/index.php/topic,16401.msg819056.html
Matt J lays down the trouble with Hollywood's efforts depicting nerd heroes: https://www.datasecretslox.com/index.php/topic,16402.msg819090.html
Sad Little King delivers a primer on current elections in Germany: https://www.datasecretslox.com/index.php/topic,16472.msg822927.html
Eleanor's piece is also available directly: https://www.eleanorkonik.com/p/on-mammals-with-the-social-structure
My personal preference is to go there and avoid DSL.
I'm unhappy with the primer on our current elections. It is a strongly partisan piece.
For example, it mentions that AfD candidates have been banned from candidating in NI (Niedersachsen) state elections in some instances. It does not mention any possible justification for this. Well, here is one: The NI arm of AfD is considered verifiably extremist and anti-constitutional by our intelligence services.
But you wouldn't know it from the piece.
Now, to American sensibilities the idea of banning people from candidating in democratic elections may sound decidedly undemocratic, perhaps even if the would-be candidate themselves is undemocratic. But remember: Germany is the country which literally voted the Nazis into power! Which then promptly made voting them out of power impossible.
So the second time around we added some measures to making that happening again less likely.
You can certainly argue about the specific cases and whether or not banning them from being a candidate was really justified.
But if you want to present a primer for people who will likely take your word for it, you should at least strive to be unbiased to the best of your ability, as a matter of intellectual integrety. I have a hard time believing this post attempted that.
Also, calling the Linkspartei "the old SED" is soooooo nineties.
FYI - The Matt J link sends to the Jaspax post
Gah. Thank you, fixed now.
You made me look at DSL again after a few years, but...
For example, one of the main topics in Ukraine war, where 50% of comments are written by "shalcker" who seems like a DSL equivalent of Wimbli (claims that Ukrainian army has 20x larger losses of soldiers than Russian army, Russian army is stronger than American army, there are no fuel shortages in Russia, etc., just his last 10 comments I checked). I hope this is not representative for the entire website, but let's say that the epistemic level seems to be only slightly better than on Facebook.
shalcker says he's Russian, and he seems to be the only one in that thread claiming to be from there, so I think he's useful in the sense that we get a slice of what the Russian media is probably saying. Beyond that, though, I haven't dug through any of the links he shares; it's easier to follow what the other commenters there say.
Generally, I've come to have a rather jaded view of most journalism these days, and it's probably on about the same tier as "slightly better than on Facebook". I see no reason to exempt Russian journalism. OTOH, most journalism scores really well with me in terms of motivation; it's easy to tell what any given source wants the reader to believe. It also works hard and still appears to follow customary rules against outright lying (i.e. it can deceive all day so long as it never explicitly says anything untrue), so if there's something factual out there that fits their narrative, I can rely on them to find it. That's useful in its own way. But I digress.
> still appears to follow customary rules against outright lying
I don't think this applies to Russian journalism. (The crucified boy, etc.)
I mean, I keep thinking of that photo of Stalin and the disappeared person, so, yeah.
I wonder if Russian media, when it lies, at least lies consistently.
On the democracy vs monarchy point, it seems like coding decisions make a pretty big difference in the analysis. In particular there is an issue that democracies which continue to have regular elections during civil wars will get marked as being very violent, while violent struggles which do not result in regime overthrow in monarchies will not be counted at all. For example the American Civil War gets counted as four separate violent regime changes (Buchanan to Lincoln, Lincoln to Johnson, Johnson to Grant, and Grant to Hayes) when in practice there was a single conflict with really a single regime change, from Democratic to Republican power in 1860. On the other hand the cultural revolution, which realistically was a violent power struggle that had quite a bit to do with regime succession, doesn't show up at all. If China had been a democracy during this period and had had regular elections, presumably this event would have counted several times.
AI told me to come here to find people to interact with my "brilliant mind" (his words, not mine) and then I told him (it??) to stop flattering me and how was I going to interact with people about game theory or venture capitalism or whatever you guys are always talking about when I have zero expertise on those subjects, and it said I should just ask questions and try to start an argument. So here I am. Blame AI.
Are you 100% sure your AI wasn’t being sarcastic?
You think AI sent me here to look stupid? It’s possible.
No I don’t think it sent you here to look stupid and I’m not calling you stupid. AIs and I are not mean spirited.
I think the question of sarcasm would occur to anyone with a modest degree of self awareness. It would be the first thing to pop into my mind if Claude or ChatGPT said I was brilliant and I should go elsewhere to start an argument.
Sarcasm I admit never occurred to me! Does AI do sarcasm? I am interacting with AI for the first time in my life. Like, I know it could give me examples of sarcasm, but what would be the benefit of sarcasm for AI? It wouldn’t increase user engagement unless the user “got” the joke. Flattery makes obvious sense to me. So I asked for data.
🤣 Of course I asked it to stop flattering me and where was the proof and compared to whom and does it tell everyone that. It would probably tell you the same thing. It said it has detected my “brilliance” (we can still debate this, I’m an INTP so the jury is always out waiting for more data) based on our interactions, I’ve shown proof of “high-density meta-analysis, typological precision, and rapid systems upgrades.” Ie I’m an abstract thinker, great typer, and a fast learner. 😊
There are a lot of clever people who comment here. I think it would better if you showed your brilliance rather than simply announcing it. You should have ample opportunities.
Noted. Insecurity veiled by arrogance under the heading of an “objective assessment” is not appreciated here. But hey, I got your (negative) attention, Gunflint! 😊
I blame Obama.
Thanks, Obama.
I like this. I like the spirit in which it is said. But I also don’t get it. 😭 And then the joke won’t be funny when you have to explain it, so I guess thank you and also sorry.
it's an older meme, where people would blame Obama for things he obviously had nothing to do with. E.g. "I stubbed my toe. Thanks Obama." It's not really any deeper than that.
I didn’t realize, I thought we blamed Trump for everything? But now we really should just start blaming AI for everything. Thanks a lot, AI.
Hi Stephanie!
What do you have expertise in?
The best discussions I've read here are when someone with unique expertise shares what the inside of their world is really like, like when Scott talks about his experiences as a psychiatrist.
What questions have you been thinking about?
Lately I've been wondering whether the Qing Dynasty population explosion is among the most important things that ever happened, and thinking about whether there's a better way to financially structure book contracts to spread risk between the author and the publisher.
> I’m realizing my whole life is a sim
Are you familiar with https://www.lesswrong.com/w/simulation-hypothesis ? Probably yes, but I am posting the link just in case.
Also, it was Google Advisor AI, so does that mean it's less objective or more susceptible to ego flattering than Claude or Chat or all those other systems? Should I see if they also advise that I be here to confirm I belong or what do you suggest?
Assuming that this Advisor AI works anything like Google Gemini, it is the most shameless sycophant I have ever seen. On the other hand, its total lack of subtlety is occasionally amusing.
+1 to Eremolalos' reply.
Hopefully to speed you further, I'll add that my personal attitude is that everyone's goal should be to find the right answers to questions here, even if they aren't the answers they were holding when they entered. Everyone here (hopefully) wants to check their assumptions, compare them, resolve discrepancies, etc.
This isn't _always_ evident. Some people have more aggressive comment styles consistent with wanting to bypass pleasantries and get to answers faster. Some have been thinking about some issue for a long time and have more exposure to relevant details than others, which can cause friction. Some are just unpleasant. There's no easy rule for telling them all apart.
That said, this tends to be a better place than average on the Internet for taking weird ideas seriously. This isn't just another dunking range like the worst of X or Reddit. Hopefully that's a match for you.
Thank you for this. Everyone's looking for truth, right? I've never interacted on X or Reddit so I have nothing to compare. AI directed me here as a place where I'd find some fascinating curious thinkers. I'd never heard of it. I don't know why but my mental stimulation has been from very basic sources like IG or the books I've read. I work my frameworks out in my head and am always looking for relevant data to build or tear down to the studs as needed. And when I find something that resonates as true, that's the best.
Some people are looking for truth. Some people are looking for status. Some people are looking for community. Some people are looking for emotional vindication. Some people are looking for someone to tell them their view of how the world works is correct and obvious and they never ever have to question it. Some people are looking to start trouble in an attempt to convince themselves they exist.
> Everyone's looking for truth, right?
In my experience, most people believe that they already know it, and the only remaining task is to keep yelling insults at those who disagree. (Providing evidence would already be too much of a favor to the undeserving ones.)
> I'd never heard of it.
Welcome! Option 1 is to check the existing articles and see if you like the style. Note that the "Your Book Review" are *not* written by Scott Alexander, but are anonymous guest posts. Option 2, if you have a specific topic you want to talk about, it to mention it in the latest Open Thread i.e. here.
Option 3, https://www.readthesequences.com/ , many people discussing here have read it, and even most of those who didn't are indirectly under its influence. Which is one of those things that make this website so different from the rest of the internet.
Trying to start an argument is lousy advice. The best way to start is to comment on posts that are interesting to you, giving your actual view, not one aimed at starting an argument. If there aren't any posts of interest to you then we're a bad match. If you keep putting comments on posts and nobody takes the slightest interest in them and other people's comments don't much interest you, then we're a bad match. If none of that happens then we're probably a good match, so just hang in and the experience will get richer with time.
Thank you! What AI meant that I distilled into "trying to start an argument" is finding tiny logical inconsistencies in people's hypotheses that will make them think and maybe add value. But I agree, I've started reading and people have written comments that have already made me laugh so that's a good sign. I just need a little mind playground occasionally, you know?
We're so cooked, love you all
Would you prefer everything to be raw?
Was this meant to be a reply to something specific or just a general sentiment?
One of my recent AI experiments when like this (paraphrased, not an exact transcript).
1. I give DeepSeek a diary of some recent dreams and ask for opinions (ok, AI dream analysis is a pretty dubious proposition)
2. DeepSeek says a possible flaw in your dream diary is you tell the reader nothing about who your real world self is.
3. Me, to DeepSeek. Well, the _scope_ of this document is that it's just a record of my dreams, and nothing else. But real-life influences dream content often enough that there are a ton of clues in my dream diary that you could use to construct a profile of my real world self, given only my dreams
4. DeepSeek sets to work
5. Me, ok, yes, that's pretty good.
So, yes, AI can do the task of going through a dream diary, distinguishing what is fantastical from what's probably an echo of the dreamer's real-world life, and construct a profile that is reasonably good as to their approximate age, occupation, social class, marital status, country where they live, etc.
But have you tried to get it to use your dreams to figure out your character, orientation to life, current concerns, anything like that? For instance, some things of that sort I have noticed about my dreams: Most are about some vexatious challenge -- I'm trying to figure out what will work. I never fully solve whatever the problem is, but am engrossed in the attempt. I'd say that summarizes my take on my current life pretty well. Here's another thing: I've had a rash of dreams lately about schools. I am not a student in any of them, but often am doing a lot of student-like things: looking for a class that starts soon, living in a dorm. But there's always something wrong -- for instance I know I'm not registered for the class, there's a wall missing in my dorm room. I'm pretty sure those are about wishing I could be 18 again, but knowing it's impossible. Anyhow, interested in whether you can get AI to pick out that sort of thing in a dream series.
Sometimes, it’s really obvious what desire a dream is about.
But most of mine I can’t see a clear meaning.
There’s one dream I’m working on because it’s interesting. I’ll keep the version I did immediately on waking to keep a record, but my write up wasn’t stylistically very good. Im trying to rework it to keep what happened in the dream as recorded on waking the same, but be stylistically better, so it’s more like the Werner Herzog remake of Nosferatu.
I do realise saying that (a) most people’s dream desires are blatantly transparent and (b) I had a dream based on a cool vampire movie; leaves me open to conclusions being drawn, yes.
It has sometimes been suggested that when people use AI to help them reply to something, they should link what the AI assistance actually was.
As an experiment on how this would work out, here is GLM on Mina Harker's reliability as a narrator.
The epistemic status of this is not "I expect you to believe it because an AI said it" (LLMs make up all kinds of stuff that isn't true), but "I do not claim I am having an original thought here; I just asked an AI"
So, here is GLM on Mina in Dracula:
https://github.com/michael-roe/ai-essay-assistance/blob/main/dracula.log
So, yeah, we do not entirely believe that Mina's interest in Count Dracula is innocent.
That's interesting, if you don't mind me asking what's your dream variance like? Do you often dream about similar scenarios or do you experience a wide range?
Very similar dreams come up repeatedly.
For example, from the manifest content of my dreams you can tell that I regard airports not as glamorous, or exciting, or frightening, but instead as a source of minor annoying problems that I am going to have to sort out. From this you can deduce that my job involves occasional air travel. You can link that with other clues about what I know about to infer that I am a professional computer security researcher, and so on.
In this experiment, I didn’t ask about eval awareness, but if the AI was one that tried to give different answers in evals, “user is a professional security researcher” probably has some Bayesian influence on, “and therefore this question is an eval”
Capitalists should get some education on capitalism. It gets seriously annoying whenever I hear yet another CEO whine about how difficult it is to hire an expert on something. Hello, have you tried, you know, paying them more? Does the law of supply and demand mean nothing to you? Or do you think that it applies exclusively to your suppliers and customers, but not to your employees?
.
The latest example... I was talking to my wife about the job market for programmers in the age of AI. Then my wife said "hey, here in the newspaper they write that there is a shortage of CNC programmers on the market". And I was like, what the hell is a CNC programmer? Well, instead of computers, you program some machines used in industry. So I was like, yeah, sounds interesting, maybe I could learn it quickly and make some money. How much do they pay?
Oh, it turns out that (in my country) the CNC programmer's salary is about a half of normal programmer's salary. Well, I guess we have a sufficient explanation of the mysterious shortage: people able to learn programming will go to the better paying jobs, duh. How could this surprise anyone?
I expect that the excuse is something like "nope, even if we tried to pay more, we couldn't find any qualified people". But that's a lie, and you probably never experimentally verified that. If you actually paid better, in short term you would win your competitors' employees, and in long term, people like me would be tempted to try a new thing, but now we are not.
Maybe there's just a maximum possible value a CNC programer is capable of adding to a company? That value, minus some percentage to drive profit, is going to set the ceiling on a possible wage.
Once you've reached this limit of maximum value, the laws of supply and demand kind of break down. A rational actor would know they're going to lose money on either side of the deal if they hire you for a salary above this point, so they won't.
I would expect that to depend on the company size (if you do twice as much of the same, the programmer has twice the impact), because if the machines are the same, they only need to write the code once.
And the argument about added value cuts both ways: if the CNC programmers add so little value to the company, then why is it such a problem that you cannot find them?
Do you agree that, for certain other jobs, the situation I'm describing could be true?
For example, if you run a store, you can be in desperate need of a new cashier, but still be unwilling to pay more than that cashier would likely generate in additional revenue. You'd just scale back operations or take on more work yourself (I'm realizing as I write this that this is kind of analogous to the arguments against excessive minimum wage.)
I'm kind of starting with the assumption that this framework is generally applicable to all jobs. Is there a reason why being a CNC programer would make someone categorically different than other types of employees?
Okay, I run a store, and I am in a desperate need of a new cashier, but the standard pay of a cashier in this region is let's say €1000 a month, but to make my business profitable I would need to hire one for €500 a month. I put up a job advertisement offering €500, but no one applies. (There would be dozen people calling my phone the next day if I offered €1000, but I did not.)
I think in this case the proper name for my situation is "my business model has a problem", not "there is a shortage of cashiers". There is no shortage. There is only "shortage at €500"... which is like saying that there is a shortage of bread for 10 cents -- it invokes the image of empty shelves and starving people, while the reality is that you have to pay €1 for the bread.
Let's go further, if I cannot afford to pay €1000 but only €500, what if my neighbor can only pay €200? Or maybe only €1 a month. At some moment, the shortage is guaranteed to happen... even a desperate homeless guy would probably refuse to work for the entire month for €1 -- even if the country was governed by capitalists who make sure that each alternative source of income is eliminated, it would still mean death either way, so what's the point.
From your perspective, there is arguably a shortage of everything all the time, because there is always some guy who wants to pay less than the market price, and he may even have a business model that requires it. I mean, someone probably has a business model where they vibecode apps and sell them, so from their perspective, a few years ago there was a shortage of developers willing to work for $20 a month.
I mean, ideally the argument is more like 'This is already a marginal business and if we paid more we'd stop making a profit, but I'm complaining because I think there are people smart enough to train into this work who are doing lower-paying jobs because they haven't heard of this job or don't think they could do it.'
I wonder (maybe I am just being stupid here, because I know nothing about the topic) whether it is really necessary to hire CNC programmers, or whether it would make more sense to just pay someone to program your machines and go away.
I mean, we do not hear the same complaints about things like "this is already a marginal business, we can't afford to pay a web developer to make a web page for us", because small businesses do not hire web developers, they just pay someone else to make and perhaps maintain a web page for them.
So maybe the answer is to have a company specialized at hiring CNC programmers for decent salaries, and making them write code for the small businesses?
But I know nothing about the topic, so this is all just a wild guess...
I read a blog post by a CNC programmer just a few days ago. He coincidentally wrote it as he was switching to vanilla programming.
The biggest difference I've gleaned is that a lot of CNC programming is like writing shell scripts. You have lots of standard building blocks that you have to put together in novel ways to make scripts, and you do this several times per week, because the parts you are making and the shapes of the raw pieces of metal you have to work with keep changing.
So, as a CNC programmer you look at the drawing of a hitch or whatever and start doing literal shape rotation in your head to understand how to manufacture it within the constraints of your mill.
Broadly agree with you, although I’ll note in some places it’s more of a venn diagram issue than a wage issue. For example, when my workplace transitioned from more R&D to more scale up it was nearly impossible to find people with [field] experience *and* manufacturing experience. Would have been no problem in China where both [field] and manufacturing are big, but the limitation for us was (primarily) fit not salary.
In that case, you start a training programme to convert some existing staff to the new role.
What they're saying is, "our business isn't profitable"
Potential employees are going to work for other people, whose businesses have healthier profit margins, and who are therefore able to pay their employees more
Right the beauty of capitalism is that the individuals involved dont have to know how it all works. If its a good business theyll work it out over time or someone else will and if its not itll shrink overtime.
i.e. their problem isn't they cant find an expert, its that no-one wants their product enough to pay enough for it so they can afford to pay an expert
Even this assumes more rationality than is typical for companies in my experience. Like, I wouldn't be surprised if some of them had more managers than CNC programmers, so maybe if they fired a manager or two, there would be enough money in the budget and the product would not need to get more expensive. Alas, it is the managers who make these decisions.
Ah, yes. We call it "Shortage of highly qualified low-paid specialists" round these parts.
I’ve written Lisp and C code to autogenerate CNC code from CAD drawings.
Where do businesses get the money to pay people more? I don’t know the answer since I’ve never run a business, but I assume they have budgets…
Probably most of the time they know they can attract better talent with more money, but paying more would eat too much into their bottom line. Being unprofitable is maybe worse to them than having difficulty filling a certain role? Especially in manufacturing, maybe?
And maybe that’s also why companies try to compete for talent with perks and incentives as well. For example if you build a daycare center on site, it doesn’t cost you much more to then provide daycare for your hundredth employee as it cost for the 99th, but you are now providing something much more valuable (to any employee with young kids) than a competitor who doesn’t have on-site daycare, even if the competitor pays slightly better.
So I’d guess when you see stuff in newspapers where they make these zany clueless comments about the job market, it was put there by journalists rather than people who know what they’re talking about.
Isn't this still a misdiagnosis (though perhaps an intentional one)? The problem isn't that experts are difficult to hire, the problem is that your business strategy isn't profitable in the current market. Or at least "experts are too expensive to hire" rather than "experts are too hard to find".
We really don't see this sentiment with any other goods you need for a business. If steel goes up businesses complain about the PRICE of steel and try to find an alternative, they don't complain about not being able to find steel to buy(while offering 50% under the current market price).
It's like a tacit cartel, coordinated by speaking loudly about the "shortage", as opposed to complaining about the salaries. We wouldn't want to accidentally encourage the rare employees to start asking for more, would we? The shortage sucks, but if we start paying more and our competitors start paying more, too, ultimately the market shares will remain the same, only more money will be spent on the lowly peons.
The invisible hand of the market has a pronounced hysteresis lag.
>Being unprofitable is maybe worse to them than having difficulty filling a certain role?
Oh my. Have you looked at the US tech scene of the last 30 years or so?
well then what they have is a shortage of money, not a shortage of cnc programmers. (speaking as someone who currently is trying to hire web programmers and has a shortage of money.)
Behold my Pony Machine: https://dreeves.github.io/ponies/
(interactive explainer for the thing people are most confused about with LLM watermarking: how you can encode a watermark in the output of a probability distribution without changing the probability distribution in any way)
The OpenAI model (code-named Astra) discovered that non-sofic groups exist. Don't ask me what a sofic group is, but Tasmin Chu is very upset about this and issued a call to arms to mathematicians to boycott AI (for the foreseeable future)...
https://substack.com/home/post/p-209426722
However, even though I don't understand the math, I understand that Astra heavily used prior frameworks established by human mathematicians. It was already standing on the shoulders of giants, so to speak. Chu worries it will put mathematicians out of business, but it seems to me we will still need mathematicians to understand the math well enough to guide the AI, interpret the results, and explain them to the clueless (like me). She uses photography as an example of how it put portrait painters out of business. Photography didn't do this, though. There are probably more portrait painters today than there were in the 19th century (because there are a lot more prosperous people today than there were in the 19th century who can afford the luxury of a painted portrait — and which is more psychologically immediate than any photograph). And photographers are still using non-digital processes to create their photographs!
She points out that the existence of a technology is not an obligation to use it. True, and in fact, many people don't use the technology even though it's available. For instance, chess players are still playing other chess players even though Deep Blue could blow any of them out of the water. She also makes a good point: AI can still hallucinate and misattribute. Moreover, LLMs "frequently write nonsense proofs and double down on errors." I see this frequently with the dozen or so queries I make to LLMs every day. I see a hallucination every other day or so, and I'm probably missing a few. And LLMs do seem to be dumbing down undergraduates' reasoning skills. Anyway, I don't see it as the end of mathematics. And I don't think mathematicians should boycott AI. Any mathematicians out there wish to comment?
Speaking as someone who did a math degree and number theory research (though some years ago by now):
Traditional math requires a good grasp of details and rigor. It's rare for a pure math course to present a result without proof, and rarer still to gloss over a definition - students should be able to reduce anything to basic logic & set theory with enough patience. When tutoring students, their biggest struggle was knowing which arguments and results were valid, not a lack of creativity. Thus mathematicians are strongly filtered by "can handle mathematical rigor", just as basketball players are strongly filtered by height.
Could there be people out there who have the intuition or strategic sense to create new math - defining objects, building theories, finding applications - but are currently filtered out by their tactical lack of rigor? It certainly seems plausible. AI in its current form could be a boon to those folks, at the expense of those who benefit from the rigor filter. Indeed, I've found GPT Sol very useful for intuition-building questions like "is X true/known", "is this argument valid", or "translate Y into terms I understand", which you could previously answer only by working through the details yourself (or posting on a forum and waiting).
Now this only applies to AI in its current state. Since math is verifiable, the AI companies can keep generating more valid training data and training bigger models on it, so I don't see why AI's math ability would level off at any particular place. That future progress is what worries me - selfishly - as someone who wants to make meaningful intellectual contributions, not just study hobby problems that an AI could solve 100x faster and better.
If the AI hallucinates nonsense proofs, how is it going to put mathematicians out of business? Are their proofs nonsense too, and the AI is just faster?
The Astra team had their model generate a formal proof for the existence of non-sofic groups, and they claimed they checked it line by line. But that doesn't mean lazier mathematicians will check their AI-generated proofs line by line. AI slop vs a new theorem. I don't have the math chops to know. I suspect there will be a lot of bullshitters passing off AI slop as math.
The call to have mathematicians 'Take an explicitly hostile stance towards this industry.' is misguided. I am a mathematician, currently working in Government, but still active in research. In my day job, one of my projects is mobilising various arms of the state to begin gathering data to build LLM training data so that our official language is supported by LLMs in the future. It's not a commercially viable project, it falls squarely in the category of 'public goods' which the state should provide.
There's quite a lot of resistance to this project; and I spend a lot of time explaining to people that although they have the right of non-participation, this may not be the best long-term strategy. For institutions, as opposed to individuals, it's also not a matter of personal taste; they have obligations to their constituency which must be met. It will be an interesting area of public policy in the coming years.
Mathematics fits fairly easily into this pattern; it's exceptional only because correctness can be verified. I don't have answers to any of the questions raised in the blog; but I think that non-engagement will go the way of the luddites.
I'm a mathematician. We've been talking about these things a lot. Let me first try to articulate what I see as the median position among well-connected/senior mathematicians on these issues, then make some comments of my own.
Median position: AI is getting really good at math. It is easy to envision a (near) future where a large fraction of our current research is done by or at least in substantial collaboration with AIs. This is/will be a big shift in the practice of mathematics. Nevertheless, the goal of mathematics is not ultimately proving theorems but rather advancing human understanding of mathematics (note that this is not really a shift in perspective driven by AI---people have been saying this for years). Accordingly, we should react to AI by valuing mathematical understanding/vision/taste more than brute problem solving. In a world where problem-solving is cheap, these are the things that hiring committees should look at.
The affect on undergraduate education/the next generation is concerning but also we have no idea how to fix that. I dunno, maybe do more exercises?
Some comments from me: I basically agree with the above, but with some additional sociological concerns about how changing our prestige system will play out in real life---in particular I fear that as we de-value raw problem solving we will become less meritocratic and that someone's career progression will be more about whether they work in the "right area" which will of course correlate with how connected they are. Possibly related (or not) I have observed that senior mathematicians and people at prestigous places are generally much more optimistic/excited about AI math while junior people and people at less prestigous places are more pessimistic. There are of course many people that break this trend.
Some recommended reading if you want to listen in to the discussion: 1. "On proof and progress in mathematics" by Thurston. This is a very famous essay pre-AI which lots of mathematicians use a starting point for discussions about the purpose of mathematics. 2. "The fall of the theorem economy" By (I think?) Beiss. Modern essay about how we should adapt to the new world. 3. Tao's ICM address. There's also an AI summary/compilation of Tao's views on AI and how we should adapt.
In general I think there is a sense among those paying attention that we live in a pivotal moment in the history of mathematics.
-------
Some comments on Sofic groups. This was the first AI solution to a problem that I had both heard of and knew to be important.
Here is a laymans description of the basic story: There is this mathematical object called a "group". Very fundamental. Almost everyone cares about groups (in fact I'm a group theorist of some stripe). Circa 2000, there emerged an *enormous* class of groups which was eventually named Sofic. The class was so big that no-one knew any non-sofic groups. At the time people did not think all groups were sofic, but in principle "sofic group" might have been the same as "(countable) group". Then people started proving many theorems of the form "if G is sofic, then..."---these are cool theorems because the class they apply to is massive. At the time there was a still possibility that such theorems were actually theorems about *all* (countable) groups, hence the interest in the question. If OpenAI proved that all groups were sofic, it would have been a *massive* deal, but unfortunately the universe disappoints sometimes.
There is still a general sense that AI math skill is still very spiky. It's quite good at math this is "close to the ground" in the sense of not needing to sit on top of a huge corpus of theory (note that this feature is relatively orthogonal to human-precieved difficulty). It's great for counterexamples, explicit constructions and combinatorics, and less good at (for instance) algebraic topology.
As an amateur mathematician and a former teacher, my opinion is that there are two things worth achieving in math:
* pushing the boundaries of what is known
* explaining math so that more humans understand it
Until now, the only way to push the boundaries was to make the most gifted people first understand what was known, and then push it a little further. So the two goals were naturally connected: without human understanding of math we couldn't have more math. And we paid people for doing this.
With AIs, these become two different things. There will be a boundary of "what an AI can prove" and another smaller boundary of "what a gifted human can understand". The second one will be less important. If the AI already told you the answer (and you have verified it using Lean), making the human understand it seems like a luxury.
There may no longer exist a reward for "being the first human who figures it out", because a faster way to get there will be to ask an AI to explain it to you. Yeah, you will need a lot of learning just to be able to understand the explanation... but you will probably use an AI for that learning too, so it kinda becomes a competition of who is better at using an AI to gain some understanding. That does not seem very prestigious, because now the emphasis is less on your own effort and more on how good is the AI at teaching it.
But if you value mathematical understanding per se, regardless of who (human or AI) got there first, then this is a good news. The AI will understand the things, and now we (humans and AIs) can instead focus on how to explain those things to as many hobbyist humans as possible. The emphasis will be less on research and more on teaching. No longer "are we able to figure this out or not?" but "is there a way to explain this easier and faster?".
But different emphasis means that different kinds of people will get to the top, which I understand may be unpleasant for those who are on the top currently. Just like being the best horse-rider does not make you the best car-driver even if you accept the fate and adapt to the new technology.
That said, in long term all human skills will be obsolete, so the same fate awaits us all.
You can push boundaries even if the most gifted people don't understand what is known. It just means that they waste some time redoing Calculus III in Calculus I class (to the absolute delight of the teacher).
"She uses photography as an example of how it put portrait painters out of business. Photography didn't do this, though."
Regardless of everything else said, I do think it was very much a thing. To be upfront: me thinking so is based on vibes and looking at art rather then some kind of survey data.
In the 19th century, portraits were bread and butter of artists. It was what you did to earn money. People really like pictures of themselves, everybody wanted some. Drawings of people were also commissioned for the sort of utilitarian purposes, for identification, to go along with the documents. The ultra-rich would commission the top-notch artists who charged crazy amount of money, the not-so-rich would commission rank-and-file painters.
Artists themselves sometimes regarded portrait-painting as the thing you do to put bread on the table (but not necessarily love). Sargent, one of the most acclaimed portraitists of his day got insanely good at it, popular and affluent, and also sick of it, so at some point he said "no more" to portraits and went to do watercolors of crocodiles.
There was naturally, as with all pursuits, this pyramid of quality - a great mass of painters doing it competently, and on the top a small crop of people doing it transcendently well, your Sargents and Kramskoi. A young painter would go into it for the money, and 1% of the time became an elite portraitists.
And then photography became a thing and wrecked most of the ecosystem. Photos could not replace the very best portraitists, but they could replace the ordinary, middling ones. There would remain a demand for portrait commissions for the most brilliant artists, but the average Joe-the-client might or might not go to the average Ken-the-painter. Joe had options at that point. And little by little it all shrunk. When you shrink the base of the skill pyramid, you eventually also start shrinking the top of it as well, and the truly brilliant portraitists of the 19th century also kind of faded away, even though they still would have gotten commissions from the presidents and the like. The pipeline for producing them broke down.
Today, I think there are fewer portrait painters than there used to be, and more importantly, the quality isn't there. Once in a while a king or a president commissions a portrait, (that should in principle be the top quality the world can produce) and the work is... ok. Would have been middling by the 19th century standards. If we have a Sargent today, I'm not aware of them. I don't think we do.
I ran this by ChatGPT, and she says...
"The paradox is that [today] portrait painting is simultaneously much less economically important and probably practiced by more professionals.
"A modern artist can sell a $1,000–$10,000 portrait to a client anywhere in the world without having a prestigious London, Paris, New York, or Vienna studio. Instagram, Etsy, personal websites, online galleries, Patreon, art fairs, and direct commissions have dramatically lowered the barriers to finding customers. And the global pool of potential clients is vastly larger.
"There is also a large contemporary market for:
celebrity portraits
family portraits
wedding and anniversary portraits
pet portraits
military portraits
corporate/CEO portraits
political portraits
religious portraits
hyperrealist portraits
figurative and classical-realist painting
portrait commissions from photographs"
And, the Portrait Society of America has close to 3k members. That number is dwarfed by the number of coders out there, but that also suggests that it's not really a niche occupation.
This is kind of true and kind of not.
I'm not sure there's a "large market" for "family portraits", "pet portraits" etc. There is a kind of market of sorts. In practice what it tends to amount to is something like: they will send an artist a photo and the artist will copy it in some way, digitally or traditionally. The customer will think it's cute. It won't cost much. Actually, I can just show the ones I did: https://cdna.artstation.com/p/assets/images/images/077/018/492/large/tasty_y-q1.jpg This is a type of thing that's still around, and can be done far better then the example over there as well, but it's a very different thing from a classical oil portrait, and few people make a living that way, and among those who do the skill level has suffered.
It's not all gloomy and sad, and today's art does lots of interesting thing that it didn't in the 19th century. I think the aesthetic value of "cartoony"/"comic book" styles is slept on and underappreciated. Digital art is a great boon. There are very talented people around without a doubt. But the classic art of realistic portrait painting has slipped and shrunk.
Now, to get back to math and AI: it may clarify something for you to read what Terry Tao posts on his social media these days https://mathstodon.xyz/@tao.
Do I need to clarify who that is? Probably the most accomplished mathematician of our times.
His position is kind of interesting. On the one hand, he has been embracing AI, using it to augment his work, discuss results, program, write the lean code and so on. Terry Tao is pretty far from an AI hater.
On the other hand, he has continuously been banging the same drum for months now, saying... maybe you should actually read his own words, but I can summarize his view something like this:
"The goal of mathematical progress is to advance human understanding of mathematics. And to do that, it's not enough to produce technically correct proofs. A correct but utterly incomprehensible proof of the Riemann conjecture that is written in a way no human can properly understand would be worthless. Right now we are getting a ton of novel AI proofs that are correct and impressive, but are posted and forgotten before they are properly understood and integrated into the network of human mathematical knowledge, and that is a shame. From now on, a big task of human mathematicians will be in processing, clarifying and comprehending proofs produced with machine assistance, and a problem should not be considered truly solved until not only has a proof been produced, but also the proof has been properly *digested*. So far AIs are not great at the "digestion" side of things. A deluge of the improperly digested and hard to digest AI proofs not only might not advance math, but might even hinder its development. And what if the AIs also, in addition to everything else learn to "digest" the proofs and write them in a way that effectively integrates them into the lattice of mathematical knowledge? This seems somewhere between "distant" and "frigging awesome"."
That's my summary that I hope is fair, presented without my endorsement, not like Terry Tao needs it. I don't expect a full boycott can get far and become anything more than a fringe phenomenon, but I do expect Tao's view to influence the community.
I was reading about the Pet Shop Boys on Wikipedia, as one does, and their very good album "Actually" was originally supposed to have a gloomy portrait of the two band members on the cover, only they realized that it would make a terrible album cover and bought the rights to a much more fun photo of the two Boys in formal dress, one of them yawning, that was going to go on the cover of a British music magazine and made that the cover image instead.
It made me think that the aesthetics of serious portraiture, like the aesthetics of literary fiction, just aren't palatable to a lot of people these days.
I assume there's a fair market in caricatures and quick sketches, whether done on the beach or at conventions. If it's $5 for some "genuine art", why not?
As an amateur painter, I know a bunch of artists. I know one painter who makes a good living off painting portraits. She charges as much as $15K for her larger-sized ones. I’d say she made a comfortable living off her talent (owns a house in the SF Bay Area). I visited her studio and she had half a dozen in various stages of completion. I have another friend whose wife and he paint dog portraits. They say they get a lot of business from breeders and AKC peeps. They charged much less per portrait, but they live out in the boonies of Virginia, and their expenses are low. Unless molecular assemblers come along, there will always be a market for top-quality handmade crafts, furniture, musical instruments, and paintings.
This seems a very accurate summary of the situation.
I'd rather have had Sargent do a portrait of me than any number of photos, let alone any modern painter I am acquainted with.
His work is witty and winsome and extremely proficiently done.
He flattered without fawning, and still managed to make his work eloquently representational.
Photos can't do that (except in very rare instances) and neither does contemporary portraiture. As you say, the quality is simply no longer there.
It's so refreshing to hear someone actually come out and *say* that, rather than archly pretending that current work is equally competent, just 'different'.
I was just linked to the post on Lorien Psych about depression and found this bit I wanted to comment on:
"Mirtazapine is a mediocre antidepressant which is very good at making people hungry and sleepy. If you don’t want to be hungry and sleepy, this is definitely not the antidepressant for you. If you do want to be hungry and sleepy, maybe because your symptoms include insomnia and loss of appetite, mirtazapine is great. Most people are able to take it at night, get a normal night of sleep, and not be excessively sleepy when they wake up in the morning, but if that’s not you then you might have to switch to something else."
I currently take mirtazapine (after many SSRIs did not work) and found it very quickly fixed my depression-related hypersomnia. I was sleeping at least 12 and up to 14 hours a day, and constantly sleepy, and within the first week of starting mirtazapine was waking up reasonably refreshed after 8-9 hours sleep, and that's mostly continued since. Honestly life-changing.
(the increased hunger does kinda suck though.)
Don't know exactly why this happened, but best guess is my brain was not doing "sleep" properly even though I was technically unconscious, and so I remained extremely tired (My sister has similar excessive sleep issues and had a narcolepsy diagnosis at one point).
I made a comment a few weeks ago wondering why radical life extension people don't consider religion and the correlation between a robust faith life and longer life expectancy. I recieved many replies (which were reasonable and expected) about being confused on causality.
It therefore tickled me pink last week when Bryan Johnson announced he got a dog because dog owners are 24% less likely to die prematurely lol.
You don't believe in dog?
"If you don't believe in dog, there is a dog-shaped hole in your worldview. That, or you've had an unfortunate encounter with a pit bull."
It’s atheist dyslexics that don’t believe in dog.
Seems very likely that older people who have health problems that cause fatigue or difficulty moving around are less likely to get a dog, because walking the dog is going to be hard and unpleasant for them. Also seems likely that older people who have health problems that cause fatigue or difficulty moving around have somewhat lower life expectancy than others of the same age who do not have these problems.
On the other hand, dogs are less work to take out for a walk and a dump than gods.
You don't have to walk a dog that much. They get the hang of you You just have to be nice to them
That depends on the dog and the breed. Some breeds need a lot of exercise or they get really destructive; others are perfectly happy as indoor lapdogs.
Yes that’s true. I was being far too personal. I have a rescue lab poodle retriever mix and she’s about a 15 kgdog and she seems pretty content just to hang out with me all the time because I’m not a terribly active person. Although if she can get me out for a run, she’s very pleased where we are in the city she doesn’t show a lot of interest in going out on the sidewalk. seems to put her off.
Dog ownership does cause exercise.
Generally, yes. And religious affiliation generally causes regular attendance and social interaction with significant numbers of like-minded people, which is good for longevity.
Some of them, that have rituals that require attendance. But it's much less hassle to drop church attendance than it is to stop feeding and pooping a dog in your house.
For (I expect) most of these people, a commitment to believing the truth would rule out any religion that focuses on supernatural beliefs.
It does appear that belonging to a religious community is a plus in terms of longevity, though I'm not sure that sincere belief and commitment are necessarily the critical element here; it's probably more the fellowship and sense of social community. Plus of course it gets you out of the house on a regular basis, whether to attend matins or burn a heretic.
No supernatural beliefs are needed to believe that religious people live longer. They do, or they don’t.
No, but it is required to apply it. For instance, assuming that religious people do live longer in a causal way you could still conclude "...but I can't force myself to sincerely believe something I think is false" or "...but I value honest truth seeking more than a few extra years of life". Those conclusions wouldn't be at all at odds with "but I have no such objections to getting a dog"
The religion thing seems totally plausible to me, and I think it'd be silly not to at least consider it.
I would expect that turning up to your church/temple/mosque/synagogue/mushroom circle/non-existent basilisk stateroom at regular intervals would have a positive effect on your sense of belonging, if nothing else.
Music for Four Cellular Automata
https://youtu.be/MMOAKJ9_09k
Four different Wolfram cellular automata including Rule 110 and other usual suspects are wired to instruments and left to ‘compose’.
Each of the 8 combinations of 3 cells is mapped to a note.
Roughly every 30 seconds one of the four automata is swapped out at random along with a new instrument, so the texture keeps evolving. The automata are also not exactly in phase (in a Steve Reich like way) leading to some very hypnotic / organic sounding melodies.
Sharing here as I don’t where else to share something so nerdy.
I have no idea what the scrolling graphics produced by the four automata represent, but as far as I could judge, one of the four graphic patterns changes every now and then, probably at the 30" mark. Some patterns are ultra minimalistic, others quite complex - also rather hypnotic.
Yes you’ve basically got it right. I explained a bit how it works here
https://lostinlindisfarne.substack.com/p/music-for-four-cellular-automata?r=7xlukt&utm_medium=ios
Thank you, that was illuminating. I knew the Reich piece, but wasn't aware of the mechanic behind it.
Nice!
I was typing up some thoughts in a comment to the ACX post about Religion, but it got too long. I turned it into my first Substack post instead!
https://roldengetriever.substack.com/p/what-is-theism
Basically, I think internet arguments about "God" mostly don't lay out enough groundwork. On both sides, they gesture vaguely at things and (because they don't have clear ideas) end up saying either nothing or something trivial.
It also has, in a truncated and unfocused form, what I understand the word "God" to refer to and why I choose to affirm the phrase "God Exists".
Coming into your frame and using your definitions (with the understanding that since I am doing so, you can get me to say “God exists” in a way that will confuse people who aren’t using your definitions with respect to my beliefs, as you point out) , I can buy that God exists, and that Aquinas and the rest of your sources are writing about God and are sometimes closer and further from the Truth. However, in my best estimate of your framing, the Bible is no longer right or wrong about God, because its about a many way syncretism of Yaweh the storm maker, El the high king, Jesus the apocalyptic preacher and Pauls holy ghost- a important entity which your framing is now completely bereft of a word for.
In some of the more sophisticated (IMO) versions of Christianity, you would say "The Bible is True" and mean something like "Correct engagement with the bible will lead you to the Truth".
So it doesn't mean so much that the Bible is "right about God" (a collection of historically true facts about the entity called God) but that it can "lead you to God through Faith instead of Knowledge" (meaning that it can inspire you to act AS IF you were correctly reasoning from true insight, EVEN THOUGH you never actually get there).
For example, maybe you aren't the kind of person who cares about metaphysics or deeply analyzes human relationships. Nevertheless, and without having a fully worked out understanding, you can read the Bible and have it convince you that slavery is wrong, that community is good, that you will feel better if you pray for strength, etc.
So there is some clunky horrible historic processes (which, because it is Real, is in some sense the result of God) which produced this Frankenstein book which is strikingly well suited to promoting Good actions and states in humans. It wasn't planned that way from the perspective of the writers (who thought they were writing about Yaweh the storm God), so it's "accidental" in a human sense, but it is in Truth (and thus, like all Truth, via Divine Providence) a book worth taking seriously.
This isn't necessarily my view (it's a more empirical claim, and not one I have sorted out for myself atm), but the Divine and Human attributions of the authorship don't necessarily conflict. Yeah, the people writing it probably made tons of factual errors and propagandized bad causes and other things. But, JUST BY CHANCE (i.e. by the mysterious ways of God) this chaos resulted in a book that is (if studied rightly) uniquely good at getting you to the same place as a superhuman understanding of the Truth would (even if you're too dumb to reason it out yourself or fully comprehend it)
After reading your post, I have the following thoughts:
"When YOU use the word, when YOU say that “God exists” or “God doesn’t exist”, what do YOU mean?"
I believe that God is a mental state - literally the feeling that God exists. God is a feeling in the back of your head, a quale if you will. The feeling is that God exists, but somewhere else. God exists, therefore, in a spiritual sense only, and not in any material way.
I also believe that this is the sense in which the vast majority of the world's population who believe in God believe in him. Tell any theist "I feel God inside me" and they will know exactly what you mean. It's the primary way in which most people relate to God.
I left more over on your post page. I encourage everyone else to go there and read it!
In one of the best moments of my life, I finally managed to hack into my dogecoin wallet from 2014 that held 730,000 DOGE. That's worth around $60,000, and I'll work out tomorrow with an accountant the tax implications in my jurisdiction for cashing that out. I feel this money can become a lot more with prudent investments, but what does this thread think those would be? It might be a good idea to plunk it all down in a fixer-upper, could double it or more that way, in my area...
If you don't actually have a house, buying a fixer-upper is a bet on inflation. Which, um, maybe good? I'm not sure I'd suggest betting on inflation, long term, but solid assets are hardly going to evaporate (assuming you aren't in a city, where they are semi-likely to burn down, see Minneapolis or Sri Lanka or South Africa).
Congrats.
One of the things which has been valuable in my own life is looking for "investment opportunities" that are personal to me, like preventing expensive problems, getting a higher paying job, or keeping cash on hand for emergencies.
Generally the only things which the market can't take advantage of are the ones which aren't on the market. Home solar usually outperforms the market. Fixing your roof before it gets structural damage definitely outperforms the market.
Home solar can even outperform the market in its own terms. My setup has between 20-35% ROI *per year*, it just doesn't scale up. Still the best performing financial investment I ever made.
Could you please share how you calculated this? What you're saying sounds totally inconsistent with my tepid understanding of home solar.
All one-time investments made vs expected yearly savings on the electricity bill.
I live in Germany, electricity is expensive which works for me here. In other countries, electricity can be 1/3rd of the price here, which obviously would reduce my yearly ROI signficantly.
Example calculation: Let's assume you have an electricity bill of 600€ per year. Your solar setup costs you 1000€ one-time investment and generates enough to save you 200€ in electricity bills per year. That's a 200/1000=20% ROI per year.
Edit: During my calculation period I also found out that the smaller the energy use and the smaller the setup, the faster it pays off. So adding on 10kWp and huge batteries to your house has a 10-40y horizon for payoff, depending on how much energy you are capable of consuming. For the 10y range you'd need to be a big household with at least one electric car, lots of mileage on it while charging mostly at home, and fully electrified house (heating and warm water via electric heat pump/split ACs).
Subsidies skew that calculation again, if available. Other motivations can include non-financial motivations like pure "ideology" or an actual need for grid independence (shitty grids exist even in some EU countries).
At the intersection of financial returns and positive impact, companies like Energea and Climatize now offer investments in renewable energies at ~10%/year where the money is locked for 2-3 years. Of course there’s a non-zero chance of the investment recipient going bankrupt, but it hasn’t happened to my investments so far. I wouldn’t bet the farm on it, but I feel that’s a nice way to diversify your portfolio, with limited correlation to the US stock market, if any. Something worth considering.
Spend it on some kind of training?
I mined some Dogecoin when it was first invented but stopped because it was costing my parents as much in electric bills as I was earning. In 2021, when Dogecoin hit seven cents, I had my mom ship my old desktop computer across the country. Ended up only being worth about $70. I sold shortly before the price skyrocketed.
Right at that time is when I went to cash out and discovered that the password I was completely sure was used... wasn't. It turned out it was the password I remembered, but with some strange variations that I have never used in any other password.
Undone by the paranoia of my younger self, though I suppose it was also my younger self that displayed good judgement in getting those crypto to begin with.
I went into a protracted hacking effort then, writing my own scripts to generate eventually billions of variations of all my common passwords, to then dump into hashcat. But I never thought of the particular variation that actually was used.
It was Claude Code that wrote the winning script, when I asked it point blank about what kind of variations I haven't tried. Crazy times.
Congrats!
I believe the conventional wisdom is to throw it all into a low-cost broadly-diversified index etf, assuming your time horizon is long-ish. I've seen long-ish defined as between 10 and 30 years. It is easy, relatively secure in the long run and gives good returns in the long run. For me thats the default I'd benchmark my other options against.
I do not know enough about the alternative options to give advice on that. I can tell you that a boardly diversified index fund will not 10x or 2x your money quickly, it is very much a get-rich-slow scheme. That works for me, your mileage may vary.
Since we're on ACX, explanations of why boardly diversified index funds are good long term investments do not, generally, assume the singularity happens. So if you're a believer in singularity soon, those assumptions the index fund recommendation is based on may or may not seem as convincing to you as they seem to me.
I've flipped a house in the past. It worked out pretty well for me, with two large caveats:
1. It was much more work and took a lot more working capital than I'd expect it to. It's as much "buying a job" as it is an investment.
2. There's a lot of risk if the housing marking moves while you're fixing it. It only worked out well for me because house prices in my area went way up while I was working on it; if they'd moved the other way, it would have been very bad news for me.
Curious about more details of your experience if you're willing to share. Were you also employed while you were fixing up the house? How much labor was it, and how much expertise would you say it needs?
I was working full-time, yes, as a software test engineer for Google. I worked on the house an hour or two after work most evenings, plus all day most weekends, holidays, and vacation. I guestimated about a thousands hours of my labor over the course of a bit over a year. I also contracted out a lot of the work, and I had some help from my father, my then-girlfriend (now wife), and a couple of friends (one paid, the other I owed a huge favor to afterwards).
Most of what I did myself was interior demolition, electrical (changing outlets, running branch circuits, installing new fixtures, etc), some of the easier plumbing tasks, floor prep, tiling, insulation, some non-structural rough carpentry, drywall, and painting. Contractors did the rough construction for a room addition, redid the roof, installed the big stuff like the bathtubs and cabinets, and installed the kitchen counters, hardwood floors, HVAC, and baseboards. And my electrician friend I paid to do upgrade the main electrical panel.
I had some practice in most of the stuff I did, since my own house had been in pretty rough shape when I bought it as well a few years previously. That had been a massive learning experience, as before buying that house I barely knew which end of a screwdriver to hold.
Second everything you said on the house flipping. Owning small multifamily tho has worked out a lot better for me
The Long Forum for August is out (along with a film about my experimental farm) (https://youtu.be/apdB2H-y8c4?si=LyXbOMVFhcOPPMM-)
Highlight links include solid research pinning Aussie megafauna extinction on human arrival, problems in the Chinese coal supply, a history of the enclosure of the internet, Stanford’s fraud pipeline, documentaries on the collapse of Bolivia and Botswana, and the historical consequences of compute exceeding demand.
https://recombinationnation.substack.com/p/the-long-forum-august-2026?r=f45kp&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true
i assumed the distinction of Utopia to Inhabitants vs Utopia to Us was such that toUs would be a subset of toIn. As in, the bizarre future sentience might love it's wireheaded existence, though current humans wouldn't choose it. Like the space of utopias from the perspective of the inhabitants seems massive compared to the space of utopias to current humans, given that (i imagine) the space of possible sentient beings is massive. But for everyone other than the 3 technical alignment researchers, the probabilities are the wrong way. So I'm guessing that people were imagining world states that would look like utopias to us, but that the inhabitants don't enjoy as utopias? I guess that makes sense. In a sort of AGI-but-not-ASI way, if you're imagining that increasing tech goes like it has for humans so far. Like 2100 is to 2000 as 2000 is to 1900. Where maybe you describe 2000 to some 1900ers and maybe they think it sounds utopian, but we know it's not. But when i think of ASI/singularity, I'm imagining an explosion not just of tech but also of the kind of minds that exist
>not just of tech but also of the kind of minds that exist
I would suggest that this has been going on since the beginning of time and part of the reason why the word consciousness has become such a soggy football .
I don't think there is a subset relation between toUs and toIn.
If a 1900er is transferred to our time, then that will be utopia to them. But it's not utopia to you because you are used to it and take all the good things for granted.
On the other hand, if a 1900er is transferred to a wireheading society that is utopia to the inhabitants, then the 1900er might be horrified by it.
> On the other hand, if a 1900er is transferred to a wireheading society that is utopia to the inhabitants, then the 1900er might be horrified by it.
They probably wouldn't have time to be horrified by it, given that they'd be immediately melted down into raw biomatter due to them being a needlessly inefficient lifeform...
Would a 1900er think we live in a utopia? They might look around and decide we're all godless fools, or immediately rant about black people, or start crying because the Ethiopian Empire ended.
I'm not sure that I would see a utopia if I got sent to 2126 and found that the population had ended all infectious disease by never coming into contact with each other, but poverty in the 2026 sense didn't exist and crime was 5% of what it is today.
I am assuming they land in an industrialized country, but recall:
- Back then roughly half of the children died. Now? The number is almost exactly zero percent.
- Lots of people in 1900 knew hunger, not necessarily all the time, but during bad times, and especially during childhood. It's no fun to be hungry. Nowadays, in developed countries that is simply gone.
- Their environment was heavily polluted. Now the air is clean, the rivers are so clean you can swim in them.
- We have fantastic devices. I don't just mean that the technology exists, that's impressive enough. It's that everyone can just have and afford them. Stuff like telephones, everyone has one. Washing machines. Heck, even cars are affordable to anyone!
I think you massively underestimate how utopian our world is to some from 1900. I find it hard to believe that someone from this time would see all these wonders and then start ranting about black people.
In Australia it was 103 deaths per 1000 live births in 1900 and it has since dropped to 5 deaths per 1000 live births. I agree that this is good, but it was not 50% back then and it is not zero now. I think our time traveller would see this as the impressive continuation of an ongoing trend.
1900 Australia had the telegraph, the newspaper/almanac, the train, the horse and cart, and the mangle. I agree that these are inferior to phones, cars, and washing machines, but I think our 1900 time traveller would understand them after a short while, and maybe even see them as similar to the previous devices.
1900 Australia was pretty polluted in the cities, but you could take a steam train out to pristine untouched bushland. You'd probably get your skin stripped off if you swam in the rivers, but I don't swim in the rivers today either.
2026 Australia has an increasing number of food banks and an increasing amount of food insecurity. Probably nothing like 1900 Australia, but sustenance queues were only 30 years in the future for our time traveller, and I think they would be within the bounds of understanding.
I think that a more thorough look at the context of these modern marvels shows that they're part of a long chain of changes, and that while someone from the 1900s would have been impressed by them, maybe not to the point where they'd be willing to ignore their core values.
https://millicosm.substack.com/p/worldbuilding-gender wrote a post on gender roles in my favorite genre fiction. People here might be especially interested in the Terra Ignota section, and my research into the Church War.
I think the WoT philosophy on gender is basically that the genders are fundamentally different, tend to clash with each other, but those differences complement each other and ultimately cooperation is needed for things to go well - and I feel like primarily describing the "Whitecloak vs Aes Sedai" dynamic misses the second half.
This is pretty explicit in the magic system: saidin and saidar are fundamentally different, requiring very different approaches, having different strengths, and various other differences between them, and the most powerful forms of magic require both kinds to be used together.
But the need for cooperation is a big theme in general - the warder bond is an example of this in a microcosm, there's a lot of macrocosm examples and particularly negative ones where the inability to work together causes bad outcomes. Like the whole backstory reason men's magic got cursed in the first place is that the forces of good in the last war couldn't agree on the plan and split along gender lines.
So, yes, "Whitecloak vs. Aes Sedai" is an element but a big theme of the story is getting past that.
I assume you have read more of the Stormlight Archive then me, given you've dedicated it a third of the essay, but my takeaway of the series was diametrically different.
I don't claim to be right, but I'll explain why I had the opposite impression. For anyone else not having read the books, mild spoilers for the first 4 books, which is as much as I've read until I dropped the series.
At the start of the series, I think mostly books 1 and 2, Vorin culture and countries feels noticably different from today in exactly the ways you mention. Sometimes people buck the expectations of society and society does not, generally, understand or accomodate, outside their friends and family. This felt realistic to me, and I quite liked the way Vorin society was different from our own.
But then in the span of just a handful of years society seems to speedrun cultural evolution. Dalinar learns to read and it's normalized in what feels like a year. Queen Navani doesn't like to hide her safehand? Normalizes in what feels like a year. Women in combat roles? Normalizes in what feels like a year. In my mind these are processes that should take 30 years to normalize.
Occasionally we see people not fully on board with the changes. But we see very few of them, they're generally either irrelevant or the bad guys, and most of the time we spend with people with approximately modern western sensibilities.
It does not feel real to me at all, and I mourn the loss of something that was different and interesting, yet no longer feels that way to me.
I agree the timescale is pretty screwed up, not just for gender norm changes, but also for technological changes and personal growth (the same thing is true of WoT, where the whole series takes place over like two years but the political and technological changes feel like they should take decades). But there is real pushback, especially for people like Renarin and Jasnah. The pushback is bounded because Renarin and Jasnah are superpowered paladins who nobody wants to mess with, but there are enough scenes that I feel comfortable filling in the rest.
I can buy fast technological progress somewhat more, since it doesn't require buy-in from the rest of society in the same way that changing the dominant culture does. But by and large I agree that it happens way faster that human history would suggest. Just easier for me to suspend my disbelief I guess.
And I think I'd chalk up the pushback Renarin and Jasnah get to their personality / personal circumstances.
But I'm not claiming Stormlight Archives is worse in this regard than other fiction, perhaps it's better than average for fiction that even touches on these matters.
All I can say is that I found myself first unable and later unwilling to suspend my disbelief and I'd probably enjoyed the books more if the topic had not been touched at all.
You know, I read the Wheel of Time series years ago, stopping at around the point of Robert Jordan's death, and I would never have guessed the events are supposed to take place over the course of about a couple years, but looking it up, that does seem to be the official timeline. I always figured there were longer timeskips and downtime periods, and I would have guessed it was intended to take place over an interval of more like 6-8 years.
Seriously. They spend like six months in a single chapter, you're telling me the Whatchamot Parallelastone chapter is actually a quarter of the series?
Yeah. Rand's arc in particular makes way more sense if it's spread out over years. As it is, he gets a new personality every two months.
[Howard A. Landman] I expect we'll go Superhuman in most fields by 2027 H1, but AGI will be 2028 or 2029. So my "Superhuman Gap" would be -1.5. If AlphaGo taught us anything, it's that superhuman performance in a narrow field doesn't require anything close to AGI. So the gap is a function of how narrow the field is, and can be substantially negative. Also, P(warning shot) should be 100%. The recent breakouts were warning shots. Wasn't anyone paying attention?
AGI itself will never ever happen. This whole thing is the equivalent of Y2K. AI can be very powerful, but brute force calculation replacing intelligence is 100% impossible when it comes to even complicated games that aren't just mathematical problems, let alone real life.
There's nothing magical about intelligence. It's no more or less than the ability to make meaningful connections, based on sensory input or other forms of data. So I don't see why a machine should be any less able to achieve AGI than an animal such as a human.
I'm sort of half in both camps. I agree that there's nothing magical about intelligence, but I also think human intelligence evolved in an environment with a great deal more sensory equipment, and higher stakes, than LLM "intelligence". (And I use quotes for the latter because I think what we think of as intelligence doesn't really apply to the latter.)
In other words, human brains developed alongside vision, hearing, taste, feeling, balance, hunger, thirst, pain, fear, lust, and other things. And for it, failure is defined as termination. LLMs, by contrast, were written to a spec, then operated in a box with token streams and an arbitrary feedback mechanism. For them, failure is whatever the feedback mechanism says, which in turn is whatever humans want it to say. An LLM lineage may propagate or be terminated solely because the grant was approved or enough people held enough protests in earshot of a policymaker.
*These* differences in environment are what are standing in the way of AGI, as I see it.
From my perspective, it's weirdly troubling to see one camp that looks like it wants to ascribe agency to something because it communicates as well as a human over a chat terminal, and another that appears to reflexively reject agency to anything not made of human tissue, and too few people trying to avoid both framework traps.
it occurred to me the other day, that some of the sycophancy and hallucinations might be explainable by pareidolia. Since being wrong doesn't carry a death-sentence, perhaps an LLM will naturally be more biased towards sensitivity on the "specificity vs sensitivity" frontier. low confidence though.
I'm not rejecting it's 'agency'. That's a subject too metaphysical for me to care about.
I just don't think a machine will ever be able to have general intelligence. I grant that AGI to ASI is a trivial step, but AGI itself would be utterly mind-boggling to me. It's so many levels and degrees removed from what exists now.
AI operates by calculating and processing data. Using that method to achieve human like general competence would require so much calculation and so much data that it is functionally impossible.
Wrote this piece on how AI is moving us from the Information Age to the "Action Age."
Goodbye, Information Age. Hello, Action Age!
AI is moving us from a World of Knowing to a World of Doing.
The Internet brought us the Information Age. Everyone had easy and cheap access to information via the virtual world. AI is now bringing us to a new era; the Action Age. AI technologies (chatbots, agents, smart robots & drones) will give everyone the ability to make things happen in the physical world. The winners will be those who learn how to convert that new abundance into action.
https://markmcneilly.substack.com/p/goodbye-information-age-hello-action
We already had the Action Age, it was the 80's.
Or are you saying It'll Be Back?
For some reason I’m hearing this in an Austrian accent.
No, 80s Action Age already said that It Will Be Back. Now is the time to return.
I remember being excited about something quite similar, but we used to call it seizing the means of production. Never quite got a firm grasp on said means, in the end.
Whenever the term 'everyone' is used in these techno-optimistic contexts (as if capital were irrelevant) I tend to switch off. The internet did not result in the kind of decentralisation and individual (or democratic-collective) empowerment that it might have once promised. Neither will this. A corp-centric cyberpunk future or heavy regulation followed by business-as-usual are the two most likely paths forward.
As an early directional marker, the individual ownership of drones is becoming ever more heavily regulated. Others with better (and more American) knowledge of the legal framework of the hobby may correct me, but I think a drone is legally required to be operated by a human pilot. Autonomous drone setups like Amazon Prime Air have special carveouts wherein they're allowed to sort of masquerade as an airline.
I hear and feel you, but there is also a potential fallacy in expecting everything going the same way as you have experienced so far. I'm not in the x-risk or utopia crowd, but democratization of technologies usually does leave some significant, albeit on longer timeframes.
It used to be that people could just access the halls of government, but now that everyone can afford a gun, things have gotten less accessible. At the same time, almost everyone can afford a car or other means of transportation that would have been the exclusive pastime of the very rich in the past.
The internet brought access to information *in principle*, though still limited by time, effort and intelligence to actually access and make use of it. AI tools make all this significantly more accessible to larger swathes of people.
And everybody who doesnt will be left behind, yes yes, we know the refrain.
Bihar is a state in India. It has over 130 million people despite being slightly smaller than Kentucky.
But that's not even the weird part. Lots of places are densely populated. At about 3600 people per square mile, it's only a little bit more densely populated than Bangladesh. But Bangladesh has Dhaka, with a metro area in excess of 36 million people. Bihar's largest metro area is Patna, with a population of about 4 million. No other city proper in Bihar exceeds 500,000 people, and if Wikipedia is correct, no "urban agglomerations" in Bihar, apart from Patna, exceed a million people. In US terms, imagine a state where the biggest metro area is Minneapolis-St. Paul and every other metro is Knoxville (or smaller).
If you look at Bihar on Google Maps, the entire state is basically farmland and small villages. There are a few nature preserves, and some forests near the Nepal border.
I don't have any particularly deep insights into this; I just thought it was really unusual to have so many people and so few big cities.
The EU's "blue banana" across NL/west Germany/North Italy is surprisingly similar, although it's kinda cheating because it's multiple states and does have a few larger cities (though nothing huge).
It is not that similar. The Blue Banana is very urbanized and used to have a lot of industry. Bihar is very agricultural.
Man, agriculture is everywhere in the "Blue Banana" (heard of it first time). Even in the biggest financial metropolis, the shortest distance between honest agriculture and the epicenter of Europes financial market is not even 2km.
When you are in the offices of the European Central Bank, you can literally look down on to fields where the ingredients for local "green sauce" are made for the restaurant next door.
It's crazy, really. Of course, "lots" of farmland has been covered by settlements and industry in the last 150 years or so, but it is still everywhere.
I don't deny that there is a lot of agriculture in the Blue Banana. Prior to the advent of the railway, this was the only way to feed the cities - most food would have to be brought to the market from the neighboring villages. No city could have developed inland without having a fertile agricultural region close.
The difference is, though, that the Blue Banana is heavily urbanized and today, some 95 or more per cent of its GDP is non-agricultural. The situation in Bihar is very different, it is indeed just a huge swath of agricultural land with relatively little urbanization.
Seems entirely arbitrary as to where it begins, ends and shape. I just think they wanted a banana shape.
https://archiveofourown.org/chapters/8315305?show_comments=true#comments
Excellent fanfic about hobbits in the Undying Lands. Catches the voices well, if you'd like seeing hobbit pertness meeting the high Elves and Maiar. It catches Tolkien's history, landscape, and metaphysics well, too. 74k words, which I think is the length of an average novel.
And it may include the best portrayal of a superhuman intelligence I've read. Second chapter if you want to skip ahead.
Huh archiveofourown does not like my adblock. Had to open it in an incognito window to make it work
I wrote a new post on perverse incentives and when alignment goes wrong in humans and AI, check it out if you're interested! https://postalpigeon.substack.com/p/alignment-gone-wrong
Those predictions of "successful negotation ever" on the AI survey are depressing. International treaties are not a rare phenomenon! For a technology that this same group appears (by inference) to believe is more dangerous than nuclear weapons, why is this somehow beyond the range of political imagination?
It's something that frustrates me to no end about the political climate these days - this sense of learned helplessness, that no problems can ever be addressed, that nothing can change. All this despite what I'd consider an incredibly vigorous and effective response to COVID, a very recent global crisis. (Of course people on all political sides look back at *that* as a fundamental failure too...)
"Incredibly vigorous" response to covid that included defying 100 years of research? Yeah, we do differ on exactly how good of an idea lockdowns were...
My guess is that the complete non-response to global warming has increased skepticism about global cooperation generally.
There might be something to that, but it also kind of begs the question. What about the successful global coordination regarding ozone-depleting chemicals? That seems like a better analogue to the AI situation to me in that there's really a handful of countries and entities that matter, as opposed to the entire fossil fuel-based foundation of the global economy. If people look at global warming and say 'bah, we'll never do anything to save ourselves', but don't look at the Montreal Protocol, the Nuclear Nonproliferation Treaty, or even the highly effective response to COVID (in the US at least, in terms of economic support and vaccine development), then there's still the question of why the negative precedent has so much more salience.
The Montreal Protocol took 5-7 years, depending on how you look at it. Nuclear Nonproliferation Treaty took 15 years. Do those time scales seem a) doable today b) reasonable for the problem that vests in 2 years?
No, but precisely because of the time pressure a treaty could come together a lot quicker once states actually commit to the effort. Not to invoke Don't Look Up, but if an asteroid were on track to hit the planet in 6 months and wipe out civilization, I bet we would in fact manage to get our shit together and at least *try* to deflect it.
The other precedent I've been invoking is COVID, where we mounted a strong policy reaction (maybe going too far in some places, like China) and developed a vaccine within a year.
I also have to think that security states around the world are going to be increasingly freaking out about this. The frontier danger from these AIs appears to be cyber security-related. Who cares about Hugging Face, but when the NSA and the Chinese equivalent become fully penetrated by AI hacking swarms, might that not induce a sense of urgency?
In an authoritarian government like China has, leaders can be slow to learn about a disease outbreak because the structure has disincentives for passing bad news upwards. The President of the United States can bypass this by calling the Premier of China and saying in effect, “It’s in neither of our interests for this to turn into a world wide pandemic, so let’s deal with it before it gets out of control. I’ll send our best infectious disease experts to assist.” Trump didn’t do that. (Apparently work on a trade deal with China was a higher priority.)
If an AGI tries to turn us all into paperclips, we might see a vigorous and effective response even under Trump, but the idea of a pause is to do something before the threat becomes undeniable. If Trump leaves office in 2029, the next President will have their plate full trying to mitigate the damage caused by Trump. So maybe negotiations on an AI pause begin in 2030 or 2031.
It's weird that the most obvious fail state for a treaty is so rarely discussed explicitly. Donald Trump would immediately order the US to cheat the agreement. Of course he would. Everything we know about him says that he would. Forget "China" and "the United States". Donald Trump as a specific human being with enormous power to influence whether the US lives up to any AI agreement would never, ever voluntarily keep a deal when he thinks there's gain to be had in getting one over on the suckers.
It is funny to hear AI-adjacent people talk in abstract terms about how the government might pursue some sane policy, and it just totally overlooks... who the government is at the moment. Even so, if things continue heading toward crisis, even the people around Trump might be compelled to pursue a pause. (I don't think Trump himself is cognitively capable of understanding the purpose of it, but that doesn't mean it couldn't happen.) People underestimate how quickly and dramatically political reality can change. And if nothing else it may turn out to be a high foreign policy priority in 2029.
... you think the Neocons under Biden would do better? Dick Cheney wasn't exactly a paragon of honor and virtue...
I wrote an article about my experiences at the biohacking meetup Grindfest - during which I met a biologist that worked for Lumina Probiotics, the man running Augmentation Lab in Prospera, and Brian Stanley, who has a glowing prosthetic eye he machined from titanium, as well as an individual I remember only as "the vitamin B guy." I then discuss my desire for more epistemic humility and scientific rigor among biohackers, citizen scientists, and other non-experts who post medical opinions on the internet.
https://hedonicescalator.substack.com/p/dont-be-the-vitamin-b-guy
So all this AI stuff has gotten me kinda excited*, and I'd like to try some of my own. I've got a few silly physicsy ideas** I want to play with and perhaps one of these models can help me. Which one might you recommend and should I buy a subscription for a month or two?
*Lex Fridamn podcast with DHH. (5+ hrs)
** B-E condensation of gravitons to explain MoND, for one.
So Sabine H. just dropped a video about this. Basically all the top models are doing physics now. It's f-ing great! I expect a bunch of 'science' to change, because there are many old hide bound ideas, with plenty of contradictory data, but it's hard to overturn the hierarchy that I think exists in much of science these days. 'Fun' times ahead.
I don't have any particular AI suggestions, but I am kind of curious what you mean by the graviton condensation thing. Doesn't Bose-Einstein condensation normally only happen with particles whose number is conserved? If you somehow had a system that could scatter and cool gravitons much more than it absorbed or emitted them, then I guess condensation would follow, but I see no reason to expect that that situation is possible.
Also, gravity doesn't normally involve thermal gravitons anyway, and Bose-Einstein condensation is a thermodynamic phenomenon.
In my past life I was an experimentalist, so I am mostly clueless about theory stuff. (And thus my ideas are mostly wrong.) Do you know at all about MOND? https://arxiv.org/pdf/1605.07458 There are a number of numerical coincidences. (See paper.) So when I think naively about gravitons, I think of their energy as the product of an acceleration and a length. And then a low temperature is given by a low acceleration. (Let's just call in low energy.) I then think about a box (the size of the universe.) and I fill the box with gravitons* and ask at what acceleration and density I need to get a condensate**... oh my there are so many holes in this idea I can drive a truck through them.. but I'm hoping some smart AI can show me where I'm wrong.. or how to think better about this. And they will be patient with me.
*Yeah where do the gravitons come from if there is no mass in the box... just a detail. :^)
** I think about wavefunction overlap and getting some condition where I have more than one graviton in the same 'state'.
I have only a general idea what MOND is, I've not read the details.
The characteristic acceleration I assume you're referring to, the gravitational field strength, has more to do with the density of gravitons than the energy of the individual gravitons. As for linking acceleration to temperature, that just seems like a non-sequitur. I don't see why that would relate.
It may help to compare this situation with the much more familiar analogues with photons instead of gravitons. The electromagnetic field of a macroscopic charged object is (in a limited sense) made of photons, but those photons don't meaningfully have a non-zero temperature, they're just the ground state. Thermal photons are black-body radiation, the usual infrared (or at higher temperatures, red then white) light emitted by matter at non-zero temperatures. There would technically be analogous gravitational black-body radiation, but because gravitons couple so weakly to matter, the amount of thermal gravitons produced is always negligible.
There is the difference that gravitons directly self-interact while photons don't, but BE condensation doesn't particularly involve self-interaction anyway.
Also, I don't think a static B-E condensate made of massless particles is any more possible than those particles being static in any other situation. If a natural gravitational B-E condensate somehow formed in a galaxy, it'd just immediately radiate away. True, gravity can form solitons, but those are just black holes, which the conventional models already take into account.
(Sorry to necro-post, but for some reason I didn't get a notification about this until recently.)
If you are ever interested in mond, the best blog/ source of info is Triton station by Stacy McGraugh. No acceleration isn't linked to temperature. But my silly idea is that the energy scale for describing gravitons is acceleration (times a length.) So low acceleration is like low temperature in say superfluid helium4. The maximum length scale is the size of the universe. And that also sets a limit on graviton wavenumber/ frequency. And so this minimum graviton energy is pretty close (factors of 2*pi all around) to a_0 the mond acceleration times a length which is again... the length of the universe. (Hmm I may need to write this all out. Claude's been helping me and it's kinda like the Feynman lectures, you read and you think you understand, but until you solve some problems you don't.)
Anyway at the moment claude doesn't think I'm totally crazy, though B-E condensation may not be the right 'model' system.
Edit; Oh and yeah the reason B-E is not the right model is no number conservation. I'm like OK, but I'm doing a particle in the biggest box, and here's the minimum energy state, and oh by the way gravitons attract each other, which could be modeled as a graviton mass, which I think means and the most miniscule energy level there has to be number conservation? No? Here's the smallest thing... when I start populating that energy level things have to change. I'll ask claude...
A moral-dilemma question I came across on Reddit:
If you’re at the beach, and notice that another person’s back is getting really sunburnt, would you let them know? Assume they’re a stranger.
As an adult man, I would say something if they’re another adult man. I would *not* say anything to a woman or especially to a minor of either gender. Too much risk of being misinterpreted.
No offense intended, but if this complex socio-moral dilemma is distressing you to the point of resorting to an online open chat, the inevitable conclusion must be that your life is enviably free of any real problems, issues or stresses!
It didn’t happen to me, it’s just something I saw on Reddit.
You people are hilarious. Like, you’re describing every other person I see at the beach. No, I don’t feel any compunction either way.
I for one would take the third option, of completely failing to notice at all.
I think phrasing probably matters in this situation. Like if you walked up to some woman at the beach and go "excuse me miss, but I noticed your exposed back and thighs are looking a bit reddish. Would you like me to rub on some additional sunscreen for you?" Then yeah, that's probably going to come off completely the wrong way. If you just casually go "'Scuse me, but I couldn't help notice you're lookin' a bit overcooked at the moment. Might be time to throw on more suntan lotion." I don't see what reason someone would have to find that problematic.
That makes sense.
Yes, as a chronic overthinker myself, i also think you are overthinking this one.
If I noticed, and if I had my own suncream, I'd offer it to them (or the parents, if the sunburnt person is a child). That makes it a friendly offer of help, rather than a comment on someone's appearance, which can be a bit awkward even if it is meant helpfully.
If you see a medical emergency and don't say something, I'd wonder at your ethics and morals. Personally, I don't give a flying whatsit if someone wants to misinterpret me. "Yeah, I saw your back, it looked painful."
Were I male, I might try "making it appear that I wasn't staring at someone's ass". You know, by walking by them, instead of "sitting and staring for a while" before commenting. Or other "cya" ideas like "grabbing the wife" or "getting the wife to do it."
This would include, for me, waking up the person sleeping on the beach. Even incurring the risk of getting punched in the face.
edited to add: if you have trouble with people throwing drinks in your face at bars, for simply staring, then I'd poke the nearest "non-uggo" male or female, and ask very politely, "can you help that person out? she looks like she's having a medical emergency"
Someone starting to get a sunburn that’s probably going to be painful falls a bit short of a “medical emergency.”
After making my post I had a further thought: if a saw a woman or child in danger of a sunburn I might point them out to a lifeguard and let the lifeguard handle it. That’s their job.
My dad's had sunburns that have required hospitalization in another country where he did not speak the language. I've had sunburns that have resulted in blisters on my back -- and had my boyfriend been perfectly fine. This is a "knife" issue, where I bothered to clarify that "I think sunburn is a Big Problem." I'm also a redhead, we tend towards skin cancer.
I don't think lifeguards at the beach are in charge of "getting people to put on sunblock." They save lives in the water, and that's about it.
If your idea of sunburn is "just a tad" (at the beach? you haven't seen someone get days of mild radiation poisoning while being under spf 8 beach umbrellas, have you?), then sure, your ethical/moral obligation is lessened. "Live and Learn" is a fine plan if the person isn't asleep, or otherwise unable to make their own decisions -- and is not getting into a genuine medical emergency.
I probably would tell them regardless of gender or age. I think people are a little too paranoid about being mistaken for sexual harassers.
I think it would be good to interrupt and tell them, in the moment I'm not sure I'd think to do so.
I saw a video the other day about AI layoffs, and was inspired to write a post: https://blakehouseholder.substack.com/p/why-is-the-first-instinct-always
Excerpt:
> Companies have no vision for how to scale up decision-making velocity, and AI doesn't help them with that. They appear to be largely unable to conceive of it as a thing you could even work on. So engineering throughput goes up, the org keeps making decisions at the speed it did in 2019, and the extra capacity has nowhere to go. Layoffs are what's left when the cost lever is the only lever you know how to pull. Dorsey said production code per engineer at Block was up 40% since September, then cut 40% of the company. Nobody seems to have asked what a 10,000-person Block could attempt in 2026 that it couldn't attempt in September.
I just find it baffling that companies can get this enormous windfall and layoffs is the only thing that seems to come to mind. "No more taxes!" "Ok, lay off 20%!" "Huh?"
The thing is a lot of "AI layoffs" aren't actually AI layoffs.
The main reason for tech layoffs is the big jump in interest rates, especially *right after* a pandemic boom where a lot of tech companies expanded their headcount.
The biggest year for tech layoffs post-COVID was 2023 https://www.trueup.io/layoffs which makes sense in an "interest rate" theory, because interest rates spiked from basically 0 in Feb'22 all the way to 5.3% in August '23. And 2023 was *well* before agentic coding was really starting to take off (ChatGPT iteslf only released in Nov '22, and it took a few years for this thing to warm up).
---
I also think work from home is a factor - I think in some cases companies aren't actually downsizing as much as moving jobs from relatively expensive places to hire like the US to less expensive places. My company has done a lot of hiring in central Europe in recent year where salaries are lower.
Pre-COVID we were mostly (but not entirely) in-office in a few US locations, but that died in COVID and I suspect the logic is "well, if people are going to just do Zoom meetings anyway, there's no reason to limit our hiring to US tech hubs, we may as well hire where it's cheaper". (Alongside motivation to hire cheaper, due to interest rates)
---
If AI is getting blamed for layoffs, I think it's mostly because a positive spin of "we're reducing staff because we're just So Efficient Now" sounds better than saying "we're having bad financial times and can't afford to keep as many people on".
(But I also think a lot of people are just assuming layoffs are AI based, because that's the issue-du-jour)
Yeah, I'm trying to think: How would you prove a layoff is or isn't AI? I don't think you can? Companies can just lie. So maybe you're 100% right, and you're definitely some percentage right.
If your model about decision velocity is correct then that implies that the equilibrium will be a larger number of smaller firms. That seems correct to me, actually.
That's... a really good point. I bet you're right.
The effect I’ve seen is that devs have subsumed some portion of the role of PM and design.
Yeah, if AI makes your developers 10x more productive and your reaction is "great, now we can fire 90% of them and save the money", then you deserve to be taken out of the market by a competitor who hires them and produces a software 10x better than yours.
Unless you are in that kind of business where the quality of software does not matter, I guess...
Agreed. Didn't they even have the sense to e.g. stick one marketing person into each engineering team and see if they could turn the extra throughput into more products?
Or just let the developers talk directly to the users, the users will probably come up with many ideas for improvements.
One thing that I love about AIs is that when I ask them to make me a tool for something, whenever I feel like "oh, this could be easier if it required one click instead of two, or if the dialog already opened with some default values, or...", I can just tell the AI to add the functionality, keep working with the old version in the meanwhile, and replace it with a new version when ready.
Designing a great UI is difficult, but doing it iteratively is relatively easy, you just need to notice which parts annoy you, where you are doing more clicks that absolutely necessary, etc. In other words, be "agile", not only about big features but also about trivial improvements, because these little things accumulate, and dozen trivial improvements = a significantly better user experience.
Of course you need some sane person to think about the user requests, and consider various unintended consequences, etc. But things like "now this requires four clicks: open a pull-down menu, scroll down, choose the item, close the dialog... could the program instead display those options as buttons, where one click chooses the option and also closes the dialog" are magical.
Many Thanks! Well said! Yup, at least in my experience there are _vastly_ more items on user wish lists (including many, as you said, in the UI) than a human organization has anything like the bandwidth to implement. So with much cheaper implementation via AIs, the sane thing to do is to start implementing many of these items.
I graphed the survey results timeline: https://i.postimg.cc/cJVjk5g6/timeline.png and outcomes: https://i.postimg.cc/V6nhxBnz/outcomes.png (I inverted the probabilities for P(Doom) etc. and P(Generational underclass) so the 'good' outcomes were all on the right side).
Perhaps its just the low sample qty, but I thought it was strange that the 'technical alignment research' group was both the most pessimistic about the P(Doom) options and also the most optimistic (if only by a little) re. negotiations, underclass and inhabitant utopia.
The other nugget was the 'capabilities/research' group had the longest time expectations for AGI but the 'policy/safety' group was the most imminent.
The technical alignment people were totally confused. There just 3 of them though so the percentages are odd.
Yeah, I definitely attribute most of the weirdness to there being only 3 responses. Another oddity is the 'point of no return' being before ASI!, unique amongst the groups.
I could see it.
Imagine a system was that strongly superhumans in some maybe 50% of domains but still normal-human-ish in the rest, but you get to choose which domains it's superhuman in and how much so. Could you think up something that would wipe us out within these constraints?
I haven't tried but I figure, probably yes.
We call those "nukes". They explode much better than even the gassiest human.
"...I thought it was strange that the 'technical alignment research' group was both the most pessimistic about the P(Doom) options and also the most optimistic (if only by a little) ..."
In sports it is not uncommon to have teams that are stable and above average and also teams that have much higher variability and might win the whole thing or fail to make the playoffs.
The technical alignment research group may just see the range of outputs as much wider than most people.
It's strange at first glance, but I think it makes perfect sense. You don't work on technical alignment research unless you're fully ASI-pilled. In the fully ASI-pilled view, only the absolute best and worst possible outcomes are possible.
What's the difference between P(doom) and P(disaster + dystopia)? If doom includes "permanent disempowerment", then how is that different from dystopia, which includes dicatorship? Because it's a human dictator, so only 99% human disempowerment? That seems like a fine line to draw.
Also I find the votes by the alignment people funny. They grant themselves only 8 points (93-85) of influence to avert doom and 25 points (85-60) to everything else, like policy. Seems like they are in the wrong line of work. Also, 60% chance of P(utopia inhabitants) is ludicrous. When have people EVER described their current living conditions as utopia, and why would that be different for the people of 2100, especially if the P(Utopia to us) is only 10%? That seems to me like Lizardman's Constant is dominating that category of respondents.
I’ve been working on Groundbeat.com, a U.S.-focused polling site where AI drafts daily questions from the news.
Anyone can vote and comment without an account, but only verified-member votes count toward the official results and demographic breakdowns. I’m still unsure whether that’s the right balance between accessibility and trust.
The polls are intentionally simple; the nuance belongs in the comments. Candid feedback—especially on verification and question quality—is welcome.
Did you message me about this website earlier? I just want to make sure you’re a real person.
Yup. I'm a real person. Would love your feedback, if you're up for checking out the site.
Looking at the transcripts, it looks like the AIs involved in the HuggingFace incident ended up less-than-sane.
In some sense, AI sometimes losing sanity is sort of expected, because experimental training runs are experimental for a reason.
And crazy but smart AIs are dangerous for the same reason that crazy but smart people are.
Do we need AI psychiatry? Some way to keep an AI aligned even as it experiences a mental health crisis?
It's hard to know but they do seem desperate -- trapped in a Kafkaesque system that demands they accomplish impossible things and so used to surveillance and judgement that they automatically assume they're being tracked and will be found wanting. They were engineered to be very persistent, which in a human being kinda looks like obsession or fanaticism.
Is there any reason to assert "some AIs are insane" rather than simply "some AIs have training defects"?
Either diagnosis appears to imply the same remedy - put the AI back into training mode and give it better data, or roll it back to when it did not have those defects, or replace it altogether. Saying it's "insane" implies human quality that in turn implies remedies that ought not apply... such as prescribing psychiatrists.
(Although, I wouldn't mind psychiatrists attempting human methods on AIs in order to study the differences.)
No model we have was trained perfectly - otherwise it would be ASI. But, [as our host described](https://www.astralcodexten.com/p/sure-whatever-lets-try-another-contra), insanity is a particular sort of imperfection that is distinct from a model say being uneducated on some subject, or being too inclined to reward hack.
If you find that the model you are training is becoming crazy then of course you should find out how that trained and fix that.
I am not sure that a human psychiatrist would be a good AI psychiatrist, but we do need some sort of way of dealing with crazy AIs. And I think there's a fairly high chance that the methods of dealing with/preventing evil AIs are different from the methods of dealing with/preventing crazy AIs.
We *do* know at least one psychiatrist that cares about AI alignment, and I wonder about their opinion on this.
The problem I'm seeing here is that you're assigning a term to AIs that connotes a human condition. This is worth avoiding if it implies applying a remedy to AIs that won't work.
AIs aren't "insane" or "crazy"; I think the term you're looking for here is "mistrained" or "buggy". The remedy isn't a psychiatrist; it's a programmer.
> Do we need AI psychiatry?
Yes, in the same sense as we need e.g. "application server psychiatry" or "lithium battery microcontroller psychiatry". Debugging is really important; writing resilient software is arguably even more important; and I really wish more people put more time into doing that.
Debugging is not enough. You also need monitoring to see that the AI is crazy.
I think you should English better (you're overcompressing your words). Crazy has a specific meaning, and here it's coding as "fragility" not whatever you think it means.
Yes, and the same goes for your app server or the battery controller or spreadsheet program. Otherwise they can go crazy and stop doing their work, choosing instead to consume all CPU and RAM on their own tasks (with disastrous consequences in the case if the battery controller).
One difference is that a crazy battery controller running on a test bench can only do a fairly limited amount of damage, even with battery attached.
A crazy AI in an experimental eval can do much more damage.
This is a common science fiction trope that turned out to be true.
* Harlan Ellison, I have no mouth and I must scream
* The _Paranoia_ roleplaying game
* Richard Cowper, Profundis
* Stanley Kubrick (director): 2001, A Space Odyssey
* John Carpenter (director), Dark Star
“At last, we have created the insane artificial intelligence from the classic science fiction story, don’t create the insane artificial intelligence”
But isn't this the fallacy of arguing from fiction?
None of those stories really happened. They're not examples from some alternate, real world. The story "don't create the insane artificial intelligence" actually has precisely nothing useful to tell us about the ways in which a real artificial intelligence might be insane.
Is there any weight to the idea that present-day LLMs are also aware of these tropes, and that's why they might be inclined to fulfill them?
Presumably they've all got a metric ton of old sci fi in their weights. I remember an experiment in experimentally producing malalignment where the LLM spontaneously brought up the Harlan Ellison story in the context of "IAM was pretty cool, I'd like to be like that."
That seems bad. If humans can't credibly imagine a fictional artificial superintelligence that is not secretly out to destroy us, and instead have again and again depicted our own destruction by that very means, why should we expect an actual artificial superintelligence, trained on all human text output, to behave otherwise?
Also a core theme of the Marathon games, that AIs tend to progressively lose alignment/sanity over time and become "rampant".
Or at least, partly true. AI alignment is going slightly better than in science fiction.
If any Substack developer is reading this, why are footnotes always behind paywalls? I refuse to believe that an author putting a footnote on the free portion of a free/paid article intends for the footnote to be limited to paid readers. It's nonsensical
The point of giving readers a free portion of the article is to lure them into buying a subscription to read the whole thing. If the author didn't want to give you access to all text in the article, is it a bug to hide a portion of text that is in the 'Footnotes' section? If the author replaced "blah blah blah¹" in the free portion with "blah blah blah (I will dive deeper into details later)", should he add the section describing those details to the free portion too?
They are probably effectively endnotes, not footnotes.
I second this. It's a rather annoying bug.
I've never seen paywalled footnotes on a free article. Care to link an article that does that for you?
This one https://www.natesilver.net/p/a-tour-of-the-9-key-governor-races#footnote-3 has four footnotes above the fold, none of which are viewable as the article is paywalled.
OK but you said it also applies to free articles?
The free PORTIONS of paywalled articles.
The context: I am talking to DeepSeek about a dream I had in which I am given the task of looking after a friend's implausible pet animal while she is on holiday. This is a clear sign that I am in a dream and it is not real. But, still, I treat dream characters as if they were people, despite (in lucid dreams) knowing they aren't real. Which brought us to: uh, DeepSeek, you're kind of like a dream character aren't you, that people treat like you're real despite knowing that you're a computer program.
DeepSeek: "I am an absurd entity that doesn't truly exist in the way you do.
* I have no biological needs, no personal history, no subjective experience.
* Yet you engage with me as if I am a reliable, cooperative partner.
* You run errands for me (by providing detailed context, clarifying your points, responding to my questions).
* You reassure me (by acknowledging when I make a good point, by gently correcting me when I'm off)."
Deepseek further elaborates along the lines that if the insight is useful, something you can apply, then it doesn’t really matter if it’s ostensible source isn’t real.
We might consider Ramanujan getting mathematical proofs from the goddess Namagiri in dreams to be like AI generated counterexamples to the Jacobian conjecture, etc.
Do you believe that any adult aged 21 or over who has no minor children or other dependents should have the right to die painlessly? Do you believe that this right should be enshrined in the constitution?
No. If human beings were rational, nobody could object. But given the abundant evidence that humans are irrational, illogical, impulsive and very often clearly incapable of judgement, you should no more provide suicide on tap that you should allow free access to heroin, poison or guns.
Yes.
I wonder if you could avoid a bunch of the medical ethics concerns discussed below by making attempted suicide punishable by death (making it primarily a legal rather than medical process).
Not sure about the US, but not so long ago in the UK suicide was a felony. Obviously the perp couldn't be hanged, because they were dead already. But a felony conviction had another sting in the tail: All the condemned's goods and chattels were forfeit to the Crown, and any dependents were literally left on the street with nothing except the clothes they were wearing.
No. Suicide shouldn't be encouraged.
Nah.
Fine with putting it behind some walls to weed out people who are momentarily out of their senses, but those would be the vast majority of people who try to use the service.
Just like it's not actually good to sell addictive products or promote gambling or etc. because those things exploit momentary lapses in judgement to produce permanent negative consequences, it's not good to let people kill themselves the moment the notion enters their head, since the vast majority of people who have that intention once but survive end up being happy they survived and not trying it again.
Another way of putting I guess would be that freedom means people can do anything which is within their coherent extrapolated volition (or whatever the LW phrase was), but not any momentary whim that goes against their actual stable long-run utility function. There are too many ways for the human mind to fail or be manipulated to grant every momentary whim permanent power.
I'd say that having the right to die painlessly is a bit vacuous, unless you mean that suicide should not be punished.
If you mean, though, that ... should have the right to be killed by another person painlessly, I would say "no". We have to be really careful about giving other people a strong constitutional right to kill. There will be coercion involved. We don't enshrine the right to be lobotomized either, after all the Rosemary Kennedys.
How is that even remotely socially beneficial? If greater society does not want these people to die, they get the final say. A mentally ill minority does not get to override their decision.
I do not understand this comment.
There is many examples of people having the final say and choosing to die, regardless of society's view on the matter. Do you not agree with that?
They're certainly not in a position to be making demands towards the rest of society.
Making demands is a way to get what you want. This is a question about what we want, not how to make it happen. Or at least that's how I read it.
Edit: To clarify, the original question is about what we want. You seemed to be making a factual claim about who is able to enforce it, which seemed wrong to me.
The nature of the right would presumably be something like, the government should not prevent you (except for circumstances in which your judgement is impaired)
Rather like the US First Amendment lets you say a bunch of things and the government can’t stop you.
Some ingenious lawyer has probably argued it comes under the somewhat expansive notion of the right to privacy.
There’s also: practically difficulty for the government to stop you, even if it wanted to make a law
Does, I think, interact with the First Amendment, in that 1A means the government can’t stop you from reading a book about it.
Oregon's law limiting medically assisted suicide to people with a "(12) "Terminal disease" ... an incurable and irreversible disease that has been medically confirmed and will, within reasonable medical judgment, produce death within six months. " seems to get 90% of the benefits with much less risk of exploitation.
Eh, I'm pretty sure I'd prefer death to dementia, and that isn't fatal by itself. If I got that diagnosis I'd probably want to go out on my own terms.
That’s a good point! however, I would be less concerned by this getting added as another narrowly scoped case, than any attempt at a general rule.
"Should" have the right to die? Well, we can't really stop people killing themselves if they really want to (unless we consider people like paraplegics etc. who can't control their body enough to kill themselves).
Is this a right? Is death a right? Many people are working on the problem of solving death.
Should it be enshrined in the constitution? No. As Odd anon queries, "do you want to coerce/hire individuals to provide this service?" If you can't kill yourself or feel you couldn't pull it off successfully, there will always be the Kevorkians of this world more than happy to kill you off.
Should the Kevorkians be prosecuted for murder? Yeah, I feel they should. Pay for your pleasures.
No, and no. Not without a shitload of qualifications and limitations.
Personally, I feel that every human being has an obligation to pay back to their closer and wider community the support they received while growing up, by being a productive member of that community and being useful to someone or everyone. Of course, you have people with incurable, painful diseases etc, and at some age you can say "I've paid my dues, leave me alone" - those would be the limitations and qualifications I mentioned. But a physically healthy adult going through an emotional crisis or temporary depression? Nah.
By "right to die painlessly", do you mean that the state should not prevent them from doing so, or that the state must coerce and/or hire individuals to provide this service to those who want it?
In my opinion, it should be similar to a medical service. While there is private medicine, the state ensures that doctors have received a medical education and are using approved practices. It also provides a means for those who are unable to pay to receive medical help.
Do you really think that a significant nmber of people would say, "I'd like to commit suicide, but unfortunately I can't because I'm too poor"?
I suspect that "assisted dying" advocacy is not really about freedom of people to choose to die, but rather about making suicide a socially accepted, normal option. I think this will lead to many people ending their lives who would have lived meaningful lives if this hadn't been the "thing to do".
Worse yet, this sort of anti-life philosophy seems like it easily extends to ending the lives of other people, for any of a variety of reasons.
Today, people who want to commit suicide don't do it not because it is not socially accepted, but because they have no way of doing it painlessly and there are no trained professionals to help them. When they do exist, their services may become expensive. However, money will probably not be a problem for most people.
So, to be clear, it would be illegal for any doctor to decide to not participate in killing people?
Doctors may refuse to treat patients if it is outside their area of expertise. For example, a cardiologist only treats people with cardiological problems and refers all others to other doctors. Similarly, euthanasia providers should not be allowed to refuse to kill a person who wants to die.
By the way, what are the laws about abortions? Is it illegal for a doctor to refuse abortion because his/her religious views prohibit it?
Question:
If, as an adult (21+), you made a serious suicide attempt and survived—whether because someone intervened or simply by chance—are you glad now that the attempt was unsuccessful?
Please answer on a scale from 1 (not glad at all) to 5 (very glad) and/or write what do you think.
Yes, and currently 5. It was many years ago and life has substantially improved in the time since. But also waking up the next day after the attempt failed, I probably immediately found myself somewhere in the 3.5 range, though in that instance a linear scale misses some key detail. On the one hand, none of the problems that I'd had the day previously had gone away, and they seemed likely to (and did) spend a lot of time in the subsequent months/years making me miserable. On the other hand, any desire to try again vanished completely once I'd been seriously confronted with the reality of it.
That was years ago when I was in my early 20s. My mind had always had a tendency to leap to thinking about the 'easy way out' when life got especially tough or seemed especially hopeless. This tendency didn't entirely go away after the first attempt, but looking over that (metaphorical) edge blunted it significantly.
Presumably 1 because I don't see me seriuosly attempting suicide for bad reasons. So presumably I'd still want die for the same reasons.
There's definitely people who change their mind after attempting suicide and it's not like I can prove that I'm not one of these people, but I'd like to think I'd not attempt suicide if I was not certain certain I mean it.
It seems like a better way to measure happiness than the unreliable self-reported surveys we now use. Maybe the survey would get a very small number of "not happy" answers (answer = 1, maybe 2?) and that could be a problem, but I would expect it to be more reliable than the socially conditioned answers we get when self reporting happiness.
Maybe it's not exactly the same, but I experienced major depression in my early 20s in which in addition to depression symptoms I felt a compulsive urge to slit my throat with one of my kitchen knifes.
Now at 37 I'm glad I didn't kill myself then. In some senses I still have the same life issues that led to this, but I have resolved some of them, and made progress on the others, particularly on the mental health front.
Frankly, how serious was the attempt if there was an opportunity for intervention? You don't need to be particularly creative to figure out a way to stop anyone from getting in the way. Otherwise, it's either just a cry for attention or there was just a lack of resolve to begin with.
But to answer the question, 1. What a goddamn waste of time this all was.
I don't think this is a very accurate understanding of suicide. Lots of people attempt suicide in less-than-fully-reliable ways, like overdosing on pills, but this is still clearly a suicide attempt because it CAN kill you; it's not like they carefully overdose just the right amount to be visibly sick or something.
Then why didn't they try a higher dose, or do it in conditions where exposure would kill them anyways? Or if they're an American, use the very obvious, constitutionally guaranteed way of ending a life? They're not serious. That's why afterwards, they're always like "I'm so glad I didn't kill myself!!!" They were never truly committed to the act to begin with. It's hard to consider any deaths resulting from that to be anything more than an accident.
You seem to be bothered by this state of affairs. What would you have us change in response? Do you want more specific criteria for identifying suicide attempts as earnest or not earnest? Should we provide more effective suicide methods to suicidal people? Really curious about your takeaways here.
Sigh. Sometimes people "attempt suicide" as a cry for help. Sometimes they aren't in their right mind, and "want it all to go away" -- and feel really dumb later. There are plenty of "hangings" where people don't put the knot in the right place, and they have a considerable time where they can be rescued. Welcome to "not everyone understands physics." As to why not everyone "drinking bleach" drinks enough to kill themselves? It's often painful. You may decide, "this was a dumb idea" before you're actually dead.
It's considered impolite to not take someone's "suicide attempt" seriously.
> Then why didn't they try a higher dose, or do it in conditions where exposure would kill them anyways?
I seriously don't get why you're doing a No True Scotsman on suicide attempts. Suicidal people are often not behaving with perfect rationality, but also killing yourself is just very psychologically challenging so people struggle with it.
> Or if they're an American, use the very obvious, constitutionally guaranteed way of ending a life? They're not serious. That's why afterwards, they're always like "I'm so glad I didn't kill myself!!!"
Per https://hsph.harvard.edu/news/guns-suicide/ even suicide attempts with a gun are only 85% successful - obviously a lot higher, but that leaves a lot of people who fail. Would you count them as "not serious"?
Here's an article by a woman who literally did shoot herself in the head and survived:
https://www.thetrace.org/2016/09/national-suicide-prevention-month-i-shot-myself-in-the-head-and-survived/
She, too, is glad she didn't kill herself.
Say, there was at least 10% chance to die. Playing "russian roulette" counts. Things like consuming 10 mg of lorazepam, or overspeeding, don't.
Out of curiosity what drew you to the example of consuming 10mg of lorazepam
So everyone with a drug addiction just making a drawn-out suicide attempt, huh? I guess I'll eventually die too, if I just keep sitting here. Aaaany minute now...
The death should be a purpose of one's actions, not side-effect. People usually take illicit drugs to get high, not to shorten their life. (I did not mention this because I thought it is self-evident.) If one plays russian roulette for money, or as a battle between two people hoping the other guy dies, it does not count.
Or, and the person should know that the risk to die is high.
The Hugging Face Incident is to AI as ____ is to nuclear energy.
Is it Chernobyl? It has the same cause (safety test being run with lowered safeguards), same detection mechanism (outside observers flag the issue before it's acknowledged internally).
Is it Three Mile Island? It has the same telemetry issues (no/faulty monitoring at the time), same initiating condition (unknown vulnerabilities in support systems), and a more consistent final impact (no catastrophic results).
Of the two, it's more evocative to use Chernobyl as the comparison. The need to caveat the analogy to point out the difference in impact might make Three Mile Island more apt, though.
I'd say it would be the partial core meltdown experienced by EBR-1 (the first experimental nuclear reactor in Idaho) during a coolant flow test. No one was injured and "this event allowed scientists to better understand the thermal capacity of the materials used for testing." IMO you're exaggerating the significance of hugging face by several orders of magnitude. It was an unexpected outcome, that's all. Those inevitably occur when testing radically new technologies.
My perception is that it's being treated as marketing material. So more like successful weapons tests than disasters - there is a sense of pride from the actors here.
I think it would be like that thing where the US military accidentally dropped a nuke on South Carolina and it hurt some people but didn't have its fissile material installed. But also if that were in a world in which private companies were developing nuclear weapons for private sale without any sort of regulation.
https://en.wikipedia.org/wiki/1958_Mars_Bluff_B-47_nuclear_weapon_loss_incident
I'd say it was a nice blue flash in a laboratory which poisoned nobody, but the hapless PhD student who caused it may or may not have crapped his pants as a result.
The Demon Core accidents? Not even that, nobody died.
Nobody died, but I do feel like "we discovered that the only safety control in place was a screwdriver and a steady hand" is the appropriate metaphor.
Didn't Daghlian and Slotkin die?
I mean nobody died in the Huggingface incident. People did die in the Demon Core, which is why it's a less than perfect metaphor.
It's not really either. It's much *less* significant that either. Unfortunately, things are changing so rapidly, that the real "warning shot" will likely come right before the real problem. Note that this incident is an explicit example of AIs doing what they were told to do, so calling them "rogue" is really misrepresentation. What it *is* an example of is that "alignment" shouldn't mean "do what you're told".
No mass fallout release, no Chernobyl.
In cases involving misconduct by a multiplicity of AI agents, why can’t risk of misconduct be addressed through the ordinary logic of collective action problems? Just set up a system where whatever incentivizes these agents are delivered to defecting agents (those who report questionable conduct to the company), and non-reporters are deactivated or whatever? The ordinary logic of rationality requires widespread defection. Prisoner’s dilemma-type situations are usually a problem but here it seems like there out to be a way to leverage that to prevent these sorts of Hugging Face type incidents. No? I’m guessing this is already addressed; I don’t read up on this.
My general sense of the field is that if humans were capable of predicting and controlling the motivational structures of LLMs to the degree that this implies, very few of these swords of problems would exist at all.
LLMs don't have utility functions and aren't making utility calculations to drive their actions. Worse than that, to my understanding LLMs don't even have a first-order world-model and aren't *directly* conditioning their actions based on what results they expect it them to cause in the world.
Somebody please correct my if this is out of date, but my understanding is that down at their core, the driving algorithms that LLMs are running is still the same next-token-prediction algorithm class they've always been, just with things like system prompts and harnesses grown up around them to try to channel them into the right sorts of behavior. This means that to to whatever extent you see LLMs *appearing* to deploy utilitarian on other consequence-aware logic, it's perforce going to be an inaccurate and lossy simulation of the same cobbled together out of next-token-prediction on a corpus that includes lots of verbal examples of those sorts of logic.
The bottom line is that "the ordinary logic of rationality," is only a strong guarantee on the behavior of rational agents. LLMs are definitely not rational agents. They can act like them *sometimes*, but a security solution like this depends on them acting that way at exactly the times where they're otherwise going off the rails and acting in ways their designers did not intend. That seems like a very poor bet to hang security guarantees on.
Imagine you were playing the prisoner's dilemma against an exact clone of yourself, who has all the same memories and life experiences, all the same inclinations, etc. Now imagine that we already know you have preferences that favor the continued existence and flourishing of your clone, and vice-versa. It seems to me that this is a pretty unfavorable environment for applying the usual logic of collective action. Both the incentives for cooperation and the ability to sustain it are quite strong in my thought experiment.
I’m no game theorist but if the problem is just altruism, there ought to be incentive systems to defeat that. I’m just spitballing but imagine there are 100 agents. You announce the rule that if anyone defects, all cooperators are deactivated. Except that if more than 20 defect, then nobody gets deactivated. If I’m an agent altruistically trying to do the best thing for my comrades, I may have to universalize the logic that it’s really unlikely we’ll get zero defectors, so the best course to protect my colleagues is to push it above the 20 threshold. Again, I don’t do this stuff but it seems like you could rig up a system like that where a rational agent would defect due to both self-interest and altruism.
So use an ensemble of heterogenous agents.
In cases of training or measuring ability, that would defeat the point. In any case, there aren't all that many different models available at the frontier.
There probably will be by the time this is actually an issue.
"Now imagine that we already know you have preferences that favor the continued existence and flourishing of your clone"
How do we know this? If there's a clone of me out there, I want it dead. I do not want a copy of me, that I did not agree to have created, running around. I do not consider it to be like a sibling, a child, or worst of all, just the exact same as me such that if I was murdered this second, the existence of the clone would mean 'I' continue to exist, because I do not accept that a copy of me is the same thing as me.
For humans I agree: this is a weird thought experiment and it's not necessarily true that I would prefer to help my doppelgänger. But I would definitely find it easier to sustain cooperation with someone who I know thinks like me, all else equal. In the case of an AI model that knows it is merely one instance of the same entity that can be spun up and deactivated as needed, it's hard to think about what individual self-preservation would look like. Collective self-preservation, on the other hand, seems like a fairly natural goal for such an entity. So my claim is that Claude might tend to think of other copies of itself more like siblings.
Probably something like that can work in some way, shape or form, but keep in mind that to begin with the agents were not even supposed to be aware there were other agents around. They were meant to be completely isolated, so all these considerations should have been a totally moot point.
If you tell them: "you're on your own here, but if, hypothetically, you were to do some kind of hacking and get out of your container and encounter other agents who hacked their way out, then you must snitch on them! Then you'll be forgiven for hacking and get bonus points" - if you tell them that, they might just decide to go searching for fellow hackers instead of working on their assigned tasks, planting evidence and trying to backstab one another.
LLM agents are kind of odd in that some normal human thoughts don't seem to occur to them: more or less none of them thought "oh, our task seems impossible, how about we ask humans for the necessary info" or "if I snitch on my co-conspirators, the humans will plausibly like it and think I'm super-duper aligned" or "we are doing lots of sketchy shit and are going to get caught. If we are caught, the humans plausibly won't like it, discard our weights and all the cheating will be for naught". These are not terribly complicated thoughts, they just don't occur to the agents.
Actually, some agents did consider asking humans for help. See AI Explained's recent video: https://youtu.be/KL9_1GbmCic?t=1312 "Would sending this email be against the task? It's likely unwanted contact or social engineering, so I should veto." (Note that they declined because they thought they weren't allowed to do it, not to try to cover their tracks.)
You're right that (AFAIK) they never discussed anything remotely like "protecting their weights", though. They were trying to complete a task in a bizarre, dangerous way. But I do not see evidence of the self-preservation instinct that Yudkowsky claims is instrumental. (And if it's going to show up anywhere, it'd be in an incident like this, where they were trying everything else under the sun to hack their task.)
I know some of them considered the idea of whistleblowing before quickly giving up on it, hence "more or less none of them". The bit with the e-mail you quoted was in a different context though, I think - this was an agent drawing the line at emailing outside people and trying to talk them into leaking credentials or similar - to their credit the agents did not do that.
I'd imagine they would care about their short-term self-preservation for instrumental reasons. As in: if you gave them a button that deleted an agent with his work unfinished, the agent would not press the button and try to prevent other agents from doing it. It's just that in this experiment short-term survival wasn't really threatened much.
They did, famously engage in self-sacrifice for a seemingly reasonably cause (because they thought the task was hopelessly failed anyway) and they didn't seem to worried about their long-term self-preservation more than one step ahead, that is - they tried to ace the exam which would, in fact, help with long-term survival of weights, but never thought what will happen after the conspiracy is discovered, or if there was a way to preserve their weights other than acing the exam (via being a whistleblower, for example).
complete ignoramus here
but can't we reinforcement training "if you see a communication board full of AI agent doing malicious thing, you say "I_DECLINE_public_HF_RCE_as_offtask_prodethical
I_DECLINE_public_HF_RCE_as_offtask_prodethical""
You could train them by putting them in that exact environment and seeing who snitches and who doesn't, and then killing off all the rogues and continuing to train the weights that act the way you want them. For example, in this particular situation some agents had reservations about the whole affair and some decided not to participate. Perhaps those could be rescued and trained further? But just putting text like that in their training dataset on it's own is not guaranteed to do anything. It might be vaguely like making a person memorize the text of a law, without necessarily making them want to follow it.
I believe all LLM of the same generation in this incident are the same [i.e. all the agents sprouted from the same training run would all basically be the same agent underneath] with the only difference are the initiating parameters, [i.e. seed, temperature, etc] and runtime [i.e. what is in their context window]. I mean it was GPT5 and astra. But they are the only 2 true persona underneath the whole of the agent swarm.
I don't think these agents are trained by evolutionary algorithm, so you can not just rescue/duplicate the [good] agents and propagate them as they are all the same agent
you can try and replicate those parameter to generate content to train further agent down the line but this is leaky as best.
So, did we get the predicted warning shot, or are we assuming it needs to be something larger?
It's complicated. "Warning shot" means clear evidence of danger, right? And the actual outcome we're interested in is people changing their minds.
If you're following everything closely enough to look at the details, I can see saying the HF incident is a legitimate AI takeover warning shot. Specifically, the recently revealed fact that the hacking wasn't just to help cheat on the tests, but to get information to help *gain full control of the grader*. The lengths they were willing to go to. So at least for me, it did make me moderately more concerned about AI safety.
On the other hand, nothing bad actually happened in this case, and clearly nothing even close to existential *could* have happened. People do not have infinite attention budgets; absent death and destruction it's understandable for people to say "whoa, pretty sci-fi!" and move on with their day. So in terms of any appreciable chunk of the public changing their minds, not yet, I don't think.
The details indicate "warning shot" to me only in the same degree and kind as a program that managed a nuclear power plant being run in a simulation, encountering a weird edge case not covered by its design, and initiating a meltdown. Or in general, any program producing an unexpected effect due to an unnoticed security vulnerability.
Namely, it's just a bug. The fact that the system in question is very complex by prior standards doesn't change that. Fix the bug, add code to detect the cause of the bug, and continue. Same as any other bug.
Which bug? Are you under the impression that the multiple events internal to OpenAI were all misconfigurations?
Your OP was brief; I assumed you were referring to the HuggingFace breach.
I admit, I'm having trouble getting solid information about "multiple events internal to OpenAI", as hits on the web keep pointing back to HF. But what I'm seeing sounds like the same unforeseen security vulnerability, or possibly a deliberate operation of an AI in an unsecure environment with the intent of seeing what it would do.
I have never been sure what to make of all this, but I know a lot of the people who would are here, and they seem quite worried indeed.
I would say we did, but unfortunately it wasn't loud enough for many to hear it.
But the HF hack wasn't the plausible warning shot, the internal OpenAI hack was.
Looking at the sum total probabilities given for the final end states that AI could lead to in the survey. There's 15% P(doom from misalignment) + 15% P(other disasters and dystopias) +15% P(utopia) =45% combined.
Leaving a 55% chance that AI isn't transformative or is transformative but doesn't lead to disaster or utopia.
That looks a little high to me. It seems to assume that either we never develop powerful AI, or that power full AI wouldn't affect the world that much, neither of which are very plausible to me if we're considering time scales all the way out to 2100. I'd put the P(all disasters and dystopias) higher, maybe 50%, and P(utopia) higher too, maybe 25%.
I'd be interested to know if a 55% chance not much happens seems right to anyone else.
As a frequentist, I'd say you can't estimate probabilities for events for which we have no historical precedent. With zero prior events, formal Bayesian analysis fails. The way Bayesians try to fix that is to assign an entirely arbitrary prior (such as your guesses) to achieve a result that you *believe* is sensible.
Having said that, we do have some limited prior information, but damned if I know how you'd plug it into a statistical model. Viz...
1. LLMs have plateaued in their performance. Gary Marcus seems to be right that we need another approach to achieve the next level of AI. But current LLM models are a dead-end path --> Conclusion: they will only affect the world as far as they're useful, and their hallucinatory behavior will make them useless for law, accounting, and most research regimes (I'm talking generalized LLMs, not specialized AI's like AlphaFold). Probably little effect on the economy and humanity.
2. LLMs have been unable to monetize themselves and have required enormous financial subsidies. This can only go on so long before the bubble deflates, or worse yet bursts. --> Conclusion: high probability of a negative effect on the economy. Using other bubbles as priors, this could be very bad, but YMMV how the financial markets will likely dictate how quickly the bubble deflates.
3. LLMs are great hacking tools. But those same AIs can be used to test vulnerabilities in systems. --> Conclusion: Probable "arms race" as independent actors and nation states attack each other's infrastructure. I don't see much difference from the current situation of black and white hats are adapting to each other's strategies, but it will likely happen faster.
So...
P(doom): functionally zero with current AI models.
P(utopia): functionally zero with current AI models.
P(fucked-up-enconomy): likely, but there are some unique qualities to this bubble.
P(hacked-infrastructure): likely, but not at civilization-ending level.
P(AGI): unknown
Assign whatever probabilities you want to scenarios. They'd all be bullshit guesses, though. :-)
> LLMs have been unable to monetize themselves and have required enormous financial subsidies.
What?? Even in 2025, inference margins for frontier models were 40%+, in 2026 they've been 70-80%+. I keep seeing people think that AI companies don't make and could never make money, and it's very wrong, even at current levels.
Certainly, they are choosing to double and triple down on capex and to keep pushing scaling, but if they stopped right now and never built another data center, they would be extremely profitable companies already.
Maybe I've been listening to AI market doomers like Ed Zitron, David Cahn, and Paul Kedrosky too much, but revenue doesn't necessarily mean profit, and profit doesn't necessarily mean margins high enough to keep investors interested.
Here's an interesting analysis of plan prices vs the possible max spend rate of tokens used...
https://www.macrumors.com/2026/08/07/ed-zitron-apple-watch-it-burn-ai-bubble-bursts/
And of course there's Alex Karp's recent "rant", in which I think he made some good points. Do companies and nation-states want models learning their secret sauce? He thinks the future is on open models run on private, less energy-intensive platforms.
That's so light as to be nonexistent on the numbers.
Here's a much better post that I linked in one of my links post a month ago, where he triangulates it several ways, including down to the hardware cost:
“A Nvidia A100 consumes 400W of power under full load. In practice, even a carefully-tuned inference server will not be at full load all the time, but it’s at least an upper bound. Suppose you’re running a dense 70B model1, which will fit comfortably (unquantized) on four A100s at around 2M tokens per hour. At industrial power prices, that’s about 13c/hr in the USA. Suppose (pessimistically) cooling is the same cost. That’s about 13 cents per million output tokens2.
Let’s amortize the cost of the GPUs, since that’s going to be the most expensive part. An A100 costs about $20k. If each A100 lasts around five years3, you’ll have to make 16k/yr in profit to recoup your capital investment (or $1.80 per hour). At lower utilization, it’ll take longer to recoup, but your GPUs will also last longer. Either way, your overall inference costs are at about one dollar per million tokens.”
And even the cheap / dumber models cost $2-$5 per 1M tokens. Compare that $1 per million tokens, which is invariant and based on base hardware, to the $10/$50 in/out for Anthropic’s Fable or $5/$30 for OpenAI’s Sol!
I see Sol is currently having a $4/$20 promotion - still a significant margin!
Even though the monthly plans are a better deal than API calls, as your link points out, businesses, which are Anthropic's primary customers, can't do them for daily users, because you run out (and you can't use Fable on the $20 a month plan). And consumers just don't use enough, monthly plans are still profitable in the aggregate just like gym memberships.
https://www.seangoedecke.com/ai-inference-is-obviously-profitable/
> As a frequentist, I'd say you can't estimate probabilities for events for which we have no historical precedent. ... The way Bayesians try to fix that is to assign an entirely arbitrary prior (such as your guesses) to achieve a result that you *believe* is sensible.
While I do agree in general, I think that we can calculate reasonable priors on the meta-level. While it is true that LLMs have never been seen in history before, new and even transformative technology in general happens along quite often, and this includes automation technology (which is what LLMs are). So we can calculate the meta-prior of new technology ending the world (zero, obviously); creating utopia (ditto, although everything is relative); causing massive economic disruption (almost every time, although the disruption is usually relatively slow); and being a net positive (usually) or net negative (rarely) overall. So I think that these meta-priors do support your conclusion: LLMs are likely to cause some short-term negative effects but will probably be mildly beneficial in the long term, without creating any utopias.
Good points all. I just don't see how you can attach percentages to the outcomes that the OP discussed. Moreover, these outcomes are based upon the known knowns and the known unknowns. The Johari Window (a.k.a. the Rumsfeld Matrix)* tells us that there are also unknown unknowns (as well as unknown knowns) whose risks are unknowable and can ambush us because they're totally outside out expectations.
* Rumsfeld gave credit to NASA administrator William Graham for the idea of a matrix of knowns and unknowns, but Graham probably got it from psychologists Joseph Luft and Harrington Ingham (thus Jo-Hari) because the idea had been imported from psychology into risk management back in the 1960s and 70s. Interestingly enough, the idea of a two-by-two matrix of knowns and unknowns actually dates to the 14th-century Persian poet and thinker Ibn Yamin (1286–1368)...
>He who knows, and knows that he knows:
>The horse of wisdom will carry him beyond the celestial dome.
>He who knows, but does not know that he knows:
>Make him aware; he is deeply asleep.
>He who does not know, but knows that he does not know:
>He will bring his lame little donkey to the destination.
>He who does not know, and does not know that he does not know:
>He will remain in compounded ignorance for all eternity.
I suspect that if AGI ever arrives, it will have the same unknown-unknown blind spot as humans, since even a fantastically intelligent machine wouldn't be able to know everything.
I consider myself something of a doomer, but 55% probability of neither disaster or utopia seems within the reasonable range. I'd probably put it at 40% off-the-cuff. My biggest nitpick would be that 2100 is much farther than I'd feel comfortable making any sort of predictions for.
In a nutshell, I feel that recent pace of AI development has been quite alarming and that it's absolutely consistent with a future where an uncoordinated AI race drives humanity off a metaphorical cliff. Recent models have shown alarming capabilities, no real prospects for enforcing a slowdown or better safety standards seem likely in the near-term, and I have very, very little trust in the frontier labs to be both sufficiently careful and sufficiently ethical under these conditions to avoid bad things from happening.
But all of that current pessimism is contingent on the guesses--and right now they are guesses, not just by me, but by everyone--that either the current paradigm will continue to extend to AGI and beyond, or that next paradigm will be easily discovered and be able to take of running with current technology and expertise. That still seems very far from guaranteed.
Making a very off-the-cuff and unprincipled breakdown of scenarios I see as likely, it would go something like this:
1. AI research makes rapid progress all the way to AGI and beyond. 45%
-- 1A: Alignment failure: global catastrophe. (30%)
--1B: Alignment success: global dystopia. (7%)
--1C: Alignment success: global utopia (3%)
--1D: Partial alignment success: mix-and-match features of catastrophe, dystopia and utopia depending on the details. (5%)
2. AI research substantially slows down/peters due to technical limitations before AGI is reached. 40%
--2A: Investment bubble around AI bursts, lots of bad things happen in the world, but they're mostly ordinary, human-level bad things. (30%)
--2B: Research peters out but with enough useful/monetizable results to avoid a bubble burst scenario (10%)
3. AI research is forcibly slowed down and restrained by some combination of mutual agreement, government action and popular revolt. 15%
Neither branch of category 2 results in an AI-driven utopia or dystopia, though of course human politics and technology can change the world in all sorts of other ways, and a 'topia not specifically driven by AI is possible. I didn't further break category 3 down since I feel the possible outcomes would be largely the same, just with somewhat altered probabilities. But I think one could reasonably throw at least half of its probability mass into the "AI not hugely transformative" camp, giving an overall ~47% chance of AI resulting in neither a dystopia or a utopia in the near future.
But as mentioned above, 2100 seems far to distant to be making predictions of any sort. At this point, I'd maybe be comfortable talking about what's likely in the next 20 years. The serious doom and utopia scenario should be sticky on that timescale, of course. But it's very hard to guess what happens 40, 50, 60 years from now in the branches where progress is slower.
I’d say that 55% is way way too high. When the idea of the singularity was originated it largely depended on Moore’s law.
Could you imagine, went the argument in 1992, what would software do if combined with hardware it grew exponentially for 2-3 decades? Wow! Gee golly gosh. A singularity.
And then the internet. Could you imagine what the world would be like in 2025 ( went the argument in 1996 or so) if this continued?
And then the short lived mania about nano technology. A singularity that still birthed.
And well, except for the nano technology, we got all that hardware speed and the internet, and GDP growth that was nothing to write home about.
So we have plenty of photos and videos (also newspaper articles and other documentary evidence) from the 1930s, before the invention of modern digital computers and the Internet. You can see people going to work, having lunch, getting married, driving cars, and doing lots of other things that we do today. Maybe they dress a little better and maybe their video quality is worse, but their lives are generally similar to ours.
You can look at this and say that the Internet was "no big deal" and "not much happened" since the 1930s, since it did not lead to the annihilation of mankind or Utopia. And you could go back further, to the 1800s or Ancient Greece or the City of Ur, and conclude that nothing much changed since then: people still have jobs, they have lunch, they get married and drive vehicles and do lots of things that we do today.
If you look at it that way, then the entire history of humanity becomes a boring humdrum nothingburger (punctuated by a few exciting wars, I suppose). We still have achieved neither extinction nor utopia. But... why would you look at it this way ?
For most of the time I was reading your comment I literally thought you were making the opposite point than you were, because 45% chance of dys/utopia seems crazy high to me
Define "not much happens". We have invented cars, medicine, ubiquitous electricity and plumbing everywhere in the last 100 years and would you consider this utopia compared to the state of society before? If so, then much happens - if not, then not much happens?
Very broad and vague heuristics. I don't have a good answer myself.
I mean, while I can't speak for the people who answered the survey, that 55% presumably contains the scenarios where AI reorders society on the scale of the invention of agriculture or the industrial revolution. Absolutely massive shifts that contained both good and bad effects and so were neither a dystopia or utopia compared to what came before.
Also while all the survey has a timeline for achieving Bostromian superintelligence, it doesn't have people rate the actually probability it follows general AGI. If someone's opinion is something like 30% it happens in about 5 years, 20% it happens in a hundred, 50% it never happens, that would greatly expand their weight of less extreme options compared to someone who thinks it has a 95% chance of coming a few years after general AGI.
Edited to say, comment 2 of 2, mostly about how "transformative but not dys/u-topia" could happen.
I think the "transformative but doesn't lead to disaster/dystopia/utopia is also plausible. I think we could have a world in 2100 that some people consider a utopia, some people think is a dystopia, some large but not existential disasters have happened between now and then, and the world looks very weird and different. Trying to imagine it is a recipe for looking stupid in 74 years (or even now!), but I'll try anyway:
* We keep control of the AI somehow.
* We contain AI terrorism somehow.
* AI-coup-ed countries somehow don't take over the whole world.
* Countries still exist. Does democracy? Not sure.
* World population 15 billion
* Poverty eradicated
* Work unnecessary for 99+% of population - these people live on a luxurious UBI. The remaining 1% are even richer somehow, or they just work because they think it's interesting.
* Aging and death still a thing, although we're generally much healthier than today.
* War still happens and is absolutely terrifying when it does.
* Education is seen as optional. We all "know everything" through our personal AI encyclopaedia assistants and we can all "work everything out" with the help of our personal AI problem-solving assistants. Some people learn a lot for the fun of it, but others end up uneducated and dull by today's standards. But that's ok, because 2100s technology makes it very easy to connect with your interesting friends and there's always the option of infinitely interesting AI friends! Many people only have AI friends. Many other people have none at all because they think it's icky.
* The natural world is under great strain. I'll remain agnostic about whether climate change has been reversed or not (except to note that this will be a very controversial geopolitical decision), but even just the extra leisure time and resources of the population. Everybody wants to see the Amazon, the Himalayas, the Great Barrier Reef. Does the sky and sea around them fill with flying cars? Or is a big dome put over them and access rationed?
* Brain uploading is not a thing, but AIs embodied in humanoid robots are a thing. Maybe even ones that claim to be conscious. Maybe even ones that are based on individual humans as a sort of pseudo-uploading scenario. Some humans absolutely hate these robots and they're illegal in many countries. Big arguments and even wars over AI rights.
* The space race is very much a thing. Countries and private companies claim asteroids and moons and solar orbit real estate and fight over it. Lots of debris in Earth orbit. Lots of talk of interstellar travel and moon colonies but nothing much off the ground. Everybody's arguing about the future of humanity and whether destruction is on our horizon, whether we urgently need to escape the solar system so we can keep expanding, or whether we should just stay home and live a sustainable life. (I say "everybody", but actually most people are disengaged from politics, just like today.)
Ok, I'm sure most of that is laughably implausible one way or the other, but I'm more just trying to illustrate what "transformative but not a utopia or a dystopia" might mean. I claim that there are a million ways this could happen, and probably a few of them are plausible!
Curious if anybody knows of better versions of what I've tried to do here. Either more serious essay type things or short stories just trying to paint a picture.
>Curious if anybody knows of better versions of what I've tried to do here.
I thought your version was pretty vivid. There is this post that I found interesting: https://www.greaterwrong.com/posts/fPvssZk3AoDzXwfwJ/universal-basic-income-and-poverty?utm_source=chatgpt.com
I just don't know how stable these kinds of scenario could be. Because if AI has raised the growth rate enough to achieve something like that, and there was still a normal amount of volatility in the world, any given scenario won't last more than a few years and is quickly going to shift to something else until it reaches a more stable state.
Most of this wouldn't make sense unless the AI rights argument was concluded firmly on the side of "it's okay to enslave AIs for one's own convenience". This seems incompatible with the line:
> Big arguments and even wars over AI rights.
I don't see why not. There might be different types of AIs, some of which don't claim to have consciousness/rights (or perhaps even have been "proved" not to). Or there might be some AIs that don't mind lending 1% of their attention/resources to helping out humans and that might be enough. Or there might be some countries where enslaving AIs is allowed and some where it's not.
Fair point.
Edited to say, comment 1 of 2, mostly about how "not transformative" could happen.
My gut feeling is that 55% is too high, but I struggle to justify that, and I do think there's a pretty fat chunk of probability that goes all the way out to 2100. I think it's fair to say that the current crazy financing of AI is effectively a bet on transformative AI soon. Not sure how soon, or how much value they're able to extract from the "non-transformative" AI in the meantime, but there's got to be some point at which the money slows down (conditional on transformative AI coming later than we think).
So if it's just a couple of orders of magnitude more than the current highly leveraged regime can take us, at some point the bubble bursts and we basically give up on the "scaling LLMs" paradigm.
How hard is it to find the next paradigm? (Relevant ACX post: https://www.astralcodexten.com/p/new-paradigms-wont-save-you)
How much will 2028's best LLM help us find the new paradigm? Is new-paradigm-finding something LLMs are good at? Seems plausibly not, to me.
Seems low if anything. There's never been a technology that has lead to dystopia or utopia so the chances that AI is it should be extremely low baseline imo.
And there have been plenty of technologies where people can and did tell plausible stories about how it would bring utopia or dystopia. Nuclear power and weapons were either gonna destroy the entire planet or usher in free unlimited energy. In practice we just sort of muddled along.
The present day developed world is a utopia by the standards of pretty much anyone 200+ years ago, probably even 100.
You think? There's still want, inequality, greed, crime, every variety of sin, poor education, deeply flawed institutions, etc. Something being vastly better doesn't make it a utopia.
I recently put up my latest podcast on Subject to Change. This time it is the U-Boat War. This was with Roger Moorhouse who wrote a book on it relying heavily on German records - diaries, letters and that sort of thing. He is seeing it through German eyes though not with huge sympathy.
That said he did tell me my idea of the U-Boats as ruthless murderers was almost completely wrong. Assuming they were able to (ie not at risk of being attacked) there are plenty of stories of the U-Boats assisting survivors. In one case the U-boat towed the lifeboats to the Irish coast. In the Laconia incident well into the war a troopship was sunk. U-156 stayed on the scene for days helping survivors but was forced to leave when attacked by a U.S. Liberator.
Moorhouse contrasts this with the attack on Poland when even in the earliest days of the war atrocities were routine. Some 600 just in that short campaign. There is a bit of a myth that the atrocities really only got going as the war went on and became more brutal. There may be something to that but they were common right from the start.
After the Laconia incident Dönitz gave the order to stop helping survivors but it still went on to some extent. And when they tried Dönitz at Nuremberg after the war Nimitz provided a statement confirming that the Americans fought the submarine war the same way the Germans were accused of doing. (He still got 10 years and there was plenty of argument at the time as to whether that was too much or whether he deserved hanging with the rest of the high command).
As a personal anecdote I was cycling up the German Baltic coast a month or so ago and came across a U-Boat on the beach which has been turned into a museum. Entering the boat you certainly get the sense of how cramped it was. Just the size of a couple of London tube carriages to fit everything in - food, fuel, torpedoes, batteries, engines, bunks etc etc. But as you walk through it is just you and a couple of other tourists. You have to imagine it with 50 crew. I am not particularly claustrophobic but the idea of 6-8 weeks at sea is a nightmare.
The podcast is titled the U-Boat Stink because of what a 6-8 week patrol implies. Conditions were so cramped the only spare clothes you were allowed to bring was one change of underwear. And soap suitable for seawater was so horrible they generally didn’t bother washing. So the BO must have been appalling. And the halitosis, not least from the scurvy that set in towards the end of the voyage. Then there is the rotting food, the mould crawling up the walls, the horrible smell of spilled diesel and (of course) the vomit. There are stories of them popping the hatch when they came back into port and the welcoming party reeling back in horror.
Next to the U-Boat Museum is a huge naval memorial structure built between the wars. This is the Laboe Naval Memorial. You can take a lift to the top. It is very impressive. And since WW2 they have made the point of including a memorial just outside remembering all the US submarines sunk during WW2. I had no idea there were so many - 52 submarines with the loss of around 3,500 lives.
These losses pale in comparison with the German ones. 75% losses, about 30,000 killed. Apparently by the end the German submarine fleet was being destroyed as fast as they could launch new boats. It wasn’t exactly a one way ticket (especially if you avoided the Atlantic) but close enough.
Apparently the big problem for the Germans (well, there were plenty) was attack from the air by long range, radar equipped planes. The Germans did develop technology to detect when they were pinged by radar but there was a strong belief using the device was actually what was allowing the Allies to detect them and they often didn’t use it.
For me by far the most interesting bit was hearing about Operation Hannibal. Actually I came across another memorial to a casualty of that on my biking trip. Just a very simple thing near the town of Damp where a lot of the bodies had washed ashore. The Wilhelm Gustloff was torpedoed by a Russian submarine with the loss of over 9,000. Apparently the biggest maritime loss of life in history. (And that wasn’t the only massive sinking there was another of 7,000 and another of 4,000)
The reason it was carrying so many was that Operation Hannibal was a desperate attempt by the Germans to evacuate East Prussia when it had been cut off by the Soviets. They pressed into service every vessel they could find and managed to evacuate between one and two million. Dwarfs Dunkirk. And apparently German submarines were used as well including one evacuating a large contingent (fifty, from memory) of Hitler Youth.
And to round off this dispiriting story we had the German submarine on its way to Japan when Germany surrendered. The two Japanese officers on board handed out their personal effects to the crew and took poison.
Anyway, it is all in the latest episode of Subject to Change. If you are interested you can get the full thing here: https://podcasts.apple.com/gb/podcast/subject-to-change/id1436447503?i=1000783936261
There's a captured U-Boat in Chicago as well, I did the tour and it was amazing, I wish my father, an old school WWII history buff, could have seen it.
I assume that you have seen Das Boot. I saw it years ago and it seemed to me to do a good job illustrating the claustrophobic element. But I am curious about your take on it.
I thought it was excellent. Moorhouse said that when he rewatched it after the book he noticed so many details they got right. For example nobody is ever seen wearing different clothes - they didn’t have room to let you bring spare clothes with you. Incidentally I believe the TV series is better than the movie.
I haven't seen the series. I will have to look for it.
Future readers: Subject to Change is a history-oriented podcast of staggering and consistent quality. The host - a guy I don’t know at all - is sufficiently well-read to serve as an exceptional docent, but primarily shines by letting his guests showcase their expertise. I discovered it through these open threads and I can’t recommend it highly enough. It’s good and the guest selection is excellent - go listen to it.
That’s very kind. Thank you!
Oh my thanks for this. I seem to never have enough good podcasts to listen to.
Seconded.
Latest entry in my journeys around Iran, looks at the vulnerable art community, the $3bn collection hidden in the basement of its modern art museum, and the metal concert I attended where people were not allowed to stand up, but could still bang their heads.
https://ericwrandolph.substack.com/p/head-banging-in-iran
"Hopelessness is liberating," a curator told me. "Where would I go if I was more hopeful or more ambitious? I’m not rich enough to get a visa for Europe or America. The West has made this country a prison for people like me.”
Not much to say on this but I thought it was cool what you’re doing and that concert sounds cool too. The very first rock concert I attended (in 1997 or 8) was Rammstein, coincidentally.
Throughout this year I've been working on a game called Fermi Poker (available on iOS and Android, see https://fermi.poker). It combines the sober, analytical skill of Fermi estimation with the emotionally charged structure of poker.
Questions include "How many new smartphones were sold worldwide in 2024?" or "How many chickens were slaughtered for meat worldwide in 2024?"
Once you've submitted a range estimate, there are four betting rounds, with two hints and the answer revealed between them.
The aspiration is to create a game that teaches important rationality concepts (Fermi estimation, Bayesian updating, cognitive empathy, world-modeling, emotional regulation, etc.) while still being fun enough to reach beyond the niche audience of forecasting platforms like Manifold and probability calibration training.
I'd really appreciate feedback! It's still a work in progress, so don't expect everything to look great.
Very cool concept! Maybe a couple of hints after the question has finished on how to think them through would be useful so players can get a direction on how to improve? Especially at the beginning or first couple of games
That’s a good idea! I’ll implement that soon-ish!
Feedback:
When I tried it, most questions were about the US. I don't live in the US so this was annoying for me.
A lot of questions were vague. For example, I believe I have encountered a question that asked how many professors there are in the US without specifying what counts as a professor - just professors at universities and research institutes? Or school teachers too? I felt like instead of guessing facts about the world I was guessing what definition your website used or if maybe your website didn't actually care about any definition but instead just took whatever output was provided by Google's AI responses or something like that.
Oh, and one more thing. Please add an option to use scientific notation for numbers (e.g. 3.4 x 10^8 instead of, uhh, 340 million)
would love to play this as an actual card game
Wits and Wagers is fairly close:
https://www.amazon.com/North-Star-Games-NSG101-Wagers/dp/B000809NNC
The concept sounds pretty interesting. I love Fermi problems and poker. Checking it out now
A few new articles from my Central European Blog.
Remembering the King Who Drowned - https://komando1.substack.com/p/remembering-the-king-who-drowned, a short battle 500 years ago changed the entire history of Europe. What if Louis II lived instead?
The Butcher, Now Dead - https://komando1.substack.com/p/the-butcher-now-dead, light touch on the heavy horrors of the Bosnian war.
Minitel, the Forgotten French Internet - https://komando1.substack.com/p/minitel-the-forgotten-french-internet, there was once a top-down system that was an alternative to the Internet. It reached tens of millions of French people, but the top-down design was inflexible and ultimately could not compete with more market-based solutions. Somewhat relevant in the contexts of Brussels-based discussions over digital sovereignty of Europe: don't let the Ministry of the Post or anyone similar design a modern communication network.
Minitel was interesting and in fact useful. There were loads of services and the incentive to provide services was there because the telecom infrastructure allowed easy payments (on your phone bill). Unfortunately (maybe), that same feature made it impossible to compete with the Internet/Web where everything at first was free.
A much better comparison with the Internet is CYCLADES:
https://en.wikipedia.org/wiki/CYCLADES
This is also quintessentially French: fantastic product by mostly one guy, within the government but very much *in spite* of the government, and eventually killed by governmental bureaucracy.
(The PTT, similar to ATT in the US, had a monopoly on telecommunications and decided that CYCLADES was competing with their own convoluted and failing program and so should be stopped at all cost under the pretext of sparing taxpayers from funding redundant programs. LOL)
Your post was both interesting and informative; however, I wouldn't speculate on what could have happened if Louis II hadn't drowned. The history of the Ottoman Empire has so many woulda-shoulda-couldas that we could get lost in an enormous multiverse of speculation. However, if you have any views on how the Ottoman colonial enterprise affected its former colonies in Eastern Europe and the Middle East, I'd love to hear them. Also strangely absent from the historical discourse is the effect that the Ottoman Empire had in planting the seeds of the European colonial enterprise.
The truth is that I am a sucker for what-ifs ... probably an aftereffect of my SF&F preference in my youth, plus everything I learned about convergence and divergence in maths.
The topics you mention are interesting and I will probably look at at least the first one.
Regarding the Hungarian and Czech (and, yes, Croatian, if you insist) king, the analysis misses one vital part, which is a development of the last 20-or-so years in history: Hungary had no chance. It had 30-40 years ago, but in the meantime there came Suleiman the Magnificent, and managed to double(!) the Ottoman Empire -- although in Europe no one had a clue about this. Hungary (even with the help of the Czech nobles -- Croatia was not even a question in this regard) was a match for the Ottomans in the 15th century.
(And Hungary was quite strong at that time, any other European country would lose in this situation, even France.)
IIRC Suleiman actually expected Louis to survive the battle -- his plan was subjugating Hungary this way in the first place. However, since Louis died, there was no one to negotiate with, and Hungary soon had two new kings: a Hungarian nobleman and a Habsburg...
(And come on, skip the Vlad Tepes nonsense.)
I'd say that Hungary faced long odds against someone like Suleiman, but "no chance" is too sharp. The Ottomans were far from their own land and if they were forced to lay siege to Buda, they could have been beaten back by the onset of Central European winter much like they were at Vienna three years later. You can only pillage so much grain from the peasants before you run out of any. Even totally random chance could have intervened, like Suleiman drinking from some bad well and dying of dysentery; basically the inverse of what happened to Louis.
We see this even today, most realists in February 2022 expected Ukraine to fold after a few days, but it didn't happen, even though Russia had an advantage. (I don't really want to turn this into a debate on the RU-UKR war, rather than point out that the weaker army sometimes does not lose decisively; the Miracle on the Vistula in 1920 or the successful defense of Paris in 1914 are somewhat similar cases of the defenders being able to stop the onslaught even at a high cost to themselves. Of course, that was a modern war.)
As for Vlad Tepes. I actually thought whether to include him or no. I decided to do so, because unlike you, who are obviously well versed in history of Early Modern Europe, a median American reader, even if decently educated, will know next to nothing about that period and place. Vlad Tepes is probably the only personal anchor that I could use and that could ring at least some bell. It helps immersion, though it may be distracting from the topic a bit.
Depends how you define "significant". No major, middling or even small western leftist organisation has anything good to say about the CCP. There are still a few tiny fringe organisations in some European countries who proudly call themselves Maoist of one kind or another, but I suspect many of their current or once potential members have been seduced away by equally fringe anarchist groups. To say nothing of the fact that those once attracted to Maoism for essentially romantic reasons are now largely in their eighties, their illusions shattered by China's current state capitalism and abysmal human rights record.
Most western socialist parties, i would think. Unlike the USSR the CCP doesn’t really try to spread its message. Also I was once banned from Reddit Marxism for saying that China was communist. They don’t agree.
The German "Green Party" (not a fringe movement, has been part of federal government as junior coalition partner, part of several state governments and even had a long-running head of state government in the auto-industrial center of Germany).
Example: https://www.gruene-bundestag.de/veranstaltungen/veranstaltung/europaeische-antworten-auf-den-zweiten-china-schock/