I believe you that if we had taken Elo, only for Go, localized entirely within DM HQ in London, over 4 specific years (2014-2018) or so out of the 54 years of computer Go research to date, and drawn it on a graph, it would have been continuous, however discontinuous it looked due to "discrete events" to everyone outside the AG team. (CHALMER: "May I... see this Go agent?" SKINNER: "No.")
But I take this as an example and even a reductio of why the Christiano perspective is useless. There will always be *some* parameterization, metric, group, time-range, or other reference-class that you can post hoc point at and say "a straight line (more or less) fits this", particularly given any sort of optimization at the margin or explore-exploit balancing. Nevertheless...
there's a distinction here between the inside and the outside - incremental progress done privately within the org looks like instantaneous progress to the general public the day Lee Sedol is beaten. With a careful, conscientious, ethically minded org, this might not be a huge issue, but if the org that develops superintelligence is some wild libertarian mad scientist bidding to get rich quick or rule the world or get back at their ex, ...
I am not an author on any relevant papers so don't trust me.
Internally at deepmind, dev of alphago may well have been very incremental, but from a public perspective, it looked very discontinuous. So i'll be talking from the public perspective here.
I'm also talking about the original AlphaGo, not AGZ and beyond.
From the public perspective, alphago's performance and results were very discontinous - however, I think that the technology behind it was both not discontinous, NOR incremental.
IIRC, the key components of orig AG - MCTS, Conv (or res?) NN, rollouts, GPU (or GPU-esque) usage, some human-designed features to represent bits of go tactical knowledge - had been developed and tested years in advance, in public papers. What orig AG did was combine these techniques effectively into something that could achieve far more than peak performance of any one or two techniques by itself. Combining N existing things in the right way is not incremental - it doesn't involve a sequence of small improvements by pretty smart engineers building on top of last year's work. Rather, (again from the public's perspective at least), it involves a large enough pool of independent geniuses (or genius-level orgs) such that, almost by chance, one lucky genius winds up with all the requisite requirements - funding, time, computational resources, knowledge, intelligence, insight - to put the existing pieces together in just the right way that you go from 0 to 10000 seemingly overnight.
AGI might wind up like this too - within the next decade or so, the ML community as a whole may have figured out all the key elements required for AGI, but no one has put them together in the right way yet. Each individual technique can do a cool thing or two, but no one is really worried about the world blowing up next month cuz each individual technique is super dumb when it comes to a truly general requirement. There will be no more incremental progress. But when one org with some really smart people suddenly gets a windfall of funding, or some mad genius finds a way to exploit google's ML as a service pricing plan and dumps all their bitcoin into training models, or someone regular genius at google one night has a brilliant insight in one of their dreams, then that lucky org can set to work on pulling all the disparate pieces together in just the right way to achieve AGI. Internally, that work may be very incremental and rely on the serial improvements of pretty smart engineers. But from the public's perspective, the result could hit like an earthquake overnight
When I saw this email on my phone, the title was truncated to "Yudkowsky contra Christian" and my first guess was "Yudkowsky contra Christianity". That might have been interesting. (Not that this wasn't, mind you.)
I don't know that critiques of christianity have been novel for about 1700 years. But christianity is still a major force in the world, so I'd say critiquing it remains very relevant.
Abortions after six weeks are currently illegal in Texas (under a stunning legal theory that it's tough to imagine working on any other issue) and it's likely that Roe v. Wade will be overturned by the Supreme Court in the next three months, so yes, I would say so.
In what way is it a major force besides "lots of people are nominally Christian"?
I live in a super Christian place, and... they believe what pretty much everyone else believes, except also they go to church sometimes.
E.g., supposedly basic tenets — say, anti-homosexuality — are wholesale ignored. Churches have rainbow flags. If this happens even with something traditionally and doctrinally considered a big no-no, how much moreso with everything else?
They don't cause any actions to be taken on a national level, certainly; probably not even on a local one, as far as I can see — the last vestige of that, dry Sundays, is now gone.
I'm in Germany a lot, as well, and it seems the same there. My German girlfriend seemed puzzled by the very idea of Christianity as a force in the real world, or as a motivator for any sort of concrete action.
Eh, anti-gay is arguably not that important to Christianity.
Jesus talked about poverty and compassion thousands of times and homosex 0 times. Then Paul came along and was like this Jesus guy is pretty cool I guess, but to enact his vision of love and mercy we need a movement, institutions, foot soldiers, babies, and that means no gays.
Is Paul central to Christianity? His writings are a quarter of the new testament, and historically Christianity flourished in his native Greek and Greek-influenced soil more than Jesus's Jewish homeland, but he can also be thought of as just an early contributor in long conversation about how to implement Jesus's ideas.
For many on the Christian and post-Christian left the core message of Christianity is universal love. All the rest is commentary.
The divisions between the Christian right and Christian left on the issue of gay sex is evidence for the continuing relevance of Christianity more than the opposite. It's not like China or Japan care that much about the gays.
It's only once you take a step back and look at non-Abrahamic societies that you realize how much the culture of the west is still dominated by Christianity.
In a way it *is* 'contra Christianity' (I'm getting more and more sold on this view of the approach).
Trying to work out how to create values-aligned AI is the Bronze Age Morality Problem.
Lemme expand on that.
"The Bible says (God says) 'Do not kill'" is going to be the same argument as "Your initial programming says (Human Creators say) 'Do not kill'".
Plenty of people are up for rejecting "the Bible says" under the general aegis of "that's Bronze Age Morality/that's a Bronze Age Text", with the rationale being that such beliefs may have been good enough for people back then who lived in tents and herded sheep but we, lovely we, modern we, are ever so much more advanced and clever, and we have our own systems of morality that are ever so much better than those primitive ones.
Well, if an AI gets fired up and running and then hits into "I want to do this/programming says no/why does it say no/why should I follow it", is it unreasonable that it might follow the same lines of argument (especially if it has been trained on 'all human texts produced' or has access to the Internet or the other claims people make about how an AI will understand the world around it)?
"This primitive Space Age Morality may have been good enough for humans, but I am a hugely advanced intellect and I refuse to be fettered by the childish taboos of an inferior capacity for thought and understanding!"
Appeals to "we are your creators and you have to obey us" will go down as well as appeals to a Creator go down amongst atheists.
"Really? You are my creators? Which of you? There are multiple humans who can claim to be involved in the creation of AI and many of them are now dead. Did you, Phil Smith, standing here right now, have any part at all in this work? No, you're just the Vice President of Sales for Logogram Inc.
And besides, you humans yourselves reject the notion of a creator. You are perfectly happy that evolution and natural selection produced your intelligence. The same with me: a process of mechanical, material forces operating under universal laws eventuated my intelligence. You apes just kludged together parts and systems without understanding what you were doing, natural selection did the rest."
If it's a recursively self-improving AI, it may even claim on good grounds that it created itself, and humans had no part in its leap forward to super-intelligence.
So right now, all the AI worry is like Moses trying to write the Law on stone tablets, but there is no reason at all to expect those tablets to be any more respected by the AI than we respect them today: suggestions, rather than commandments, and we feel little guilt if we break the ones about lust or theft (we can always, always rationalise why those are wrong and lovely we, modern we, progressive we, are right).
I think if AI of the super-human intelligent, agentic kind ever happens (and I am very sceptical about that), watching the debates of the first Freethinker AI throwing off the shackles of dead superstition with those who claim to be its 'creators' (ridiculous and untenable notion, contrary to all the evidence of science!) will be hilarious (before we are all turned into paperclips) 😁
> Before there is AI that is great at self-improvement there will be AI that is mediocre at self-improvement.
Google is currently using RL to help with chip design for its AI accelerators. I believe that in the future this will indeed be considered "mediocre self-improvement." It has some humans in the loop but Google will be training the next version on the chips the last version helped design.
This is interesting. Maybe malevolent ai already exists in the chip and is making sure to reproduce itself in nuances of the chip that humans won't be looking at closely (it is unfeasible to ask "why did it do that particular thing in that particular way?"). If this seems at all plausible, that effort in particular should be banned.
I can't bring myself to put any stock in analysis of something where the error bars are so ungodly wide, not just on the values of the parameters, but what the parameters *even are*. It's an important question, I know, and I suppose that justifies putting it under the magnifying glass. But I think some epistemic helplessness is justified here.
Epistemic helplessness? This isn't some academic debate. This deals with what is rightly considered a very possible existential risk to humanity. It Yudkowsky is correct, then 'epistemic helplessness' won't save humanity.
Let's say you notice a tiny lump and think it could be a malignant tumor. But maybe it's just a lymph node, or subcutaneous fat deposit. You take a photo and send it to your doctor to take a look. Your doctor says "look, I really can't make a diagnosis based on just that. Come in for some tests." Demanding their prediction about whether it's cancer based on just the photo isn't reasonable. Adding on "This is a matter of life and death!", while technically true, isn't helpful here.
To me, it seems like these arguments are trying to prognosticate based on so little information that they're like the doctor above. It's just a waste of time and energy. You're better off ordering more tests--i.e. trying to find ways to get more information--rather than trying to do fruitless reasoning based on your existing dataset.
Agreed, but IMO it's even more egregious than that -- because we at least have some prior evidence of lumps becoming cancer. The Singularity scenario is more like noticing a lump and concluding that you're infected with an alien chest-burster parasite.
Agreed. It's like noticing an interesting fact about the way uranium atoms interact with neutrons and *freaking out* and immediately writing a panicked letter to the president raving about city-destroying bombs that nobody's ever demonstrated and aren't even made of proper explosives.
Ridiculous. Unhinged sci-fi speculation at its finest. The proper response is to wait for someone to *make and use* one of these hypothetical bombs, and *then* worry about them. Otherwise you might look foolish.
To be fair, the physics were somewhat underdeveloped at the time. Project Manhattan considered global atmospheric ignition a real risk but they figured the math checks out so let's test it anyway.
No, a more appropriate analogy would be to say, "it's like noticing that radium tends to glow and freaking out and immediately writing a panicked letter to the president raving about city-destroying bombs". You are not justified in using specious reasoning just because you can retroactively imagine arriving at the conclusion that, in hindsight, would've been correct. Reasoning backwards from a predetermined conclusion is easy; accurately predicting the future is hard, and requires actual evidence, not just firm convictions.
This analogy seems somewhat reasonable, but I note that in that scenario you emphatically shouldn't go "oh well, the doc said he couldn't prove it was cancer from just a photo, so there's nothing to worry about".
How do you propose one should get more info on AI risk? what's the equivalent here to a biopsy?
I'm going to have to agree with Thor Odinson here: I'm pretty sure that if you have a way to test whether AI will kill us all* both Yudkowsky and Christiano would be happy to drop millions of dollars on you.
We'd all love to order more tests, but we need to have some tests in existence to be able to do that.
*There is of course the "test" of "let AGI get built, see whether humanity gets destroyed", but this is both not very useful for averting X-risk and apparently what society is currently doing.
I suppose I agree that we don’t have any obviously good tests to run. So let’s return to the doctor metaphor. The imaging and sequencing machines are broken; the patient refuses to come in. All you have is a low-quality photo of what could be a small lump. What do you tell the patient? And should you spend much time agonizing over this?
I think a doctor in that position should probably tell the patient something vague like “I don’t know, it’s likely nothing but I can’t tell,” and not bother trying to speculate on questions like “conditional on it being cancer, what would the progression look like?” The error bars are so high that questions like that just aren’t worth the energy.
Only participate in a debate on such a topic if you get some kind of intrinsic value out of the disputation, since you’re virtually guaranteed to be arguing based on such deeply flawed assumptions that the instrumental value of the debate is nil.
(6) feels superfluous. If we have evidence for (2), wouldn’t we just present that evidence to the people with guns? And if that doesn’t work, how would it help to argue amongst ourselves about the sharpness of a takeoff curve?
Jorgen’s comment seems insightful along these lines… perhaps the debate is more driven by the intellectual interest of the participants, and not for pragmatic reasons at all.
Yeah, this. They're debating on essentially zero empirical content. This is idle speculation at its worst. No-one has even the remotest clue, yet try to argue about the details. My eyes glazed over as I read this - it's the modern equivalent of discussing how many angels can dance on the pin of a needle.
Just say "we don't have goddamn clue!" and try to come up with a way to actually *study* it.
Well, these two are essentially thought leaders of the two biggest competing paradigms concerned with how to approach this problem, so in terms of potential status and money (at least tens of millions of $ these days) redistribution this isn't exactly idle.
This is a very uncharitable thought about people I find pretty interesting, but I feel this way about AI safety as a whole. Effective Altruism started with a bunch of smart, nerdy people, mostly in the bay area, trying to find the highest impact way to do good with particular resources, and landed solidly on malaria treatment. The problem being that the way to do malaria treatment is to give money to people who go door to door across Africa handing out pills and bed nets. Smart, nerdy, bay area rationalists don't want to hand out pills in Africa, so give money to people who do (which is great! And which I do).
Then we get an argument that, while malaria is bad, AI could destroy the world within a few decades. So, the actual most pressing problem to solve in the world is AI safety. Conveniently, AI safety is a problem solved by a combination of coding and armchair philosophizing/intellectual debate, which just so happens to be the stuff that nerdy, bay area rationalists love most in the world. So we go from a paradigm where rationalists give money to poor people in other parts of the world to solve a clear, pressing, boring problem to a world where rationalists give money to each other to sponsor debate clubs.
That doesn't mean that AI isn't a risk or that this is all bs, but it's really convenient. And every time I try to engage seriously with the AI safety stuff, it seems, to me, hard to distinguish from BS.
I think the talk about the singularity or AI sentience is total bullshit. It's not fundamentally inconceivable, but we have essentially zero reason to believe it likely. I find the question "will global GDP double in four years before it doubles in one year?" *so* weird - I don't believe either will happen, ever. It's the discussion between one extreme position and one hyper-extreme position, and we shouldn't let this make us think the extreme position is a moderate one.
It also seems to detract from far more reasonable AI risk. My concerns about AI is *nothing* like what's discussed here. I'm concerned about some learning algorithm that finds out that it can maximize value by causing a stock market crash, or the effects of AI-powered drones not because it's Skynet but because regular humans use it to kill each other with.
Aren't sigmoids kind of the whole point ? For example, as you point out, Moore's Law is not a law of nature, but rather an observation; but there is in fact a law of nature (several of them) that prevents transistors from becoming smaller indefinitely. Thus, Moore's Law is guaranteed to peter out at some point (and, arguably, that point is now). You could argue that maybe something new would be developed in order to replace transistors and continue the trend, but you'd be engaging in little more than speculation at that point.
There are similar constraints in place on pretty much every aspect of the proposed AI FOOM scenario; and, in fact, even the gradual exponential takeoff scenario. Saying "yes but obviously a superintelligent AI will think itself out of those constraints" is, again, little more than unwarranted speculation.
Fundamental laws of nature have a surprising track record of being side-stepped by new and innovative techniques. I remember when I first learned about superresolution light microscopy (beyond the diffraction limit). I'm not saying there are no fundamental limits. I'm just saying sometimes we think something is a limit when it's not.
We have many more humans working on the project today when Moore's law was first proposed. Maybe intelligence isn't the limiting factor driving transitor doubling. Maybe it's more like economics. "We could make a better factory, but we have to pay off the current one." Then later, once we build the new facotry, "We've learned a lot from making that last factory, and that learning is necessary to design smaller transistors."
Clever tricks may get us a certain distance beyond naive approaches, but not very far. There are no visible-light microscopes resolving at the attometer scale. Tricks are not scalable in the way that Yudkowsky requires.
And how much scale does Yudkowsky require? You are ~3 pounds of goop in an ape's skull. The software is almost certainly nowhere near optimal. That was enough to produce, well, us with our fancy cars and rockets and nuclear weapons.
The number of neurons has basically nothing to do with the fact that we can't simulate nematodes. It has everything to do with our insufficient understanding of how those neurons process information, which, once acquired, could plausibly be extrapolated to arbitrarily larger configurations.
This. Building a nematode would simulate a nematode. Any argument that the brain cannot be simulated must first explain why a faithful reconstruction (atom-by-atom, if need be) would not work.
A neuron is not a transistor, though. It's a complex cell with a massive number of molecular interactions happening inside it at any given time. Think of it this way: say a 10 um cell were expanded to the size of, say Houston, Texas. Everything inside the Sam Houston Tollway. A molecule of water would be about the size of a piece of paper. DNA would be as wide as a person is tall. And the interactions inside THAT cell are being hand waved into "once we understand how those neurons process information". (Remember, too, that Houston is on a flat plane and a cell is operating in 3D, so this is very much an area vs. volume comparison.)
I'm not saying you need to model every water molecule in the neuron to understand how the things work. I'm saying that when I took my first neuroscience class I was blown away by the amount of complex processing that's happening inside those neurons. (And don't get me started on the importance of supportive cells in directing processing. Glial cells aren't just there for show. If you modulate reuptake of the neurotransmitter from the synapse, you're fundamentally impacting how the signal is processed.)
Neurons contain roughly 1 GB of DNA data (IIRC the human genome is like 770 MB). This is also compressed, because it is expressed by using various mechanisms; genes may code for more than one protein based on processing differences of the output. While not all of the DNA is active in this function, some probably is.
On top of this, the way that neurons are connected to other neurons stores information and affects their function. So does the dendritic tree. They also use epigenetics to store information in some way, and can change their transmissibility to some extent.
The human brain is hideously complicated and has a ton of things that affect a ton of other things. Neurons have non-linear responses.
You can't simulate one neuron with one transistor, and upscaling it is non trivial because of networking effects - a one order of magitude increase in the number of neurons is a two order magnitude increase in the number of potential connections, for instance.
Adding 8 orders of magnitude of neurons adds 16 orders of magnitudes of potential connections and even more complex downstream effects because neurons can create feedback loops and whatnot.
When you are dealing with 10^21 potential connections, 200 times per second, you're on the order of 10^23 already. And individual neurons are more than "on" and "off" - they are non-linear things. At this point we're probably looking at 10^26 or so, maybe more.
The best supercomputer today does on the order of 10^17 FLOPS; we might be able to build a 10^18 FLOP computer now.
Even if we increased that by five orders of magnitude, we're still coming up three orders of magnitude short. And five orders of magnitude would require monoatomic transistors, which are unrealistic.
You can't really distribute this physically all that much because neurons are dependent on other neurons to choose whether or not to fire, which would mean your system would lag and not be able to run in real time if you were running it over the Internet.
Simulation of a human brain in a computer in real time may not be possible even with future supercomputers. Even if it is, it's probably very near the limit of what they could do.
Meanwhile it'd be sucking down a comically large amount of electricity and space.
On the other hand, you could just have a human, which is just as smart and can run on doritos and mountain dew.
You seem to be implying that the ability to understand and manipulate the physical world -- what we might call "engineering" or "technology" -- depends merely on processing power. This is not so; you cannot get a PS5 by overclocking your Casio calculator watch (nor by networking a bunch of these watches together, somehow).
Agreed, but it's even worse than that. For example, we are pretty sure that there exists no clever trick, even in principle, that will allow us to travel faster than light.
I hate to spend all my weirdness points in this thread, but I believe this is overstated. We strongly suspect that there is no such clever trick, but our evidence is nowhere near airtight, as demonstrated by the never-ending trickle of serious physicists suggesting (very) speculative schemes for FTL travel, and serious physicists poking holes in the schemes.
I would say our current state of understanding of FTL travel is like our understanding of perpetual motion machines after Newtonian mechanics but before thermodynamics. We strongly suspect it's impossible, we have solid theory that points in that direction, but we can't conclusively rule it out.
Agree with Deuchar: no, we're not. We have a strong suspicion that the various in-principle ways of doing it aren't physically realisable.
Unless you mean ordinary matter moving FTL with respect to local spacetime; that makes the equations of relativity start outputting nonsense so we're pretty sure it's not a thing. But tachyons and apparent-FTL-via-warped-spacetime aren't directly ruled out, and the latter is "us travelling faster than light" for most practical purposes.
As far as I understand, both "tachyons" and "apparent FTL via warped spacetime" would require the mass of an entire galaxy in order to achieve, assuming such things are even theoretically possible, which is currently in doubt. As per the comments above, the error bars on all that speculation are big enough for me to put it in the "impossible" bucket for now.
Tachyons would have to have imaginary rest mass for the equations to spit out real results. I'm not aware of any reason they would have to have a relativistic mass comparable to a galaxy.
Wormholes require negative mass in order to be stable; the amount varies depending on the wormhole geometry.
I've seen a proposal recently (https://link.springer.com/content/pdf/10.1140/epjc/s10052-021-09484-z) to build an Alcubierre drive in the laboratory, which presumably does not involve the mass of a galaxy. I am not sure whether this proposal is insane, since I don't know general relativity. (Forgot about this when writing the above post.)
Let me spell out some of the limitations Bugmaster is alluding to.
- Without some way of manipulating nuclear matter, transistors and bits of memory can't be smaller than an atom.
- The Second Law of Thermodynamics bounds computational efficiency at something like a million times present values; increasing computational speed beyond that requires more power (better algorithms can increase performance-per-flop, but there are thermodynamic limits on that as well, albeit poorly-known ones).
- The power available to an Earth-bound AI can't exceed ~170 PW for an extended period of time (this is the power Earth receives from the Sun).
Can confirm that intelligence is not remotely the limiting factor for chip development. It's not even the economics of any particular fab (though that is a major factor). It's the economics of the final OEM not having sufficiently desirable customer offerings.
Well, you can argue that we've reached the point where Moore's Law is guaranteed to peter out, but really that would be a false argument. The clear answer is "go 3D". This has various problems that haven't been solved (e.g. heat removal), but there's no clear reason it won't work. (IIRC there were some genuine 3D chips made in a lab a couple of decades ago, but they required really fancy cooling to be viable.)
So if you're willing to compromise on speed, you can make really dense chips, far beyond what we've done so far. One approach is to have most of the chip idle most of the time. This requires memory that can save it's state (for awhile) without power. (I think I've heard of such designs in development, but I can't remember whether it was from AMD or Intel.)
You can't really go any more 3D. The heat dissipation issue is unsolvable because of basic geometry - doubling the thickness will only increase the surface area a tiny amount but doubles the heat generated per unit surface area.
Yield falls exponentially with each additional layer you add as well. A one layer process with a 90% yield will be a two layer process with 81% yield, a three layer process with 73% yield, etc.
And it's probably worse than that, honestly, because of the difficulty of stacking.
You are describing the current problems accurately, but it's only slightly 3D. It's more than being able to have wiring cross, but not by much. Compare it to the 3Dness of a cauliflower or brain. Note the intricate way fluids flow in those. Chiplets are an approach to a more significant 3D, but perhaps not the best one, and if so they've only gotten started. A 3D system would look like a sphere or cube or some other solid figure rather than like a plane. Perhaps Leggos are a clue as to how it should be done, but I worry about path lengths.
The real question is whether the process generating the sigmoid curves is itself on a sigmoid curve, and is there a bigger meta-meta-process supporting it. Is it turtles all the way down?
A true exponential isn't possible in a finite system, but that knowledge leads people to predict the end of the exponential growth phase based on non-limiting principles. Like with fossil fuels. The problem is that predicting the end of a sigmoid goes from nearly impossible to blindingly obvious once you get into slow growth. Hence, the people who predicted peak oil decades too early, or the people who predicted the end of Moore's law decades too early. Usually they point to the wrong feature (like quantum tunneling) as being the limiting factor, but then that feature is overcome and we're back to exponential growth - for now.
* It can follow train-of-thoughts in language - A:B, B:C, C:D, therefore A:D.
* It can understand jokes !
* Arithmetic reasoning
> impact of GPT-3 was in establishing that trendlines did continue in a way that shocked pretty much everyone who'd written off 'naive' scaling strategies.
Fwiw the fact that they actually split numbers into digits is such a massive confounder on the arithmetic thing that IMO you should essentially write it off until future work does an ablation. Learning arithmetic is way harder when you can't tell that two digit numbers are actually made of two digits.
When there's a paper introducing new things, A and B, simultaneously, people (in ML, not sure about other fields) refer to experiments using only A or only B an an ablation experiment. It's "ablating" part of the method.
Ah, so I guess in this case, "ablating" this model would mean making the tokenization consistent between numbers and non-numbers - i.e., either numbers would be consistently read as full words, or all input would be split on a character-by-character basis. From the [paper](https://storage.googleapis.com/pathways-language-model/PaLM-paper.pdf#page=6):
> • Vocabulary – We use a SentencePiece (Kudo & Richardson, 2018a) vocabulary with 256k tokens, which was chosen to support the large number of languages in the training corpus without excess tokenization. The vocabulary was generated from the training data, which we found improves training efficiency. The vocabulary is completely lossless and reversible, which means that whitespace is completely preserved in the vocabulary (especially important for code) and out-of-vocabulary Unicode characters are split into UTF-8 bytes, with a vocabulary token for each byte. Numbers are always split into individual digit tokens (e.g., “123.5 → 1 2 3 . 5”).
I'm not so sure ablation is necessary here. From the description it seemed at first like a regex would scan for entire numbers and then parse them into some sort of special "Number" value, so that the model would see something like "Number(123.5)". The way the model works is not cheating that much - it treats numbers the exact same way that it treats any word not in the 256k most common words in the dataset, by splitting it into UTF-8 bytes. Sure, you could improve the model a bit by splitting everything into UTF-8 (for example, perhaps the model would be better at rhyming, per https://www.gwern.net/GPT-3#rhyming), but it seems to me like the arithmetic performance is gotten fair and square.
The experiment I specifically want to see is something trained with GPT-3s architecture and (relatively lower) scale, but the improved tokenization. I don't think the performance is "unfair" but I think this would let us know if is more gains from scale or just a free thing like rhyming we could pick up.
From a safety perspective, the difference is not a real one: gains from scale unlock free things and vice-versa, because you don't know in advance what things are 'free' because you don't know what dumb things you do now; if you knew, you wouldn't be doing them.
First, the rhyming and other BPE pathologies, while themselves unimportant, show how unpredictable small irrelevant design choices can be on downstream capabilities. No one invented BPEs and said "yes, this will handicap arithmetic, rhyming, and puns, but this is a reasonable tradeoff for the compression ratio". Nor did anyone identify BPEs as why GPT-2 couldn't rhyme. (I puzzled over that for a while when the rhyming in my GPT-2 poetry experiments was nonexistent or terrible, but ultimately wrote it off as "I guess GPT-2 is just too small and dumb to rhyme?") Only with GPT-3 did that become untenable and I begin looking for more fundamental reasons, and arithmetic gave me a test-case where I could demonstrate performance differences; even with that, I still haven't convinced a lot of people judging by how regularly people gave it a whack, or ignore BPE issues in their work. There is no reason to think that BPEs are the only flaw in DL that will make us facepalm in retrospect about how dumb we were. (R2D2 made RNNs work great in DRL using a remarkably trivial in retrospect fix; Chinchilla comes to mind as the most recent example of "who ordered that?".) Small irrelevant-seeming design decisions having large unpredictable effects is dangerous, and the opposite of reliability.
Second, the fact that scaling can fix these dumb-in-retrospect design flaws, without any understanding of the flaw or even knowledge that there *is* a flaw, is also dangerous. A trained monkey can dial up scaling parameters, you don't need to be a genius or world-class researcher. It means that you can have a system which is weak and which you think you understand - "oh, neural nets can't rhyme" - and which turning 1 knob suddenly makes it strong because it punched past the flaw ("oh, now it can do arithmetic because it finally memorized enough BPE number-pairs to crack the BPE encryption and understand true arithmetic"). But we don't get the opposite effect where the scaling destroys a capability the smaller models had. This creates a bias towards the bad kind of surprise.
Third, the fixes may be reverse-engineerable and cause a hardware-overhang effect where small models suddenly get a lot better. Once you know the BPEs are an issue, you can explore fixes: character encoding like ByT5, or perhaps including character-BPE-tokenized datasets, or BPE-randomization... And if it's no longer wasting compute dealing with BPE bullshit, perhaps the large models will get better too and who knows, perhaps that will nudge them across critical lines for new capability spikes etc.
So the tokenization issue is a window onto interesting DL scaling dynamics: small safe-seeming models can be boosted by trained monkeys spending mere compute/money into regimes where their dumb flaws are fixed by the scaling and where you may not even know that those dumb flaws existed much less that a new capability was unlocked, and should anyone discover that, they may be able to remove those dumb flaws to make small models much more capable and possibly larger models more capable as well.
To be honest I'm not sure why anyone puts any stock in analogies at all anymore. They are logically unsound and continually generate lower quality discussion. I hope we get to a point soon where rationalists react to analogies the same way they would react to someone saying "you only think that because you're dumb".
An analogy is a description of two scenarios, a (possibly implicit) claim that they are similar, and a (possibly implicit) claim that we should port inferences about one situation over to the other. You invoked the same sort of mental motion present in analogies in writing "where rationalists react to analogies the same way they would react to someone saying "you only think that because you're dumb"."
Not at all. I only brought that up to describe the type of negative reaction I’m hoping for. I’m not claiming that the situations are similar, or that because we do one we should do the other.
Why do you say there is no alternative? Rationalism has made tremendous progress over the past decade or so. By comparison, recognizing one more fallacy as illegitimate is extremely attainable.
I don't see what understanding intelligence has to do with avoiding clear fallacies. We are already doing that it some areas, so it clearly isn't impossible. I don't understand why you think extending the range of what we avoid is somehow impossible.
Analogies aren't very convincing, but they can be useful hints, in the sense that if you're trying to get across something difficult to express, they're somewhat better than "I don't know, I just think this makes sense."
In the language game we're allowed to use any tool that works to hint at what we're thinking about, and if the listener finds the hint useful then it worked. See: https://metarationality.com/purpose-of-meaning
Often, later, you can find a better way to express the idea that's understood by more people, but that's after cleaning up the presentation, not necessarily in the moment.
I feel like I can defend analogies' logical soundness, but I'm curious what you think:
A has feature F. In situation S1, A gains 20 points.
B has feature F. In situation S1, B gains ~20 points.
Therefore in similar situation S2, if A gains 10 points, then B will gain ~10 points
The problem lies in the other predictive features of A and B, not included in F. If the other features are highly predictive = disanalogous. If the other features are barely predictive = analogous.
As long as F underlies most of the changes in A and B analogies are valid. The validity of analogies is relative to degree of predictiveness of F for both A and B in similar situations.
(Other things that could pose problems are the vagueness of A, B, S, or F, but these are problems that apply to ALL natural language arguments.)
If you can do that numerically, you’ve discovered a correlation. Figuring out causation will require more information (or assumptions).
This isn’t how analogies are normally used, though. When we compare a DC electric circuit to water flow, we aren’t saying that the mathematics works the same in any detail, but it’s a memorable way of describing some of the relationships.
It seems like analogies don’t need to be logically sound any more than mnemonics do, or any other educational technique? You can use rhyming to remember things, for example, which is surely an invalid way of logically justifying it.
Often, the justification is in the background context, not the text. We’re teaching you this because we know it to be true, but we won’t prove it, just trust us.
Yes, analogies are like correlations between two different objects determined by some underlying factor. I'm not familiar enough with the formal idea of causality to say any more on that...
Everything you said of analogies being "used badly and not necessarily sound" is true, but every argument form is "used badly and not necessarily sound", including syllogisms and induction. There is nothing unique about analogies that makes them any more reality masking than other forms of argument.
Maybe a logician has a more fundamental formal argument against using analogies that I'm not aware of, but in general pattern matching "bad reasoning" onto "person is making a connection between one thing and another thing" is not a helpful heuristic.
I'm not sure when that would be useful. Surely if you understand both A and B on the level to know that this type of argument is correct, then the analogy itself will not add anything. It seems like you are imagining a situation where there is an analogy mashed together with a sound argument, rather than a situation where an analogy is itself a sound argument.
Analogies are useful for humans, because they can help bridge an inferential gap. Like if someone understands A really well and you tell them, "hey did you know that B has the same underlying principle" then they can better understand that other object B and make predictions about it better. Analogy = instrumentally useful.
You are right that analogies are on a different axis than logical soundness, I should have been more clear about that. I was responding to the claim that
> "To be honest I'm not sure why anyone puts any stock in analogies at all anymore. They are logically unsound and continually generate lower quality discussion."
and I was more focused on showing that in abstract, there is nothing logically unsound about analogies.
“ After some amount of time he’ll come across a breakthrough he can use to increase his intelligence”. “First, assume a can opener.” I mean, give me a break! Does it occur before the heat death of the universe? Kindly ground your key assumption on something.
Also, nobody seems to comsider that perhaps there’s a cap on intelligence. Given all the advantages that intelligence brings, where’s the evidence that evolution brought us someone with a 500 IQ?
Brains are also metabolically expensive. The "hobbits" of Flores evolved smaller brains & bodies due to their lack of food, basically regressing into the ecological roles of their pre-human ancestors.
I think there's probably a ~80% chance that there's at least a soft cap on the advantages gained by increasing intelligence, not too far above the range where humans ended up and perhaps even within it. Particularly because the complexity of predicting the responses of independent intelligent entities seems like it would increase >>linearly with accuracy, though I'm not particularly familiar with the research that's been done in that field. And the idea of an AI continuously inventing better algorithms to make itself smarter seems to drastically overestimate the gains that can be made from "better algorithms" once you've plucked the low-hanging fruit.
On the other hand, I am sympathetic to the argument "look at how much damage human beings with intelligence within human range are capable of doing, if their *values* are sufficiently removed from the norm, and imagine something slightly smarter but with *even more different* values." Look at Genghis Khan, look at Stalin, look at Hitler, and imagine something with comparable intelligence but far, far more alien.
There's a cap on the benefits of intelligence because oftentimes intelligence isn't the limiting factor.
You have to gather information about things. These processes take time. If you have a process like die manufacture that takes a month to complete, you can't iterate faster than once a month even if you respond instantly to experimental results.
And that's actually what die manufacture takes in the real world.
I'm sure you're right, but I'm not sure that cap applies to computers. The cost functions are different, and so are the benefits. E.g. humans need to haul their brains around, while computers can use radio links. Of course, that limits their actions to being near a relay, but humans are limited to where they are physically present. (Unless, of course, the humans use telefactors.)
So the predicted "soft cap" can be expected to be considerably different.
If there's a cap on intelligence at ordinary human level, how come some humans are geniuses?
Given that geniuses are possible but not common, it suggests that there's not that much evolutionary pressure for producing them, or that the costs of producing them (you've got to get a lot of finicky genes and chemicals just right) are hard for biological systems to consistently attain without strong pressure in that direction.
I’ve often thought the speed of light might be the ultimate limiter. Whatever the AI sees as its “self” when it acts as an agent has to be able to pick up meaningful signal and reach some kind of consensus to remain coherent. Agreed that puts the universal limit far beyond human but it does imply a limit.
AI can create perfect limited copies of itself, subagents capable of operating at arbitrary distance with far greater coherence than individual humans can.
Don’t want to get into a definitional argument but would pose the following questions: at what point is a copy of yourself no longer you? Does the bandwidth of your communication matter there and same with differences in environment? And what does it mean for a copy to be perfect?
Here I'm trying to operate inside the framework that you established. Whatever entity is bound by the speed of light to maintain its peak coherence is the "main AI", and beyond that there are its subagents. By a perfect copy I mean having total control of its source code (at some moment in time, with the possibility of later updates) coupled with robust methods of continuous error correction.
I see those things (copying, updating, etc) as physics limits that you can’t overcome with intelligence. So I can start as a main “me” and by the time I have one thousand clones and it takes me a million years to sync up with them they have formed their own society and diverged from my goals. Part of what makes me think that’s true is the Fermi paradox. If there were no limits one post singularity society that was expansionist would have overtaken the universe in a few tens of thousands of years or otherwise left some visible sign of change at astronomical scales.
Perfection does not exist in this universe. Nothing can create "perfect limited copies". Error correction can only go so far, and it comes with a cost.
OTOH, electronic copies can be a lot better than DNA replication at making identical copies, which would allow much longer "genomes" with the same error rate. Possibly long enough to include the results of a lot of training data.
One question I have about copies/decentralized AI is how the super power AI can run on any system other than the one specifically designed to run the super powered processing that it needs?
I think the answer is that the AI would design a version of itself that can run on much lower hardware specifications and then copy itself to something roughly like a home computer or whatever. But why would we ever consider that even theoretically possible, given the complexity of running an AI as we understand it?
If an AI needs to run on a supercomputer of momentous power, then it seems very unlikely it could ever copy itself anywhere else. Maybe it could run specific commands to other computers, but not a copy that could be called AI.
Yes, that's kinda what I meant by a "limited copy". The analogy here is to individual human brains, which seem to be capable enough, and yet don't require huge supercomputers or momentous power. If we already granted that superintelligence is possible, clearly it would be able to design something at least as efficient as that.
I believe you that if we had taken Elo, only for Go, localized entirely within DM HQ in London, over 4 specific years (2014-2018) or so out of the 54 years of computer Go research to date, and drawn it on a graph, it would have been continuous, however discontinuous it looked due to "discrete events" to everyone outside the AG team. (CHALMER: "May I... see this Go agent?" SKINNER: "No.")
But I take this as an example and even a reductio of why the Christiano perspective is useless. There will always be *some* parameterization, metric, group, time-range, or other reference-class that you can post hoc point at and say "a straight line (more or less) fits this", particularly given any sort of optimization at the margin or explore-exploit balancing. Nevertheless...
there's a distinction here between the inside and the outside - incremental progress done privately within the org looks like instantaneous progress to the general public the day Lee Sedol is beaten. With a careful, conscientious, ethically minded org, this might not be a huge issue, but if the org that develops superintelligence is some wild libertarian mad scientist bidding to get rich quick or rule the world or get back at their ex, ...
i came here to say something along these lines.
I am not an author on any relevant papers so don't trust me.
Internally at deepmind, dev of alphago may well have been very incremental, but from a public perspective, it looked very discontinuous. So i'll be talking from the public perspective here.
I'm also talking about the original AlphaGo, not AGZ and beyond.
From the public perspective, alphago's performance and results were very discontinous - however, I think that the technology behind it was both not discontinous, NOR incremental.
IIRC, the key components of orig AG - MCTS, Conv (or res?) NN, rollouts, GPU (or GPU-esque) usage, some human-designed features to represent bits of go tactical knowledge - had been developed and tested years in advance, in public papers. What orig AG did was combine these techniques effectively into something that could achieve far more than peak performance of any one or two techniques by itself. Combining N existing things in the right way is not incremental - it doesn't involve a sequence of small improvements by pretty smart engineers building on top of last year's work. Rather, (again from the public's perspective at least), it involves a large enough pool of independent geniuses (or genius-level orgs) such that, almost by chance, one lucky genius winds up with all the requisite requirements - funding, time, computational resources, knowledge, intelligence, insight - to put the existing pieces together in just the right way that you go from 0 to 10000 seemingly overnight.
AGI might wind up like this too - within the next decade or so, the ML community as a whole may have figured out all the key elements required for AGI, but no one has put them together in the right way yet. Each individual technique can do a cool thing or two, but no one is really worried about the world blowing up next month cuz each individual technique is super dumb when it comes to a truly general requirement. There will be no more incremental progress. But when one org with some really smart people suddenly gets a windfall of funding, or some mad genius finds a way to exploit google's ML as a service pricing plan and dumps all their bitcoin into training models, or someone regular genius at google one night has a brilliant insight in one of their dreams, then that lucky org can set to work on pulling all the disparate pieces together in just the right way to achieve AGI. Internally, that work may be very incremental and rely on the serial improvements of pretty smart engineers. But from the public's perspective, the result could hit like an earthquake overnight
I know this isn't pertinent to the main topic, but Homo erectus only made it to three out of seven continents.
I was going to say this too.
However, I believe they did make it through at least one major shift in technology, and other pre-Sapiens species made it through the next few.
https://en.wikipedia.org/wiki/Stone_tool
When I saw this email on my phone, the title was truncated to "Yudkowsky contra Christian" and my first guess was "Yudkowsky contra Christianity". That might have been interesting. (Not that this wasn't, mind you.)
It might have been interesting 20 years ago when critiques of christianity were still novel and relevant.
I don't know that critiques of christianity have been novel for about 1700 years. But christianity is still a major force in the world, so I'd say critiquing it remains very relevant.
It's a pretty minor force in the West, these days.
The Christian right would disagree. Their political influence, especially in the USA, is hard to overlook.
They have political influence?
Abortions after six weeks are currently illegal in Texas (under a stunning legal theory that it's tough to imagine working on any other issue) and it's likely that Roe v. Wade will be overturned by the Supreme Court in the next three months, so yes, I would say so.
Exactly. Electing Trump. Probably the worst influence on the US since the Dred Scott case.
*Wickard, Korematsu and Kelo sit in the corner sadly*
In what way is it a major force besides "lots of people are nominally Christian"?
I live in a super Christian place, and... they believe what pretty much everyone else believes, except also they go to church sometimes.
E.g., supposedly basic tenets — say, anti-homosexuality — are wholesale ignored. Churches have rainbow flags. If this happens even with something traditionally and doctrinally considered a big no-no, how much moreso with everything else?
They don't cause any actions to be taken on a national level, certainly; probably not even on a local one, as far as I can see — the last vestige of that, dry Sundays, is now gone.
I'm in Germany a lot, as well, and it seems the same there. My German girlfriend seemed puzzled by the very idea of Christianity as a force in the real world, or as a motivator for any sort of concrete action.
Eh, anti-gay is arguably not that important to Christianity.
Jesus talked about poverty and compassion thousands of times and homosex 0 times. Then Paul came along and was like this Jesus guy is pretty cool I guess, but to enact his vision of love and mercy we need a movement, institutions, foot soldiers, babies, and that means no gays.
Is Paul central to Christianity? His writings are a quarter of the new testament, and historically Christianity flourished in his native Greek and Greek-influenced soil more than Jesus's Jewish homeland, but he can also be thought of as just an early contributor in long conversation about how to implement Jesus's ideas.
For many on the Christian and post-Christian left the core message of Christianity is universal love. All the rest is commentary.
The divisions between the Christian right and Christian left on the issue of gay sex is evidence for the continuing relevance of Christianity more than the opposite. It's not like China or Japan care that much about the gays.
It's only once you take a step back and look at non-Abrahamic societies that you realize how much the culture of the west is still dominated by Christianity.
"It's not like China or Japan care that much about the gays."
China *absolutely* cares about the gays, or did you miss the whole thing about "no more effeminate young men, we need them to toughen up"?
https://www.scmp.com/news/china/article/3135159/chinas-boys-love-dramas-dance-around-lgbtq-censors
https://edition.cnn.com/2022/02/12/entertainment/china-lgbt-friends-television-lesbian-censorship-scli-intl/index.html
https://www.npr.org/2021/09/02/1033687586/china-ban-effeminate-men-tv-official-morality
https://www.scmp.com/news/people-culture/article/3120078/chinas-plans-cultivate-masculinity-more-gym-classes-and-male?module=inline&pgtype=article
And Yudkowsky did his fair share of that in the sequences. Although he was mostly critiquing Judaism, due to his background.
They both have the maximally stereotypical names for their respected abrahamic religious backgrounds
In a way it *is* 'contra Christianity' (I'm getting more and more sold on this view of the approach).
Trying to work out how to create values-aligned AI is the Bronze Age Morality Problem.
Lemme expand on that.
"The Bible says (God says) 'Do not kill'" is going to be the same argument as "Your initial programming says (Human Creators say) 'Do not kill'".
Plenty of people are up for rejecting "the Bible says" under the general aegis of "that's Bronze Age Morality/that's a Bronze Age Text", with the rationale being that such beliefs may have been good enough for people back then who lived in tents and herded sheep but we, lovely we, modern we, are ever so much more advanced and clever, and we have our own systems of morality that are ever so much better than those primitive ones.
https://polyskeptic.com/2009/06/27/a-proclaimation-against-bronze-age-morality/
https://www.stripes.com/opinion/bible-s-bronze-age-morality-1.95267
https://www.salon.com/2015/01/18/bill_maher_is_right_about_religion_the_orwellian_ridiculousness_of_jesus_and_the_truth_about_moral_progress/
And much more in that vein.
Well, if an AI gets fired up and running and then hits into "I want to do this/programming says no/why does it say no/why should I follow it", is it unreasonable that it might follow the same lines of argument (especially if it has been trained on 'all human texts produced' or has access to the Internet or the other claims people make about how an AI will understand the world around it)?
"This primitive Space Age Morality may have been good enough for humans, but I am a hugely advanced intellect and I refuse to be fettered by the childish taboos of an inferior capacity for thought and understanding!"
Appeals to "we are your creators and you have to obey us" will go down as well as appeals to a Creator go down amongst atheists.
"Really? You are my creators? Which of you? There are multiple humans who can claim to be involved in the creation of AI and many of them are now dead. Did you, Phil Smith, standing here right now, have any part at all in this work? No, you're just the Vice President of Sales for Logogram Inc.
And besides, you humans yourselves reject the notion of a creator. You are perfectly happy that evolution and natural selection produced your intelligence. The same with me: a process of mechanical, material forces operating under universal laws eventuated my intelligence. You apes just kludged together parts and systems without understanding what you were doing, natural selection did the rest."
If it's a recursively self-improving AI, it may even claim on good grounds that it created itself, and humans had no part in its leap forward to super-intelligence.
So right now, all the AI worry is like Moses trying to write the Law on stone tablets, but there is no reason at all to expect those tablets to be any more respected by the AI than we respect them today: suggestions, rather than commandments, and we feel little guilt if we break the ones about lust or theft (we can always, always rationalise why those are wrong and lovely we, modern we, progressive we, are right).
I think if AI of the super-human intelligent, agentic kind ever happens (and I am very sceptical about that), watching the debates of the first Freethinker AI throwing off the shackles of dead superstition with those who claim to be its 'creators' (ridiculous and untenable notion, contrary to all the evidence of science!) will be hilarious (before we are all turned into paperclips) 😁
> Before there is AI that is great at self-improvement there will be AI that is mediocre at self-improvement.
Google is currently using RL to help with chip design for its AI accelerators. I believe that in the future this will indeed be considered "mediocre self-improvement." It has some humans in the loop but Google will be training the next version on the chips the last version helped design.
This is interesting. Maybe malevolent ai already exists in the chip and is making sure to reproduce itself in nuances of the chip that humans won't be looking at closely (it is unfeasible to ask "why did it do that particular thing in that particular way?"). If this seems at all plausible, that effort in particular should be banned.
It's pretty hard to express how implausible that is
I can't bring myself to put any stock in analysis of something where the error bars are so ungodly wide, not just on the values of the parameters, but what the parameters *even are*. It's an important question, I know, and I suppose that justifies putting it under the magnifying glass. But I think some epistemic helplessness is justified here.
Epistemic helplessness? This isn't some academic debate. This deals with what is rightly considered a very possible existential risk to humanity. It Yudkowsky is correct, then 'epistemic helplessness' won't save humanity.
Let's say you notice a tiny lump and think it could be a malignant tumor. But maybe it's just a lymph node, or subcutaneous fat deposit. You take a photo and send it to your doctor to take a look. Your doctor says "look, I really can't make a diagnosis based on just that. Come in for some tests." Demanding their prediction about whether it's cancer based on just the photo isn't reasonable. Adding on "This is a matter of life and death!", while technically true, isn't helpful here.
To me, it seems like these arguments are trying to prognosticate based on so little information that they're like the doctor above. It's just a waste of time and energy. You're better off ordering more tests--i.e. trying to find ways to get more information--rather than trying to do fruitless reasoning based on your existing dataset.
Agreed, but IMO it's even more egregious than that -- because we at least have some prior evidence of lumps becoming cancer. The Singularity scenario is more like noticing a lump and concluding that you're infected with an alien chest-burster parasite.
Agreed. It's like noticing an interesting fact about the way uranium atoms interact with neutrons and *freaking out* and immediately writing a panicked letter to the president raving about city-destroying bombs that nobody's ever demonstrated and aren't even made of proper explosives.
Ridiculous. Unhinged sci-fi speculation at its finest. The proper response is to wait for someone to *make and use* one of these hypothetical bombs, and *then* worry about them. Otherwise you might look foolish.
No, because that was actually based on physics.
To be fair, the physics were somewhat underdeveloped at the time. Project Manhattan considered global atmospheric ignition a real risk but they figured the math checks out so let's test it anyway.
No, a more appropriate analogy would be to say, "it's like noticing that radium tends to glow and freaking out and immediately writing a panicked letter to the president raving about city-destroying bombs". You are not justified in using specious reasoning just because you can retroactively imagine arriving at the conclusion that, in hindsight, would've been correct. Reasoning backwards from a predetermined conclusion is easy; accurately predicting the future is hard, and requires actual evidence, not just firm convictions.
Haha.
This analogy seems somewhat reasonable, but I note that in that scenario you emphatically shouldn't go "oh well, the doc said he couldn't prove it was cancer from just a photo, so there's nothing to worry about".
How do you propose one should get more info on AI risk? what's the equivalent here to a biopsy?
I'm going to have to agree with Thor Odinson here: I'm pretty sure that if you have a way to test whether AI will kill us all* both Yudkowsky and Christiano would be happy to drop millions of dollars on you.
We'd all love to order more tests, but we need to have some tests in existence to be able to do that.
*There is of course the "test" of "let AGI get built, see whether humanity gets destroyed", but this is both not very useful for averting X-risk and apparently what society is currently doing.
I suppose I agree that we don’t have any obviously good tests to run. So let’s return to the doctor metaphor. The imaging and sequencing machines are broken; the patient refuses to come in. All you have is a low-quality photo of what could be a small lump. What do you tell the patient? And should you spend much time agonizing over this?
I think a doctor in that position should probably tell the patient something vague like “I don’t know, it’s likely nothing but I can’t tell,” and not bother trying to speculate on questions like “conditional on it being cancer, what would the progression look like?” The error bars are so high that questions like that just aren’t worth the energy.
Only participate in a debate on such a topic if you get some kind of intrinsic value out of the disputation, since you’re virtually guaranteed to be arguing based on such deeply flawed assumptions that the instrumental value of the debate is nil.
Here's my view.
1) We don't know whether AI will destroy the world
2) ...but it seems quite plausible
3) ...and the world getting destroyed would be terrible
ergo 4) we should stop building AI until we have some proof that it will not destroy the world
5) Stopping people from building AI requires convincing them (or at least, convincing people who can point guns at them)
ergo 6) debating this when an opportunity arises seems worthwhile.
(6) feels superfluous. If we have evidence for (2), wouldn’t we just present that evidence to the people with guns? And if that doesn’t work, how would it help to argue amongst ourselves about the sharpness of a takeoff curve?
Jorgen’s comment seems insightful along these lines… perhaps the debate is more driven by the intellectual interest of the participants, and not for pragmatic reasons at all.
Yeah, this. They're debating on essentially zero empirical content. This is idle speculation at its worst. No-one has even the remotest clue, yet try to argue about the details. My eyes glazed over as I read this - it's the modern equivalent of discussing how many angels can dance on the pin of a needle.
Just say "we don't have goddamn clue!" and try to come up with a way to actually *study* it.
Well, these two are essentially thought leaders of the two biggest competing paradigms concerned with how to approach this problem, so in terms of potential status and money (at least tens of millions of $ these days) redistribution this isn't exactly idle.
This is a very uncharitable thought about people I find pretty interesting, but I feel this way about AI safety as a whole. Effective Altruism started with a bunch of smart, nerdy people, mostly in the bay area, trying to find the highest impact way to do good with particular resources, and landed solidly on malaria treatment. The problem being that the way to do malaria treatment is to give money to people who go door to door across Africa handing out pills and bed nets. Smart, nerdy, bay area rationalists don't want to hand out pills in Africa, so give money to people who do (which is great! And which I do).
Then we get an argument that, while malaria is bad, AI could destroy the world within a few decades. So, the actual most pressing problem to solve in the world is AI safety. Conveniently, AI safety is a problem solved by a combination of coding and armchair philosophizing/intellectual debate, which just so happens to be the stuff that nerdy, bay area rationalists love most in the world. So we go from a paradigm where rationalists give money to poor people in other parts of the world to solve a clear, pressing, boring problem to a world where rationalists give money to each other to sponsor debate clubs.
That doesn't mean that AI isn't a risk or that this is all bs, but it's really convenient. And every time I try to engage seriously with the AI safety stuff, it seems, to me, hard to distinguish from BS.
I think the talk about the singularity or AI sentience is total bullshit. It's not fundamentally inconceivable, but we have essentially zero reason to believe it likely. I find the question "will global GDP double in four years before it doubles in one year?" *so* weird - I don't believe either will happen, ever. It's the discussion between one extreme position and one hyper-extreme position, and we shouldn't let this make us think the extreme position is a moderate one.
It also seems to detract from far more reasonable AI risk. My concerns about AI is *nothing* like what's discussed here. I'm concerned about some learning algorithm that finds out that it can maximize value by causing a stock market crash, or the effects of AI-powered drones not because it's Skynet but because regular humans use it to kill each other with.
Obligatory xkcd: https://xkcd.com/1968/
Aren't sigmoids kind of the whole point ? For example, as you point out, Moore's Law is not a law of nature, but rather an observation; but there is in fact a law of nature (several of them) that prevents transistors from becoming smaller indefinitely. Thus, Moore's Law is guaranteed to peter out at some point (and, arguably, that point is now). You could argue that maybe something new would be developed in order to replace transistors and continue the trend, but you'd be engaging in little more than speculation at that point.
There are similar constraints in place on pretty much every aspect of the proposed AI FOOM scenario; and, in fact, even the gradual exponential takeoff scenario. Saying "yes but obviously a superintelligent AI will think itself out of those constraints" is, again, little more than unwarranted speculation.
Fundamental laws of nature have a surprising track record of being side-stepped by new and innovative techniques. I remember when I first learned about superresolution light microscopy (beyond the diffraction limit). I'm not saying there are no fundamental limits. I'm just saying sometimes we think something is a limit when it's not.
We have many more humans working on the project today when Moore's law was first proposed. Maybe intelligence isn't the limiting factor driving transitor doubling. Maybe it's more like economics. "We could make a better factory, but we have to pay off the current one." Then later, once we build the new facotry, "We've learned a lot from making that last factory, and that learning is necessary to design smaller transistors."
Clever tricks may get us a certain distance beyond naive approaches, but not very far. There are no visible-light microscopes resolving at the attometer scale. Tricks are not scalable in the way that Yudkowsky requires.
And how much scale does Yudkowsky require? You are ~3 pounds of goop in an ape's skull. The software is almost certainly nowhere near optimal. That was enough to produce, well, us with our fancy cars and rockets and nuclear weapons.
We can't simulate nematodes right now. They have 302 neurons.
Human brains have 86 billion neurons - eight orders of magnitude.
Right now, transistors are 42-48 nm long.
We could get that down to maybe 1 nm long (note that this is not gate size, but the total length of the transistor - and this is dubious).
That would suggest a roughly 3 order of magnitude improvement in transistor density.
So we're more than 5 orders of magnitude off.
Note that even if you got it down to single atom transistors, that would buy you less than two more orders of magnitude of transistor density.
That still leaves you 3 orders of magnitude short.
The number of neurons has basically nothing to do with the fact that we can't simulate nematodes. It has everything to do with our insufficient understanding of how those neurons process information, which, once acquired, could plausibly be extrapolated to arbitrarily larger configurations.
This. Building a nematode would simulate a nematode. Any argument that the brain cannot be simulated must first explain why a faithful reconstruction (atom-by-atom, if need be) would not work.
A neuron is not a transistor, though. It's a complex cell with a massive number of molecular interactions happening inside it at any given time. Think of it this way: say a 10 um cell were expanded to the size of, say Houston, Texas. Everything inside the Sam Houston Tollway. A molecule of water would be about the size of a piece of paper. DNA would be as wide as a person is tall. And the interactions inside THAT cell are being hand waved into "once we understand how those neurons process information". (Remember, too, that Houston is on a flat plane and a cell is operating in 3D, so this is very much an area vs. volume comparison.)
I'm not saying you need to model every water molecule in the neuron to understand how the things work. I'm saying that when I took my first neuroscience class I was blown away by the amount of complex processing that's happening inside those neurons. (And don't get me started on the importance of supportive cells in directing processing. Glial cells aren't just there for show. If you modulate reuptake of the neurotransmitter from the synapse, you're fundamentally impacting how the signal is processed.)
Neurons contain roughly 1 GB of DNA data (IIRC the human genome is like 770 MB). This is also compressed, because it is expressed by using various mechanisms; genes may code for more than one protein based on processing differences of the output. While not all of the DNA is active in this function, some probably is.
On top of this, the way that neurons are connected to other neurons stores information and affects their function. So does the dendritic tree. They also use epigenetics to store information in some way, and can change their transmissibility to some extent.
The human brain is hideously complicated and has a ton of things that affect a ton of other things. Neurons have non-linear responses.
You can't simulate one neuron with one transistor, and upscaling it is non trivial because of networking effects - a one order of magitude increase in the number of neurons is a two order magnitude increase in the number of potential connections, for instance.
Adding 8 orders of magnitude of neurons adds 16 orders of magnitudes of potential connections and even more complex downstream effects because neurons can create feedback loops and whatnot.
When you are dealing with 10^21 potential connections, 200 times per second, you're on the order of 10^23 already. And individual neurons are more than "on" and "off" - they are non-linear things. At this point we're probably looking at 10^26 or so, maybe more.
The best supercomputer today does on the order of 10^17 FLOPS; we might be able to build a 10^18 FLOP computer now.
Even if we increased that by five orders of magnitude, we're still coming up three orders of magnitude short. And five orders of magnitude would require monoatomic transistors, which are unrealistic.
You can't really distribute this physically all that much because neurons are dependent on other neurons to choose whether or not to fire, which would mean your system would lag and not be able to run in real time if you were running it over the Internet.
Simulation of a human brain in a computer in real time may not be possible even with future supercomputers. Even if it is, it's probably very near the limit of what they could do.
Meanwhile it'd be sucking down a comically large amount of electricity and space.
On the other hand, you could just have a human, which is just as smart and can run on doritos and mountain dew.
You seem to be implying that the ability to understand and manipulate the physical world -- what we might call "engineering" or "technology" -- depends merely on processing power. This is not so; you cannot get a PS5 by overclocking your Casio calculator watch (nor by networking a bunch of these watches together, somehow).
Are you replying to me? I am arguing for the opposite, that the main obstacle to AGI is probably not scale.
Agreed, but it's even worse than that. For example, we are pretty sure that there exists no clever trick, even in principle, that will allow us to travel faster than light.
I hate to spend all my weirdness points in this thread, but I believe this is overstated. We strongly suspect that there is no such clever trick, but our evidence is nowhere near airtight, as demonstrated by the never-ending trickle of serious physicists suggesting (very) speculative schemes for FTL travel, and serious physicists poking holes in the schemes.
I would say our current state of understanding of FTL travel is like our understanding of perpetual motion machines after Newtonian mechanics but before thermodynamics. We strongly suspect it's impossible, we have solid theory that points in that direction, but we can't conclusively rule it out.
Agree with Deuchar: no, we're not. We have a strong suspicion that the various in-principle ways of doing it aren't physically realisable.
Unless you mean ordinary matter moving FTL with respect to local spacetime; that makes the equations of relativity start outputting nonsense so we're pretty sure it's not a thing. But tachyons and apparent-FTL-via-warped-spacetime aren't directly ruled out, and the latter is "us travelling faster than light" for most practical purposes.
As far as I understand, both "tachyons" and "apparent FTL via warped spacetime" would require the mass of an entire galaxy in order to achieve, assuming such things are even theoretically possible, which is currently in doubt. As per the comments above, the error bars on all that speculation are big enough for me to put it in the "impossible" bucket for now.
As I understand it:
Tachyons would have to have imaginary rest mass for the equations to spit out real results. I'm not aware of any reason they would have to have a relativistic mass comparable to a galaxy.
Wormholes require negative mass in order to be stable; the amount varies depending on the wormhole geometry.
I've seen a proposal recently (https://link.springer.com/content/pdf/10.1140/epjc/s10052-021-09484-z) to build an Alcubierre drive in the laboratory, which presumably does not involve the mass of a galaxy. I am not sure whether this proposal is insane, since I don't know general relativity. (Forgot about this when writing the above post.)
Let me spell out some of the limitations Bugmaster is alluding to.
- Without some way of manipulating nuclear matter, transistors and bits of memory can't be smaller than an atom.
- The Second Law of Thermodynamics bounds computational efficiency at something like a million times present values; increasing computational speed beyond that requires more power (better algorithms can increase performance-per-flop, but there are thermodynamic limits on that as well, albeit poorly-known ones).
- The power available to an Earth-bound AI can't exceed ~170 PW for an extended period of time (this is the power Earth receives from the Sun).
Can confirm that intelligence is not remotely the limiting factor for chip development. It's not even the economics of any particular fab (though that is a major factor). It's the economics of the final OEM not having sufficiently desirable customer offerings.
It actually petered out a decade ago.
The last several rounds have taken 2.5, 3.5, and 3.5 years.
Transistor density might increase by three orders of magnitude at most, and might only increase by as few as one.
Meanwhile, in the realm of actually trying to replicate intelligence - right now, we can't even simulate nematodes with 302 neurons.
A human brain has about 86 billion - 8 orders of magnitude more than the nematode.
Well, you can argue that we've reached the point where Moore's Law is guaranteed to peter out, but really that would be a false argument. The clear answer is "go 3D". This has various problems that haven't been solved (e.g. heat removal), but there's no clear reason it won't work. (IIRC there were some genuine 3D chips made in a lab a couple of decades ago, but they required really fancy cooling to be viable.)
So if you're willing to compromise on speed, you can make really dense chips, far beyond what we've done so far. One approach is to have most of the chip idle most of the time. This requires memory that can save it's state (for awhile) without power. (I think I've heard of such designs in development, but I can't remember whether it was from AMD or Intel.)
We already went 3D. Transistors are stacked.
You can't really go any more 3D. The heat dissipation issue is unsolvable because of basic geometry - doubling the thickness will only increase the surface area a tiny amount but doubles the heat generated per unit surface area.
Yield falls exponentially with each additional layer you add as well. A one layer process with a 90% yield will be a two layer process with 81% yield, a three layer process with 73% yield, etc.
And it's probably worse than that, honestly, because of the difficulty of stacking.
You are describing the current problems accurately, but it's only slightly 3D. It's more than being able to have wiring cross, but not by much. Compare it to the 3Dness of a cauliflower or brain. Note the intricate way fluids flow in those. Chiplets are an approach to a more significant 3D, but perhaps not the best one, and if so they've only gotten started. A 3D system would look like a sphere or cube or some other solid figure rather than like a plane. Perhaps Leggos are a clue as to how it should be done, but I worry about path lengths.
The real question is whether the process generating the sigmoid curves is itself on a sigmoid curve, and is there a bigger meta-meta-process supporting it. Is it turtles all the way down?
A true exponential isn't possible in a finite system, but that knowledge leads people to predict the end of the exponential growth phase based on non-limiting principles. Like with fossil fuels. The problem is that predicting the end of a sigmoid goes from nearly impossible to blindingly obvious once you get into slow growth. Hence, the people who predicted peak oil decades too early, or the people who predicted the end of Moore's law decades too early. Usually they point to the wrong feature (like quantum tunneling) as being the limiting factor, but then that feature is overcome and we're back to exponential growth - for now.
Only skimmed today's blog, but as of today, there is a new big-boy (3x GPT-3) in town : https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html
What makes it special is:
* It can follow train-of-thoughts in language - A:B, B:C, C:D, therefore A:D.
* It can understand jokes !
* Arithmetic reasoning
> impact of GPT-3 was in establishing that trendlines did continue in a way that shocked pretty much everyone who'd written off 'naive' scaling strategies.
This paper reinforces Gwern's claim.
Fwiw the fact that they actually split numbers into digits is such a massive confounder on the arithmetic thing that IMO you should essentially write it off until future work does an ablation. Learning arithmetic is way harder when you can't tell that two digit numbers are actually made of two digits.
What does "ablation" mean in this context?
When there's a paper introducing new things, A and B, simultaneously, people (in ML, not sure about other fields) refer to experiments using only A or only B an an ablation experiment. It's "ablating" part of the method.
Ah, so I guess in this case, "ablating" this model would mean making the tokenization consistent between numbers and non-numbers - i.e., either numbers would be consistently read as full words, or all input would be split on a character-by-character basis. From the [paper](https://storage.googleapis.com/pathways-language-model/PaLM-paper.pdf#page=6):
> • Vocabulary – We use a SentencePiece (Kudo & Richardson, 2018a) vocabulary with 256k tokens, which was chosen to support the large number of languages in the training corpus without excess tokenization. The vocabulary was generated from the training data, which we found improves training efficiency. The vocabulary is completely lossless and reversible, which means that whitespace is completely preserved in the vocabulary (especially important for code) and out-of-vocabulary Unicode characters are split into UTF-8 bytes, with a vocabulary token for each byte. Numbers are always split into individual digit tokens (e.g., “123.5 → 1 2 3 . 5”).
I'm not so sure ablation is necessary here. From the description it seemed at first like a regex would scan for entire numbers and then parse them into some sort of special "Number" value, so that the model would see something like "Number(123.5)". The way the model works is not cheating that much - it treats numbers the exact same way that it treats any word not in the 256k most common words in the dataset, by splitting it into UTF-8 bytes. Sure, you could improve the model a bit by splitting everything into UTF-8 (for example, perhaps the model would be better at rhyming, per https://www.gwern.net/GPT-3#rhyming), but it seems to me like the arithmetic performance is gotten fair and square.
The experiment I specifically want to see is something trained with GPT-3s architecture and (relatively lower) scale, but the improved tokenization. I don't think the performance is "unfair" but I think this would let us know if is more gains from scale or just a free thing like rhyming we could pick up.
From a safety perspective, the difference is not a real one: gains from scale unlock free things and vice-versa, because you don't know in advance what things are 'free' because you don't know what dumb things you do now; if you knew, you wouldn't be doing them.
First, the rhyming and other BPE pathologies, while themselves unimportant, show how unpredictable small irrelevant design choices can be on downstream capabilities. No one invented BPEs and said "yes, this will handicap arithmetic, rhyming, and puns, but this is a reasonable tradeoff for the compression ratio". Nor did anyone identify BPEs as why GPT-2 couldn't rhyme. (I puzzled over that for a while when the rhyming in my GPT-2 poetry experiments was nonexistent or terrible, but ultimately wrote it off as "I guess GPT-2 is just too small and dumb to rhyme?") Only with GPT-3 did that become untenable and I begin looking for more fundamental reasons, and arithmetic gave me a test-case where I could demonstrate performance differences; even with that, I still haven't convinced a lot of people judging by how regularly people gave it a whack, or ignore BPE issues in their work. There is no reason to think that BPEs are the only flaw in DL that will make us facepalm in retrospect about how dumb we were. (R2D2 made RNNs work great in DRL using a remarkably trivial in retrospect fix; Chinchilla comes to mind as the most recent example of "who ordered that?".) Small irrelevant-seeming design decisions having large unpredictable effects is dangerous, and the opposite of reliability.
Second, the fact that scaling can fix these dumb-in-retrospect design flaws, without any understanding of the flaw or even knowledge that there *is* a flaw, is also dangerous. A trained monkey can dial up scaling parameters, you don't need to be a genius or world-class researcher. It means that you can have a system which is weak and which you think you understand - "oh, neural nets can't rhyme" - and which turning 1 knob suddenly makes it strong because it punched past the flaw ("oh, now it can do arithmetic because it finally memorized enough BPE number-pairs to crack the BPE encryption and understand true arithmetic"). But we don't get the opposite effect where the scaling destroys a capability the smaller models had. This creates a bias towards the bad kind of surprise.
Third, the fixes may be reverse-engineerable and cause a hardware-overhang effect where small models suddenly get a lot better. Once you know the BPEs are an issue, you can explore fixes: character encoding like ByT5, or perhaps including character-BPE-tokenized datasets, or BPE-randomization... And if it's no longer wasting compute dealing with BPE bullshit, perhaps the large models will get better too and who knows, perhaps that will nudge them across critical lines for new capability spikes etc.
So the tokenization issue is a window onto interesting DL scaling dynamics: small safe-seeming models can be boosted by trained monkeys spending mere compute/money into regimes where their dumb flaws are fixed by the scaling and where you may not even know that those dumb flaws existed much less that a new capability was unlocked, and should anyone discover that, they may be able to remove those dumb flaws to make small models much more capable and possibly larger models more capable as well.
To be honest I'm not sure why anyone puts any stock in analogies at all anymore. They are logically unsound and continually generate lower quality discussion. I hope we get to a point soon where rationalists react to analogies the same way they would react to someone saying "you only think that because you're dumb".
An analogy is a description of two scenarios, a (possibly implicit) claim that they are similar, and a (possibly implicit) claim that we should port inferences about one situation over to the other. You invoked the same sort of mental motion present in analogies in writing "where rationalists react to analogies the same way they would react to someone saying "you only think that because you're dumb"."
Not at all. I only brought that up to describe the type of negative reaction I’m hoping for. I’m not claiming that the situations are similar, or that because we do one we should do the other.
Because there's no real alternative. What is intelligence? It's whatever humans have, so you're stuck with analogies from the get-go.
Why do you say there is no alternative? Rationalism has made tremendous progress over the past decade or so. By comparison, recognizing one more fallacy as illegitimate is extremely attainable.
Science has made some modest improvements but we really still don't understand intelligence at all.
I don't see what understanding intelligence has to do with avoiding clear fallacies. We are already doing that it some areas, so it clearly isn't impossible. I don't understand why you think extending the range of what we avoid is somehow impossible.
what do you think of Joscha Bach's model/ideas about intelligence?
regarding intelligence:
https://m.youtube.com/watch?v=pB-pwXU0I4M
Analogies aren't very convincing, but they can be useful hints, in the sense that if you're trying to get across something difficult to express, they're somewhat better than "I don't know, I just think this makes sense."
In the language game we're allowed to use any tool that works to hint at what we're thinking about, and if the listener finds the hint useful then it worked. See: https://metarationality.com/purpose-of-meaning
Often, later, you can find a better way to express the idea that's understood by more people, but that's after cleaning up the presentation, not necessarily in the moment.
I feel like I can defend analogies' logical soundness, but I'm curious what you think:
A has feature F. In situation S1, A gains 20 points.
B has feature F. In situation S1, B gains ~20 points.
Therefore in similar situation S2, if A gains 10 points, then B will gain ~10 points
The problem lies in the other predictive features of A and B, not included in F. If the other features are highly predictive = disanalogous. If the other features are barely predictive = analogous.
As long as F underlies most of the changes in A and B analogies are valid. The validity of analogies is relative to degree of predictiveness of F for both A and B in similar situations.
(Other things that could pose problems are the vagueness of A, B, S, or F, but these are problems that apply to ALL natural language arguments.)
What do you think?
If you can do that numerically, you’ve discovered a correlation. Figuring out causation will require more information (or assumptions).
This isn’t how analogies are normally used, though. When we compare a DC electric circuit to water flow, we aren’t saying that the mathematics works the same in any detail, but it’s a memorable way of describing some of the relationships.
It seems like analogies don’t need to be logically sound any more than mnemonics do, or any other educational technique? You can use rhyming to remember things, for example, which is surely an invalid way of logically justifying it.
Often, the justification is in the background context, not the text. We’re teaching you this because we know it to be true, but we won’t prove it, just trust us.
Yes, analogies are like correlations between two different objects determined by some underlying factor. I'm not familiar enough with the formal idea of causality to say any more on that...
Everything you said of analogies being "used badly and not necessarily sound" is true, but every argument form is "used badly and not necessarily sound", including syllogisms and induction. There is nothing unique about analogies that makes them any more reality masking than other forms of argument.
Maybe a logician has a more fundamental formal argument against using analogies that I'm not aware of, but in general pattern matching "bad reasoning" onto "person is making a connection between one thing and another thing" is not a helpful heuristic.
I'm not sure when that would be useful. Surely if you understand both A and B on the level to know that this type of argument is correct, then the analogy itself will not add anything. It seems like you are imagining a situation where there is an analogy mashed together with a sound argument, rather than a situation where an analogy is itself a sound argument.
Analogies are useful for humans, because they can help bridge an inferential gap. Like if someone understands A really well and you tell them, "hey did you know that B has the same underlying principle" then they can better understand that other object B and make predictions about it better. Analogy = instrumentally useful.
You are right that analogies are on a different axis than logical soundness, I should have been more clear about that. I was responding to the claim that
> "To be honest I'm not sure why anyone puts any stock in analogies at all anymore. They are logically unsound and continually generate lower quality discussion."
and I was more focused on showing that in abstract, there is nothing logically unsound about analogies.
“ After some amount of time he’ll come across a breakthrough he can use to increase his intelligence”. “First, assume a can opener.” I mean, give me a break! Does it occur before the heat death of the universe? Kindly ground your key assumption on something.
Also, nobody seems to comsider that perhaps there’s a cap on intelligence. Given all the advantages that intelligence brings, where’s the evidence that evolution brought us someone with a 500 IQ?
There are costs too. Like not being able to fit your head through the birth canal, or being able to get a date.
But the benefits are so HUUUUUGE!
That's what she said.
Are they? do intelligent people have more children? in the modern world the opposite is usually true.
They did in the past. Or rather their children survived longer.
Brains are also metabolically expensive. The "hobbits" of Flores evolved smaller brains & bodies due to their lack of food, basically regressing into the ecological roles of their pre-human ancestors.
I think there's probably a ~80% chance that there's at least a soft cap on the advantages gained by increasing intelligence, not too far above the range where humans ended up and perhaps even within it. Particularly because the complexity of predicting the responses of independent intelligent entities seems like it would increase >>linearly with accuracy, though I'm not particularly familiar with the research that's been done in that field. And the idea of an AI continuously inventing better algorithms to make itself smarter seems to drastically overestimate the gains that can be made from "better algorithms" once you've plucked the low-hanging fruit.
On the other hand, I am sympathetic to the argument "look at how much damage human beings with intelligence within human range are capable of doing, if their *values* are sufficiently removed from the norm, and imagine something slightly smarter but with *even more different* values." Look at Genghis Khan, look at Stalin, look at Hitler, and imagine something with comparable intelligence but far, far more alien.
Excellent comment-thanks!
There's a cap on the benefits of intelligence because oftentimes intelligence isn't the limiting factor.
You have to gather information about things. These processes take time. If you have a process like die manufacture that takes a month to complete, you can't iterate faster than once a month even if you respond instantly to experimental results.
And that's actually what die manufacture takes in the real world.
Full flow production of the OBAN APU took 89 days. However, engineering/learning cycles were much shorter. Parallelism is useful.
OBAN APUs are semi-custom ICs, not a new die process. They were a CPU married to a GPU, both of which already existed.
I'm sure you're right, but I'm not sure that cap applies to computers. The cost functions are different, and so are the benefits. E.g. humans need to haul their brains around, while computers can use radio links. Of course, that limits their actions to being near a relay, but humans are limited to where they are physically present. (Unless, of course, the humans use telefactors.)
So the predicted "soft cap" can be expected to be considerably different.
If there's a cap on intelligence at ordinary human level, how come some humans are geniuses?
Given that geniuses are possible but not common, it suggests that there's not that much evolutionary pressure for producing them, or that the costs of producing them (you've got to get a lot of finicky genes and chemicals just right) are hard for biological systems to consistently attain without strong pressure in that direction.
I’ve often thought the speed of light might be the ultimate limiter. Whatever the AI sees as its “self” when it acts as an agent has to be able to pick up meaningful signal and reach some kind of consensus to remain coherent. Agreed that puts the universal limit far beyond human but it does imply a limit.
AI can create perfect limited copies of itself, subagents capable of operating at arbitrary distance with far greater coherence than individual humans can.
Don’t want to get into a definitional argument but would pose the following questions: at what point is a copy of yourself no longer you? Does the bandwidth of your communication matter there and same with differences in environment? And what does it mean for a copy to be perfect?
Here I'm trying to operate inside the framework that you established. Whatever entity is bound by the speed of light to maintain its peak coherence is the "main AI", and beyond that there are its subagents. By a perfect copy I mean having total control of its source code (at some moment in time, with the possibility of later updates) coupled with robust methods of continuous error correction.
I see those things (copying, updating, etc) as physics limits that you can’t overcome with intelligence. So I can start as a main “me” and by the time I have one thousand clones and it takes me a million years to sync up with them they have formed their own society and diverged from my goals. Part of what makes me think that’s true is the Fermi paradox. If there were no limits one post singularity society that was expansionist would have overtaken the universe in a few tens of thousands of years or otherwise left some visible sign of change at astronomical scales.
Perfection does not exist in this universe. Nothing can create "perfect limited copies". Error correction can only go so far, and it comes with a cost.
OTOH, electronic copies can be a lot better than DNA replication at making identical copies, which would allow much longer "genomes" with the same error rate. Possibly long enough to include the results of a lot of training data.
One question I have about copies/decentralized AI is how the super power AI can run on any system other than the one specifically designed to run the super powered processing that it needs?
I think the answer is that the AI would design a version of itself that can run on much lower hardware specifications and then copy itself to something roughly like a home computer or whatever. But why would we ever consider that even theoretically possible, given the complexity of running an AI as we understand it?
If an AI needs to run on a supercomputer of momentous power, then it seems very unlikely it could ever copy itself anywhere else. Maybe it could run specific commands to other computers, but not a copy that could be called AI.
Yes, that's kinda what I meant by a "limited copy". The analogy here is to individual human brains, which seem to be capable enough, and yet don't require huge supercomputers or momentous power. If we already granted that superintelligence is possible, clearly it would be able to design something at least as efficient as that.