I claim that you've made a dozen bad life decisions relating to this in the past week, and that several forms of therapy are built on top of it (though they wouldn't frame it in those terms).
The ugh fields post linked above has a lot of examples of bad life decisions. In terms of therapy, more speculative, and I'll write more on it later, but I think a lot of blocks in therapy relate to something like "It is easier and more fun to come up with a theory of why I shouldn't have to feel bad about this problem, than to solve it", which I think is this same idea of reward operating on epistemics instead of action.
That is, if there's a problem (you are poor), and your brain can solve it by either coming up with some kind of theory in which it's rewarding (rich people suck, they've all sold out, my poverty actually makes me a good person) or by working hard to get more money, and your brain is indifferent between getting reward by epistemic changes or by real-world changes, making the real-world changes is going to be a really tough sell. My guess is it's much more complicated than this, but I do think this is part of the dynamic.
Ok, but you didn't say it was "untethered from the physical nuts and bolts of MY daily life", you said it was from "daily life". A reasonable reading of this is that you think it doesn't apply to the daily life of any significant number of people.
Can people really stop feeling something because they have a theory? Or is your model really that people *cannot* stop feeling bad and that's *why* they have to solve the real problem?
Regarding procrastination and ugh fields, here's an interesting and relevant question: Do animals procrastinate? I don't know the answer, but it certainly seems quite possible to study it. For instance, squirrels gathering nuts for winter: Are they at all like students studying, i.e. gathering information, for an exam? Do squirrels, like students, often laze around til the deadline is near, then put out a burst of effort that exhausts them and sometimes does not permit gathering of a sufficient store of the resource in question? Maybe squirrels' behavior with nuts isn't the best thing to study, since the nuts don't land on the ground at a steady rate all fall but instead all drop over a short span of time. But there have to be some situations where animals really do have the option to procrastinate. And if there isn't, we could create on artifically in the lab with rats.
I thought your original comment was a little hostile, but I really appreciated your follow-up comments to me (that you've since deleted) and also wanted to reply that I'm sympathetic to what you described as part of your impulse to comment, tho I think having discussions like those in this post is a perfectly fine hobby to have :)
I don't think it was a cheap shot. I think your underlying point was important and sound, and it resonated with me. It *is* a little worrying when the best and brightest young folks are seriously debating how many angels can dance on the head of a pin *and* seem at least somewhat unaware that the question is a priori frivolous. One can't help thinking someone ought to say something. Recall the wry popular quote: "If everybody contemplates the infinite instead of fixing the drains, many of us will die of cholera." There's a lot of fixing the drains stuff that still needs to get done in 2022.
If I had to guess, I'd say you regretted that you expressed it in a tone that came closer to contempt than bemusement, which unnecessarily antagonizes and derails your point. Fair enough, but I could wish you had rephrased it rather than eliminated it.
Cleverly worded cheap shots are deletable? I assume that includes sardonic, obscenity-free ad hominem. What a world we live in where judiciously phrased, non-vulgar insults are considered outside the bounds of reasoned discussion. We are all snowflakes now, aren't we?
Gunflint deleted their own comment. I didn't consider it "outside the bounds of reasoned discussion" – I, and several others, directly discussed their comment!
I'm pretty sensitive to the price of pasta, potatoes, flour, and rice. I'm somewhat aware of the price of meat. But I don't live on milk. It is something I buy rather sporadically, if I get a craving for something I don't eat regularly - rice pudding, or cold cereal.
I suspect that milk consumption in the US is excessive due to PR from the dairy industry driven by...motivated reasoning.
Anyway, I was wondering if Scott has talked to anyone who is diagnosed with schizophrenia recently, and what he's observed or assumes to be their relationship with "motivated reasoning". Sub-question is, do people with schizophrenia go to therapy?
Something I found interesting as a difference in perspective between generations that may get lost in time: I've heard/read people scorn skim milk as unpleasant, watery, and blue colored. At the same time, products of a family farm are idealized as wholesome, rich, organic, unprocessed. But someone I knew that grew up on a farm in the 1940s, didn't like whole milk, considered it greasy and distasteful. Why? Conditioning. On the farm, the fat was always skimmed off, otherwise you wouldn't have butter and cream.
Possibly they didn’t like it because it wasn’t homogenized. Whole milk is actually worse to drink when the fat floats in greasy globs instead of being smoothly mixed in.
Me neither. If he lived on a farm he may have drunk milk from his own cows, and that milk wouldn’t necessarily be homogenized. That’s just what comes to mind when I hear milk described as “greasy.”
It's probably got a lot to do with what you're accustomed to. Skim milk is more watery than whole milk, but which you actually like better is subjective. If you're used to a certain level of thickness, different levels are probably going to taste wrong.
(On a related note, sugar-sweetened soda gives me heartburn but diet soda doesn't. After drinking only diet soda for a while, regular soda now tastes too syrupy for me.)
Certainly from the few people I know with schizophrenia, many of them go to therapy - there's a lot of useful technique to be learned re: how to calm down enough to not act on the more dangerous of the unshared experiences, how to survive the anhedonia side effects of the antipsychotics, etc.
>how to calm down enough to not act on the more dangerous of the unshared experiences
I think this advice could be harmful to anyone who takes it at face value. There's an inherent runaway positive feedback loop. If a consumer discloses that they are concerned about anything, then they have conceded there might be a danger, that they aren't sure of the level, and irrevocably put judgement of it in the hands of others who cannot read minds. Sharing with someone who has professional responsibilities to take action, on balance, might be worse even than talking to a lay friend.
I can't imagine any way to fix the dichotomy between suffering alone from an illness and overreaction/misunderstanding other than coming up with an objective way to determine states of mind based on biological markers. If there is one thing I would like to see in my lifetime it's something analogous to blood sugar and A1C tests, only for depression, suicidality, aggression, psychosis, etc.
Obviously it could be used in harmful ways and many (especially paranoid) people would be fearful of a neo-eugenicist movement, but I think one has to come to terms with how harmful and intractable it is to *not* have any objective measurement to base life and death decisions on.
And I do not believe that good treatments can be developed for anything that does not even have a real definition based on measurement.
I think he's saying locksmiths had to approach strangers in the dark (who were maybe locked out of their car, or are maybe hoping to rob him) and this is a scary action
In a nutshell, for me at the very least, the Catholic faith solves this problem. From a skeptical outside perspective, ignore the deistic epistemology entirely and focus on the phenomology.
One benefit of faith is that it makes your utility function non-monotonic and flexible. It does this by putting a presumably conscious entity at the peak of the hierachy of values (think virtue ethics here). So you get an ultimate determiner of value that is pretty rigid but can decide between competing values. If Christ isn't this for you, well, I enjoy sharing this experience with 1 billion + people
¯\(°_o)/¯
24 hour later edit:
A few responders to this post seem to be operating on an assumption that no one is capable of being very intelligent, well-informed, intellectually honest, and religious simultaneously. While I don't agree with this perspective, even though it may be a strawman, I suspect many of the benefits of this way of thinking could apply to eg HPMOR fans who want to consider the actions of their favorite character rather than the deity I choose to follow. I think good fiction is invaluable, it allows for an idealized and situationally unique perspective, to which one might consider themselves an "apprentice" in a way that they would balk at doing for a real person. I think holding yourself to a higher ideal is generally a great thing, even if it's not mine (and from where I'm sitting, if my assessment is correct an honest and courageous attempt at this will have the same ultimate endpoint one way or another).
(PS I have cried more than one at HPMOR Harry and his interactions with dementors and phoenixes, it's better than the original IMO)
I can say more, but I did preface it by saying nutshell. I'll try some additional bullet points for now and if more clarity is needed I can provide it later.
- utility functions that aren't monotonic are susceptible to "Dutch booking" ie a form of exploiting cycles in values
-having a rigid hiearchy of values is basically deontology, which can lead you to ratting out a friend in a Kantian murder mystery because you aren't clever enough to be evasive rather than lie; if lying is always wrong, and saving your friend is usually right given the circumstances, this can lead to ineffective consumption of congnitive resources on intractable problems (also a critique of much utilitarianism)
-if you assume virtue ethics, then you have the company of roughly a billion people also asking the question "what would Jesus do?"
I mean if being tractable is the only thing you care about, why not just base your ethics around something simple, like maximizing the amount of hydrogen in the universe or increasing the amount of entropy? Most people care more about their ethics actually being right than they do about them being easy to follow.
Right, this is why I asked to ignore deistic epistemology. If you want Catholic apologetics I'm not your man, I'll just say my personal process of developing my faith has included a lot of wrestling with God which is also an ongoing process at times.
I agree with Paul above, and I don't see how your comment addresses his point. Also, your initial argument sounded something like "Catholicism simplifies ethics", but this comment sounds like even that might not be the case.
Tractable perhaps, but Catholic ethics is anything but a simplification of the problem. Tbh if you think that I question if you're making a good faith faith effort to understand my position.
As to bringing it back to original point, I suspect having an overarching moral/ethical/ideological system is an extreme benefit to motivational and decisional paralysis, which to me does seem to address the gist of the problem with "motivated reasoning" and reinforcement learners.
Well, if you are in a reflexive equilibria such that choosing a utility function involving maximizing H2 and choosing one that maximizes human happiness are indifferent, then your re probably better off going with the first. The problem is that that is not where we find ourselves, in that we prefer happiness to H2 is there is nothing we really can, not to metion should, do about it. Whereas if having a simple tractable utility function (doing what God says maximizes utility) that still seems to conform to human values (God says to do things I generally agree are good) then that would, initially, seem to solve the problem quite neatly, by being both tractable and ethical.
Of course, there are plenty of other objections to that paradigm (“how do I actually know what God wants me to do here? If God’s depress are isomorphic to what I already think of as good, why is this elaborate utility function necessary, and if they aren’t, then doesn’t that create mor problems?” But I think this particular objection is answerable.
My point wasn’t to provide a good answer, but merely to explain why this objection didn’t make sense. I think if you work it out the “Christianity (or any deistic religion with certain qualities) solves certain self referential problems in ethics and motivation” could work, but you would have to do far more work to get there than OP did.
>having a rigid hiearchy of values is basically deontology, which can lead you to ratting out a friend in a Kantian murder mystery because you aren't clever enough to be evasive rather than lie; if lying is always wrong, and saving your friend is usually right given the circumstances, this can lead to ineffective consumption of congnitive resources on intractable problems (also a critique of much utilitarianism
I think you need to be clearer about what rigid means. Utilitarianism doesn't approximate deontology so long as utility is finite, because a sufficient number of specks has more negative utility than torture.
The problem of hell and comparing infinities something I wrestled with for a long time, and has led me to some difficulties considering heresy. I don't think there's a consensus on the topic among more interested, expert, and intelligent theologians than my amateur musings and I defer to them.
I'd be quite interested in hearing your thoughts about the problem of hell.
When I was a christian, I simply didn't believe that hell existed. It just felt completely against my personal experience of the divine. The Benevelent God, whose existence I directly felt and saw, would have never created such a place. It contradicted everything on multiple levels. So I just never even thought about it as a philosophical problem
Later I understood that all my religious experiences was not my communication with God, but a glimpse into my own values and ethics. And this allowed me to think about the problem of hell and be really terrified. Not of the cruelty of God, of course, but of people who simultaneously claim that their ethics comes from their religion and that somebody being eternally tortured for finite amount of evildoing is part of this religion. If ones ethics can accept hell it can justify literally anything.
How do you, deal with the existence of hell? This is the mode of thinking that has never been possible for me, so I'm very currious.
I spent the last few years as agnostic and anti-religion and recently converted to Christianity. One of my big gripes with Christianity was the idea of hell until I came to the conclusion that popular understanding of heaven and hell doesn’t reflect what’s actually spelled out in scripture.
The way I understand it now is that God is Love and Hell isn’t a place, it is the absence of love. God doesn’t send anyone to hell but we instead choose Love or it’s absence.
Closest analogy I can come up with: you’re dating the perfect significant other. You lie, apologize, they forgive you. You cheat, apologize, they forgive you. But you continue cheating and lose the remorse. They call off the wedding and you are stuck for the rest of your life with the agony of losing the perfect spouse. That is Hell.
Important to note that this also means a deeply Loving/selfless Hindu, Zoroastrian, Atheist, etc already knows God and will spend eternity with God
I don't really have any original or personal thoughts on the topic. What I'll say is, for most things I seem to side with Thomists, but on this question I do "dare to hope", since it's the most legitimate seeming option that's available to me.
I'm going to post this here and rely on some security through obscurity with my Google profile, I will come back and delete it once someone comments reminding me to do so (please do this is you've got the link saved). I don't want to share early drafts that I haven't at least presented to clergy, but this is my most sincere answer to how I (currently and after attempting to study the issues, but without any claim to expertise) deal with squaring my faith with my understanding of the world.
> a sufficient number of specks has more negative utility than torture.
Does the majority of utilitarians actually agree with this? I seem to recall that there was plenty of argument on LW on this, and no clear consensus emerged.
I wonder if this also ties in to self-identity. I remember reading about if kids are praised for being smart, they can start to self-identify as smart and so become reluctant to try new things (since they will initially be bad at them, which clashes with their self-identity). Maybe that was bunk but I think I can see it a little bit in my own life.
If we start to self-identify as being right, or smart, it might make it more painful to update our beliefs because being wrong in the past could clash with that identity. I could see the same thing happening with virtue too (although I've read another Christian say the view is more that everyone is a sinner, which if practiced might prevent that).
For now, I try to think of the truth as a model. The world is too complex to totally understand, but we imagine simplified models in order to navigate it and find patterns. It would be unusual to be upset at finding a better layout for a model train set, since in that context the goal is usually to *have* the best model rather than to *be the best at building models*.
I like this train of thought. One thing I've found helpful is to not only think, what would Jesus do? But also, what would my "saint self" do. This approach has pros and cons, but fortunately the Catholic approach makes both methods isomorphic.
I once heard someone (I think Leah Libresco but I'm not highly confident in this attribution) describe the "communion of saints" as an attempt to provide cross-temporal bayesian agreement on theological questions. I really like this framing, as it justifies the variety and sometimes imperfect nature of recognized saints. Few if any saints were virtuous their entire lives, and some of the most notable we're quite sinful (a great example here is the apostle Paul, who prior to his conversion persecuted Christians).
Another great benefit, imo, is that Christ was incarnate in a particular time and place, which though his example was indeed perfect, has limited utility in informing people in other contexts. So saints, while not perfect, at least by the end led sufficiently holy lives for the Church to confidently state that they're in heaven, which is a great element for a faith that is living to have as examples.
The "everyone is a sinner" thing is really good for solving that sort of problem. My own experience with Christianity was that it solved a lot of circular reasoning.
Suppose I'm depressed. I could solve the problems that are exacerbating that depression, but then I'd have to think hard about depressing things, which I obviously don't want to do. The actual truth is that, because of the aforementioned debilitating depression, at this moment I do indeed suck and am doing bad things with my life. But if everyone's a sinner and Jesus loves me anyways, then I have motivation to examine my life with clear eyes. No matter what I find, it won't decrease Christ's opinion of me (even though it will temporarily reduce my own opinion of myself). Then I can make the changes that allow me to love myself like Christ does.
Buddha's four noble truths (notably the one about "everything sucks all the time just cause") can achieve a similar end
I'm a little confused -- what does the problem have to do with ethics ? I suppose you could say that the fear of opening your tax book because you know you'll find taxes in it, could be labeled as "sloth", but it's a bit of a stretch.
When you write "and religious", my question is, what is "religious"? It's kind of a cliche that people supposedly like to say they're spiritual but not religious. I don't know if you are using that sense of "religious" or even what the cliche really means.
I'm not religious in the sense of belonging to any community, attending regular services, preferring one type of Christianity, and so on. That doesn't seem directly related to being intelligent or honest.
I was raised by people who I think were essentially atheists but probably would've abhorred the label. I think, but am not sure, that they came from their respective very religious backgrounds and were turned off by hypocrisy and bad behavior of religious people. I had no bar mitzvah or baptism or confirmation or anything. But I was given a Bible and I read a fair amount of it, plus later on stuff by C.S. Lewis, a biography of a saint, etc.
Recently, I've been talking to someone with a Catholic background, and it perplexes me that they seem to not be very familiar with the Bible, to believe in all sorts of new-agey things that seem to me in conflict with Catholicism, but they don't seem to be consciously hostile to it either.
When I started reading about the Inquisition, while I don't accept the *premises*, the theological debate over whether and which magic or astrology is compatible with Christianity makes some sense to me within a closed system. Normal people are oblivious or think of witch trials and/or Monty Python, though, right?
Conversely, contemporary magical beliefs like the law of attraction completely confuse me as to why someone would accept them, and not even see a conflict with traditional beliefs. But I don't think it's normal to analyze things like a Talmud scholar.
I guess where I'm meandering to, is that religious (or Christian) can mean many things, and I'm doubtful that there is even an intersection between what is it logical for it to mean and what it is normal for it to mean.
There you go, that's exactly what I'm talking about. Sure, that is "standard", but from what I read it excludes most Catholics! And my anecdote is not inconsistent with that.
Also, I notice you don't even include "reading the Bible".
"Most Catholics worldwide disagree with church teachings on divorce, abortion and contraception and are split on whether women and married men should become priests, according to a large new poll released Sunday and commissioned by the U.S. Spanish-language network Univision."
Is this the way to like scary movies? Not “I like to be scared,” but “I like the sequence of feeling scared and then realizing I’m totally safe/heroically safe at home”?
I definitely think part of the appeal of horror (whether film, game, or attraction) is the ability to feel fear in a controlled environment where no ACTUAL danger can happen to you*. Of course, people have their own personal tolerances for this, and some people have their risk-assessment feedback set to the point that even simulated peril is unacceptable.
I've heard this a lot, but never with any evidence I find convincing. I think it might be the kind of thing that sounds nice so no one wants to reject it.
My experience with horror is more like spicy food. It's literally activating pain receptors, but so detached from pain, and so mixed up with a particular sensory context, that the ostensibly bad stimulus becomes more of a unique overtone creating a richer more textured flavor. (Yes, I know some people eat spicy food just to show off their pain tolerance, but I don't think many horror film junkies are like this)
Another interpretation is that humans are actually pretty bad at distinguishing emotions, and so, for example, heightened emotional states are equally confused. Being excited is not that different from being terrified.
This could also be an example of reinforcement. The thinking is: 'You were in danger, and you did something to avoid harm, so you should feel pretty darn good about whatever it was. Here are some endorphins'.
It's similar as with hot food. Your pain sensores get triggered, but nothing bad happen to you. But the pain triggers the fight or flight response, that make you feel awake and fit.
So over time your brain asociate this kind of pain, with the fun feeling of being awake an fit, but no danger, so you start to like the pain.
The "bad day" example is meant as a hypothetical scenario in which a singular bad experience would worsen the lion detection circuitry if reinforcement learning was applied to it.
Right, but it hinges on the detection actually being negative, whereas noticing a danger before it hurts you is generally exciting and stimulating, a positive experience..
It's entirely hypothetical. For this example it does not matter what would likely happen, but what *could* happen. I'd agree that it is not a particularly good example.
I think in the lion-in-corner-of-eye example a lot of people would freeze, which also explains the tax behavior.
It makes sense from an evolutionary perspective, where if a predator doesn't spot you they'll eventually leave you alone, but the IRS doesn't work that way.
The IRS is kind of weird. US law can subject you to criminal penalties for not submitting an honest tax return, but not for refusing to send the IRS money. However, if you *do* refuse to pay them, the IRS does get to take your stuff to get what it's owed plus interest and penalties.
My mailbox definitely projects an "Ugh Field" for me. I know I need to check it, but it only ever brings me junk mail or problems. So every time I think "I should check the mail" another part of me is thinking "Do I have time to solve problems right now? Do I want to? No and no."
Huh, this is an un-realized advantage of not having street delivery and instead only having a PO box. Lots of my packages (which I ordered and want/am looking forward do) got to the PO box, so the only way to get them is to pretty regularly check my mail.
This in no way outweighs the inconvenience of needing to drive 15 minutes whenever I have something I need/just to check the mail, but it _is_ an advantage I suppose.
Not exactly applicable, but you made me think of this:
Napoleon had a policy of ignoring incoming letters for three weeks. Most things "requiring" his attention didn't actually need him to do anything. Thus, many things just resolved themselves by the time he got around to reading the letter.
I find Plantinga's argument strange, in the sense that in most situations, having a belief closely aligned with the observed phenomenon seems clearly advantageous. To my knowledge, systematic discrepancies between belief and phenomenon correspond to relatively rare cases where both (1) it is difficult to determine what is true and (2) there is an inbalance between the costs of the different ways of being wrong (ie. if you are not sure whether A or B is true, and wrongly believing A is much better than wrongly believing B then believing A is better, even if B is slightly more probable).
The crux of the argument is that evolution should plausibly produce adaptive behaviors, but not truthful beliefs. It might be that truthful beliefs are adaptive, but not necessarily so, and evolution would only reinforce truthful beliefs that are adaptive but not ones that aren't. So if there are truthful beliefs that aren't adaptive, can we trust that our minds can find those truths? How can we be sure that truthful beliefs tend to be more adaptive than non-truthful beliefs when the thing we use to make that determination, our mind, is the very thing we're trying to determine the accuracy of? If the problem was that you were not sure whether a scientific instrument was reading accurately, you can't determine that by using the instrument in question alone.
Really the argument boils down to: if the human mind is the creation (regardless of method of creation, could still be evolution) of a rational mind trying to create more rational minds, then it makes sense that the human mind is rational. If the human mind is the product of blind forces that are only aimed at increasing fitness, then we can expect the human mind to be very fit for reproduction but have no reason to believe it is necessarily rational. But if the human mind is irrational, then we have no reason to believe it is the product of blind forces because that belief is also the product of an irrational mind.
It certainly seems that our minds our rational, in that we can understand concepts and think about them and come to true beliefs in a way that, say, a chicken can't. Given that data point (human mind seems able to come to true beliefs), the "designed to be able to do that" model fits more parsimoniously than the "maybe if you select for only fitness you'll get rational minds in some cases." It's not exactly a knock down argument, and you can certainly disagree with it rationally.
I have a vague idea of an argument for why intelligence isn't adaptive past a certain point. Intelligent people are afraid of small probabilities and unknown unknowns. But these are where intellect is least useful, and analysis is most sensitive to assumptions. People who aren't as intelligent and aren't as imaginative, are more likely to just do stuff until they are stopped or killed. That could be systematically better for a population even if not for an individual. Because it explores phase space that is inaccessible to anyone who must *understand* what they are doing to do it.
Suppose there are 1,000 people facing some threat to all of them, and they can each do something with a 1% chance of success and a 99% chance of death. If they're all intelligent, then they won't do the thing until it becomes clear that the alternative is certain death. At which point it might be too late. But if they're fearless and oblivious, and all do the thing, then 10 people will survive, and carry on their risk-taking genes.
Someone smarter than me could put it more rigorously, but I feel like intuitively that analytic pessimists do not take optimal amounts of risk from an evolutionary perspective.
Empirically, evolution isn't producing runaway intelligence, so there must be some logical reason it doesn't increase fitness, right?
This assumes a risk distribution, not too unreasonably, like the one in which we evolved, where it is easy for me to get myself killed, possible for me to get my entire band wiped out, but nearly impossible for me to wipe out all humans or all life. We have changed our environment.
It's still easy enough to get oneself killed, and easier to wipe out humanity, than thousands of years ago. What the modern world has reduced, I think, is the effect of natural disasters that aren't global. And it's increased the individual benefit of being analytical, but, I think, only to a point.
It could also be the case that intelligence does increase fitness but that the cost of a bigger brain outweighs that fitness increase. That would be analogous to the explanation I'd use for why men don't have bigger muscles and why women don't have bigger breasts, despite the obvious fitness advantages.
After thinking about this, I don't understand either:
(1) Why it's "obvious" that those things would be better, or;
(2) How decomposing a reduction in fitness into a small increase and a bigger decrease is meaningful rather than an arbitrary choice applicable to everything.
I can't see any other explanation than sexual selection for the existence of D cups, and I can't see any other explanation than natural selection for the persistence of A cups despite this sexual selection, and to me these things both seem obvious.
If some variation affects fitness through multiple mechanisms, I think that understanding those mechanisms separately will give a better understanding of that variation. Experimentally, those mechanisms can be manipulated separately.
Using a heuristic that seeks truth without trying to filter on adaptivity might be more adaptive in the long run. It’s hard enough trying to figure out what's true. Is figuring out what is adaptive easier?
I thought that properly, something being "adaptive" means that it spreads through a population over time. Leading to death or preventing biological reproduction doesn't necessarily prevent a progression of changing statistics. I like that way of defining evolutionary fitness because it avoids mixing in human values and remains abstract and general.
I was answering the part about "figuring out" what is adaptive. I thought was misleading, because organisms do not "figure it out" which of course does not in the least prevent adaptation from hapening.
And yes, of course, adaptation optimizes the number of produced copies of genes, and not the fact that the individual bearing these gens die or not, though the two are usually quite strongly related!
"The crux of the argument is that evolution should plausibly produce adaptive behaviors, but not truthful beliefs."
I completely agree with the argument, but I think it has little consequence, in the sense that I expect evolution has produced "reasonably" true beliefs in most cases (and no beliefs at all in many situations in our modern lives!) because true beliefs are generally adaptive.
There are many examples where evolution has produced beliefs that are not perfectly true, because when there is uncertainty it is often better to be wrong in one direction (e.g., predator detection), but I know of no examples of "very false" beliefs constructed by evolution.
I don't think so. We have many "slightly false" beliefs that are not insulated from analysis at all. For example evolution gave us beliefs that spiders are dangerous, but we are very well able to determine than this is false for many spider species. The beliefs produced by evolution are expectations or emotions, it is perfectly possible to analyse them (though it can of course be difficult to act rationnally even when we are concious that a given belief is false).
> It might be that truthful beliefs are adaptive, but not necessarily so, and evolution would only reinforce truthful beliefs that are adaptive but not ones that aren't.
I've seen this claim often, but it's never accompanied by a plausible example that would make it convincing. It's basically asserting that there's no known reason why evolved adaptations would correspond to truth, and thus concludes that evolved adaptations may in fact imply false beliefs.
Not knowing the reason does not entail such a reason doesn't exist, so that argument just doesn't follow though. It's a premature conclusion at best.
The crux of the argument is that divine creation should plausibly produce behaviors that are in line with God's will, but not truthful beliefs. It might be that truthful beliefs are in line with God's will, but not necessarily so, and divine creation would only reinforce truthful beliefs that are in line with God's will but not ones that aren't. So if there are truthful beliefs that aren't in line with God's will, can we trust that our minds can find those truths? How can we be sure that truthful beliefs tend to be more in line with God's will than non-truthful beliefs when the thing we use to make that determination, our mind, is the very thing we're trying to determine the accuracy of? If the problem was that you were not sure whether a scientific instrument was reading accurately, you can't determine that by using the instrument in question alone.
Really the argument boils down to: if the human mind is the result of an evolutionary process which selects for rationality, then it makes sense that the human mind is rational. If the human mind is the product of intelligent design carrying out some inscrutable divine plan, then we can expect the human mind to be very fit for carrying out God's plan but have no reason to believe it is necessarily rational. But if the human mind is irrational, then we have no reason to believe it is the product of intelligent design because that belief is also the product of an irrational mind.
It certainly seems that our minds our rational, in that we can understand concepts and think about them and come to true beliefs in a way that, say, a chicken can't. Given that data point (human mind seems able to come to true beliefs), the "evolved to be able to do that" model fits more parsimoniously than the "maybe God wills there to be rational minds in some cases." It's not exactly a knock down argument, and you can certainly disagree with it rationally.
This is actually a pretty good counterargument I haven't heard of before. At least in the case of evolution you have good reason to believe that reproductive ability correlates with truthfinding. There is no reason to believe an omnipotent entity wants to create rational minds.
The main objection, is that it is at least plausible that a rational mind could be designed by a rational mind: it is harder to see how a rational mind could come about through irrational processes that are not aimed at rationality in any case. You can certainly object that a rational mind might design an irrational mind if they wanted to, but at least it isn't mysterious where the rationality came from if they designed a rational mind.
I never said that. I said one side was at least plausible on its face, but the other side is a bit harder to see if it is plausible. Do you disagree that a rational mind designing another rational mind is plausible? Do you disagree that it's more obviously plausible than a non-rational process that isn't aimed at producing rationality producing one?
As for the rules portion of his argument, I think Plantinga's argument makes more sense of a counter-argument against 1970s era rules-based AI. Which is fair enough, since that kind of AI is no longer seen as an effective tool with which to model human behavior anyways.
I think Plantinga is a bit binary in terms of labeling things 'rational' vs. 'irrational.'
Don't put too much by my summary: I don't think he'd necessarily use the labels "rational" and "irrational" in his formal argument, I just grabbed them as easy words to use in a short explanation of the argument. Obviously what it means to be "rational" in this sense requires a lot of defining of terms.
Yes, yes! I totally agree! I think that is the usual mistake that mathematician/philosophers do when dealing with living things : they consider that "very slightly false" is identical to false, whereas in biology, the "truthfulness" is better considered quantitatively, with "very slightly false" in practice very similar to "correct".
Yeah, in his actual argument, Plantinga deliberately avoids the binaries of 'true', 'false', 'rational' and 'irrational'. He talks about a type of belief he calls a "defeater": a belief which makes other beliefs seem less likely, but in some other way than by directly disproving them.
I'm thinking something like: This post takes first steps towards some kind of taxonomy of neurological skepticism. The Plantinga argument relies on (something like) reinforcement learning governing the whole brain. But if this post is right, that's not the case; the lion country plan part is, but the visual cortex isn't. So you could insulate naturalism from Plantingean skepticism if you could securely place it in the kind of thinking not governed by reinforcement learning.
For social creatures who evolved in small groups, challenging the group consensus could create a sort of negative hedonic reinforcement learning. There's little to be gained from sharing your opinion, even if it's true, if everyone else is going to get upset at you for saying it. So, best not think too hard about whether what your community thinks is true is actually true, and just go with the flow.
I try not to fall too hard into the cliche of "constantly explain complex brain functionality with simple comparisons to something from my field", but with the field in question being machine learning, it's impossible to resist these comparisons, especially as of the last few years. Too many subsets of our functionality seem so analogous to the way many artificial neural networks work!
This example also gets more interesting if we add other fundamental ML concepts in, e.g. learning rates (what magnitude of events correspond to what kinds of belief updates? are there some areas gradients are passed to more strongly than others, and if so what changes and modulates this?), weight freezing (at some point we have to learn to recognize basic objects and patterns - at what point(s) is this most adaptable, and which parts of it are immutable as an adult?), and of course backprop and other hyper-parameters, which are already interesting enough to contrast on their own. This also reminds me of a deepmind paper I saw yesterday https://deepmind.com/blog/article/grid-cells in which they construct an ANN similar to grid cells https://en.wikipedia.org/wiki/Grid_cell
There is a relevant concept of "catastrophic forgetting" when fine-tuning a model where it is a serious concern that training on recent data may "unlearn" knowledge gained from earlier data, losing the capacity for generalization.
Another very different but also relevant concept is the idea of "negative class" for classifiers where e.g. if you would train an image classifier to distinguish between different types of butterflies, you might also want to include a class of random non-butterfly images to teach the classifier what images should be ignored.
I think that the decreasing learning rate concept is strongly related to the explore vs exploit tradeoff which has a solid basis in game theory (multi-armed bandit model) and applied in reinforcement learning, and also observed in biology ("you can't teach an old dog new tricks") and changes in human behavior as we age.
I mean, a decreasing learning rate is adaptive (according to game theory) so we should expect to see that in evolved systems, and we do.
One of 5 or so places in the brain that can get a dopamine burst when a bad thing happens (opposite of the usual) is closely tied to inferotemporal cortex (IT). I talked about it in "Example 2C" here - https://www.lesswrong.com/posts/jrewt3rLFiKWrKuyZ/big-picture-of-phasic-dopamine#Example_2C__Visual_attention Basically, as far as I can tell, IT is "making decisions" about what to attend to within the visual scene, and it's being rewarded NOT for "things are going well in life", but rather for "something scary or exciting is happening". So from IT's own narrow perspective, noticing the lion is very rewarding. (Amusingly, "noticing a lion" was the example in my blog post too!)
Turning to look at the lion is a type of "orienting reaction", I think. I'm not entirely sure of the details, but I think orienting reactions involve a network of brain regions one of which is IT. The superior colliculus (SC) is involved here too, and SC is ALSO not part of the "things are going well in life" RL system—in fact, SC is not even in the cortex at all, it's in the brainstem.
So yeah, basically, looking at the lion mostly "isn't reinforceable", or to the extent that it is "reinforceable", it's being reinforced by a different reward signal, one in which "scary" is good, as far as I understand right now.
Deciding to open an email, on the other hand, has basically nothing to do with IT or superior colliculus, but rather involves high-level decision-making (dorsolateral prefrontal cortex maybe?), and that bran region DOES get driven by the main "things are going well in life" reward signal.
Offering a clarification here: I don't believe that the IT cortex receives a dopamine burst when bad things happen. The paper you linked to in the Less Wrong post (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC38733/) correctly identifies that IT is an input to and output of the *basal ganglia*, which is a loop from cortex -> striatum -> nigra -> thalamus -> cortex. But that's not the same as saying that IT receives lots of dopamine input (it does not). So there isn't really a problem here in terms of dopamine training higher-order visual areas/rewarding when bad things happen. (The tail of the striatum, and potentially other aversive hotspots, is another question. Those regions tend to drive avoidance and not approach, so it's not really fair to say noticing the lion is rewarding in this case, despite phasic dopamine in these areas.)
Thanks! In my defense I didn't say that IT has a dopamine burst. What I said was, well, maybe it's a bit confusing out of a particular context. So here's the context.
There are a bunch of parallel cortico-basal ganglia-thalamocortical loops, throughout the cerebrum. I think for certain purposes, we should treat "one single loop" as a unit that is trained (by RL or supervised learning) to do one particular thing. See https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.379.6154&rep=rep1&type=pdf . In that paper, referring to their Fig. 5, they talk about "column-wise thinking" as the traditional way of thinking, where you say what does the striatum do? what does the cortex do? Etc. They are proposing an alternative, namely "row-wise thinking": What does this loop do? What does that loop do? And that's where this particular comment is coming from.
It turns out that IT and the tail of the caudate are looped up into the same cortico-basal ganglia-thalamocortical loops—IT is the cortex stop and tail of caudate is the striatum stop of the same loops. Therefore (in this way of thinking) if the tail of the caudate is getting aversive dopamine bursts, that's basically a mechanism for manipulating what IT will do.
You're still welcome to disagree with that perspective of course, but hopefully at least you understand it a bit now.
Yes, thank you for the clarification! Sorry if I misinterpreted your original comment. I'm a big fan of the "row-wise" thinking you espouse. However, I don't think there's good evidence either way for what effect DA has on the cortical sites to which BG loops return. Indeed, I don't think there's a good account for why these are loops at all! Certainly, the easiest way to think about it is that cortex is providing the "state" input to the RL algorithm, and DA is training the weights of corticostriatal synapses to compute value (or threat, in the case of the caudate tail). But there has been interesting work in RL asking how can we harness more information from RPE signals to train better state representations (see e.g. speculation in Dabney et al., Nature, 2020). Of course, in ML, you just backpropagate this information so it's no big deal. In the brain, backpropagation is tricky or impossible, so maybe the BG gets around this limitation by somehow forward-propagating the info all the way around the BG loop. But there isn't really a plausible account of how that would happen either. By the way, this is much more general than just IT cortex; primary visual and auditory cortex project to tail of striatum, motor and sensory cortex to dorsolateral striatum, etc. See Hintiryan et al., 2016 and Hunnicutt et al., 2016 for much more than you wanted to know.
tl;dr I think we don't yet know nearly enough to say what effect, if any, DA has on sensory cortices that provide input to the striatum.
Re. "IT is "making decisions" about what to attend to within the visual scene, and it's being rewarded NOT for "things are going well in life", but rather for "something scary or exciting is happening": I would be interested in anything ": Do you know of any brain systems which relate this to aesthetics (cast as preferences about what we attend to), curiosity, or the fun of problem-solving?
Not sold on the "visual-cortex-is-not-a-reinforcement-learner" conclusion. If the objective is to maximize total reward (the reinforcement learning objective), then surely having your day ruined by spotting a tiger is better than ignoring the tiger and having your day much more ruined by being eaten by said tiger. (i.e.: visual cortex is "clever" and has incurred some small cost now in order to save you a big cost). Total reward is the same reason humans will do any activities with delayed payoffs.
Sounds like a plausible experiment would be something like: have somebody show you pictures of circles. You report whether it looks like a circle or a square (be honest!) Each time you say circle, the person gives you an electric shock. See whether, after long enough, you start genuinely seeing squares.
My guess is this never happens - do you guess the opposite?
I'm not sure why you're bringing up trolley problems. Phil and I have different scientific theories about how the brain works; it's hardly unfair to test them by experiment.
I just did the experiment on myself (using "hitting myself very hard" as a standin for electric shocks) and got the predicted result.
I don't think this was necessary (except in order to win this argument) though - I think it's a *good* thought experiment, in the sense that just clarifying what the experiment would be makes everyone agree on how it would turn out. Essentially I was asking Phil "How does your theory survive the fact that in this particular concrete situation we both agree X would happen?" It's *possible* he could say "I disagree that would be the result", but then I would have learned something interesting about his model!
That's the wrong timescale for your visual system to change. Changing so fast would have been unstable. A small learning rate doesn't mean the objective isn't a reinforcement-learning one.
Trolley problems are actually extremely common and can be found almost everywhere - the law, the medical system, engineering. It's probably the most useful thought experiment of all time.
The thing everyone in the world discussed for two years straight ("should we lock down the economy to stop the virus?") was essentially a form of the problem.
> the overwhelming majority of cases, it's just that the person making the analogy wants what they want and presenting the situation as a trolley problem is an effective method by which to emotionally manipulate their audience into agreeing with them.
I don't agree. A feature of the problem is that there's *no* obviously correct answer. Ethical arguments can be advanced for either course of action.
I take the unrealistic aspect of a trolley problem to be knowing exactly what happens given each choice, and only being in doubt about the ethical weught.
This reminds me of that Star Trek: The Next Generation episode where Picard is tortured for days by Cardassians, who are trying to get him to say the wrong number when asked "How many lights are there". At the end of the episode, he tells RIker that he actually saw the number they wanted him to say.
I thought it's kind of mainstream science that evolution has 'programmed' us to see things that aren't there if it's good for us. For example, here's a quote from Why Buddhism is True:
"Suppose you’re hiking through what you know to be rattlesnake terrain, and suppose you know that only a year ago, someone hiking alone in this vicinity was bitten by a rattlesnake and died. Now suppose there’s a stirring in the brush next to your feet. This stirring doesn’t just give you a surge of fear; you feel the fear that there is a rattlesnake near you. In fact, as you turn quickly toward the disturbance and your fear reaches its apex, you may be so clearly envisioning a rattlesnake that, if the culprit turns out to be a lizard, there will be a fraction of a second when the lizard looks like a snake. This is an illusion in a literal sense: you actually believe there is something there that isn’t there; in fact, you actually “see” it.
These kinds of misperceptions are known as “false positives”; from natural selection’s point of view, they’re a feature, not a bug. Though your brief conviction that you’ve seen a rattlesnake may be wrong ninety-nine times out of a hundred, the conviction could be lifesaving the other one time in a hundred. And in natural selection’s calculus, being right 1 percent of the time in matters of life or death can be worth being wrong 99 percent of the time, even if in every one of those ninety-nine instances you’re briefly terrified."
Seeing a square when there's really a circle is a pretty extreme example (which I guess is why it got called a 'trolley problem'). But seeing a rattlesnake when there's a weird-shaped stick in the grass (and the penalties for false positives and false negatives are vastly different) seems pretty plausible to me.
This is not an adequate comparison. "genuinely seeing square" would do nothing to change the electric shocks; *claiming* you saw them would, which is certainly what you would have learned to do.
I think Phil's disagreement comes rather from thinking in a different time scale. The total reward maximitazion model is correct, but it works in evolutionary time: a visual cortex that correctly identifies objetcs is "reinforced" by evolution because this maximizes total reward. This is precisely what originates the *epistemic* architecture. But on the lifetime of a person, however, the visual cortex does not do any reinforcement-learning (at least after infancy).
Speculations about microscopic processes a billion miles away are a complete waste of time in the long run. We have the prameters of the Standard Model that 'explains the properties of atoms molecules that enable the construction of cells the visual-cortex is filled with, yet we can't get our most powerful computers to accurately predict the behavior of a single molecule of H2O. Considering the fact a single cell is built from hundreds or thousands of molecules far more complex, and that the behavior of everything at the core is a probabilistic quantum mechanical information process, I might be wise to listen to those with knowledge about God the Creator and life's purpose than scientists who's god is random mutation.
"That is why we're screwed. That number. That number will doom us."
"I hate that number. Can we run away from it?"
"No. It's a number. It represents an immutable fact."
"Can we fight it?"
"No. <...> What?"
"Sorry. Evolving a threat response over a half-million years on the African savannah hasn't really left me with any good mechanisms for dealing with a threatening number."
I think all of the supposed discrepanices with modeling the brain as a hedonic reinforcement learning model can be explained with standard ML and economics.
If you do a lot of research on epistemic facts related to your political beliefs, the first order consequence is often that you spend hours doing mildly unpleasant reading, then your friends yell at you and call you a Nazi.
In the case of doing your taxes or the lion, that unpleasantness is modulated by the much larger unpleasantness of being sued by the IRS and/or eaten alive by a lion. So there's a normal tradeoff between costs (filing taxes is boring, seeing lions is scary) and benefits (not being sued or devoured).
But in the case of political beliefs, the costs are internalized (your friends hate you) and the benefits are diffuse (1 vote out of 160 million for different policy outcome). So it's no wonder that people aren't motivated to have a scout mindset.
People are pretty good at avoiding sources of stress, especially ones that offer no obvious benefit. I can't really imagine why anyone should go become a scout given the tradeoffs. Maybe it's a noble sacrifice you make for the greater good?
I don't think that's obvious. The scout mindset is almost definitionally being less confident about your knowledge. Soldiers seem confident too, that's kind of the point.
I think that's only one route to competence. Plenty of people are very confident without being competent specifically because they have the soldier mindset.
I don't think even Julia Galef would claim that a scout mindset makes you more likely to have babies (which is all 'effectiveness' means from an evolutionary perspective)
Because if you become good at predicting social and economic trends that everyone else thinks is wild pie-in-the-sky nonsense but actually happen, you get words like "disrupter" and "avant-garde" attached to you, attain social status and financial success, and ultimately be able to have a kid with Grimes before she divorces you and decides she's a Marxist now.
Now this I could believe with the caveat that you have to already be a billionaire to get "disrupter" or "avant garde". The rest of us just get "crazy" or "weird".
And of course then there's views that aren't pie in the sky, just ordinary run-of-the-mill views of the opposite tribe. Those get the label "Nazi" or "Marxist" depending on which tribe you defect to.
If you're a Bitcoin millionaire, people aren't going to call you "crazy" (unless you start trying to argue for the AOC to be abolished, cannibalism to be legalized, or something else so far outside the Overton Window that no amount of money can shield you from social censure). "Crazy" requires you to flap your arms, jump off the barn, and break your legs. If you actually FLY, you're an eccentric visionary. I realize it's really really easy to imagine that social classes are an iron-clad law of the universe and people will still treat you as Joe Schmoe from Buffalo even if your bank account hits 7+ figures, but (in America, at least), like 70% of social class is attached to your net worth.
>Politics
I hope you aren't looking for someone on a random blog's comment section to give a complete solution to solve political disagreement. I was just making the point that there are, in fact, reasons to break from the herd besides pure altruism.
I'm not trying to focus on the title, just the argument being made. I believe that all of the supposed discrepanices with modeling the brain as a hedonic reinforcement learning model can be explained with standard ML and economics.
I have a prior: Being eaten alive by a lion would be extremely painful and end my life before I can procreate. I've assigned 100% probability to that prior. The brain is able to backpropagate to my visual cortex that a False Positive on a lion is scary, but low risk (let's say loss value = 1.0), but a False Negative on a lion is life-ending (let's say loss value = 10000.0). Given those incentives, your visual cortex optimizes lion recognition for high recall and reasonable precision.
I don't see why there's any contradiction between hedonic reinforcement learning (learn not to get eaten alive, extremely low hedonic state) and what the visual cortex does.
His point is that sometimes in life we do the thing that doesn't happen with our eyes: our eyes always see the lion, even if we really do not want the lion to be there. Your explanation of why that is makes sense: if we don't see the lion, we die. Yet at the same time in other areas of our lives there is a problem, it really will negatively impact us if we don't do something about it, and yet instead of seeing the problem our minds work hard to find a way to convince ourselves the problem doesn't exist. So why does it do that? That's the question, not why our eyes always see the lion. It's why our minds sometimes try to tell us the lion we see isn't really there because we don't want it to be.
Is that the point? The post seems to presuppose that there are "reinforcable" and "non-reinforcable" parts of the brain. I don't see any need for that. You can perfectly explain the function of the neural cortex and every other part of the brain through hedonic reinforcement learning.
I originally supposed that he was asking "why does our brain work for the lion, but not for reasoning about politics?", so I answered that question with what I think is a logical, mathematical view based on internalized costs and externalized (diffuse) benefits. I think that's the answer to your question "Why does our brain not see the problem when it will negatively impact us?" My answer is that being wrong about politics *doesn't* negatively impact you because the costs/benefits of being right/wrong are shared across hundreds of millions of people. But the costs/benefits of agreeing/disagreeing with your friends accrue only to you.
Because the future is uncertain, and the further away it is, the more uncertain, and while a lion in the bush will eat you *right now* almost all the things we procrastinate about are lions that may or may not eat us some time from now, assuming our fearful reasoning about the future is correct, which it often isn't. The fact that our emotional centers routinely dial down threats which are more hypothetical and uncertain is hardly surprising, and clearly adaptive. The fact that we have warring agents in our awareness that struggle to gain consensus on threat-level estimates that do not agree with each other is also unsurprising, given normal conscious experience.
I rephrased the first sentence of my original post to not focus on the title and instead summarize my argument. My argument is that all of the supposed discrepanices with modeling the brain as a hedonic reinforcement learning model can be explained with standard ML and economics.
"This question - why does the brain so often confuse what is true vs. what I want to be true?"
Going back to first principles of natural selection one would presume the human brain to be:
(a) **well adapted** to discern true facts that have positive impacts on reproductive fitness (e.g. identifying actual lions, learning which hunting or gathering techniques work best, etc.);
(b) **well adapted** to engage in useful self-deceptions that also have positive impacts on reproductive fitness. (e.g., believing that your tribes socially constructed superstitions and political rules are "true" so that you fit in and get along.
(c) **Non-adapted** for determining true facts that might make your life more financially profitable, enjoyable and stress-free but that don't have any direct impacts on your hunter-gatherer reproductive fitness. (e.g., realizing that you shouldn't procrastinate on your taxes, or shouldn't worry about things you can't change.)
Sadly, the human brain is evolved to make more surviving humans, not to make us happy or successful in a modern capitalist economy. I think the happy/high-achieving people are probably those who are more successful in somehow tricking their brains to move category (c) issues into category (a). If only you can convince yourself that the search for absolute truth is a life or death hunt that will keep you from starving to death and instead allow you to have sex with the hot cave woman https://youtu.be/gSYmJur0Npw?list=PLVVuOIA1lowEKOVGZgf4_GsmH51Lu4511&t=68.
I thought motivated reasoning was.... Reasoning with a motivation. Reasoning, but, WITH AN AGENDA!! DUN DUN DUNN... Like, you want something, so you spend a lot of time entertaining counterfactuals which seem like they could be pieces of plans you could make which lead to you getting the thing you want, as opposed to Perfect Unmotivated Reasoning: reasoning done entirely for the sake of Truth, and not utility.
I thought motivated reasoning meant reasoning with ulterior motives
Well, yeah, it is. The problem is that you can use reason to come to lots of non-true beliefs, provided your main motivation is not finding the truth. That's what Scotts trying to puzzle out: why exactly do succeed at tricking ourselves that a real problem isn't real just because we don't want it to be?
"Maybe thinking about politics - like doing your taxes - is such a novel modality that the relevant brain networks get placed kind of randomly on a bunch of different architectures, and some of them are reinforceable and others aren’t. Or maybe evolution deliberately put some of this stuff on reinforceable architecture in order to keep people happy and conformist and politically savvy. "
I wonder if it is not simpler to just consider a dichotomy between tasks we have evolved for (e.g., learning to speak) and those we have not (e.g., learning to read), rather than the epistemic/behavioural dichotomy. Detecting threatening animals is clearly an evolved ability, while doing one's taxes is not at all. This could mean that "doing one's taxes" will not be done automatically, and will depend on many factors, including of course the pleasantness of the task (and also the tendency to ignore the future, trust in the group, anxiety, etc.).
I don't think it's mysterious why we don't like doing our taxes, I think it's mysterious why this leads to behavior like not checking whether our taxes need to get done, or refusing to open your email because you're worried there will be a difficult task in it.
Even if you hate email, I think the "rational" action is to open the email, check if there's any message which is so important it overrides your desire not to do it, and *then* procrastinate. But that's not how most people experience this.
I of course agree that the rational action is to open the email and get the information (which I find a bit hard to do...). But well, we already know that we do not act very rationally.
It seems to me that evolution has given us specific rules for situations that were both common and important during our evolution (avoiding the predator, indeed), and very general rules for all other situations. Among these general rules, there is clearly the acquisition of information, primates are notoriously curious, and of course the avoidance of annoyances. But we could not get a rule like "evaluate the situation and choose the behavior according to the best predicted option". So we "choose" a behavior according to the impulse that turns out to be the strongest, and many people procrastinate to open that email.
Because our behavior usually does not result from rational considerations but
I think you're thinking about this too much from a model where everybody is a good-faith Mistake Theorist.
In a mistake theory model, it's a mystery why people fail to update their beliefs in response to evidence that they're wrong. If the only penalty for being wrong is the short term pain of realising that you'd been wrong, then what you've written makes sense.
I think that most people tend to be conflict theorists at heart, though, using mistake theory as a paper-thin justification for their self interest. When I say "Policy X is objectively better for everybody", what I mean is "Policy X is better for me and people I like, or bad for people I hate, and I'm trying to con you into supporting it".
There's no mystery, in this model, why people are failing to update their "Policy X is objectively better" argument based on evidence that Policy X is objectively worse; they never really cared whether Policy X was objectively better in the first place, they just want Policy X.
I think there are three things: honest mistakes, honest conflicts, and bias - with this last being a state in which you "honestly believe" (at least consciously) whatever is most convenient for you.
If a rich person says the best way to help the economy is cutting taxes on rich people, or a poor person says the best way to help the economy is to stimulate spending by giving more to the poor, it's possible that they're thinking "Haha, I'm going to pull one over on the dumb people who believe me". But it also seems like even well-intentioned rich/poor people tend to be more receptive to the arguments that support their side, and genuinely believe them.
I don't think honest mistakes or honest conflicts need much of an explanation, but bias seems interesting and important and worth figuring out.
Is the conventional explanation unsatisfactory? That people are more convincing when they argue for their position honestly, and so it's beneficial for them to become biased in ways that favor their interests.
The question is, what is the mechanism? If I was offered a billion dollar incentive to sincerely believe I have three hands, I still wouldn't be able to do it. I can't alter beliefs on command. That makes the bias idea harder to explain.
That's because such offers presumably weren't common in the ancestral environment in which our brains evolved.
As far as I can tell, we're pretty far off of being able to establish the exact ways in which high-level adaptive strategies are implemented in the brain. For these purposes, it's an extremely complex black box, which also isn't too amenable to high-precision controlled experiments.
Thank you for the comments on this post. I was having a hard time figuring out what the post was about. I'm not sure why we ignore the Ugh fields, I sometimes see these balance scales in my head, on one side is the badness of not taking care of my taxes, and on the other side is the slight goodness of ignoring this problem for another day. The badness keeps building and eventually I have to open the dang letter.
Then why do many people make decisions that are objectively bad for them and the people they care about, are GOOD for the people they hate, and then actively defend these decisions unto the trenches for decades?
I'm referring to people who, as an example, vote for tax breaks for the wealthy and increased taxes for their own income bracket who, when asked WHY they do this, don't have a satisfactory answer beyond "My strawman conception of an enemy tribe wants that and I'll be damned if I ever agree with them on anything." (Yes, you can accuse this itself of being a strawman, but these people really do exist, and in sufficient numbers that calling it a "strawman" is a bit deceptive. You see them on both sides of the aisle, usually yelling at each other on social media). This isn't behavior that can be explained purely by calculated zero-sum games, because EVEN BY THEIR OWN CALCULUS they're making decisions that damage their own side (salt-of-the-earth Real Americans who work with their hands and backs) and empower their enemies (rich, effete latte-sipping Coastal Elites) based on weird ideological principles.
I mean, you could certainly say that the above descriptions are just branding those people consciously adopt, but that doesn't explain the core issue here- I'm not sure if conflict theory is willing to believe in tactically shooting yourself in the foot.
I knew exactly where you were going with it and responded accordingly. I've not met a single person who bemoans people "voting against their own interests" who wasn't entirely convinced that their preferences are derived from universal and timeless moral rightness and that anyone opposed to their preferences are stupid (poor people), evil (rich people), or both.
Yes, I do in fact think people should be paid a living wage and shouldn't be condemned to a life of misery, hunger, or dependence on intoxicants as an alternative to suicide, regardless of their genetics. And I do, in fact, think people who argue that Moloch-sacrifice is good and that inquiries into bringing the number down as close to zero as we can should be sneered at are evil.
If someone kept hitting themselves in the head with a hammer, complaining loudly every time he got hit, you would, in fact, call him an idiot- or else determine that he really likes the idea of dying via TBI, but I don't pretend this is some kind of just world where people are making the rational decision to be systemically discriminated against, not make any money, get hooked on painkillers, and die in their own waste. If you think my attitude comes from a sneering condescension towards the working poor instead of being one of them, that's your problem- unless you want to say you have access to my mental states.
>If someone kept hitting themselves in the head with a hammer, complaining loudly every time he got hit, you would, in fact, call him an idiot
No one did or said this, stop fantasizing.
>If you think my attitude comes from a sneering condescension towards the working poor instead of being one of them, that's your problem
I think your attitude comes from an unwarranted confidence in the correctness of your beliefs - which is pretty damn funny given that I knew the direction your diatribe was going to go based on how banal your "insight" is.
For my part, I assume in the overwhelming majority of cases that people make decisions for themselves for good reasons, where "good" is defined by the decision-making person him/herself. I don't presume to be able to define another person's preferences for them.
This model explains why people would make this kind of mistake when it benefits them, but it doesn't explain examples like procrastinating on your taxes.
If this is true, then maybe Cognitive Behavioral Therapy is just using your "basically epistemic" brain regions to retrain your "basically behavioral" brain regions. Like, trick yourself into ending up in a better hedonic state than you used to -> reinforcement learning counter updates towards "yay". That could explain why it's so crazy effective despite being "just" talk therapy.
I know less about brains and psychology than just about every other commenter out here, am I on to something here?
I don't think anybody really knows anything about brains and psychology at this point -- at least when it comes to complex behaviors and beliefs. For example, this behavioral vs. epistemic brain dichotomy sounds like one of those models that appeals to pattern-seekers but I doubt there is any way to prove it's useful or accurate with concrete data.
Consider that AI programmers will set up a self-teaching model for their programs and then set it lose to evolve a way to solve a particular problem. But once it's taught itself to solve the problem you can't go back and figure out exactly how it learned or how it logically gets the right answer. So why would we think we could reverse engineer our bio-brain's 3-billion year old self-learned chemical algorithms.
I think it is wrong-footed to proceed as if reasoning always directly connects your brain to the phenomenon that you are reasoning about. Instead, the process is social. You decide what to believe based on who you believe. You decide who to believe mostly based on who you would like to get approval from (or to get imaginary approval from). If in your imagination you would rather hear Joe Rogan say you're a great guy than hear Anthony Fauci say you're a great guy, then when they differ you will work harder to discredit what Fauci says than what Rogan says. And conversely. That is what puts the motivation into motivated reasoning.
I'm trying to make sense of this. First, I'm going to rename '[people] who you would like to get approval from' as 'desired approvers', because it makes the following sentences more comprehensible.
Does your model imply a connection between how vividly you imagine your desired approvers, and how easily you reason? And does it imply a connection between how aware you are of which beliefs are endorsed by your desired approvers, and how easily you reason?
If so, consider people with only a nebulous sense of their desired approvers (as individuals and/or as belief-holders). Your model seems to imply that such people will tend to have more trouble with reasoning than other people do. And the rare truly isolated person, with minimal media and no religion, family, or friends, would have a great deal of trouble with reasoning. Is that what you expect? Have I misunderstood?
Thanks for the recommendation. I've been seeing that book and Henrich's book on WEIRD mentioned, and was leaning toward reading the latter, but now I'm motivated to read both.
I guess reinforcement learning is all about good sense of balance between exploration and exploitation. I feel that when someone proposes a new theory to me it's like asking me to explore some new train. And I am less willing to do exploration when I am low on resources - to limit variance, which could kill me, when I am close to bankruptcy, I prefer the steady income of my current best model.
It seems like that would be because AlphaStar was learning to win games, and won more games when it checked what was in the fog of war. We're trying not to be eaten by lions, but unfortunately we rarely get more than one crack at it.
Oh I didn't mean to say evading predators wasn't an option! Just that if you closed your eyes and pretended the lion wasn't there (perhaps making lots of noise to drown out its footsteps), you wouldn't get a chance to learn from your mistake. Evolution, of course, would.
I think you can go a level deeper into this idea - there obviously aren't *actually* parts of the brain that are just magically epistemic, really just about everything in our CNS learns how to do its function to some degree or another. E.g., even the lowest levels of your visual cortex learn how the visual field fits together, how to handle lines of different directions and kinds, and etc. Probably most of that kind of learning is wrapped up and finalized by the time you're past the toddler stage, but it means that even these basic "epistemic" functions formed under a reward function of some kind. We would hope that reward function is something like surprise-minimization (and/or some free-energy whatsit that I don't understand,) but the brain is messy and full of poorly enforced boundaries, so the reward function could also be partially coming from extrinsic signals unrelated to its nominal function- perhaps of the form "if I become aware of X, I feel really bad suddenly."
The upshot of all of which *could be* that particular kinds of bad/traumatic upbringing in very early life could predispose someone towards motivated reasoning, by mis-training "epistemic" parts of the brain. That approaches the kind of thing that could be testable!
Just a thought, maybe one of many causes for this phenomenon:
Shifting a worldview isn't zero cost - when you shift your worldview many posteriors need to update (which means every situation and corresponding action which might have been 'memoized' before needs to undergo an update process which takes extra time and energy as it is compared against the new worldview).
I use the term worldview here but I guess it applies to anything you've taken as an assumption and might need to shift. I think energy and response time costs might play a part alongside hedonic reward.
This is also something I care a lot about, but maybe it serves our social animal nature more than our predictive fitness. Hedonic states are evolutionarily wired to confer at least four functions - they feel good, "energize" behavior, confer unique psychosocial strengths (motivation, humor, creativity, willpower, etc) and resilience to stress. These resources are useful are highly useful in the buffeting winds of the social world, so hedonic states becomes a meta-goal independent of their original evolutionary function. Originally, evolution creates a cybernetic system where anticipation of positive outcomes generates hedonic states to motivate behavior, but when people can coopt the benefits of the hedonic states for social functions, suddenly social evolutionary logic motivates individuals toward a positivity bias to generate hedonic resilience for its own sake.
You can extend this logic to other hedonic states like social connection or social status and people suddenly appear in relief as highly motivated to maintain tribal ideologies build on ingroup biases and attribution biases and positivity biases all to generate hedonic resilience. Suddenly the "madness" of doing the same thing and expecting a different result, of maintaining beliefs that are costly, of staying with ingroups that drag us down or of competing in self-defeating games makes sense - people are trapped on a treadmill of attempting to generate short-term hedonic resilience as a social resource, particularly as unlinking from predictive accuracy generates negative consequences and chronic stress, creating even more deficits of hedonic resilience and more need, cyclically. A pernicious spiral dynamic for social animals.
The beautiful part of this particular rabbit hole is when you realise motivated reasoning is basically the same concept as original sin, which is a combination of social optimisation and the psychological neoteny that allows societies to scale beyond the Dunbar number.
I think this is a bit unfair on motivated reasoning.
We train people at motivated reasoning for years; here's the conclusion, now argue for it, is basically the standard school writing assignment.
Then we get into the real world, and discover the vast majority of people don't care about the same things we care about, and if we need to convince them of something that is important to us, we have to argue for it from a set of values that were not used to arrive at it.
And truth isn't even that valuable for most people, and is in many cases actively harmful to them, compared to white lies. The weird behavior isn't motivated reasoning, it's being able to accept the truth with an awareness of bad things being constant sources of suffering.
>What is the connection between being trained to do something and *not knowing when you are doing it*?
In traditional martial arts, the goal of practicing the exact same motions hundreds or thousands of times a day, every day, for YEARS, is so that you stop *deciding* to do the motion and just *do* it. You see someone, and instead of your thought process being a complex analysis of everything happening step by step, you just make the decision to draw your sword and kill this man and everything else is just reflex. All the mechanics stop being a conscious thing you think about, and the basics are so deeply buried that being an instructor is its own separate set of skills from just being a practitioner of the blade. A friend of mine who's an aikido instructor spends most of his time in aikido form from the waist down because he spends so much of his time with his feet at 90-degree angles from each other that his limbic system now defaults to that.
This is also true in the sciences: PhD's in astrophysics tend to have MORE difficulty explaining basic physics concepts to laymen than someone who's just consulted a Wikipedia article despite the former being infinitely more qualified to give the explanation than the latter, because things like the basic Laws of Motion are so deeply ingrained in the former's mind that they almost never never consciously access that information.
I see absolutely zero reason why making people argue from conclusion to premise for the vast majority of their formative years wouldn't result in arguing from conclusion to premise becoming a habit.
"Then we get into the real world, and discover the vast majority of people don't care about the same things we care about, and if we need to convince them of something that is important to us, we have to argue for it from a set of values that were not used to arrive at it."
But all people do in debates with strangers online every day is try this strategy and fail to change peoples' minds.
When have you ever seen someone convince anyone of anything by appealing to their model of alien values?
The only way I am aware that anyone has ever convinced anyone is by first having insight into their own values and explaining them such that shared humanity is recognized.
Empathizing with someone who you've dehumanized up front is as impossible as...some mathematically impossible thing. The fallacy that's ever-present is the idea that lack of empathy can go one way rather than being symmetrical.
But I'm doubtful that convincing people has anything to do with the games normally played with strangers online.
I've been convinced by somebody appealing to my (to them alien) values; the argument they created was more convincing from the perspective of my own values than my own. I've convinced other people by doing the same. It does require one or ideally both parties being pretty good at either stating their own values, or correctly identifying somebody else's, which is pretty rare.
For the thing of people arguing from what they think other peoples' values are in general, though, it happens constantly, it's just often hard to see because we see the values other people argue from, not the values they actually hold.
Once you notice it, however, it's hard to stop noticing it, and it explains why so many arguments are basically pointless. The standard pointless argument goes basically like this: I think you have value X; I argue from a position starting with value X. You observe my arguments, and think they mean I hold value Y, which is related to but subtly different from value X; you argue from a position starting with value Y.
If I happen to notice you seem to actually be arguing from value Y, I change my arguments; now you think I'm a hypocrite who is contradicting myself. Which, in the sense of "Saying one thing but believing another" might be true, but is missing the humanity of what is actually going on. If, on the other hand, I notice that you aren't actually arguing from value X, but don't identify value Y, the argument turns into a semantic debate.
Or neither of us notice the values are different, and we have an argument in which we spend the entire time arguing past one another, on the basis of values that neither of us actually have, and both of us feel like we've "won" the argument, and the other party just won't acknowledge it. That seems to be the most common case.
It's all quite amusing when you start to notice it, really.
Having separate reinforcement and epistemic learners would be the elegant solution. There's also the ugly hack, which is to make "there might be a lion" even scarier than "there is a lion" so that checking is hedonically rewarded with "at least now I know".
Successful horror movie directors can confirm evolution went for the ugly hack, as usual.
Why do we need "there might be a lion" to be scarier than "there is a lion", which would be an "ugly hack"? We only need it to be scary enough to do damage to the hedonic state. Then checking offers a chance to restore the state --- there might not be a lion after all! If the lion probability appears so high that checking is not promising for that purpose, then surely it makes sense for the hedonic state to be very badly damaged anyway, without checking.
To me your point, the hedonic implications of "there might be a lion", puts into question Scott's explanation of wishful thinking, as the objection should in principle extend to the other scenarios in the post. Can we not just say instead (in cases not explained by virtue signalling) that the human novel ability to imagine counterfactual scenarios affords a cheating opportunity, kept within limits by evolution but not completely neutralised, namely enjoying the rewards via imagination? See also (a very dense page of text, I understand only parts of it): picoeconomics.org/HTarticles/BBScoms/BBSMcKayBelief.html
This seems very wrong to me. Visual systems can clearly learn. A veteran spy agency field agent, veteran NFL quarterback, NBA point guard, infantry commander, F1 driver, pilot, whatever, can all see things and read a scene much better after years of experience than they could beforehand.
The problem with your contrived example is the negative hedonic nudge from getting spooked by something that looks like a lion but isn't one is a tiny signal, dwarfed by the gigantic evolved-in prior that things that look like a lion need to be properly identified and avoided. It isn't that the visual system learns any differently. You're just comparing it to realms where signal strength is unclear and there may be no instinctual priors at all. You can throw inputs at a model with randomly initialized weights and watch it very quickly update its predictions, and then throw exactly the same inputs at another model with exactly the same learning algorithm but weights already stuck in a local minimum, and it won't budge at all. The learning algorithms are still the same. Human brains aren't randomly initialized models. For "is that a lion," we have very strong priors. For "when should I do my taxes," we don't.
I think the problem is that we have a very good idea of how you'd train, say, a visual system to be good at seeing things: predictive processing! But that stops working at the high level, and so for behavior-like things you want a different training method. It may well be that the whole brain has one learning algorithm that somehow strikes a balance between the two, but it's not a silly question to consider.
As for all the stuff about confirmation bias? I'm with you on that
In my long running commentary theme of 'missing the point' of what you're talking about.
I.
The 'I'm right about politics and the other side sabotaged our program' is a fairly poor proxy for your current argument because it is true. When you have endless think tanks writing articles about how to sabotage the other side for decades and they brag about it every chance they get along with funding offical and unofficial controlled opposition, abuse of law enforcement, selling political favour for censorship actions by media companies, etc.
All to get their ideas across and suppress the other side's ideas....then it would fall into the rational side of ideas to think rather than the wrongthink side you describe of running away from a picture of a lion on a box of cereal in a grocery store.
When the FBI and republican senators and fundraisers and think tanks and such were all colluding against the civil rights movement on a multi-pronged front and the FBI had a floor to ceiling stack of papers tracking the peaceful anti-war activist, reverend, and civil rights activist MLK to harass him, blackmail him, and eventually assassinate him.
Along with many thousands and thousands of other prominent activists over a multi decade span on time...I'd say it is pretty reasonable to think that one side is using legal, illegal, offical, and unofficial channels and every funding mechanism they have to push their ideology.
When you go through the events on the day of Malcom X's assassination and you see how the FBI had contact and was tracking them and how they and the police uncharacteristically pulled their protection/monitoring team from the event hall that day when they'd been showing up in force to all his other events for months....it doesn't take a genius to see the kinds of sabotage you've dismissed as poor quality thinking.
If that's what they were doing then...what is happening now? Does anyone think that after Snowden exposed the NSA etc. that their spying was actually reduced? Nay, it has certainly advanced to a degree far higher and more sophisticated than what they could do before as technology has progressed.
The idea of GOP sabotage of government is so well understood that it has existed in parody for decades. When Bush appointed anti-UN extremist Bolton to the US's top UN post or put his veterinarian etc. in charge of a large bureaucracy....or when Trump literally failed to bother appointing new heads for hundreds of offices of government causing a general slowdown and disruption, etc.
Or when shows like Parks and Recreation can parody the very widespread right wing rural middle aged white guy's genuine perspective to 'starve the beast' and intentionally mismanage the bureaucracy in order to show how bad it is....what is a person to think? That they're being an irrational fool?
II.
On the main subject of the post and it's intended topic, I doubt there would be any physical element of the brain we could identify in terms of when our ideas and learned reactions to the world are correct or incorrect. This is because 'reality' is not represented in the brain - there is no 'truth' there against which our ideas form around or avoid in proper or in proper brain processes.
A large part of this is somewhat random in terms of what any individual experience and learns....not to mention that in evolutionary terms the main thing that is causing this confusion with lions in your example is about representational depictions of reality. In our history as a species if you saw a lion it was probably a lion, not a realistic image of a lion.
If the occasional child got a bit frightened at night as the firelight flickered on the cave art depiction of a lion...well that too is and advantage as you WANT to instil a generalised fear of lions in the children as you pass on the stories of your people and their knowledge of the landscape and how to survive. If a tiny handful of people end up 'misfiring' in their learning process in that context, then it is not going to be very harmful very often and evolution will have little to say in terms of selecting brains for or against any strange learning errors which pop up.
Evolution has the tool of fecundity and death. Expose them all to reality and let god sort em out to paraphrase the more common idea of kill em all and let god sort em out phrase.
Having a collection of data points that suggest some conspiratorial behavior exists in reality is different from believing in a specific, perhaps overbroad, conspiracy theory.
Would you reject out of hand the idea that multiple possibilities for explaining known facts exist that are not disproven, and even multiple conspiracies could coexist in reality?
Furthermore, to the extent there is one or more big conspiracies that are durable and effective, the information you get is more likely to be cover or chaff.
The perspective isn't primarily neurological, but if you haven't already read The Righteous Mind by Jonathan Haidt, that has a lot to say about motivated reasoning surrounding things like politics to the extent that politics often involves moral commitments and involvement in a community defined by its moral commitments.
A very poor explanation is something like poliltics = religion because they both define communities of people with similar values, and saying your party's policies don't work is a lot like saying that your religion's sacraments don't work.
The underlying intuition here about reinforcement learning is incorrect.
> Plan → higher-than-expected hedonic state → do plan more
No, it's: higher (relative to other actions) hedonic future state *conditioned on current state*. The conditioning is crucial. Conditioned on there being a lion, RL says you should run away because it's better than not running away.
It gets tricky with partial observability, because you don't know the state on which you have to condition. So instead, says RL theory (not so much practice, which is a shame), you can condition on your belief-state, *but only if it's the Bayesian belief-state*. If you're not approximately Bayesian, you get into the kind of trouble the post mentions.
But being Bayesian is the RL-optimal thing to do. You get to the best belief state possible: if there's a lion, you want to believe there's a lion, litany-of-Tarsky style. The visual cortex could, in principle, be incentivized to recognized lions through RL.
I suspect people don't open IRS letters not because their RL is fundamentally broken, but because their reward signal is broken. They truly dislike IRS letters, and the pain it causes to open one is truly more than their expected value. People probably also underestimate the probability and cost of a bad IRS letter, but that's due to poor estimation, not poor deduction from that estimation.
Perhaps it's easier to see in organizations, where you can tell the components (individuals) apart. It's sometimes hard to tell apart the bearer-of-bad-news from the instigator-of-bad-news. This disincentivizes bearers, who might be mistaken for instigators. With enough data, you can learn to tell them apart. Until you do, disincentivizing bearers to the extent that they really could be instigators is the optimal thing to do.
Regarding the lion, isn't the issue that few agents ever experience negative utility from not running away from it? Seeing as you die immediately after, there's no time for reinforcement learning to do anything.
Everybody does some amount of motivated reasoning, but there's probably some systematic variation in propensity to motivated reasoning and it's probably possible to measure that and from that infer some ideas about the evolutionary tradeoffs involved. I slightly suspect it comes down to whether your ancestors won more of their bread by dealing with things or dealing with people and the latter would relatively increase motivated reasoning -- so a verbal tilt should correlate with motivated reasoning. Less motivated reasoning may have made great^20 grandpa a more competent farmer, but more motivated reasoning may have made him less likely to reach taboo conclusions and be shunned or burned as a heretic.
1. the environment is different -- people aren't getting burned as heretics so much anymore. They just get kicked off of twitter and fired from woke corporations.
2. Value discordance. We value truth more highly than the blind idiot god of inclusive-fitness-maximizing.
3. Evolutionary restrictions. The brain is such a rube-goldberg kludge that it's not clear evolution could have eliminated motivated reasoning even if that were strongly beneficial to inclusive fitness.
> but I am neither a deity, nor a human that has designed an AI, nor do I have the least interest in manipulating other people, or intelligences, or indeed myself.
I mean, to each their own, but the link mostly asks interesting questions about how minds work. Being interested in that sort of thing doesn't require being a deity, AI engineer, or manipulative.
I swear I'm not just trying to argue you into liking an article, but now I'm more confused. I assumed you were referencing that, but did you think it was intended as a how-to guide? Or is this a principled stance against thought experiments that couldn't be performed as actual experiments by mortals? Again, if you don't find the subject interesting you don't find the subject interesting, I'm just confused.
Perhaps "action planning" is the primary domain of reinforcement learning, whereas perception is primarily achieved via self-supervision / free energy minimization / etc?
... and "turn my head 45 degrees to the right" perhaps isn't an action plan but rather part of free energy minimization.
Like others in this thread, I also wonder if there are different scales of reward-prediction-based learning.
Dopamine mediates all sorts of critical brain functions, including the ability to initiate and cease movement (thinking about Parkinson's). I assume that means that dopamine plays a broader role than just reward prediction, but OTOH could "initiation and cessation of movement" be thought of as micro reward-prediction-based learning?
Is there a reward system scale equivalent to "edge detection" in the visual cortex? I.e. what is the simplest unit reward mediated activity of the brain? If that is similarly low-level, then the question might not be "what parts of the brain are reinforceable vs not" and rather "what parts of the brain are reinforceable AND implicated in higher order thinking of which we're cognitively aware?"
This is perhaps the most hotly debated question in the neuroscience of reinforcement learning right now. Is dopamine always some kind of (generalized) prediction error, just acting on different state representations and/or different timescales? Or is it doing fundamentally different things in different regions? I lean towards the former, but it's very far from settled.
Perhaps this is far too basic, but what if we look at decision-making algos being sorted into "survival" vs. "thrival," where the surviving decision tree overrides the thriving decision tree?
So, turning one's head when they see a yellow blob in periphery might threaten ruin one's hedonistic state (anti-thriving), but since surviving to see more hedonistic days vs. less overrides hedonistic/thriving decision tree?
Thanks Scott, really good discussion. The idea that the brain uses two types of reinforcement learning (Epistemic and Hedonic) is really, really important. It has deep implications for the structure of the brain, and the ways we should try to model it. What you call the epistemic network reinforces associations based on correctly predicting the outcomes of agent actions, even if those agent actions have negative outcomes. Hedonic reinforcement works as a controller of behaviour, reinforcing possible agent actions based on predicted hedonic states associated with the epistemic outcomes. To model it, you have to see them as two separate networks working in tandem, one caring only about the (Bayesian approximate) accuracy of its' predictions, and the other caring only about the hedonic consequences of those predictions.
Your point, which had never occurred to me, is that this implies there will be cases where agents will actively avoid seeking new information where experience has taught them that this would have negative hedonic effects. It's a real insight I think. But if it's true, and I think it is, then you have to assume that the epistemic/hedonic dichotomy is an architectural principle of all animal minds.
Another interesting thing is that when you try to model a reinforcement learning neural network that operates according to this epistemic/hedonic architectural principle, it's very difficult, and you end up with something that doesn't look much like the sort of models I see in the ML literature.
The way I think about it is there are 4 levels of feedback loop that biologicals systems use to process information.
Level one is evolution itself: does it kill you or not? individual organisms don't process this, but life as a whole gains information this way.
Level two is reflex: You have various sensors for light, chemicals, touch etc, and you automatically react by like, flinching away from pain or a dangerous chemical stimulus.
Level three is reinforcement learning, to either gain reactions to stimuli that are non-reflexive, or to gain control over reflexes and not react. This is like when you learn to like a flavor because despite the bad taste, it has provided you with good nutrition in the past, or something like that.
Level 4 is memetic and abstract. This is having an information system that is able to learn from communication, observation, and reasoning, to a greater or lesser extent. It enables us to hear phrases like "If you go into those woods, a tiger will eat you"
Each of these plays into lower levels given enough time and evolution. Sufficient repetition of abstract memes will train you into automatic responses to certain stimuli even if you have no actual experience, eg flinching away from poisons that you have never tasted yourself.
Things like paying your taxes stays extremely abstract for most people and thus is not particularly good at compelling action.
Re the visual cortex part - in humans, there’s cortisol to help out - if you see something scary enough, the cortisol faucet goes on and later recall becomes more difficult/memories harder to access. That seems to me like the biochemical cheat for the situation described - if the visual cortex does sometimes learn by reinforcement, but in situations of heightened threat that would clear the “don’t look” bar, cortisol makes the cortex not retain it.
That is to say, there are two different kinds of "module" that are relevant to this problem as you described, but they're not RL and other; they're both other. The learning parts are not precisely speaking reinforcement learning, at least not by the algorithm you described. They're learning the whole map of value, like a topographic map. Then the acting parts find themselves on the map and figure out which way leads upward toward better outcomes.
More precisely then: The brain learns to predict value and acts on the gradient of predicted value.
The learning parts are trying to find both opportunities and threats, but not unimportant mundane static facts. This is why, for example, people are very good at remembering and obsessing over intensely negative events that happened to them -- which they would not be able to do in the RL model the post describes! We're also OK at remembering intensely positive events that happened to us. But ordinary observations of no particular value mostly make no lasting impression. You could test this by a series of 3 experiments, in each of which you have a screen flash several random emoji on screen, and each time a specific emoji is shown to the subject, you either (A) penalize the subject such as with a shock, or (B) reward the subject such as with sweet liquid when they're thirsty, or (C) give the subject a stimulus that has no significant magnitude, whether positive or negative, such as changing the pitch of a quiet ongoing buzz that they were not told was relevant. I'd expect subjects in both conditions A and B to reliably identify the key emoji, whereas I'd expect quite a few subjects in condition C to miss it.
By learning associates with a degree of value, whether positive or negative, it's possible to then act on the gradient in pursuit of whatever available option has highest value. This works reliably and means we can not only avoid hungry lions and seek nice ripe bananas, but we also do compare two negative or two positives and choose appropriately: like whether you jump off a dangerous cliff to avoid the hungry lion, or whether you want to eat the nice ripe banana yourself or share it with your lover to your mutual delight. The gradient can be used whether we're in a good situation or a bad one. You could test this by adapting the previous experiment: associate multiple emoji with stimuli of various values (big shock, medium shock, little shock, plain water, slightly sweet water, more sweet water, various pitch changes in a background buzz), show two screens with several random emoji, and the subject receives the effect of the first screen unless they tap the second. I'd expect subjects to learn to act reliably to get the better of the two options, regardless of sign, and to be most reliable when the magnitude difference is large.
For an alternative way of explaining this situation, see Fox's comment, which I endorse.
OK, now to finally get around to motivated reasoning. The thoughts that will be promoted to your attention for action are those that are the predicted to lead to the best value. You can roughly separate that into two aspects as "salience = probability of being right * value achieved if right". Motivated reasoning happens when the "value achieved if right" dominates the "probability of being right". And well, that's pretty much always, in abstract issues where we don't get clear feedback on probabilities. The solution for aspiring skeptics is to heap social rewards on being right and using methods that help us be more right. Or to stick to less abstract claims. You could test this again by making the emojis no longer a certainty of reward/penalty, but varying probabilities.
Source: I trained monkeys to do neuroscience experiments.
"Maybe thinking about politics - like doing your taxes - is such a novel modality...." ???
Dunno, chimpanzees and all the other non-solitary primates seem to be quite keen on politics, so this modality must have been around for the past 15 million years, or even longer, if you are willing to stretch the meaning of the word. Richard Wrangham's "The Goodness Paradox" strongly supports the idea that we have a very fine, evolved grasp of politics. It's not about being happy or conformist but yes, being savvy enough to viciously cut the other guys off at the knees at just the right moment and yet look quite saintly while doing it, is just what Mr Darwin ordered. The life of Aspies was always precarious on the veldt.
I dunno. These models seem so widly oversimplified trying to tease out the logic strikes me as about as productive as Aristotle trying to theorize about how to improve metallurgy using only the idea that everything is made of Fire, Air, Water and Earth in varying proportions.
I mean, you've described feedback structure that could be implemented with a dozen transistors. What are the other 79,999,999,988 neurons for? Not to mention even the feedback structures we know at the most basic biochemical or sensory processing level are already far more complex. We've got delayed feedback, feedback that is added and subtracted and differentiated. We have edge and motion detecting feedback circuits. We have feedback that is integrated, often with nonlinear kernels, and for all we know there's FFTs being taken and all kinds of feedback circuits operating in transform space, too.
That's not to even mention the biggest elephant in the room, which is that we have no idea what the intermediate metrics of cognitive success actually are. Sure, optimizing survival of the related gene group is the ultimate metric of success -- but how does that translate to more immediate measures of optimal individual decision and action, the kinds of things that can be optimized by an individual brain in real time? The fact that the bookstores and libraries groan with pop-psych self-help books, and pro libraries under slightly more polysyllabic versions of same, is pretty clear evidence that we have at best a tiny handful of scattered clues.
Meh, I agree with Carl here. Your model of the brain amounts to "it does reinforcement learning based on the outputs of some blackbox" and then dump 99.9999999% of neurons into the blackbox. But even this super-simplified model seems wrong, so you amend it to "only some parts of the brain do reinforcement learning, and the rest is total blackbox".
I don't think this is useful; I don't think it provides real insight rather than a fake feeling of having insight. What prediction about the brain can you make now that you know this theory that you couldn't make before?
I suspect you're factually wrong about that. I think you attribute way too much challenge to simple object segmentation of the world, which even very small brains in very primitive animals can do with ease, and way too little to the subtle agency decisions which human beings appear to spend 98% of their time and brainpower struggling to get right -- all of which involve exactly this complex web of feedback between remembered decisions and outcomes which lies at the heart of your issue.
I think it unlikely a priori that our vast painful existential struggles are the result of the last 0.000001% of the neurons struggling to finish the last 0.000001% of the general cognitive task. I rather suggest objectively taking the measure of the world through our senses is the *easy* task, occupying a very modest fraction of our brainpower, and deciding what we can and want to do with that model is the hard task. That is, I take the fact that most of us experience the bulk of our brain wattage being used to figure out what we should do next at face value -- that it's saying *that's* the hard task, the one that takes tens of billions of neurons.
Which suggest *whatever* the network that links our prior experience to our current decisions, it is something that requires billions of neurons to implement. So it's not something that can be plausibly modeled by a handful of logic gates or a flowchart that fits on one 11x18 page. Indeed, if I had to guess, I would guess an accurate model is extremely difficult to even understand -- since if it *were* less than nearly impossible to intuit, we would have done so long ago and be talking confidently and precisely about how to build intelligent awarenss from biochemical transistors as simple as a neuron (so long as you have enough of them), instead of waving our hands vaguely and incoherently about what "intelligence" and "awareness" and "intention" even mean.
Now that I think about it: another piece of strong empirical evidence is that we can *already* program computers with far simpler computing machinery than we have between our ears to "categorize things and figure out which things cause other things" (or at least are well-correlated). Doing exactly that -- to whom does this face belong? What is its age? If our user said X and Y just now, what is the next word Z going to be? -- are among the most salient modern triumphs of machine intelligence.
But having idiosyncratic self-generated goals and purposes, having intention, making decisions -- things that even rodents can do -- we have absolutely no idea how to duplicate in silicon. Unless one wants to invoke strange conspiracy theories or weirdly selective idiocy, that's pretty reasonable indirect evidence that having the experience of agency and successfully exercising it is the hard (programming) task, and just recognizing and categorizing patterns in reality is the easy one.
I disagree. If there is any lesson to be learned from machine learning, then it is that a large number of neurons is really helpful, both for supervised learning and reinforcement learning.
And for your point that a lot of tasks can be done by small brains, it also misses part of the truth. The number of neurons in animals is not just determined by intelligence, but also by body mass. An elephant has more neurons than a human. A lot of large whales have more than 10^10 neurons in their sensory areas. The statement "perception and classification can be done with few neurons" is not wrong, but evolution ignores this beautiful truth and uses lots of neurons on these tasks if the animal can afford it.
Well, in the first place, I don't think there are any important lessions to be learned from machine learning, because there's no evidence it has any relationship to real learning. There's not much you can learn about how to build high-performance airplanes by watching hawks, because the two devices solve similar problems in very different ways. So far as I can see, it remains to be proven that human brains and neural networks using TensorFlow have any useful relationship to each other.
Secondly, I suggest even computers argue against your point that more is always better. Which is better, RISC or CISC? It's not obvious. Which is better, more processors and parallel programming, or boosting scalar speed of fewer processors? Also not obvious. Which is more efficent among human organizations, small fast-moving groups that make a lot of small mistakes, or large complex groups with many layers of error-correcting management that only make great big mistakes? Et cetera. Basically, being large introduces problems of communication and organization that can overwhelm any gain from dividing the workload more finely.
And finally, we don't know what all the neurons in a whale do. Indeed, my impression is that the estimate of the social interactions and social intelligence of whales have steadily risen over the decades. They may need bigger brains because they have far more complex social interactions than, say, Diplodocus did (famous for having a huge body and a small brain).
I’ve always thought of that motivated reasoning as a way to economize compute power. Your brain says: Something bad may happen, I can’t do anything about it, my avoidance of detecting it doesn’t much change the odds of it happening, so I’m not going to think about it because if I’m going to have any hope I need to explore other stuff. The bit about not wanting to see the lion because it will give you a bad day I think only practically works if the lion future is competing with another future with a higher goal. “The tribe is starving. Janice remembered gathering some berries by the river a ways back. She took our remaining food to give her strength to run. There are lions there now and she may get eaten, but if she doesn’t gather the berries and remember exactly where they were to get back in time the tribe dies.” So the lion looker module gets downgraded against the berry finder module.
But at the edges where these things are fuzzy, as they usually are in real life, with lower stakes and overlapping error bars, this probably becomes procrastination/willful ignorance.
When I see a picture of a large spider, i get really creeped out, even though it is just a picture. Isn't this a lowering of my hedonic state? If I close the book with the spider picture, it gets removed from my visual cortex, thus causing my hedonic state to increase. The same is true if I saw an actual spider and freaked out and ran away. Running away removes it from my cortex, increasing my hedonic state. Why does this need to be reinforcement learning, isn't it just an evolved instinctual reaction to scary animals?
I usually prefer "reinforcement learning" to "evolved instinctual reaction" because instinctual reactions are really really hard and there aren't enough genes and brain structures to store all of them. My guess is evolution is under strong pressure to minimize the number of instincts and use reinforcement learning for whatever it can.
i think there was also a post at some point about fear and reinforcement ( i think about dogs)... even if being scared of the lion is learned, doesnt running away still reinforce, because you are removing the fear. isnt your day *saved* and not ruined by running away? you no longer have to be in the presence of the lion/dog
As a neuroscientist studying reinforcement learning, I feel like I ought to have something for you here. As you know, we broadly think that reinforcement learning in the brain is mediated by dopamine. So one way of framing the question "which parts of the brain are / aren’t reinforceable" is "which parts of the brain do / do not receive dopamine input." Interestingly, the brain regions with the most dense dopamine innervation are the striatum (input nucleus of the basal ganglia, dealing a lot with movement and motivated behaviors) and to a lesser degree the prefrontal cortex (dealing with executive function / decision-making). Notably, sensory cortices do *not*, by and large, receive dopamine input. It strikes me that this relatively cartoonish diagram of the dopamine system maps on quite well to your intuition that "behavioral" regions should be reinforceable and "epistemic" regions should not, with executive regions somewhere in between.
"But suppose you see a lion, and your visual cortex processes the sensory signals and decides “Yup, that’s a lion”. Then you have to freak out and run away, and it ruins your whole day. That’s a lower-than-expected hedonic state! If your visual cortex was fundamentally a reinforcement learner, it would learn not to recognize lions (and then the lion would eat you). So the visual cortex (and presumably lots of other sensory regions) doesn’t do hedonic reinforcement learning in the same way."
I think this logic is wrong. Standard RL algorithms learn to maximize *long-term* summed reward, not just the reward from the immediate next state. (If they only learned to maximize the immediate next reward, they'd be awful at most tasks.) So a visual cortex that learned via RL would learn to recognize lions because, even though it leads to a worse state immediately, it avoids a huge punishment down the road. I agree with you that sensory cortex isn't implementing RL, but this is not a good argument for why not.
Rather than a purely "is reinforceable" vs "isn't reinforceable" distinction I suspect the difference has more to do with the relevant timescales for reinforcement. In the foot injury case, we'd have a very fast gait correction reinforcement loop trying to minimize essentially instantaneous pain. In the lion country case it sounds like something slightly longer timescale -- we make a plan to go to lion country and then learn maybe a few hours later that the plan went poorly so we shouldn't make such plans again. In the taxes case it's much longer term, it might take years before the IRS manages to garnish your wages, though you'll still eventually likely get real consequences. Politics on the other hand, often cost/reward is so diffuse and long-term that I suspect the only reason anyone is ever right about difficult policy issues is because the cognitive processes that correctly evaluate them happen to be useful for other reasons. The vision example I think is a mistake of timescale; a vision system which learned to not see something unpleasant would get a much worse net reward when you don't avoid the lion you could have seen and subsequently get mauled.
I'm coming at this from the ML side so I'm out of my depth biologically, but perhaps we have different relevant biological RL processes with different timescales? Eg, pain for ultra-short timescale reinforcement, dopamine for short-to-medium timescale reinforcement, and some higher-level cognitive processes for medium-to-long timescale reinforcement.
One observation: You can't avoid a lion trying to eat you by refusing to look at it. But you might be able to avoid another lecture from your neighbor Ug Bob about how you haven't made the proper blood sacrifices to Azathoth in a while, if you refuse to make eye contact with him and keep walking.
That is to say, huge parts of our brains developed in order to process social reality. And social reality, unlike physical reality, actually does change based on what information you have (and what information other people know you have, or what information you know other people know you have...). So controlling the timing of when you are seen by people around you to acquire certain information likely does have some degree of survival benefit. And the parts of our brains that learned how to do that are probably the ones that are involved in reading letters from the IRS today.
I think the major problem here is that from an individual perspective the "system" you are learning about is so frigging noisy and generally large/complex that "the answer" to any given scenario is only valid across a large population. Maybe we can slowly increase the percentage from say 60 to 70 to 80 to 90 to even 95 some day but 5% of all people is a catastrophically large number.
Perhaps the visual cortex is more of a reinforcement learner than you're giving it credit for. Children have to be taught to look both ways before they cross the street. I know that when I started working in a lab, highly-significant translucent specks in a vial of water didn't stand out to me the way they do now. In the lion example, as soon as the yellowish blob was seen - or even as soon as the hunter stepped out onto the Savannah where they knew lions existed - they'd be having worse anxiety not checking if the blob is a lion than they would if they looked directly at the blob.
It takes a lot of learning to carve out significant features from our visual field, build habits about what to pay attention to and what to ignore, and learn to coordinate our bodies in order to execute these sensory routines. That learning seems to happen via reward mechanisms, either self-administered or delivered by others.
In the lion case, it seems like anxiety is the reinforcement learning that forces people to look at the yellow blob. So how do people come to have that anxiety? I suspect it is socially reinforced. Children are punished for being careless and rewarded for being perceptive, and warned about specific dangers until their brain punishes them with more anxiety for carelessness than for caution.
I notice that with the taxes and email examples, these are often activities done in solitude, and where the direct dangers afflict only adults. It may be that there's not enough social reinforcement to instill sufficient anxiety for not doing one's taxes or checking one's email in order to overcome the natural unpleasantness of checking Quickbooks or gmail. Therefore, a whole class of problems becomes a source of akrasia.
Another explanation for these two examples in particular is that it might be that the threat of bankruptcy or losing one's job isn't actually as bad as it seems. Perhaps people are, in effect if not in conscious intention, checking to see if they can keep their job without checking their email, or somehow get out of paying their taxes (perhaps long enough to see a bump in their income).
So maybe we don't need to resort to a whole alternative brain architecture for special cases where reinforcement learning seems to predict some unrealistic behavior. Maybe instead, it's simpler to assume reinforcement learning, and figure out the simplest explanation for how reinforcement learning might produce the outcomes we observe. Then you could test this explanation empirically, and try testing several iterations of more complex explanations. If you try hard and fail to find an empirically verifiable reinforcement-based explanation for a behavior that seems contrary to the 100%-reinforcement hypothesis, then you start thinking that there must be some alternative brain architecture.
Humans don't have "decision-making algorithms." Humans are not computers. It's really embarrassing to read stuff written by people who don't understand that.
You should really read The Master and His Emissary. It does a lot of deep diving and presents many satisfying approaches to answering the question you have. The author is a psychiatrist and also has a great deal of expertise when it comes to studying the brain physically, both in humans and in monkeys. This makes him able to use hard science to approach the issue while still recognizing that humans are not computers. It's an incredible book.
Hi Scott. You might be interested in the works of Robert Trivers and Hugo Mercier on the hidden role of persuasion in shaping reasoning. My colleague Moshe Hoffman and I also address this in a chapter of our upcoming book. Lmk if it’d be helpful for me to send further details on anything.
Trivers was hugely influential in my own thinking about this topic. Perhaps people here would benefit from a little explanation of what he says. And what's this book?
Oh, and the TLDR: People are often motivated to persuade others of something, regardless of the truth. They're more convincing when they themselves believe that thing (for instance, because there's no risk of accidentally revealing the truth). This is the Trivers story, which the others touch or build upon. In our case, we add a bit of game theory to the mix.
Uh, it's an additional point, rather than a counterpoint. There are various drivers of motivated reasoning. I think that persuasion is one very important one--probably the most important one--and wanted to add it to the discussion.
(FWIW, ATM, my guess is learning is probably not an important driver, though it was an intriguing suggestion, and I could easily imagine being convinced otherwise. I think I'd want to see more evidence that learning can explain features of motivated reasoning that persuasion and/or standard social psych stories cannot.)
This seems like a back-propagation error, or rather, back-propogation seems like the mechanism which decides how much to update “avoid looking for lions” vs “avoid lion-infested savanas” vs “carry a pointier stick”.
There was a paper posted to lw recently showing that predictive coding can implement back-prop, maybe this is a place where evolution's handy trick to simulate back-prop using local information breaks down.
It does not seem clear reinforcement learning is being "misapplied" as there might be no alternative and human reinforcement learning is simply far from optimal as it does not recurse the decision tree at all like reinforcement learning algorithms, but instead generalizes from memorized valent experiences with a patchwork of adaptive computational fragments like a babbling large language model. For accurately computing the utility of doing the taxes, you'd have to descend very deep in the decision tree until you arrive at the relatively narrow rewarding branch in the far future—a computation that is infeasible in our wetware. So a better title may have been "Motivated Reasoning As Suboptimal Reinforcement Learning Messing Up Our Epistemology".
As much as I wanted to believe this when I first read it (because it seems more parsimonious at first glance than my current model of this sort of thing) I'm going to roll to disbelieve because this model does *not* predict that people will feel better after admitting the thing they're avoiding isn't going to go away, and the ugh field will disappear.
ISTM if the issue were purely reinforcement learning, resistance shouldn't disappear upon admitting what you're avoiding admitting.
Now, is it something that affects AI design? Not going to argue one way or the other. But for humans this seems to be missing whatever element(s) predict the collapse of avoidance once you "admit" that what you're doing isn't working. ISTM the Litany of Gendlin wouldn't do anything useful if knowledge-avoidance were as simple as reinforcement learning.
I'm surprised you count this as a point against. I can't trivially make myself want to do unpleasant things (like check my tax status) by reciting the Litany of Gendlin. I always though of it as more aspirational.
The litany of Gendlin needs application, e.g. asking yourself if not checking the taxes is actually helping, or mentally predicting what will happen if you don't and then it turns out to be bad anyway.
More specifically, my observation both personally and of my clients is that the brain seems to be treating problems as if they can be avoided by avoiding the situation that reminds you of the potential problem. That is, it's treating the negative future as if it's conditional upon your perception or possibly *admission* of the problem as existing.
Or to put it another way, "if I don't admit it, I can pretend it's not true". Breaking this conditional prediction by admitting that if it's bad, it's already bad -- oh wait, crap, that's not the Litany of Gendlin, that's the other one... Tarski?
Anyway -- admitting that the problem is going to be there *anyway* seems to remove the option of "pretend it's not true if I don't look" from the planning subsystem, allowing less-favored but more useful plans to proceed.
I've been teaching something somewhat like this as a generalized anti-ugh-field technique, more specifically involving imagining that you've *already experienced* the ugh you're anticipating, so that avoiding it isn't perceived as an *option*.
Granted, it only works on ugh fields where the ugh is generated by the task itself: i.e. if doing taxes is unpleasant, then imagining it as having happened already (and unavoidable), will get one past it, IME. But if the ugh field is about something tied to completion of the task (or possible failure thereof), then imagining *doing* the thing isn't really going to help, nor will it address meta-issues to the whole process.
But for stuff like, "I don't really want to get up from this warm bed", it works great: just imagine you're *already* cold.
The type of ugh where one is avoiding *finding out* something because of an expected unpleasant answer seems to work the same way, i.e. imagining you already found out it's bad.
ISTM this would not *ever* work, though, if the underlying mechanism was some parts running on the wrong storage machinery, vs. just being simple prospect theory.
That is, the model I'm using here is that prospect theory means we compare plans of action by generating a diff between the expected future and the status quo, and so pretending the status quo already includes the negative update removes the negative ranking on the action producing the ugh.
(The "bug" in the brain here is just that the act of checking the maybe-negative thing is being treated as responsible for the projected negative future state, where in the present state the thing is not-bad-to-my-knowledge and the future state is probably-bad. And people seem to do this a lot.)
So, ISTM that just using known/understood concepts of how the brain rates actions based on predicted outcomes (and how it handles risk and probabilities) is sufficient to generate both ugh fields and avoiding checking on things that might be bad.
To be fair it doesn't rule out your hypothesis either. It just seems more parsimonious, especially since this model works without needing any prior reinforcement: just having information that *suggests* something will be bad will be enough to put an ugh on checking it, after all, even if every other time you checked it, it was good.
For example, checking your account balance regularly for years with no problems, then thinking you spent a lot of money and it might be low: you could generate an ugh field without any specific reinforcement having happened at all. (Unless you count what happens each time you think about it from then on -- but the first time it wouldn't have been conditioned as such.)
"why does the brain so often confuse what is true vs. what I want to be true?"
Are you applying this question to yourself or other people? I'm not able to relate to this question if I apply it to myself, but it seems plausible when I apply it to others.
Introspecting while keeping the intent of the question and applying it personally, I think I would rephrase the question as "why does my lizard-brain/base-animal cognition so often win against my metacognition?"
I think that rephrased question is answered by the basic fact that metacognitively I have no motivation. The reason I do anything is for base animal motivations. I mean, even the reason I use metacognition is from the fear of making a mistake or the gratification of being right.
Using the tax example (and assuming I didn't do my taxes), I think the reason for me not doing my taxes would be that my base-cognition would fear the act of doing taxes to the point of shutting down my metacognition in that domain. My motivation would be shifted to playing video games, data binging, or other distractions (to keep my metacognition from generating possible futures that my base-cognition fears).
I really don't buy the "Then you have to freak out and run away, and it ruins your whole day. That’s a lower-than-expected hedonic state!". You know what's an even lower-than-expected hedonic state? Getting mauled by a lion (or run over by a truck). It's so much lower that in many cases, it's permanently zero. I don't buy that a behaviour that involves *not* investigating potential existential threats to survive long enough. Especially one that is just as, if not even more relevant today than previously (how many lions did a primitive man encounter, and how many trucks could a modern man walk in front of?)
This post made me wonder, given the good logic of having a bad day after seeing a lion, why do we seem to get so much enjoyment from seeing lions (and large animals in general) up close? People love zoos and for a sizeable number the attraction seems sufficient to sink a lot of money into safaris etc. It feels like this is more than just enjoyment of nature's majesty a la an ocean view or a sunset?
Is this some kind of vestigial love of the hunt, at odds with (and perhaps made all the more thrilling by) the countervailing fear of a mauling?
Most people outright don't like spiders, I wonder if we simply haven't evolved the same excuse-making-fascination with them that we have with big cats?
Regarding "not turning the head", I see a big positive value in self-reflection and/or therapy. In a behavioural scenario, you want to avoid bad outcomes. But in an epistemic scenario, you want to say "seeing a lion is bad, but noticing it quickly is positive and I did a good job". Reinforcing this kind of thinking may help people to rescue some trapped priors.
I feel like you’re creating 2 cases out of a single algorithm. Let us say your brain is only optimizing sum of net future rewards (Return). Now if you miss a lion at the peripheral, there’s a chance you’ll die so your brain will focus on that. Once you see the lion, your hedonism will reduce for a short while, but compared to the chance of outright dying the reward is much higher when you spot the lion than when you don’t, so it makes sense that your brain chooses that option. On the other hand what is the net reward for knowing you are wrong and changing your prior. Your hedonism increases a bit because of your internal satisfaction of knowing you are right, but this can be smaller than the decrease for feeling like an idiot. In fact the lessee the chance that that piece of information affects you directly, the lessee the lifetime reward of correcting yourself of that piece of information. This way i don’t think the dichotomy you claim exists
I think this is likely to come off as very naive, but I work in a very different field so please excuse me.
I find this kind of question- where we can think about thought in some way that combines decision theory, psychiatry, potentially artificial intelligence and different models of consciousness, to be like *the most important and interesting* thing- are there any groups of people that do research on this? Could I come at this kind of problem from being a practicing psychiatrist?
I think the most useful search term would be "computational psychiatry", if you have more specific questions feel free to email me at scott@slatestarcodex.com
“ This question - why does the brain so often confuse what is true vs. what I want to be true?”
I somewhat regularly find the need to talk myself out of something. If my dream jobs comes along and it falls through at the last minute, I comfort myself by saying, “That’s a sign that it would have sucked.” And that’s more comforting to believe than that it would have been everything I dreamed it could be. So I go with that.
But I know I’m doing it. Do most people not know “on the inside” that they are doing that?
When someone goes to Destin, FL and not St Barths or they buy a Camry and not a Lexus and say it’s just as good or say their super successful friend isn’t really living his best life - they know (at some level) that they are just telling themselves comforting lies, right?
I've thought for a while that something like this might happen around the "cognitive dissonance" feeling. That's an unpleasant feeling, and you get it more if you look for info that goes against current belief. Thus, this punishment would breed confirmation bias.
There is 2 things that make this more complex imho :
Immediate feedback is much more efficient at reinforcement than delayed one. So people not doing things to avoid pain is clearly happening, but much much more for immediate pain than future one. The more future and abstract the pain (or gain) is, the less reinforcing effect it will have. People will certainly avoid walking on an injured foot because it hurt, maybe not look in a dark corner that may hide a lion, but certainly not hesitate to do their tax in fear of a having to pay more than they could in 6 month.
On the other hand, there is a kind of meta reinforcement system active for abstract / delayed stuff : you directly enjoy (or hate) some mental activities, not directly linked to practical consequences. I constantly delay doing my taxes because I hate all administrative paperwork, not because I fear bad surprises. If anything, it's not doing them /rush them that had lead to bad surprises (and will probably do so in the future)... On the other hand, me enjoying math and logic games certainly helped getting my degrees easily... But i did it because i enjoyed this kind of mental activities, not because i saw that doing more of it helped with my studies...
Delayed reinforcement demand a conscious effort, and doesn't work so well (for me, but i usually have a relatively long time preference so i would be surprised if on average future reward /punishment have a lot of reinforcement effect )
Meta reward where mental activity itself activate the brain reward system seems a stronger effect. Then it remain to see why some mental activities are enjoyed more than others, and how and why it varies across people...
In a post-rational world full of valence-charged facts, it is impossible to have a rational (or even civil) discussion. Being immersed in our post-rational era means we must acknowledge ‘lived experience’ without being able to opine on how it impacts our common goal of ‘equal opportunity for all’ or how we can work as a pluralistic society with differing definitions of “The Good.” Makes me wonder what will happen to the humanities like literature and theatre studies (founded on fiction and using empathy-like mechanisms for exploring differing definitions of The Good), but that’s for another time.
The greatest danger is believing only your opponents are infected with Humanistic Truth. While I hate to admit it, I am basically intuitive, not rational. Just like all people, practically all the time, I make hasty judgments. These are not based on reason. My false reason comes from the hidden operations of cognitive predispositions and a two-track brain. Instead of my mind acting as a judge weighing the facts, it acts as a press secretary seeking to justify my beliefs.
That’s right, “humans are not designed to practice reason. We are designed to make arguments that aim to support our preconceived conclusions, not yours,” The Righteous Mind: Why Good People are Divided by Politics and Religion.
Once a social value has been mapped onto the biological disgust mechanism in social interactions (e.g., valence-charged facts), then it is impossible to have a reasoned discussion about any topic (from racism, inequality, BREXIT, or the election of President Trump). We need a new narrative to allow us to deploy our critical thinking skills and address the growing inequity in our society. I find the best new narratives use allegory and metaphor deployed through humor that must be grasped and not explained.
I think Hansonian signalling/far mode is at work, as usual. People hear you say you believe X. Belief X indicates that you value Y. They value Y, so they socialize with you more. The system that caused you to say X is an adaptation to this. It really just doesn't enter into the question whether X is true.
Suppose someone says "I believe the solution to violent crime is that we need more creativity in public schools" or "I believe the solution to violent crime is that we need more religion in public schools". This does actually vaguely correspond to an empirically testable statement. But that's really far from the point - in a deep sense, nobody cares if it is true or not, they (the speaker and listeners) just care about what it indicates about the person who has said it. If it's false, well, at least this person is right-thinking and therefore someone they want to socialize with.
Does it really ruin your day to turn your head and see a lion? Or does it save the day to be warned against the lion so you can go on living? Does it ruin your day to see that you're not going to make the tax deadline unless you get down to working on your taxes, or are you glad to remember that you can avert a fine? Does it really hurt you to find out that you were wrong about politics, or does it please you to find out why you were wrong and why you can now confidently make a better choice? When I'm learning new things, it's painful, but afterward I'm so glad I went through the pain.
To me the difference between the lion case and the taxes case is something like - how quickly are you going to get feedback on your decision/beliefs? In the lion case, you can't actually avoid learning in short order if there is a lion, because it will probably eat you. In the taxes case, you can avoid it for a pretty long time! Short-term bias is a pretty normal factor in how humans make decisions and it seems pretty applicable here too.
Wait. I'm firmly in "the brain's architecture is complex and heterogeneous and it's probably a requirement for intelligence" camp, but there's no need to assume additional complexity and specialization to solve this particular problem.
Predictive coding framework, as I understand it, says the brain tries to fit expectation (top-down signals from its model of the world) to reality (bottom-up signals from its inputs), and ultimately, to minimize the mismatch between the two. This alone basically covers the "behavioral"/"epistemic" distinction already. Visual cortex tells you that your model is wrong (a yellow blob where you don't expect it), which triggers an action to resolve the mismatch (by checking more closely). That's the only mechanism you need. There's no distinct "hedonic reinforcement learning", there are only "hedonic inputs" from your body, which get processed the same way all inputs do. And there's no distinct "what is true" discovery in the brain, it's literally all about wanting its internal model of the world to be true - and to achieve that, it chooses from different, but potentially equally valid strategies like changing the model, acting so that the inputs change to a better match, or disregarding inputs and substituting its own.
The reason this procedure may cause problems is threefold:
1. That the inputs are imprecise. (Turning your head towards a yellow blob is the right action in clear daylight, but on cloudy night it will likely fail to penetrate the darkness and resolve the confusion, not to mention all the false alarms you'll be needing to process. In terms of detail/clarity/readability, stress, etc., signals from your body are more like the latter case.)
2. That overruling the inputs requires learned experience. (Children do in fact need to specifically learn that closing their eyes does not get rid of the distressing object they see. But we're practicing seeing all day, every day, from the day we are born, we've had plenty of time to find and learn correct heuristics. For taxes, not so much, and for many stressful situations in general, taking your mind off the source of distress and waiting until you calm down is in fact the correct action to take.)
3. That the brain optimizes for the inputs, and not directly for acquiring the best possible model of reality. (So, it will decide to check your taxes if and only if it concludes it will help you reduce your stress. The rational argument that resolving the source of your stress is the obviously correct action to take is merely one of the pieces of evidence to consider, actual practical experience with stress input is another. Say, I'm a depressed ADHD sufferer with a lot of experience in being late on payments, and my learned strategy of "wait for a reminder, hopefully get around to reading it right away, make a single money transfer for the right amount" does in fact reduce stress compared to "search for the bill, check your bank statement to see if you've paid it, manually count how much you're overdue, add penalty/interest, etc. etc.")
I would take issue with your argument at line 1, basically, where you assume -- and it is an assumption, one worth pondering -- that the metric to be optimized is congruence between physical measurement and interior model. That the brain is spending lots of effort trying to make its interior model of physical events match the outside real world.
In the first place, I think that problem was solved by rodents about 300 million years ago, and is something we can pretty much take for granted in the human brain. I doubt there are that many neurons devoted to that particular task. Doesn't mean they don't screw up, but they do it sufficiently rarely that there hasn't been any strong evolutionary pressure on it for the lifetime of our species. We're really, really good at interpreting sensory data and predicting dangerous events in the physical world. That's why even quite modest intelligence people can learn to drive fast on crowded highways or fly fighter jets in dogfights, and readily do as well or better than the best computers with the cleverest programs. It's not like chess at all.
What we've skipped over is tghe question of what the apex task of the brain is. Like all organs, it's selected for the fitness of the closely related gene group (which often but not all the time means survival of the individual). So the question is: what is the most critical and difficult task for the brain that touches on survival?
Your assumption seems to be "modeling the physical world to detect threats." I don't think that' s obvious. In fact, I'd say a much more plausible guess is "modeling the social world of other human beings to detect threats and opportunities." We are a highly social species, and the threats to our survival and prosperity almost always come to us from social interactions, not direct physical threats. The IRS is far more dangerous to us than lions, these days, and that has been true for at least 100,000 years, and perhaps since Australopithecus. (Even if we point out lions are a real problem for early hominids, it is equally true that the single best defense against lions in general is to be a member in good standing of a tribe of other humans with whom you can construct a common defense. You're far more likely to be die young from being rejected by the tribe than being eaten by a lion.)
But if Job #1 of the human brain is to optimize the response of the individual to social threat and opportunity, it is not at all obvious that this necessarily implies what we would typically call accuracy in modeling the physical (or even social) world. We already know there are plenty of functional delusions: each mother believes her own individual child is the best in the world -- the bravest, most handsome, most capable, sweetest tempered -- and this is clearly a very functional delusion. If parents evaluated the worth of their children, versus other children, in as objective a way as a 3rd party or computer, it would be a disaster. The same can readily be said about the perceptions spouses have of each other, about the perceptions of followers for a leader, perceptions of the citizens of the nation, et cetera. We can bring in religion: the feeling that God is watching you is clearly functional, most of the time. We can even adduce survival in highly stressful situations: those who have studied the survival of people under extreme circumstances (prisoners of war or gulags, lost at sea or in the wilderness) note that the survivors tend to share a highly functional delusion: that they *will* survive, even in the face of daunting odds, and that by planning and perservance they can overcome extreme bad luck. The belief in this unlikely possibility is *what* enables them to do what needs to be done to boost the probability of survival.
So if we *assume* most or all of the complex things the brain is doing are necessarily related to accurate prediction in the physical world, and fail to take into account that it might be (I would say almost always actually is) trying to optimize the social interactions of the individual, we are quite likely to go astray in our subsequent reasoning.
People often say things like "Thank God I noticed that lion in time!". I think the premise that noticing and escaping from a lion is a negative experience could use examination.
Expanding a little more, a reinforcement learning system can't be solely defined by the method. The reward schedule is a part of the system that can't be separated from it.
I get a particular kick out of having opinions that contrast to some sort of orthodoxy or are rare. The fitness motivation for rare beliefs is that they have a high potential to be unusually valuable. Everyone says there's a lion to the north but I am motivated and got up three hours early to find out there's five to the south and we should probably go east or west. I want to be that guy
But rare opinions are baseline low hedonic. Being or being percieved to be the only person who is wrong is more bad than being the only person who is right is good
What's my special kick then? I am unusually confident in my ability to be the only right person. Instead of fear of non-conformity I feel the appeal of trying to demonstrate that I have a unique value that could save us from five lions
Regardless of the quality of my actual opinions a key ability I have gained in this respect is to contrast my opinion against people not in my sub-tribe and also against people who are in my sub-tribe. I can direct the motivation to some degree as I choose
Not in my sub-tribe are people who still fear the north lion. But there's also people now in my sub-tribe who agree with me and say "this is why we should keep going north"
Here's where the hedonic motivation loses a lot of steam for most everyone. My desire to be specially correct can also be directed against the "go north" people who agree with me. I'm like "actually with five south lions and one north we should go north by northwest". Now I have something that is prospectively valuable to the whole tribe but is also prospectively agreeable to both sub-tribes. Here's a hedonic (and hopefully practical) north star - finding the rare belief that could be attractive to two opposed sub-tribes
It's notable that this general willingness to be of unusual value is perhaps unprecedentedly reinforced by western culture. We are the red nosed reindeer looking for foggy nights. The underlying hedonic motivator has been overlayed with a large dose of stories about how there is a very delayed and potentially more optimal social outcome
It's notable that Rudolph's situation is not hedonically ideal for him. You can only be so happy. If it's rare you should not typically be able to negotiate routine hedonic losses to reach the rare outcome, even if the group's sum of well being is improved on net. I think it takes some combination of social encouragement plus unconventional personal characteristics to get there as often as you'd like. In my case I have both a very high and low self regard. I'm very confident in my ability to be right in opposition to a chosen group. So I am on average less sensitive to near term low hedonic experiences related to being different. On the other hand I have something like low overall self regard or low expectation of group regard. So I have a lower expectation and ability to experience near term high hedonic group acceptance states that aren't exceptional. I have a lower barrier to the path to unusual prospectively high hedonic states and I have a stronger desire or need to achieve them to feel I have truly gained a secure status within the group
I am not very attracted to the explanation that taxes and politics are novel problems compared to finding lions in the wild. Early humans were probably exposed to politics like situations that posed threats to them: "Grug thinks Bagug is spreading lies about Grug to tribe", "What if red-rock tribe is about to sack our village to take berries?", "What if bagug hunts bigger game than Grug and thus surpasses Grug in social capital within tribe?"(I guess this last one would probably be less eloquently put by our paleolithic ancestors). Not only a bunch of political situations would be present in their lives, given the current data on the levels of violence between tribes of early humans, I think it is fair to assume that those political threats were an even more immediate and lethal to them compared to our current political quarrels. Illustratively it is fair to assume that Grug is much less safe having and Ugh moment about the red-rock tribe coming to sack his village for berries than some college student having and Ugh moment about going out to vote for who he deems to be the most fiscally responsible candidate in his local senate election. For me the much more obvious explanation is that we feel like not paying our taxes in time or losing some election are not problems in the present but in the future. I want to iterate that I am agreeing with the main idea of motivated reasoning as mis-applied reinforcement learning, I just feel like the explanation of why some problems are put in the Ugh category and others are dealt with surges of adrenaline are easily explainable by time preference. I mean, when the SWAT breaks your door after 10 years of unpaid taxes you'll probably not have much time to have an Ugh moment, but those 10 years were really sweet 10 Ugh years.
Love is healthy confirmation bias. We choose a spouse, we have children, we love, protect, and nurture them. If our family life is a source of happiness, we assume the person we married is uniquely compatible as a spouse and co-parent. It's probably not true, but it can't be proven otherwise.
> Motivated reasoning is the tendency for people to believe comfortable lies, like “my wife isn’t cheating on me” or “I’m totally right about politics, the only reason my program failed was that wreckers from the other party sabotaged it”.
I think a large part of the reason people believe things like these is they are crony beliefs, that is, beliefs that one has because they are socially useful not useful in connection with the non-social physical world. <https://meltingasphalt.com/crony-beliefs/>
So if someone thinks their wife isn't cheating on them, they might come across as more confident, which will help them socially.
Of if they have the same political views as all their friends, they won't get ostracised for being a non-conformist.
"But suppose you see a lion, and your visual cortex processes the sensory signals and decides “Yup, that’s a lion”. Then you have to freak out and run away, and it ruins your whole day. That’s a lower-than-expected hedonic state! If your visual cortex was fundamentally a reinforcement learner, it would learn not to recognize lions (and then the lion would eat you)."
I find this sentence quite baffling. I think you are conflating reinforcement with goodness or pleasantness. But I don't see any reason why reinforcement on a neurological level needs to have any kind of emotional valence tied to it. Brain regions are reinforcement learners that are being reinforced to do something USEFUL, not necessarily something that feels hedonicly pleasurable. The visual cortex isn't reinforced to make you see butterflies and rainbows, it is reinforced to turn complex noisy data into compressed, regularized, USEFUL data that other brain regions can do useful things with. Pleasure / goodness arise as a meta-heuristic to direct behavior in useful directions like survival and reproduction. From a biological perspective pleasure is a tool, not an end.
A visual cortex that recognizes a lion and saves your life will be reinforced because that outcome was useful to your biological imperative for survival and reproduction. Evolution doesn't give a damn that it made you feel bad to perceive that lion, it cares that you located and evaded a threat and survived to reproduce.
The Elephant in the Brain¹ explains Motivated Reasoning quite differently, and more disturbingly:
Your brain has many parts that operates semi independently, and many of them are not observable to the "you" that's reading this.
One of them we can call your Moral Center. It makes you genuinely want to do The Right Thing, as defined by some rules evolution probably generated to solve eons of iterated prisoner's dilemmas.
Another part, which has no name, rationally, unsentimentally and subconsciously determines what would be in the best interest for you to do. Sometimes that is to do good things for other people, so they'll reward you. Other times it's to do terrible things to other people so you can take their stuff.
Things like the latter don't sit well with the Moral Center. So your brain has another module, sometimes called the Press Secretary, which comes up with a good enough sounding reason for doing the terrible thing that the Moral Center, which isn't very smart, will accept. So you do the terrible thing, reap the rewards, and feel good about your superior morals.
And that's how at least one kind of Motivated Reasoning works.
You might be interest in a paper by Firestone and Scholl (2016). "Cognition does not affect perception: Evaluating the evidence for “top-down” effects". One of their arguments is similar to yours: perception is essentially epistemic, and so it would be really really Bad Design if cognition was able to influence it in any substantive way.
But the visual cortex doesn't learn to see lions, it learns to *see*. And learning to see -- that is, learning to see *everything*, trees, lions, refrigerators -- could, quite plausibly, be achieved by hedonic reinforcement. Which would come not from the emotional valence of the objects in the visual field, but from the satisfaction of building a good working model of the correspondence between the visual field and haptic space.
I think you might be understating the importance of social relations to this question. None of these calculations are made at the individual level. IOW I can't wonder "is the blur a lion?" without wondering, probably below the level of awareness, what other people think about the blur. If I yell "lion!" and it's not, will people make fun of me and cancel my invite to the reindeer games? Or maybe it's a lion but I know all self-respecting members of our party know that the Great Leader eliminated the lion threat last year. Is it worth yelling "lion!" if I then lose all my friends and allies? (Sure, if the lion is running toward me, but otherwise, maybe not.) This is the root idea of Dan Kahan's work on this, of course: That people value good relations with others in their group far more than they value epistemic accuracy. So there may be a good deal more simple reinforcement happening here, on the order of "say there is no lion"-->all my friends cheer and all my enemies scowl-->feel good.
I claim that you've made a dozen bad life decisions relating to this in the past week, and that several forms of therapy are built on top of it (though they wouldn't frame it in those terms).
The ugh fields post linked above has a lot of examples of bad life decisions. In terms of therapy, more speculative, and I'll write more on it later, but I think a lot of blocks in therapy relate to something like "It is easier and more fun to come up with a theory of why I shouldn't have to feel bad about this problem, than to solve it", which I think is this same idea of reward operating on epistemics instead of action.
That is, if there's a problem (you are poor), and your brain can solve it by either coming up with some kind of theory in which it's rewarding (rich people suck, they've all sold out, my poverty actually makes me a good person) or by working hard to get more money, and your brain is indifferent between getting reward by epistemic changes or by real-world changes, making the real-world changes is going to be a really tough sell. My guess is it's much more complicated than this, but I do think this is part of the dynamic.
Ok, but you didn't say it was "untethered from the physical nuts and bolts of MY daily life", you said it was from "daily life". A reasonable reading of this is that you think it doesn't apply to the daily life of any significant number of people.
Can people really stop feeling something because they have a theory? Or is your model really that people *cannot* stop feeling bad and that's *why* they have to solve the real problem?
Regarding procrastination and ugh fields, here's an interesting and relevant question: Do animals procrastinate? I don't know the answer, but it certainly seems quite possible to study it. For instance, squirrels gathering nuts for winter: Are they at all like students studying, i.e. gathering information, for an exam? Do squirrels, like students, often laze around til the deadline is near, then put out a burst of effort that exhausts them and sometimes does not permit gathering of a sufficient store of the resource in question? Maybe squirrels' behavior with nuts isn't the best thing to study, since the nuts don't land on the ground at a steady rate all fall but instead all drop over a short span of time. But there have to be some situations where animals really do have the option to procrastinate. And if there isn't, we could create on artifically in the lab with rats.
Anyone here know anything about this?
We all know that grasshoppers procrastinate and ants don't.
> The ugh fields post linked above has a lot of examples of bad life decisions. In terms of therapy, more speculative, and I'll write more on it later
If you want to do a post of ugh fields I would greatly appreciate it.
This is the kind of comment that makes me really curious about what a deleted post used to say.
It was just a cleverly worded cheap shot, inaccurately implying this discussion doesn’t have practical value.
I thought your original comment was a little hostile, but I really appreciated your follow-up comments to me (that you've since deleted) and also wanted to reply that I'm sympathetic to what you described as part of your impulse to comment, tho I think having discussions like those in this post is a perfectly fine hobby to have :)
To remove any mystery here, I’ll say that part of it was a feeling of “Don’t these fellas have any real problems?”
I don't think it was a cheap shot. I think your underlying point was important and sound, and it resonated with me. It *is* a little worrying when the best and brightest young folks are seriously debating how many angels can dance on the head of a pin *and* seem at least somewhat unaware that the question is a priori frivolous. One can't help thinking someone ought to say something. Recall the wry popular quote: "If everybody contemplates the infinite instead of fixing the drains, many of us will die of cholera." There's a lot of fixing the drains stuff that still needs to get done in 2022.
If I had to guess, I'd say you regretted that you expressed it in a tone that came closer to contempt than bemusement, which unnecessarily antagonizes and derails your point. Fair enough, but I could wish you had rephrased it rather than eliminated it.
The tone was way off base. I don’t want to ruin an apology with an explanation.
Cleverly worded cheap shots are deletable? I assume that includes sardonic, obscenity-free ad hominem. What a world we live in where judiciously phrased, non-vulgar insults are considered outside the bounds of reasoned discussion. We are all snowflakes now, aren't we?
I’ve been trying to eliminate snark from my conversations. Serves no purpose if you’re seriously trying to communicate.
Gunflint deleted their own comment. I didn't consider it "outside the bounds of reasoned discussion" – I, and several others, directly discussed their comment!
Ok, great. Sorry for assuming.
I'm pretty sensitive to the price of pasta, potatoes, flour, and rice. I'm somewhat aware of the price of meat. But I don't live on milk. It is something I buy rather sporadically, if I get a craving for something I don't eat regularly - rice pudding, or cold cereal.
I suspect that milk consumption in the US is excessive due to PR from the dairy industry driven by...motivated reasoning.
Anyway, I was wondering if Scott has talked to anyone who is diagnosed with schizophrenia recently, and what he's observed or assumes to be their relationship with "motivated reasoning". Sub-question is, do people with schizophrenia go to therapy?
Something I found interesting as a difference in perspective between generations that may get lost in time: I've heard/read people scorn skim milk as unpleasant, watery, and blue colored. At the same time, products of a family farm are idealized as wholesome, rich, organic, unprocessed. But someone I knew that grew up on a farm in the 1940s, didn't like whole milk, considered it greasy and distasteful. Why? Conditioning. On the farm, the fat was always skimmed off, otherwise you wouldn't have butter and cream.
Possibly they didn’t like it because it wasn’t homogenized. Whole milk is actually worse to drink when the fat floats in greasy globs instead of being smoothly mixed in.
I haven't ever seen non-homogenized milk in a grocery store.
Me neither. If he lived on a farm he may have drunk milk from his own cows, and that milk wouldn’t necessarily be homogenized. That’s just what comes to mind when I hear milk described as “greasy.”
You can buy non homogenised milk as a specialty item in a few shops. I've had it.
It's probably got a lot to do with what you're accustomed to. Skim milk is more watery than whole milk, but which you actually like better is subjective. If you're used to a certain level of thickness, different levels are probably going to taste wrong.
(On a related note, sugar-sweetened soda gives me heartburn but diet soda doesn't. After drinking only diet soda for a while, regular soda now tastes too syrupy for me.)
Certainly from the few people I know with schizophrenia, many of them go to therapy - there's a lot of useful technique to be learned re: how to calm down enough to not act on the more dangerous of the unshared experiences, how to survive the anhedonia side effects of the antipsychotics, etc.
>how to calm down enough to not act on the more dangerous of the unshared experiences
I think this advice could be harmful to anyone who takes it at face value. There's an inherent runaway positive feedback loop. If a consumer discloses that they are concerned about anything, then they have conceded there might be a danger, that they aren't sure of the level, and irrevocably put judgement of it in the hands of others who cannot read minds. Sharing with someone who has professional responsibilities to take action, on balance, might be worse even than talking to a lay friend.
I can't imagine any way to fix the dichotomy between suffering alone from an illness and overreaction/misunderstanding other than coming up with an objective way to determine states of mind based on biological markers. If there is one thing I would like to see in my lifetime it's something analogous to blood sugar and A1C tests, only for depression, suicidality, aggression, psychosis, etc.
Obviously it could be used in harmful ways and many (especially paranoid) people would be fearful of a neo-eugenicist movement, but I think one has to come to terms with how harmful and intractable it is to *not* have any objective measurement to base life and death decisions on.
And I do not believe that good treatments can be developed for anything that does not even have a real definition based on measurement.
Doing one's taxes is "untethered from the physical nuts and bolts of daily life"?
You did qualify that your comment was "probably unfair" but I guess I'm still confused why you bothered commenting at all.
What is the connection between locksmithing and going toward strangers in the dark?
I think he's saying locksmiths had to approach strangers in the dark (who were maybe locked out of their car, or are maybe hoping to rob him) and this is a scary action
In a nutshell, for me at the very least, the Catholic faith solves this problem. From a skeptical outside perspective, ignore the deistic epistemology entirely and focus on the phenomology.
One benefit of faith is that it makes your utility function non-monotonic and flexible. It does this by putting a presumably conscious entity at the peak of the hierachy of values (think virtue ethics here). So you get an ultimate determiner of value that is pretty rigid but can decide between competing values. If Christ isn't this for you, well, I enjoy sharing this experience with 1 billion + people
¯\(°_o)/¯
24 hour later edit:
A few responders to this post seem to be operating on an assumption that no one is capable of being very intelligent, well-informed, intellectually honest, and religious simultaneously. While I don't agree with this perspective, even though it may be a strawman, I suspect many of the benefits of this way of thinking could apply to eg HPMOR fans who want to consider the actions of their favorite character rather than the deity I choose to follow. I think good fiction is invaluable, it allows for an idealized and situationally unique perspective, to which one might consider themselves an "apprentice" in a way that they would balk at doing for a real person. I think holding yourself to a higher ideal is generally a great thing, even if it's not mine (and from where I'm sitting, if my assessment is correct an honest and courageous attempt at this will have the same ultimate endpoint one way or another).
(PS I have cried more than one at HPMOR Harry and his interactions with dementors and phoenixes, it's better than the original IMO)
Can you say more here? Having trouble relating this to the post.
I can say more, but I did preface it by saying nutshell. I'll try some additional bullet points for now and if more clarity is needed I can provide it later.
- utility functions that aren't monotonic are susceptible to "Dutch booking" ie a form of exploiting cycles in values
-having a rigid hiearchy of values is basically deontology, which can lead you to ratting out a friend in a Kantian murder mystery because you aren't clever enough to be evasive rather than lie; if lying is always wrong, and saving your friend is usually right given the circumstances, this can lead to ineffective consumption of congnitive resources on intractable problems (also a critique of much utilitarianism)
-if you assume virtue ethics, then you have the company of roughly a billion people also asking the question "what would Jesus do?"
I mean if being tractable is the only thing you care about, why not just base your ethics around something simple, like maximizing the amount of hydrogen in the universe or increasing the amount of entropy? Most people care more about their ethics actually being right than they do about them being easy to follow.
Right, this is why I asked to ignore deistic epistemology. If you want Catholic apologetics I'm not your man, I'll just say my personal process of developing my faith has included a lot of wrestling with God which is also an ongoing process at times.
I agree with Paul above, and I don't see how your comment addresses his point. Also, your initial argument sounded something like "Catholicism simplifies ethics", but this comment sounds like even that might not be the case.
Tractable perhaps, but Catholic ethics is anything but a simplification of the problem. Tbh if you think that I question if you're making a good faith faith effort to understand my position.
As to bringing it back to original point, I suspect having an overarching moral/ethical/ideological system is an extreme benefit to motivational and decisional paralysis, which to me does seem to address the gist of the problem with "motivated reasoning" and reinforcement learners.
Well, if you are in a reflexive equilibria such that choosing a utility function involving maximizing H2 and choosing one that maximizes human happiness are indifferent, then your re probably better off going with the first. The problem is that that is not where we find ourselves, in that we prefer happiness to H2 is there is nothing we really can, not to metion should, do about it. Whereas if having a simple tractable utility function (doing what God says maximizes utility) that still seems to conform to human values (God says to do things I generally agree are good) then that would, initially, seem to solve the problem quite neatly, by being both tractable and ethical.
Of course, there are plenty of other objections to that paradigm (“how do I actually know what God wants me to do here? If God’s depress are isomorphic to what I already think of as good, why is this elaborate utility function necessary, and if they aren’t, then doesn’t that create mor problems?” But I think this particular objection is answerable.
What's the answer then? I'm not asking to be tedious. I don't see a good answer.
My point wasn’t to provide a good answer, but merely to explain why this objection didn’t make sense. I think if you work it out the “Christianity (or any deistic religion with certain qualities) solves certain self referential problems in ethics and motivation” could work, but you would have to do far more work to get there than OP did.
>having a rigid hiearchy of values is basically deontology, which can lead you to ratting out a friend in a Kantian murder mystery because you aren't clever enough to be evasive rather than lie; if lying is always wrong, and saving your friend is usually right given the circumstances, this can lead to ineffective consumption of congnitive resources on intractable problems (also a critique of much utilitarianism
I think you need to be clearer about what rigid means. Utilitarianism doesn't approximate deontology so long as utility is finite, because a sufficient number of specks has more negative utility than torture.
The problem of hell and comparing infinities something I wrestled with for a long time, and has led me to some difficulties considering heresy. I don't think there's a consensus on the topic among more interested, expert, and intelligent theologians than my amateur musings and I defer to them.
I'd be quite interested in hearing your thoughts about the problem of hell.
When I was a christian, I simply didn't believe that hell existed. It just felt completely against my personal experience of the divine. The Benevelent God, whose existence I directly felt and saw, would have never created such a place. It contradicted everything on multiple levels. So I just never even thought about it as a philosophical problem
Later I understood that all my religious experiences was not my communication with God, but a glimpse into my own values and ethics. And this allowed me to think about the problem of hell and be really terrified. Not of the cruelty of God, of course, but of people who simultaneously claim that their ethics comes from their religion and that somebody being eternally tortured for finite amount of evildoing is part of this religion. If ones ethics can accept hell it can justify literally anything.
How do you, deal with the existence of hell? This is the mode of thinking that has never been possible for me, so I'm very currious.
I spent the last few years as agnostic and anti-religion and recently converted to Christianity. One of my big gripes with Christianity was the idea of hell until I came to the conclusion that popular understanding of heaven and hell doesn’t reflect what’s actually spelled out in scripture.
The way I understand it now is that God is Love and Hell isn’t a place, it is the absence of love. God doesn’t send anyone to hell but we instead choose Love or it’s absence.
Closest analogy I can come up with: you’re dating the perfect significant other. You lie, apologize, they forgive you. You cheat, apologize, they forgive you. But you continue cheating and lose the remorse. They call off the wedding and you are stuck for the rest of your life with the agony of losing the perfect spouse. That is Hell.
Important to note that this also means a deeply Loving/selfless Hindu, Zoroastrian, Atheist, etc already knows God and will spend eternity with God
I don't really have any original or personal thoughts on the topic. What I'll say is, for most things I seem to side with Thomists, but on this question I do "dare to hope", since it's the most legitimate seeming option that's available to me.
https://www.youtube.com/watch?v=dmsa0sg4Od4
I'm going to post this here and rely on some security through obscurity with my Google profile, I will come back and delete it once someone comments reminding me to do so (please do this is you've got the link saved). I don't want to share early drafts that I haven't at least presented to clergy, but this is my most sincere answer to how I (currently and after attempting to study the issues, but without any claim to expertise) deal with squaring my faith with my understanding of the world.
https://docs.google.com/document/d/1ItOgM-zI5Ybtp_4TTg4xB23BdNryQhIF-z5hUlX4MxE/edit?usp=drivesdk
> a sufficient number of specks has more negative utility than torture.
Does the majority of utilitarians actually agree with this? I seem to recall that there was plenty of argument on LW on this, and no clear consensus emerged.
I wonder if this also ties in to self-identity. I remember reading about if kids are praised for being smart, they can start to self-identify as smart and so become reluctant to try new things (since they will initially be bad at them, which clashes with their self-identity). Maybe that was bunk but I think I can see it a little bit in my own life.
If we start to self-identify as being right, or smart, it might make it more painful to update our beliefs because being wrong in the past could clash with that identity. I could see the same thing happening with virtue too (although I've read another Christian say the view is more that everyone is a sinner, which if practiced might prevent that).
For now, I try to think of the truth as a model. The world is too complex to totally understand, but we imagine simplified models in order to navigate it and find patterns. It would be unusual to be upset at finding a better layout for a model train set, since in that context the goal is usually to *have* the best model rather than to *be the best at building models*.
I like this train of thought. One thing I've found helpful is to not only think, what would Jesus do? But also, what would my "saint self" do. This approach has pros and cons, but fortunately the Catholic approach makes both methods isomorphic.
Adding another 2c to this:
I once heard someone (I think Leah Libresco but I'm not highly confident in this attribution) describe the "communion of saints" as an attempt to provide cross-temporal bayesian agreement on theological questions. I really like this framing, as it justifies the variety and sometimes imperfect nature of recognized saints. Few if any saints were virtuous their entire lives, and some of the most notable we're quite sinful (a great example here is the apostle Paul, who prior to his conversion persecuted Christians).
Another great benefit, imo, is that Christ was incarnate in a particular time and place, which though his example was indeed perfect, has limited utility in informing people in other contexts. So saints, while not perfect, at least by the end led sufficiently holy lives for the Church to confidently state that they're in heaven, which is a great element for a faith that is living to have as examples.
The "everyone is a sinner" thing is really good for solving that sort of problem. My own experience with Christianity was that it solved a lot of circular reasoning.
Suppose I'm depressed. I could solve the problems that are exacerbating that depression, but then I'd have to think hard about depressing things, which I obviously don't want to do. The actual truth is that, because of the aforementioned debilitating depression, at this moment I do indeed suck and am doing bad things with my life. But if everyone's a sinner and Jesus loves me anyways, then I have motivation to examine my life with clear eyes. No matter what I find, it won't decrease Christ's opinion of me (even though it will temporarily reduce my own opinion of myself). Then I can make the changes that allow me to love myself like Christ does.
Buddha's four noble truths (notably the one about "everything sucks all the time just cause") can achieve a similar end
I'm a little confused -- what does the problem have to do with ethics ? I suppose you could say that the fear of opening your tax book because you know you'll find taxes in it, could be labeled as "sloth", but it's a bit of a stretch.
When you write "and religious", my question is, what is "religious"? It's kind of a cliche that people supposedly like to say they're spiritual but not religious. I don't know if you are using that sense of "religious" or even what the cliche really means.
I'm not religious in the sense of belonging to any community, attending regular services, preferring one type of Christianity, and so on. That doesn't seem directly related to being intelligent or honest.
I was raised by people who I think were essentially atheists but probably would've abhorred the label. I think, but am not sure, that they came from their respective very religious backgrounds and were turned off by hypocrisy and bad behavior of religious people. I had no bar mitzvah or baptism or confirmation or anything. But I was given a Bible and I read a fair amount of it, plus later on stuff by C.S. Lewis, a biography of a saint, etc.
Recently, I've been talking to someone with a Catholic background, and it perplexes me that they seem to not be very familiar with the Bible, to believe in all sorts of new-agey things that seem to me in conflict with Catholicism, but they don't seem to be consciously hostile to it either.
When I started reading about the Inquisition, while I don't accept the *premises*, the theological debate over whether and which magic or astrology is compatible with Christianity makes some sense to me within a closed system. Normal people are oblivious or think of witch trials and/or Monty Python, though, right?
Conversely, contemporary magical beliefs like the law of attraction completely confuse me as to why someone would accept them, and not even see a conflict with traditional beliefs. But I don't think it's normal to analyze things like a Talmud scholar.
I guess where I'm meandering to, is that religious (or Christian) can mean many things, and I'm doubtful that there is even an intersection between what is it logical for it to mean and what it is normal for it to mean.
I'd go with a pretty standard usage: attend church and confession with some regularity, and deferring to the hierarchy on spiritual matters.
There you go, that's exactly what I'm talking about. Sure, that is "standard", but from what I read it excludes most Catholics! And my anecdote is not inconsistent with that.
Also, I notice you don't even include "reading the Bible".
"Most Catholics worldwide disagree with church teachings on divorce, abortion and contraception and are split on whether women and married men should become priests, according to a large new poll released Sunday and commissioned by the U.S. Spanish-language network Univision."
https://www.washingtonpost.com/national/pope-francis-faces-church-divided-over-doctrine-global-poll-of-catholics-finds/2014/02/08/e90ecef4-8f89-11e3-b227-12a45d109e03_story.html
What if it would make your day great to see the lion because you’d run away and then feel like a hero?
I think this is the wrong level on which to engage with the example.
Is this the way to like scary movies? Not “I like to be scared,” but “I like the sequence of feeling scared and then realizing I’m totally safe/heroically safe at home”?
I definitely think part of the appeal of horror (whether film, game, or attraction) is the ability to feel fear in a controlled environment where no ACTUAL danger can happen to you*. Of course, people have their own personal tolerances for this, and some people have their risk-assessment feedback set to the point that even simulated peril is unacceptable.
*Provided nothing goes severely wrong
I've heard this a lot, but never with any evidence I find convincing. I think it might be the kind of thing that sounds nice so no one wants to reject it.
My experience with horror is more like spicy food. It's literally activating pain receptors, but so detached from pain, and so mixed up with a particular sensory context, that the ostensibly bad stimulus becomes more of a unique overtone creating a richer more textured flavor. (Yes, I know some people eat spicy food just to show off their pain tolerance, but I don't think many horror film junkies are like this)
Another interpretation is that humans are actually pretty bad at distinguishing emotions, and so, for example, heightened emotional states are equally confused. Being excited is not that different from being terrified.
See also: roller coasters
This could also be an example of reinforcement. The thinking is: 'You were in danger, and you did something to avoid harm, so you should feel pretty darn good about whatever it was. Here are some endorphins'.
It's similar as with hot food. Your pain sensores get triggered, but nothing bad happen to you. But the pain triggers the fight or flight response, that make you feel awake and fit.
So over time your brain asociate this kind of pain, with the fun feeling of being awake an fit, but no danger, so you start to like the pain.
The "bad day" example is meant as a hypothetical scenario in which a singular bad experience would worsen the lion detection circuitry if reinforcement learning was applied to it.
Right, but it hinges on the detection actually being negative, whereas noticing a danger before it hurts you is generally exciting and stimulating, a positive experience..
It's entirely hypothetical. For this example it does not matter what would likely happen, but what *could* happen. I'd agree that it is not a particularly good example.
I think in the lion-in-corner-of-eye example a lot of people would freeze, which also explains the tax behavior.
It makes sense from an evolutionary perspective, where if a predator doesn't spot you they'll eventually leave you alone, but the IRS doesn't work that way.
arguably the IRS does work that way, but they have more object permanence than your average predator
You made my day. That's one of the things I love about this community. Interesting, funny and (because?) so on-point
Tax advice: if you can’t see the IRS, the IRS can’t see you, so invest in an eyepatch and stop paying taxes.
You can probably avoid the IRS by playing dead. By that I mean, faking your own death and moving to Brazil.
+1 for the freeze hypothesis
The IRS is kind of weird. US law can subject you to criminal penalties for not submitting an honest tax return, but not for refusing to send the IRS money. However, if you *do* refuse to pay them, the IRS does get to take your stuff to get what it's owed plus interest and penalties.
My mailbox definitely projects an "Ugh Field" for me. I know I need to check it, but it only ever brings me junk mail or problems. So every time I think "I should check the mail" another part of me is thinking "Do I have time to solve problems right now? Do I want to? No and no."
Huh, this is an un-realized advantage of not having street delivery and instead only having a PO box. Lots of my packages (which I ordered and want/am looking forward do) got to the PO box, so the only way to get them is to pretty regularly check my mail.
This in no way outweighs the inconvenience of needing to drive 15 minutes whenever I have something I need/just to check the mail, but it _is_ an advantage I suppose.
I get my packages delivered to my office, which makes my mailbox even worse. Not even a chance of a cool package!
Not exactly applicable, but you made me think of this:
Napoleon had a policy of ignoring incoming letters for three weeks. Most things "requiring" his attention didn't actually need him to do anything. Thus, many things just resolved themselves by the time he got around to reading the letter.
I am writing this comment in large part to avoid looking at my inbox.
This reminds me very much of Plantinga's evolutionary argument against naturalism. https://en.wikipedia.org/wiki/Evolutionary_argument_against_naturalism#Plantinga's_1993_formulation_of_the_argument
In case that's helpful.
I find Plantinga's argument strange, in the sense that in most situations, having a belief closely aligned with the observed phenomenon seems clearly advantageous. To my knowledge, systematic discrepancies between belief and phenomenon correspond to relatively rare cases where both (1) it is difficult to determine what is true and (2) there is an inbalance between the costs of the different ways of being wrong (ie. if you are not sure whether A or B is true, and wrongly believing A is much better than wrongly believing B then believing A is better, even if B is slightly more probable).
The crux of the argument is that evolution should plausibly produce adaptive behaviors, but not truthful beliefs. It might be that truthful beliefs are adaptive, but not necessarily so, and evolution would only reinforce truthful beliefs that are adaptive but not ones that aren't. So if there are truthful beliefs that aren't adaptive, can we trust that our minds can find those truths? How can we be sure that truthful beliefs tend to be more adaptive than non-truthful beliefs when the thing we use to make that determination, our mind, is the very thing we're trying to determine the accuracy of? If the problem was that you were not sure whether a scientific instrument was reading accurately, you can't determine that by using the instrument in question alone.
Really the argument boils down to: if the human mind is the creation (regardless of method of creation, could still be evolution) of a rational mind trying to create more rational minds, then it makes sense that the human mind is rational. If the human mind is the product of blind forces that are only aimed at increasing fitness, then we can expect the human mind to be very fit for reproduction but have no reason to believe it is necessarily rational. But if the human mind is irrational, then we have no reason to believe it is the product of blind forces because that belief is also the product of an irrational mind.
It certainly seems that our minds our rational, in that we can understand concepts and think about them and come to true beliefs in a way that, say, a chicken can't. Given that data point (human mind seems able to come to true beliefs), the "designed to be able to do that" model fits more parsimoniously than the "maybe if you select for only fitness you'll get rational minds in some cases." It's not exactly a knock down argument, and you can certainly disagree with it rationally.
I have a vague idea of an argument for why intelligence isn't adaptive past a certain point. Intelligent people are afraid of small probabilities and unknown unknowns. But these are where intellect is least useful, and analysis is most sensitive to assumptions. People who aren't as intelligent and aren't as imaginative, are more likely to just do stuff until they are stopped or killed. That could be systematically better for a population even if not for an individual. Because it explores phase space that is inaccessible to anyone who must *understand* what they are doing to do it.
Suppose there are 1,000 people facing some threat to all of them, and they can each do something with a 1% chance of success and a 99% chance of death. If they're all intelligent, then they won't do the thing until it becomes clear that the alternative is certain death. At which point it might be too late. But if they're fearless and oblivious, and all do the thing, then 10 people will survive, and carry on their risk-taking genes.
Someone smarter than me could put it more rigorously, but I feel like intuitively that analytic pessimists do not take optimal amounts of risk from an evolutionary perspective.
Empirically, evolution isn't producing runaway intelligence, so there must be some logical reason it doesn't increase fitness, right?
This assumes a risk distribution, not too unreasonably, like the one in which we evolved, where it is easy for me to get myself killed, possible for me to get my entire band wiped out, but nearly impossible for me to wipe out all humans or all life. We have changed our environment.
It's still easy enough to get oneself killed, and easier to wipe out humanity, than thousands of years ago. What the modern world has reduced, I think, is the effect of natural disasters that aren't global. And it's increased the individual benefit of being analytical, but, I think, only to a point.
“ easier to wipe out humanity” was my point, though maybe not much of one.
Evolution is slow. Why isn’t intelligence more culturally adaptive? Or maybe it is, and we’re not noticing?
It could also be the case that intelligence does increase fitness but that the cost of a bigger brain outweighs that fitness increase. That would be analogous to the explanation I'd use for why men don't have bigger muscles and why women don't have bigger breasts, despite the obvious fitness advantages.
After thinking about this, I don't understand either:
(1) Why it's "obvious" that those things would be better, or;
(2) How decomposing a reduction in fitness into a small increase and a bigger decrease is meaningful rather than an arbitrary choice applicable to everything.
I can't see any other explanation than sexual selection for the existence of D cups, and I can't see any other explanation than natural selection for the persistence of A cups despite this sexual selection, and to me these things both seem obvious.
If some variation affects fitness through multiple mechanisms, I think that understanding those mechanisms separately will give a better understanding of that variation. Experimentally, those mechanisms can be manipulated separately.
Using a heuristic that seeks truth without trying to filter on adaptivity might be more adaptive in the long run. It’s hard enough trying to figure out what's true. Is figuring out what is adaptive easier?
"Is figuring out what is adaptive easier?" It is very easy. All living organisms do that very efficiently by dying when badly adapted.
I thought that properly, something being "adaptive" means that it spreads through a population over time. Leading to death or preventing biological reproduction doesn't necessarily prevent a progression of changing statistics. I like that way of defining evolutionary fitness because it avoids mixing in human values and remains abstract and general.
I was answering the part about "figuring out" what is adaptive. I thought was misleading, because organisms do not "figure it out" which of course does not in the least prevent adaptation from hapening.
And yes, of course, adaptation optimizes the number of produced copies of genes, and not the fact that the individual bearing these gens die or not, though the two are usually quite strongly related!
"The crux of the argument is that evolution should plausibly produce adaptive behaviors, but not truthful beliefs."
I completely agree with the argument, but I think it has little consequence, in the sense that I expect evolution has produced "reasonably" true beliefs in most cases (and no beliefs at all in many situations in our modern lives!) because true beliefs are generally adaptive.
There are many examples where evolution has produced beliefs that are not perfectly true, because when there is uncertainty it is often better to be wrong in one direction (e.g., predator detection), but I know of no examples of "very false" beliefs constructed by evolution.
"I know of no examples of "very false" beliefs constructed by evolution."
If there were such, they would have to be well insulated from analysis somehow, thus, one would not expect to be aware.
I don't think so. We have many "slightly false" beliefs that are not insulated from analysis at all. For example evolution gave us beliefs that spiders are dangerous, but we are very well able to determine than this is false for many spider species. The beliefs produced by evolution are expectations or emotions, it is perfectly possible to analyse them (though it can of course be difficult to act rationnally even when we are concious that a given belief is false).
I was assuming there's an important semantic distinction between "slightly" and "very". What does "very false" mean, as opposed to "slightly false"?
Yes, "a wizard did it" is a very parsimonious explanation, to a certain type of person.
> It might be that truthful beliefs are adaptive, but not necessarily so, and evolution would only reinforce truthful beliefs that are adaptive but not ones that aren't.
I've seen this claim often, but it's never accompanied by a plausible example that would make it convincing. It's basically asserting that there's no known reason why evolved adaptations would correspond to truth, and thus concludes that evolved adaptations may in fact imply false beliefs.
Not knowing the reason does not entail such a reason doesn't exist, so that argument just doesn't follow though. It's a premature conclusion at best.
The crux of the argument is that divine creation should plausibly produce behaviors that are in line with God's will, but not truthful beliefs. It might be that truthful beliefs are in line with God's will, but not necessarily so, and divine creation would only reinforce truthful beliefs that are in line with God's will but not ones that aren't. So if there are truthful beliefs that aren't in line with God's will, can we trust that our minds can find those truths? How can we be sure that truthful beliefs tend to be more in line with God's will than non-truthful beliefs when the thing we use to make that determination, our mind, is the very thing we're trying to determine the accuracy of? If the problem was that you were not sure whether a scientific instrument was reading accurately, you can't determine that by using the instrument in question alone.
Really the argument boils down to: if the human mind is the result of an evolutionary process which selects for rationality, then it makes sense that the human mind is rational. If the human mind is the product of intelligent design carrying out some inscrutable divine plan, then we can expect the human mind to be very fit for carrying out God's plan but have no reason to believe it is necessarily rational. But if the human mind is irrational, then we have no reason to believe it is the product of intelligent design because that belief is also the product of an irrational mind.
It certainly seems that our minds our rational, in that we can understand concepts and think about them and come to true beliefs in a way that, say, a chicken can't. Given that data point (human mind seems able to come to true beliefs), the "evolved to be able to do that" model fits more parsimoniously than the "maybe God wills there to be rational minds in some cases." It's not exactly a knock down argument, and you can certainly disagree with it rationally.
This is actually a pretty good counterargument I haven't heard of before. At least in the case of evolution you have good reason to believe that reproductive ability correlates with truthfinding. There is no reason to believe an omnipotent entity wants to create rational minds.
The main objection, is that it is at least plausible that a rational mind could be designed by a rational mind: it is harder to see how a rational mind could come about through irrational processes that are not aimed at rationality in any case. You can certainly object that a rational mind might design an irrational mind if they wanted to, but at least it isn't mysterious where the rationality came from if they designed a rational mind.
"My preferred theory is plausible and yours isn't, because I say so" isn't an argument.
I never said that. I said one side was at least plausible on its face, but the other side is a bit harder to see if it is plausible. Do you disagree that a rational mind designing another rational mind is plausible? Do you disagree that it's more obviously plausible than a non-rational process that isn't aimed at producing rationality producing one?
As for the rules portion of his argument, I think Plantinga's argument makes more sense of a counter-argument against 1970s era rules-based AI. Which is fair enough, since that kind of AI is no longer seen as an effective tool with which to model human behavior anyways.
I think Plantinga is a bit binary in terms of labeling things 'rational' vs. 'irrational.'
Don't put too much by my summary: I don't think he'd necessarily use the labels "rational" and "irrational" in his formal argument, I just grabbed them as easy words to use in a short explanation of the argument. Obviously what it means to be "rational" in this sense requires a lot of defining of terms.
Yes, yes! I totally agree! I think that is the usual mistake that mathematician/philosophers do when dealing with living things : they consider that "very slightly false" is identical to false, whereas in biology, the "truthfulness" is better considered quantitatively, with "very slightly false" in practice very similar to "correct".
Yeah, in his actual argument, Plantinga deliberately avoids the binaries of 'true', 'false', 'rational' and 'irrational'. He talks about a type of belief he calls a "defeater": a belief which makes other beliefs seem less likely, but in some other way than by directly disproving them.
It's always difficult to determine truth if truth means correspondence to reality. We can check predictiveness and usefulness but not correspondence.
I go with structural realism : predictiveness and usefulness probably imply some kind of correspondence.
https://plato.stanford.edu/entries/structural-realism/
I'm thinking something like: This post takes first steps towards some kind of taxonomy of neurological skepticism. The Plantinga argument relies on (something like) reinforcement learning governing the whole brain. But if this post is right, that's not the case; the lion country plan part is, but the visual cortex isn't. So you could insulate naturalism from Plantingean skepticism if you could securely place it in the kind of thinking not governed by reinforcement learning.
Which I suppose is the whole point of the scientific method: to insulate the truth-deriving system from anybody's personal hedonic reinforcement.
For social creatures who evolved in small groups, challenging the group consensus could create a sort of negative hedonic reinforcement learning. There's little to be gained from sharing your opinion, even if it's true, if everyone else is going to get upset at you for saying it. So, best not think too hard about whether what your community thinks is true is actually true, and just go with the flow.
That would certainly explain a lot.
The Emperor's New Clothes being the canonical example.
I try not to fall too hard into the cliche of "constantly explain complex brain functionality with simple comparisons to something from my field", but with the field in question being machine learning, it's impossible to resist these comparisons, especially as of the last few years. Too many subsets of our functionality seem so analogous to the way many artificial neural networks work!
This example also gets more interesting if we add other fundamental ML concepts in, e.g. learning rates (what magnitude of events correspond to what kinds of belief updates? are there some areas gradients are passed to more strongly than others, and if so what changes and modulates this?), weight freezing (at some point we have to learn to recognize basic objects and patterns - at what point(s) is this most adaptable, and which parts of it are immutable as an adult?), and of course backprop and other hyper-parameters, which are already interesting enough to contrast on their own. This also reminds me of a deepmind paper I saw yesterday https://deepmind.com/blog/article/grid-cells in which they construct an ANN similar to grid cells https://en.wikipedia.org/wiki/Grid_cell
Can I ask a tangentially related question. How often does it become a practical concern that you have to teach your models to ignore some of the data?
There is a relevant concept of "catastrophic forgetting" when fine-tuning a model where it is a serious concern that training on recent data may "unlearn" knowledge gained from earlier data, losing the capacity for generalization.
Another very different but also relevant concept is the idea of "negative class" for classifiers where e.g. if you would train an image classifier to distinguish between different types of butterflies, you might also want to include a class of random non-butterfly images to teach the classifier what images should be ignored.
I think that the decreasing learning rate concept is strongly related to the explore vs exploit tradeoff which has a solid basis in game theory (multi-armed bandit model) and applied in reinforcement learning, and also observed in biology ("you can't teach an old dog new tricks") and changes in human behavior as we age.
I mean, a decreasing learning rate is adaptive (according to game theory) so we should expect to see that in evolved systems, and we do.
I pretty much agree with everything you said.
One of 5 or so places in the brain that can get a dopamine burst when a bad thing happens (opposite of the usual) is closely tied to inferotemporal cortex (IT). I talked about it in "Example 2C" here - https://www.lesswrong.com/posts/jrewt3rLFiKWrKuyZ/big-picture-of-phasic-dopamine#Example_2C__Visual_attention Basically, as far as I can tell, IT is "making decisions" about what to attend to within the visual scene, and it's being rewarded NOT for "things are going well in life", but rather for "something scary or exciting is happening". So from IT's own narrow perspective, noticing the lion is very rewarding. (Amusingly, "noticing a lion" was the example in my blog post too!)
Turning to look at the lion is a type of "orienting reaction", I think. I'm not entirely sure of the details, but I think orienting reactions involve a network of brain regions one of which is IT. The superior colliculus (SC) is involved here too, and SC is ALSO not part of the "things are going well in life" RL system—in fact, SC is not even in the cortex at all, it's in the brainstem.
So yeah, basically, looking at the lion mostly "isn't reinforceable", or to the extent that it is "reinforceable", it's being reinforced by a different reward signal, one in which "scary" is good, as far as I understand right now.
Deciding to open an email, on the other hand, has basically nothing to do with IT or superior colliculus, but rather involves high-level decision-making (dorsolateral prefrontal cortex maybe?), and that bran region DOES get driven by the main "things are going well in life" reward signal.
Offering a clarification here: I don't believe that the IT cortex receives a dopamine burst when bad things happen. The paper you linked to in the Less Wrong post (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC38733/) correctly identifies that IT is an input to and output of the *basal ganglia*, which is a loop from cortex -> striatum -> nigra -> thalamus -> cortex. But that's not the same as saying that IT receives lots of dopamine input (it does not). So there isn't really a problem here in terms of dopamine training higher-order visual areas/rewarding when bad things happen. (The tail of the striatum, and potentially other aversive hotspots, is another question. Those regions tend to drive avoidance and not approach, so it's not really fair to say noticing the lion is rewarding in this case, despite phasic dopamine in these areas.)
Thanks! In my defense I didn't say that IT has a dopamine burst. What I said was, well, maybe it's a bit confusing out of a particular context. So here's the context.
There are a bunch of parallel cortico-basal ganglia-thalamocortical loops, throughout the cerebrum. I think for certain purposes, we should treat "one single loop" as a unit that is trained (by RL or supervised learning) to do one particular thing. See https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.379.6154&rep=rep1&type=pdf . In that paper, referring to their Fig. 5, they talk about "column-wise thinking" as the traditional way of thinking, where you say what does the striatum do? what does the cortex do? Etc. They are proposing an alternative, namely "row-wise thinking": What does this loop do? What does that loop do? And that's where this particular comment is coming from.
It turns out that IT and the tail of the caudate are looped up into the same cortico-basal ganglia-thalamocortical loops—IT is the cortex stop and tail of caudate is the striatum stop of the same loops. Therefore (in this way of thinking) if the tail of the caudate is getting aversive dopamine bursts, that's basically a mechanism for manipulating what IT will do.
You're still welcome to disagree with that perspective of course, but hopefully at least you understand it a bit now.
Yes, thank you for the clarification! Sorry if I misinterpreted your original comment. I'm a big fan of the "row-wise" thinking you espouse. However, I don't think there's good evidence either way for what effect DA has on the cortical sites to which BG loops return. Indeed, I don't think there's a good account for why these are loops at all! Certainly, the easiest way to think about it is that cortex is providing the "state" input to the RL algorithm, and DA is training the weights of corticostriatal synapses to compute value (or threat, in the case of the caudate tail). But there has been interesting work in RL asking how can we harness more information from RPE signals to train better state representations (see e.g. speculation in Dabney et al., Nature, 2020). Of course, in ML, you just backpropagate this information so it's no big deal. In the brain, backpropagation is tricky or impossible, so maybe the BG gets around this limitation by somehow forward-propagating the info all the way around the BG loop. But there isn't really a plausible account of how that would happen either. By the way, this is much more general than just IT cortex; primary visual and auditory cortex project to tail of striatum, motor and sensory cortex to dorsolateral striatum, etc. See Hintiryan et al., 2016 and Hunnicutt et al., 2016 for much more than you wanted to know.
tl;dr I think we don't yet know nearly enough to say what effect, if any, DA has on sensory cortices that provide input to the striatum.
Re. "IT is "making decisions" about what to attend to within the visual scene, and it's being rewarded NOT for "things are going well in life", but rather for "something scary or exciting is happening": I would be interested in anything ": Do you know of any brain systems which relate this to aesthetics (cast as preferences about what we attend to), curiosity, or the fun of problem-solving?
Not sold on the "visual-cortex-is-not-a-reinforcement-learner" conclusion. If the objective is to maximize total reward (the reinforcement learning objective), then surely having your day ruined by spotting a tiger is better than ignoring the tiger and having your day much more ruined by being eaten by said tiger. (i.e.: visual cortex is "clever" and has incurred some small cost now in order to save you a big cost). Total reward is the same reason humans will do any activities with delayed payoffs.
Sounds like a plausible experiment would be something like: have somebody show you pictures of circles. You report whether it looks like a circle or a square (be honest!) Each time you say circle, the person gives you an electric shock. See whether, after long enough, you start genuinely seeing squares.
My guess is this never happens - do you guess the opposite?
I'm not sure why you're bringing up trolley problems. Phil and I have different scientific theories about how the brain works; it's hardly unfair to test them by experiment.
I'd sign up for it. I'm curious what would happen: I don't think I would start seeing squares, but it would be nice if that was confirmed.
What if this reinforcement learning could only happen during a limited developmental window? Would you be willing to sign up your toddler?
I just did the experiment on myself (using "hitting myself very hard" as a standin for electric shocks) and got the predicted result.
I don't think this was necessary (except in order to win this argument) though - I think it's a *good* thought experiment, in the sense that just clarifying what the experiment would be makes everyone agree on how it would turn out. Essentially I was asking Phil "How does your theory survive the fact that in this particular concrete situation we both agree X would happen?" It's *possible* he could say "I disagree that would be the result", but then I would have learned something interesting about his model!
That's the wrong timescale for your visual system to change. Changing so fast would have been unstable. A small learning rate doesn't mean the objective isn't a reinforcement-learning one.
Trolley problems are actually extremely common and can be found almost everywhere - the law, the medical system, engineering. It's probably the most useful thought experiment of all time.
The thing everyone in the world discussed for two years straight ("should we lock down the economy to stop the virus?") was essentially a form of the problem.
> the overwhelming majority of cases, it's just that the person making the analogy wants what they want and presenting the situation as a trolley problem is an effective method by which to emotionally manipulate their audience into agreeing with them.
I don't agree. A feature of the problem is that there's *no* obviously correct answer. Ethical arguments can be advanced for either course of action.
I would totally sign up for it, as long as the shock was non-fatal and left no permanent damage. Color me abnormal, I guess.
I take the unrealistic aspect of a trolley problem to be knowing exactly what happens given each choice, and only being in doubt about the ethical weught.
I'm pretty sure this was an episode of Star Trek-- four or five lights?
Itself based on OBrians torture of Smith in nineteen eighty four.
Now I'm wondering whether and to which extent the famous basketball gorilla experiments would give different results in live action than on video.
http://theinvisiblegorilla.com/gorilla_experiment.html
This reminds me of that Star Trek: The Next Generation episode where Picard is tortured for days by Cardassians, who are trying to get him to say the wrong number when asked "How many lights are there". At the end of the episode, he tells RIker that he actually saw the number they wanted him to say.
I thought it's kind of mainstream science that evolution has 'programmed' us to see things that aren't there if it's good for us. For example, here's a quote from Why Buddhism is True:
"Suppose you’re hiking through what you know to be rattlesnake terrain, and suppose you know that only a year ago, someone hiking alone in this vicinity was bitten by a rattlesnake and died. Now suppose there’s a stirring in the brush next to your feet. This stirring doesn’t just give you a surge of fear; you feel the fear that there is a rattlesnake near you. In fact, as you turn quickly toward the disturbance and your fear reaches its apex, you may be so clearly envisioning a rattlesnake that, if the culprit turns out to be a lizard, there will be a fraction of a second when the lizard looks like a snake. This is an illusion in a literal sense: you actually believe there is something there that isn’t there; in fact, you actually “see” it.
These kinds of misperceptions are known as “false positives”; from natural selection’s point of view, they’re a feature, not a bug. Though your brief conviction that you’ve seen a rattlesnake may be wrong ninety-nine times out of a hundred, the conviction could be lifesaving the other one time in a hundred. And in natural selection’s calculus, being right 1 percent of the time in matters of life or death can be worth being wrong 99 percent of the time, even if in every one of those ninety-nine instances you’re briefly terrified."
Seeing a square when there's really a circle is a pretty extreme example (which I guess is why it got called a 'trolley problem'). But seeing a rattlesnake when there's a weird-shaped stick in the grass (and the penalties for false positives and false negatives are vastly different) seems pretty plausible to me.
This is not an adequate comparison. "genuinely seeing square" would do nothing to change the electric shocks; *claiming* you saw them would, which is certainly what you would have learned to do.
I think Phil's disagreement comes rather from thinking in a different time scale. The total reward maximitazion model is correct, but it works in evolutionary time: a visual cortex that correctly identifies objetcs is "reinforced" by evolution because this maximizes total reward. This is precisely what originates the *epistemic* architecture. But on the lifetime of a person, however, the visual cortex does not do any reinforcement-learning (at least after infancy).
Speculations about microscopic processes a billion miles away are a complete waste of time in the long run. We have the prameters of the Standard Model that 'explains the properties of atoms molecules that enable the construction of cells the visual-cortex is filled with, yet we can't get our most powerful computers to accurately predict the behavior of a single molecule of H2O. Considering the fact a single cell is built from hundreds or thousands of molecules far more complex, and that the behavior of everything at the core is a probabilistic quantum mechanical information process, I might be wise to listen to those with knowledge about God the Creator and life's purpose than scientists who's god is random mutation.
Relevant webcomic:
https://i.imgur.com/kFqkPEb.jpeg
"That."
"That?"
"That is why we're screwed. That number. That number will doom us."
"I hate that number. Can we run away from it?"
"No. It's a number. It represents an immutable fact."
"Can we fight it?"
"No. <...> What?"
"Sorry. Evolving a threat response over a half-million years on the African savannah hasn't really left me with any good mechanisms for dealing with a threatening number."
"That is also why we're screwed."
Betting money on things seems like one way to push your brain more toward the less-hedonically-reinforceable regime.
I think all of the supposed discrepanices with modeling the brain as a hedonic reinforcement learning model can be explained with standard ML and economics.
If you do a lot of research on epistemic facts related to your political beliefs, the first order consequence is often that you spend hours doing mildly unpleasant reading, then your friends yell at you and call you a Nazi.
In the case of doing your taxes or the lion, that unpleasantness is modulated by the much larger unpleasantness of being sued by the IRS and/or eaten alive by a lion. So there's a normal tradeoff between costs (filing taxes is boring, seeing lions is scary) and benefits (not being sued or devoured).
But in the case of political beliefs, the costs are internalized (your friends hate you) and the benefits are diffuse (1 vote out of 160 million for different policy outcome). So it's no wonder that people aren't motivated to have a scout mindset.
People are pretty good at avoiding sources of stress, especially ones that offer no obvious benefit. I can't really imagine why anyone should go become a scout given the tradeoffs. Maybe it's a noble sacrifice you make for the greater good?
I don't think that's obvious. The scout mindset is almost definitionally being less confident about your knowledge. Soldiers seem confident too, that's kind of the point.
> Confidence comes from competence
I think that's only one route to competence. Plenty of people are very confident without being competent specifically because they have the soldier mindset.
If scout mindset were that effective, I think evolution would have made it more common.
I don't think even Julia Galef would claim that a scout mindset makes you more likely to have babies (which is all 'effectiveness' means from an evolutionary perspective)
Yes, and I'll emphasize that this was the case prior to the invention of birth control as well.
Because if you become good at predicting social and economic trends that everyone else thinks is wild pie-in-the-sky nonsense but actually happen, you get words like "disrupter" and "avant-garde" attached to you, attain social status and financial success, and ultimately be able to have a kid with Grimes before she divorces you and decides she's a Marxist now.
Now this I could believe with the caveat that you have to already be a billionaire to get "disrupter" or "avant garde". The rest of us just get "crazy" or "weird".
And of course then there's views that aren't pie in the sky, just ordinary run-of-the-mill views of the opposite tribe. Those get the label "Nazi" or "Marxist" depending on which tribe you defect to.
>The rest of us just get "crazy" or "weird".
If you're a Bitcoin millionaire, people aren't going to call you "crazy" (unless you start trying to argue for the AOC to be abolished, cannibalism to be legalized, or something else so far outside the Overton Window that no amount of money can shield you from social censure). "Crazy" requires you to flap your arms, jump off the barn, and break your legs. If you actually FLY, you're an eccentric visionary. I realize it's really really easy to imagine that social classes are an iron-clad law of the universe and people will still treat you as Joe Schmoe from Buffalo even if your bank account hits 7+ figures, but (in America, at least), like 70% of social class is attached to your net worth.
>Politics
I hope you aren't looking for someone on a random blog's comment section to give a complete solution to solve political disagreement. I was just making the point that there are, in fact, reasons to break from the herd besides pure altruism.
There are some, very limited, reasons to break from the herd in specific highly limited circumstances.
The title of this post is "Motivated Reasoning As Mis-Applied Reinforcement Learning". I'm not missing it, I'm trying to explain what it is.
I'm not trying to focus on the title, just the argument being made. I believe that all of the supposed discrepanices with modeling the brain as a hedonic reinforcement learning model can be explained with standard ML and economics.
I have a prior: Being eaten alive by a lion would be extremely painful and end my life before I can procreate. I've assigned 100% probability to that prior. The brain is able to backpropagate to my visual cortex that a False Positive on a lion is scary, but low risk (let's say loss value = 1.0), but a False Negative on a lion is life-ending (let's say loss value = 10000.0). Given those incentives, your visual cortex optimizes lion recognition for high recall and reasonable precision.
I don't see why there's any contradiction between hedonic reinforcement learning (learn not to get eaten alive, extremely low hedonic state) and what the visual cortex does.
His point is that sometimes in life we do the thing that doesn't happen with our eyes: our eyes always see the lion, even if we really do not want the lion to be there. Your explanation of why that is makes sense: if we don't see the lion, we die. Yet at the same time in other areas of our lives there is a problem, it really will negatively impact us if we don't do something about it, and yet instead of seeing the problem our minds work hard to find a way to convince ourselves the problem doesn't exist. So why does it do that? That's the question, not why our eyes always see the lion. It's why our minds sometimes try to tell us the lion we see isn't really there because we don't want it to be.
Is that the point? The post seems to presuppose that there are "reinforcable" and "non-reinforcable" parts of the brain. I don't see any need for that. You can perfectly explain the function of the neural cortex and every other part of the brain through hedonic reinforcement learning.
I originally supposed that he was asking "why does our brain work for the lion, but not for reasoning about politics?", so I answered that question with what I think is a logical, mathematical view based on internalized costs and externalized (diffuse) benefits. I think that's the answer to your question "Why does our brain not see the problem when it will negatively impact us?" My answer is that being wrong about politics *doesn't* negatively impact you because the costs/benefits of being right/wrong are shared across hundreds of millions of people. But the costs/benefits of agreeing/disagreeing with your friends accrue only to you.
Because the future is uncertain, and the further away it is, the more uncertain, and while a lion in the bush will eat you *right now* almost all the things we procrastinate about are lions that may or may not eat us some time from now, assuming our fearful reasoning about the future is correct, which it often isn't. The fact that our emotional centers routinely dial down threats which are more hypothetical and uncertain is hardly surprising, and clearly adaptive. The fact that we have warring agents in our awareness that struggle to gain consensus on threat-level estimates that do not agree with each other is also unsurprising, given normal conscious experience.
Love it. Great thinking. Really well put.
I rephrased the first sentence of my original post to not focus on the title and instead summarize my argument. My argument is that all of the supposed discrepanices with modeling the brain as a hedonic reinforcement learning model can be explained with standard ML and economics.
"This question - why does the brain so often confuse what is true vs. what I want to be true?"
Going back to first principles of natural selection one would presume the human brain to be:
(a) **well adapted** to discern true facts that have positive impacts on reproductive fitness (e.g. identifying actual lions, learning which hunting or gathering techniques work best, etc.);
(b) **well adapted** to engage in useful self-deceptions that also have positive impacts on reproductive fitness. (e.g., believing that your tribes socially constructed superstitions and political rules are "true" so that you fit in and get along.
(c) **Non-adapted** for determining true facts that might make your life more financially profitable, enjoyable and stress-free but that don't have any direct impacts on your hunter-gatherer reproductive fitness. (e.g., realizing that you shouldn't procrastinate on your taxes, or shouldn't worry about things you can't change.)
Sadly, the human brain is evolved to make more surviving humans, not to make us happy or successful in a modern capitalist economy. I think the happy/high-achieving people are probably those who are more successful in somehow tricking their brains to move category (c) issues into category (a). If only you can convince yourself that the search for absolute truth is a life or death hunt that will keep you from starving to death and instead allow you to have sex with the hot cave woman https://youtu.be/gSYmJur0Npw?list=PLVVuOIA1lowEKOVGZgf4_GsmH51Lu4511&t=68.
I thought motivated reasoning was.... Reasoning with a motivation. Reasoning, but, WITH AN AGENDA!! DUN DUN DUNN... Like, you want something, so you spend a lot of time entertaining counterfactuals which seem like they could be pieces of plans you could make which lead to you getting the thing you want, as opposed to Perfect Unmotivated Reasoning: reasoning done entirely for the sake of Truth, and not utility.
I thought motivated reasoning meant reasoning with ulterior motives
Well, yeah, it is. The problem is that you can use reason to come to lots of non-true beliefs, provided your main motivation is not finding the truth. That's what Scotts trying to puzzle out: why exactly do succeed at tricking ourselves that a real problem isn't real just because we don't want it to be?
"Maybe thinking about politics - like doing your taxes - is such a novel modality that the relevant brain networks get placed kind of randomly on a bunch of different architectures, and some of them are reinforceable and others aren’t. Or maybe evolution deliberately put some of this stuff on reinforceable architecture in order to keep people happy and conformist and politically savvy. "
I wonder if it is not simpler to just consider a dichotomy between tasks we have evolved for (e.g., learning to speak) and those we have not (e.g., learning to read), rather than the epistemic/behavioural dichotomy. Detecting threatening animals is clearly an evolved ability, while doing one's taxes is not at all. This could mean that "doing one's taxes" will not be done automatically, and will depend on many factors, including of course the pleasantness of the task (and also the tendency to ignore the future, trust in the group, anxiety, etc.).
I don't think it's mysterious why we don't like doing our taxes, I think it's mysterious why this leads to behavior like not checking whether our taxes need to get done, or refusing to open your email because you're worried there will be a difficult task in it.
Even if you hate email, I think the "rational" action is to open the email, check if there's any message which is so important it overrides your desire not to do it, and *then* procrastinate. But that's not how most people experience this.
I of course agree that the rational action is to open the email and get the information (which I find a bit hard to do...). But well, we already know that we do not act very rationally.
It seems to me that evolution has given us specific rules for situations that were both common and important during our evolution (avoiding the predator, indeed), and very general rules for all other situations. Among these general rules, there is clearly the acquisition of information, primates are notoriously curious, and of course the avoidance of annoyances. But we could not get a rule like "evaluate the situation and choose the behavior according to the best predicted option". So we "choose" a behavior according to the impulse that turns out to be the strongest, and many people procrastinate to open that email.
Because our behavior usually does not result from rational considerations but
I think you're thinking about this too much from a model where everybody is a good-faith Mistake Theorist.
In a mistake theory model, it's a mystery why people fail to update their beliefs in response to evidence that they're wrong. If the only penalty for being wrong is the short term pain of realising that you'd been wrong, then what you've written makes sense.
I think that most people tend to be conflict theorists at heart, though, using mistake theory as a paper-thin justification for their self interest. When I say "Policy X is objectively better for everybody", what I mean is "Policy X is better for me and people I like, or bad for people I hate, and I'm trying to con you into supporting it".
There's no mystery, in this model, why people are failing to update their "Policy X is objectively better" argument based on evidence that Policy X is objectively worse; they never really cared whether Policy X was objectively better in the first place, they just want Policy X.
I think there are three things: honest mistakes, honest conflicts, and bias - with this last being a state in which you "honestly believe" (at least consciously) whatever is most convenient for you.
If a rich person says the best way to help the economy is cutting taxes on rich people, or a poor person says the best way to help the economy is to stimulate spending by giving more to the poor, it's possible that they're thinking "Haha, I'm going to pull one over on the dumb people who believe me". But it also seems like even well-intentioned rich/poor people tend to be more receptive to the arguments that support their side, and genuinely believe them.
I don't think honest mistakes or honest conflicts need much of an explanation, but bias seems interesting and important and worth figuring out.
Is the conventional explanation unsatisfactory? That people are more convincing when they argue for their position honestly, and so it's beneficial for them to become biased in ways that favor their interests.
The question is, what is the mechanism? If I was offered a billion dollar incentive to sincerely believe I have three hands, I still wouldn't be able to do it. I can't alter beliefs on command. That makes the bias idea harder to explain.
That's because such offers presumably weren't common in the ancestral environment in which our brains evolved.
As far as I can tell, we're pretty far off of being able to establish the exact ways in which high-level adaptive strategies are implemented in the brain. For these purposes, it's an extremely complex black box, which also isn't too amenable to high-precision controlled experiments.
Selection. Economists who believe things convenient things for rich people get more connections, media exposure, grants, etc.
(In the conflict theory worldview; I have no idea how econ academia works IRL)
Thank you for the comments on this post. I was having a hard time figuring out what the post was about. I'm not sure why we ignore the Ugh fields, I sometimes see these balance scales in my head, on one side is the badness of not taking care of my taxes, and on the other side is the slight goodness of ignoring this problem for another day. The badness keeps building and eventually I have to open the dang letter.
Then why do many people make decisions that are objectively bad for them and the people they care about, are GOOD for the people they hate, and then actively defend these decisions unto the trenches for decades?
It's likely that your definitions of objectively good/objectively bad are good examples of what Melvin is describing.
I'm referring to people who, as an example, vote for tax breaks for the wealthy and increased taxes for their own income bracket who, when asked WHY they do this, don't have a satisfactory answer beyond "My strawman conception of an enemy tribe wants that and I'll be damned if I ever agree with them on anything." (Yes, you can accuse this itself of being a strawman, but these people really do exist, and in sufficient numbers that calling it a "strawman" is a bit deceptive. You see them on both sides of the aisle, usually yelling at each other on social media). This isn't behavior that can be explained purely by calculated zero-sum games, because EVEN BY THEIR OWN CALCULUS they're making decisions that damage their own side (salt-of-the-earth Real Americans who work with their hands and backs) and empower their enemies (rich, effete latte-sipping Coastal Elites) based on weird ideological principles.
I mean, you could certainly say that the above descriptions are just branding those people consciously adopt, but that doesn't explain the core issue here- I'm not sure if conflict theory is willing to believe in tactically shooting yourself in the foot.
>I'm referring to poor people...
I knew exactly where you were going with it and responded accordingly. I've not met a single person who bemoans people "voting against their own interests" who wasn't entirely convinced that their preferences are derived from universal and timeless moral rightness and that anyone opposed to their preferences are stupid (poor people), evil (rich people), or both.
Yes, I do in fact think people should be paid a living wage and shouldn't be condemned to a life of misery, hunger, or dependence on intoxicants as an alternative to suicide, regardless of their genetics. And I do, in fact, think people who argue that Moloch-sacrifice is good and that inquiries into bringing the number down as close to zero as we can should be sneered at are evil.
If someone kept hitting themselves in the head with a hammer, complaining loudly every time he got hit, you would, in fact, call him an idiot- or else determine that he really likes the idea of dying via TBI, but I don't pretend this is some kind of just world where people are making the rational decision to be systemically discriminated against, not make any money, get hooked on painkillers, and die in their own waste. If you think my attitude comes from a sneering condescension towards the working poor instead of being one of them, that's your problem- unless you want to say you have access to my mental states.
>If someone kept hitting themselves in the head with a hammer, complaining loudly every time he got hit, you would, in fact, call him an idiot
No one did or said this, stop fantasizing.
>If you think my attitude comes from a sneering condescension towards the working poor instead of being one of them, that's your problem
I think your attitude comes from an unwarranted confidence in the correctness of your beliefs - which is pretty damn funny given that I knew the direction your diatribe was going to go based on how banal your "insight" is.
For my part, I assume in the overwhelming majority of cases that people make decisions for themselves for good reasons, where "good" is defined by the decision-making person him/herself. I don't presume to be able to define another person's preferences for them.
This model explains why people would make this kind of mistake when it benefits them, but it doesn't explain examples like procrastinating on your taxes.
I agree, which is why I think that procrastinating on your taxes has a different mechanism to maintaining self-interested political beliefs.
If this is true, then maybe Cognitive Behavioral Therapy is just using your "basically epistemic" brain regions to retrain your "basically behavioral" brain regions. Like, trick yourself into ending up in a better hedonic state than you used to -> reinforcement learning counter updates towards "yay". That could explain why it's so crazy effective despite being "just" talk therapy.
I know less about brains and psychology than just about every other commenter out here, am I on to something here?
I don't think anybody really knows anything about brains and psychology at this point -- at least when it comes to complex behaviors and beliefs. For example, this behavioral vs. epistemic brain dichotomy sounds like one of those models that appeals to pattern-seekers but I doubt there is any way to prove it's useful or accurate with concrete data.
Consider that AI programmers will set up a self-teaching model for their programs and then set it lose to evolve a way to solve a particular problem. But once it's taught itself to solve the problem you can't go back and figure out exactly how it learned or how it logically gets the right answer. So why would we think we could reverse engineer our bio-brain's 3-billion year old self-learned chemical algorithms.
I think it is wrong-footed to proceed as if reasoning always directly connects your brain to the phenomenon that you are reasoning about. Instead, the process is social. You decide what to believe based on who you believe. You decide who to believe mostly based on who you would like to get approval from (or to get imaginary approval from). If in your imagination you would rather hear Joe Rogan say you're a great guy than hear Anthony Fauci say you're a great guy, then when they differ you will work harder to discredit what Fauci says than what Rogan says. And conversely. That is what puts the motivation into motivated reasoning.
I'm a fan of your idea of (mostly) 'social epistemology', but surely reasoning isn't _entirely_ social, especially for things like 'doing taxes'.
I'm trying to make sense of this. First, I'm going to rename '[people] who you would like to get approval from' as 'desired approvers', because it makes the following sentences more comprehensible.
Does your model imply a connection between how vividly you imagine your desired approvers, and how easily you reason? And does it imply a connection between how aware you are of which beliefs are endorsed by your desired approvers, and how easily you reason?
If so, consider people with only a nebulous sense of their desired approvers (as individuals and/or as belief-holders). Your model seems to imply that such people will tend to have more trouble with reasoning than other people do. And the rare truly isolated person, with minimal media and no religion, family, or friends, would have a great deal of trouble with reasoning. Is that what you expect? Have I misunderstood?
I think that is correct. Knowledge is social. See Joseph Henrich, "The Secret of our Success"
Thanks for the recommendation. I've been seeing that book and Henrich's book on WEIRD mentioned, and was leaning toward reading the latter, but now I'm motivated to read both.
How did AlphaStar learn to overcome the fear of checking what's covered by the fog of war?
I like how this explanation meshes with:
- don't increase the pressure, lower the resistance
- Leave A Line Of Retreat
- exposure therapy
- Street Epistemology
I guess reinforcement learning is all about good sense of balance between exploration and exploitation. I feel that when someone proposes a new theory to me it's like asking me to explore some new train. And I am less willing to do exploration when I am low on resources - to limit variance, which could kill me, when I am close to bankruptcy, I prefer the steady income of my current best model.
It seems like that would be because AlphaStar was learning to win games, and won more games when it checked what was in the fog of war. We're trying not to be eaten by lions, but unfortunately we rarely get more than one crack at it.
Oh I didn't mean to say evading predators wasn't an option! Just that if you closed your eyes and pretended the lion wasn't there (perhaps making lots of noise to drown out its footsteps), you wouldn't get a chance to learn from your mistake. Evolution, of course, would.
I think you can go a level deeper into this idea - there obviously aren't *actually* parts of the brain that are just magically epistemic, really just about everything in our CNS learns how to do its function to some degree or another. E.g., even the lowest levels of your visual cortex learn how the visual field fits together, how to handle lines of different directions and kinds, and etc. Probably most of that kind of learning is wrapped up and finalized by the time you're past the toddler stage, but it means that even these basic "epistemic" functions formed under a reward function of some kind. We would hope that reward function is something like surprise-minimization (and/or some free-energy whatsit that I don't understand,) but the brain is messy and full of poorly enforced boundaries, so the reward function could also be partially coming from extrinsic signals unrelated to its nominal function- perhaps of the form "if I become aware of X, I feel really bad suddenly."
The upshot of all of which *could be* that particular kinds of bad/traumatic upbringing in very early life could predispose someone towards motivated reasoning, by mis-training "epistemic" parts of the brain. That approaches the kind of thing that could be testable!
Just a thought, maybe one of many causes for this phenomenon:
Shifting a worldview isn't zero cost - when you shift your worldview many posteriors need to update (which means every situation and corresponding action which might have been 'memoized' before needs to undergo an update process which takes extra time and energy as it is compared against the new worldview).
I use the term worldview here but I guess it applies to anything you've taken as an assumption and might need to shift. I think energy and response time costs might play a part alongside hedonic reward.
This is also something I care a lot about, but maybe it serves our social animal nature more than our predictive fitness. Hedonic states are evolutionarily wired to confer at least four functions - they feel good, "energize" behavior, confer unique psychosocial strengths (motivation, humor, creativity, willpower, etc) and resilience to stress. These resources are useful are highly useful in the buffeting winds of the social world, so hedonic states becomes a meta-goal independent of their original evolutionary function. Originally, evolution creates a cybernetic system where anticipation of positive outcomes generates hedonic states to motivate behavior, but when people can coopt the benefits of the hedonic states for social functions, suddenly social evolutionary logic motivates individuals toward a positivity bias to generate hedonic resilience for its own sake.
You can extend this logic to other hedonic states like social connection or social status and people suddenly appear in relief as highly motivated to maintain tribal ideologies build on ingroup biases and attribution biases and positivity biases all to generate hedonic resilience. Suddenly the "madness" of doing the same thing and expecting a different result, of maintaining beliefs that are costly, of staying with ingroups that drag us down or of competing in self-defeating games makes sense - people are trapped on a treadmill of attempting to generate short-term hedonic resilience as a social resource, particularly as unlinking from predictive accuracy generates negative consequences and chronic stress, creating even more deficits of hedonic resilience and more need, cyclically. A pernicious spiral dynamic for social animals.
The beautiful part of this particular rabbit hole is when you realise motivated reasoning is basically the same concept as original sin, which is a combination of social optimisation and the psychological neoteny that allows societies to scale beyond the Dunbar number.
I think this is a bit unfair on motivated reasoning.
We train people at motivated reasoning for years; here's the conclusion, now argue for it, is basically the standard school writing assignment.
Then we get into the real world, and discover the vast majority of people don't care about the same things we care about, and if we need to convince them of something that is important to us, we have to argue for it from a set of values that were not used to arrive at it.
And truth isn't even that valuable for most people, and is in many cases actively harmful to them, compared to white lies. The weird behavior isn't motivated reasoning, it's being able to accept the truth with an awareness of bad things being constant sources of suffering.
Yeah, rationality is rarely the optimal strategy.
>We train people at motivated reasoning for years; here's the conclusion, now argue for it, is basically the standard school writing assignment.
What is the connection between being trained to do something and *not knowing when you are doing it*?
Perhaps the additional ingredient required is a lack of consequences for being unaware.
>What is the connection between being trained to do something and *not knowing when you are doing it*?
In traditional martial arts, the goal of practicing the exact same motions hundreds or thousands of times a day, every day, for YEARS, is so that you stop *deciding* to do the motion and just *do* it. You see someone, and instead of your thought process being a complex analysis of everything happening step by step, you just make the decision to draw your sword and kill this man and everything else is just reflex. All the mechanics stop being a conscious thing you think about, and the basics are so deeply buried that being an instructor is its own separate set of skills from just being a practitioner of the blade. A friend of mine who's an aikido instructor spends most of his time in aikido form from the waist down because he spends so much of his time with his feet at 90-degree angles from each other that his limbic system now defaults to that.
This is also true in the sciences: PhD's in astrophysics tend to have MORE difficulty explaining basic physics concepts to laymen than someone who's just consulted a Wikipedia article despite the former being infinitely more qualified to give the explanation than the latter, because things like the basic Laws of Motion are so deeply ingrained in the former's mind that they almost never never consciously access that information.
I see absolutely zero reason why making people argue from conclusion to premise for the vast majority of their formative years wouldn't result in arguing from conclusion to premise becoming a habit.
"Then we get into the real world, and discover the vast majority of people don't care about the same things we care about, and if we need to convince them of something that is important to us, we have to argue for it from a set of values that were not used to arrive at it."
But all people do in debates with strangers online every day is try this strategy and fail to change peoples' minds.
When have you ever seen someone convince anyone of anything by appealing to their model of alien values?
The only way I am aware that anyone has ever convinced anyone is by first having insight into their own values and explaining them such that shared humanity is recognized.
Empathizing with someone who you've dehumanized up front is as impossible as...some mathematically impossible thing. The fallacy that's ever-present is the idea that lack of empathy can go one way rather than being symmetrical.
But I'm doubtful that convincing people has anything to do with the games normally played with strangers online.
I've been convinced by somebody appealing to my (to them alien) values; the argument they created was more convincing from the perspective of my own values than my own. I've convinced other people by doing the same. It does require one or ideally both parties being pretty good at either stating their own values, or correctly identifying somebody else's, which is pretty rare.
For the thing of people arguing from what they think other peoples' values are in general, though, it happens constantly, it's just often hard to see because we see the values other people argue from, not the values they actually hold.
Once you notice it, however, it's hard to stop noticing it, and it explains why so many arguments are basically pointless. The standard pointless argument goes basically like this: I think you have value X; I argue from a position starting with value X. You observe my arguments, and think they mean I hold value Y, which is related to but subtly different from value X; you argue from a position starting with value Y.
If I happen to notice you seem to actually be arguing from value Y, I change my arguments; now you think I'm a hypocrite who is contradicting myself. Which, in the sense of "Saying one thing but believing another" might be true, but is missing the humanity of what is actually going on. If, on the other hand, I notice that you aren't actually arguing from value X, but don't identify value Y, the argument turns into a semantic debate.
Or neither of us notice the values are different, and we have an argument in which we spend the entire time arguing past one another, on the basis of values that neither of us actually have, and both of us feel like we've "won" the argument, and the other party just won't acknowledge it. That seems to be the most common case.
It's all quite amusing when you start to notice it, really.
>That seems to be the most common case.
I think it's good to set an example of admitting uncertainty and talking about emotions and experiences as the basis of beliefs.
Having separate reinforcement and epistemic learners would be the elegant solution. There's also the ugly hack, which is to make "there might be a lion" even scarier than "there is a lion" so that checking is hedonically rewarded with "at least now I know".
Successful horror movie directors can confirm evolution went for the ugly hack, as usual.
Why do we need "there might be a lion" to be scarier than "there is a lion", which would be an "ugly hack"? We only need it to be scary enough to do damage to the hedonic state. Then checking offers a chance to restore the state --- there might not be a lion after all! If the lion probability appears so high that checking is not promising for that purpose, then surely it makes sense for the hedonic state to be very badly damaged anyway, without checking.
To me your point, the hedonic implications of "there might be a lion", puts into question Scott's explanation of wishful thinking, as the objection should in principle extend to the other scenarios in the post. Can we not just say instead (in cases not explained by virtue signalling) that the human novel ability to imagine counterfactual scenarios affords a cheating opportunity, kept within limits by evolution but not completely neutralised, namely enjoying the rewards via imagination? See also (a very dense page of text, I understand only parts of it): picoeconomics.org/HTarticles/BBScoms/BBSMcKayBelief.html
This seems very wrong to me. Visual systems can clearly learn. A veteran spy agency field agent, veteran NFL quarterback, NBA point guard, infantry commander, F1 driver, pilot, whatever, can all see things and read a scene much better after years of experience than they could beforehand.
The problem with your contrived example is the negative hedonic nudge from getting spooked by something that looks like a lion but isn't one is a tiny signal, dwarfed by the gigantic evolved-in prior that things that look like a lion need to be properly identified and avoided. It isn't that the visual system learns any differently. You're just comparing it to realms where signal strength is unclear and there may be no instinctual priors at all. You can throw inputs at a model with randomly initialized weights and watch it very quickly update its predictions, and then throw exactly the same inputs at another model with exactly the same learning algorithm but weights already stuck in a local minimum, and it won't budge at all. The learning algorithms are still the same. Human brains aren't randomly initialized models. For "is that a lion," we have very strong priors. For "when should I do my taxes," we don't.
I think the problem is that we have a very good idea of how you'd train, say, a visual system to be good at seeing things: predictive processing! But that stops working at the high level, and so for behavior-like things you want a different training method. It may well be that the whole brain has one learning algorithm that somehow strikes a balance between the two, but it's not a silly question to consider.
As for all the stuff about confirmation bias? I'm with you on that
In my long running commentary theme of 'missing the point' of what you're talking about.
I.
The 'I'm right about politics and the other side sabotaged our program' is a fairly poor proxy for your current argument because it is true. When you have endless think tanks writing articles about how to sabotage the other side for decades and they brag about it every chance they get along with funding offical and unofficial controlled opposition, abuse of law enforcement, selling political favour for censorship actions by media companies, etc.
All to get their ideas across and suppress the other side's ideas....then it would fall into the rational side of ideas to think rather than the wrongthink side you describe of running away from a picture of a lion on a box of cereal in a grocery store.
When the FBI and republican senators and fundraisers and think tanks and such were all colluding against the civil rights movement on a multi-pronged front and the FBI had a floor to ceiling stack of papers tracking the peaceful anti-war activist, reverend, and civil rights activist MLK to harass him, blackmail him, and eventually assassinate him.
Along with many thousands and thousands of other prominent activists over a multi decade span on time...I'd say it is pretty reasonable to think that one side is using legal, illegal, offical, and unofficial channels and every funding mechanism they have to push their ideology.
When you go through the events on the day of Malcom X's assassination and you see how the FBI had contact and was tracking them and how they and the police uncharacteristically pulled their protection/monitoring team from the event hall that day when they'd been showing up in force to all his other events for months....it doesn't take a genius to see the kinds of sabotage you've dismissed as poor quality thinking.
If that's what they were doing then...what is happening now? Does anyone think that after Snowden exposed the NSA etc. that their spying was actually reduced? Nay, it has certainly advanced to a degree far higher and more sophisticated than what they could do before as technology has progressed.
The idea of GOP sabotage of government is so well understood that it has existed in parody for decades. When Bush appointed anti-UN extremist Bolton to the US's top UN post or put his veterinarian etc. in charge of a large bureaucracy....or when Trump literally failed to bother appointing new heads for hundreds of offices of government causing a general slowdown and disruption, etc.
Or when shows like Parks and Recreation can parody the very widespread right wing rural middle aged white guy's genuine perspective to 'starve the beast' and intentionally mismanage the bureaucracy in order to show how bad it is....what is a person to think? That they're being an irrational fool?
II.
On the main subject of the post and it's intended topic, I doubt there would be any physical element of the brain we could identify in terms of when our ideas and learned reactions to the world are correct or incorrect. This is because 'reality' is not represented in the brain - there is no 'truth' there against which our ideas form around or avoid in proper or in proper brain processes.
A large part of this is somewhat random in terms of what any individual experience and learns....not to mention that in evolutionary terms the main thing that is causing this confusion with lions in your example is about representational depictions of reality. In our history as a species if you saw a lion it was probably a lion, not a realistic image of a lion.
If the occasional child got a bit frightened at night as the firelight flickered on the cave art depiction of a lion...well that too is and advantage as you WANT to instil a generalised fear of lions in the children as you pass on the stories of your people and their knowledge of the landscape and how to survive. If a tiny handful of people end up 'misfiring' in their learning process in that context, then it is not going to be very harmful very often and evolution will have little to say in terms of selecting brains for or against any strange learning errors which pop up.
Evolution has the tool of fecundity and death. Expose them all to reality and let god sort em out to paraphrase the more common idea of kill em all and let god sort em out phrase.
Having a collection of data points that suggest some conspiratorial behavior exists in reality is different from believing in a specific, perhaps overbroad, conspiracy theory.
Would you reject out of hand the idea that multiple possibilities for explaining known facts exist that are not disproven, and even multiple conspiracies could coexist in reality?
Furthermore, to the extent there is one or more big conspiracies that are durable and effective, the information you get is more likely to be cover or chaff.
The perspective isn't primarily neurological, but if you haven't already read The Righteous Mind by Jonathan Haidt, that has a lot to say about motivated reasoning surrounding things like politics to the extent that politics often involves moral commitments and involvement in a community defined by its moral commitments.
A very poor explanation is something like poliltics = religion because they both define communities of people with similar values, and saying your party's policies don't work is a lot like saying that your religion's sacraments don't work.
The underlying intuition here about reinforcement learning is incorrect.
> Plan → higher-than-expected hedonic state → do plan more
No, it's: higher (relative to other actions) hedonic future state *conditioned on current state*. The conditioning is crucial. Conditioned on there being a lion, RL says you should run away because it's better than not running away.
It gets tricky with partial observability, because you don't know the state on which you have to condition. So instead, says RL theory (not so much practice, which is a shame), you can condition on your belief-state, *but only if it's the Bayesian belief-state*. If you're not approximately Bayesian, you get into the kind of trouble the post mentions.
But being Bayesian is the RL-optimal thing to do. You get to the best belief state possible: if there's a lion, you want to believe there's a lion, litany-of-Tarsky style. The visual cortex could, in principle, be incentivized to recognized lions through RL.
I suspect people don't open IRS letters not because their RL is fundamentally broken, but because their reward signal is broken. They truly dislike IRS letters, and the pain it causes to open one is truly more than their expected value. People probably also underestimate the probability and cost of a bad IRS letter, but that's due to poor estimation, not poor deduction from that estimation.
Perhaps it's easier to see in organizations, where you can tell the components (individuals) apart. It's sometimes hard to tell apart the bearer-of-bad-news from the instigator-of-bad-news. This disincentivizes bearers, who might be mistaken for instigators. With enough data, you can learn to tell them apart. Until you do, disincentivizing bearers to the extent that they really could be instigators is the optimal thing to do.
Regarding the lion, isn't the issue that few agents ever experience negative utility from not running away from it? Seeing as you die immediately after, there's no time for reinforcement learning to do anything.
Presumably getting eaten by a lion reinforces the gene pool to beware lions.
Everybody does some amount of motivated reasoning, but there's probably some systematic variation in propensity to motivated reasoning and it's probably possible to measure that and from that infer some ideas about the evolutionary tradeoffs involved. I slightly suspect it comes down to whether your ancestors won more of their bread by dealing with things or dealing with people and the latter would relatively increase motivated reasoning -- so a verbal tilt should correlate with motivated reasoning. Less motivated reasoning may have made great^20 grandpa a more competent farmer, but more motivated reasoning may have made him less likely to reach taboo conclusions and be shunned or burned as a heretic.
In the context of Gwern's Algernon argument (https://www.gwern.net/Drug-heuristics#loopholes), all three of the loopholes might apply to the project of reducing motivated reasoning.
1. the environment is different -- people aren't getting burned as heretics so much anymore. They just get kicked off of twitter and fired from woke corporations.
2. Value discordance. We value truth more highly than the blind idiot god of inclusive-fitness-maximizing.
3. Evolutionary restrictions. The brain is such a rube-goldberg kludge that it's not clear evolution could have eliminated motivated reasoning even if that were strongly beneficial to inclusive fitness.
I see Steve Byrnes has already commented, but in case you haven't/aren't planning to read his stuff on this topic, I'd recommend checking it out. E.g. https://www.alignmentforum.org/posts/frApEhpyKQAcFvbXJ/reward-is-not-enough
I'm going to substitute AI safety in general for Richard Ngo for "you people":
> Is what you people are trying to do train an AI, like one trains (raises) a child, to do the right thing?
Maybe? There's lots of different people with lots of different ideas and with lots of different thoughts.
"Is it even *possible* to train an AI to do the right thing?" is one of the questions that's up for debate.
Another question under debate "Is training like a child even the right framework to use to think about this?".
> always thought AI safety was about "Can we still shut it off, or keep it contained?"
That's part of it, but not all of it.
> but I am neither a deity, nor a human that has designed an AI, nor do I have the least interest in manipulating other people, or intelligences, or indeed myself.
I mean, to each their own, but the link mostly asks interesting questions about how minds work. Being interested in that sort of thing doesn't require being a deity, AI engineer, or manipulative.
I swear I'm not just trying to argue you into liking an article, but now I'm more confused. I assumed you were referencing that, but did you think it was intended as a how-to guide? Or is this a principled stance against thought experiments that couldn't be performed as actual experiments by mortals? Again, if you don't find the subject interesting you don't find the subject interesting, I'm just confused.
No worries! Looks like I was the literal one in this exchange anyway.
Interesting question.
Perhaps "action planning" is the primary domain of reinforcement learning, whereas perception is primarily achieved via self-supervision / free energy minimization / etc?
... and "turn my head 45 degrees to the right" perhaps isn't an action plan but rather part of free energy minimization.
Like others in this thread, I also wonder if there are different scales of reward-prediction-based learning.
Dopamine mediates all sorts of critical brain functions, including the ability to initiate and cease movement (thinking about Parkinson's). I assume that means that dopamine plays a broader role than just reward prediction, but OTOH could "initiation and cessation of movement" be thought of as micro reward-prediction-based learning?
Is there a reward system scale equivalent to "edge detection" in the visual cortex? I.e. what is the simplest unit reward mediated activity of the brain? If that is similarly low-level, then the question might not be "what parts of the brain are reinforceable vs not" and rather "what parts of the brain are reinforceable AND implicated in higher order thinking of which we're cognitively aware?"
This is perhaps the most hotly debated question in the neuroscience of reinforcement learning right now. Is dopamine always some kind of (generalized) prediction error, just acting on different state representations and/or different timescales? Or is it doing fundamentally different things in different regions? I lean towards the former, but it's very far from settled.
Perhaps this is far too basic, but what if we look at decision-making algos being sorted into "survival" vs. "thrival," where the surviving decision tree overrides the thriving decision tree?
So, turning one's head when they see a yellow blob in periphery might threaten ruin one's hedonistic state (anti-thriving), but since surviving to see more hedonistic days vs. less overrides hedonistic/thriving decision tree?
Thanks Scott, really good discussion. The idea that the brain uses two types of reinforcement learning (Epistemic and Hedonic) is really, really important. It has deep implications for the structure of the brain, and the ways we should try to model it. What you call the epistemic network reinforces associations based on correctly predicting the outcomes of agent actions, even if those agent actions have negative outcomes. Hedonic reinforcement works as a controller of behaviour, reinforcing possible agent actions based on predicted hedonic states associated with the epistemic outcomes. To model it, you have to see them as two separate networks working in tandem, one caring only about the (Bayesian approximate) accuracy of its' predictions, and the other caring only about the hedonic consequences of those predictions.
Your point, which had never occurred to me, is that this implies there will be cases where agents will actively avoid seeking new information where experience has taught them that this would have negative hedonic effects. It's a real insight I think. But if it's true, and I think it is, then you have to assume that the epistemic/hedonic dichotomy is an architectural principle of all animal minds.
Another interesting thing is that when you try to model a reinforcement learning neural network that operates according to this epistemic/hedonic architectural principle, it's very difficult, and you end up with something that doesn't look much like the sort of models I see in the ML literature.
The way I think about it is there are 4 levels of feedback loop that biologicals systems use to process information.
Level one is evolution itself: does it kill you or not? individual organisms don't process this, but life as a whole gains information this way.
Level two is reflex: You have various sensors for light, chemicals, touch etc, and you automatically react by like, flinching away from pain or a dangerous chemical stimulus.
Level three is reinforcement learning, to either gain reactions to stimuli that are non-reflexive, or to gain control over reflexes and not react. This is like when you learn to like a flavor because despite the bad taste, it has provided you with good nutrition in the past, or something like that.
Level 4 is memetic and abstract. This is having an information system that is able to learn from communication, observation, and reasoning, to a greater or lesser extent. It enables us to hear phrases like "If you go into those woods, a tiger will eat you"
Each of these plays into lower levels given enough time and evolution. Sufficient repetition of abstract memes will train you into automatic responses to certain stimuli even if you have no actual experience, eg flinching away from poisons that you have never tasted yourself.
Things like paying your taxes stays extremely abstract for most people and thus is not particularly good at compelling action.
Re the visual cortex part - in humans, there’s cortisol to help out - if you see something scary enough, the cortisol faucet goes on and later recall becomes more difficult/memories harder to access. That seems to me like the biochemical cheat for the situation described - if the visual cortex does sometimes learn by reinforcement, but in situations of heightened threat that would clear the “don’t look” bar, cortisol makes the cortex not retain it.
The brain trains on magnitude and acts on sign.
That is to say, there are two different kinds of "module" that are relevant to this problem as you described, but they're not RL and other; they're both other. The learning parts are not precisely speaking reinforcement learning, at least not by the algorithm you described. They're learning the whole map of value, like a topographic map. Then the acting parts find themselves on the map and figure out which way leads upward toward better outcomes.
More precisely then: The brain learns to predict value and acts on the gradient of predicted value.
The learning parts are trying to find both opportunities and threats, but not unimportant mundane static facts. This is why, for example, people are very good at remembering and obsessing over intensely negative events that happened to them -- which they would not be able to do in the RL model the post describes! We're also OK at remembering intensely positive events that happened to us. But ordinary observations of no particular value mostly make no lasting impression. You could test this by a series of 3 experiments, in each of which you have a screen flash several random emoji on screen, and each time a specific emoji is shown to the subject, you either (A) penalize the subject such as with a shock, or (B) reward the subject such as with sweet liquid when they're thirsty, or (C) give the subject a stimulus that has no significant magnitude, whether positive or negative, such as changing the pitch of a quiet ongoing buzz that they were not told was relevant. I'd expect subjects in both conditions A and B to reliably identify the key emoji, whereas I'd expect quite a few subjects in condition C to miss it.
By learning associates with a degree of value, whether positive or negative, it's possible to then act on the gradient in pursuit of whatever available option has highest value. This works reliably and means we can not only avoid hungry lions and seek nice ripe bananas, but we also do compare two negative or two positives and choose appropriately: like whether you jump off a dangerous cliff to avoid the hungry lion, or whether you want to eat the nice ripe banana yourself or share it with your lover to your mutual delight. The gradient can be used whether we're in a good situation or a bad one. You could test this by adapting the previous experiment: associate multiple emoji with stimuli of various values (big shock, medium shock, little shock, plain water, slightly sweet water, more sweet water, various pitch changes in a background buzz), show two screens with several random emoji, and the subject receives the effect of the first screen unless they tap the second. I'd expect subjects to learn to act reliably to get the better of the two options, regardless of sign, and to be most reliable when the magnitude difference is large.
For an alternative way of explaining this situation, see Fox's comment, which I endorse.
OK, now to finally get around to motivated reasoning. The thoughts that will be promoted to your attention for action are those that are the predicted to lead to the best value. You can roughly separate that into two aspects as "salience = probability of being right * value achieved if right". Motivated reasoning happens when the "value achieved if right" dominates the "probability of being right". And well, that's pretty much always, in abstract issues where we don't get clear feedback on probabilities. The solution for aspiring skeptics is to heap social rewards on being right and using methods that help us be more right. Or to stick to less abstract claims. You could test this again by making the emojis no longer a certainty of reward/penalty, but varying probabilities.
Source: I trained monkeys to do neuroscience experiments.
Are there any books or sources you might point to that would go more into detail on these two different maps?
I'm wondering particularly how they are arranged, for example, and how their communication is coordinated.
Is that value map always operating? How stable is it? Is there some other map for mundane facts, separate from the value map?
Thanks in advance :)
"Maybe thinking about politics - like doing your taxes - is such a novel modality...." ???
Dunno, chimpanzees and all the other non-solitary primates seem to be quite keen on politics, so this modality must have been around for the past 15 million years, or even longer, if you are willing to stretch the meaning of the word. Richard Wrangham's "The Goodness Paradox" strongly supports the idea that we have a very fine, evolved grasp of politics. It's not about being happy or conformist but yes, being savvy enough to viciously cut the other guys off at the knees at just the right moment and yet look quite saintly while doing it, is just what Mr Darwin ordered. The life of Aspies was always precarious on the veldt.
I dunno. These models seem so widly oversimplified trying to tease out the logic strikes me as about as productive as Aristotle trying to theorize about how to improve metallurgy using only the idea that everything is made of Fire, Air, Water and Earth in varying proportions.
I mean, you've described feedback structure that could be implemented with a dozen transistors. What are the other 79,999,999,988 neurons for? Not to mention even the feedback structures we know at the most basic biochemical or sensory processing level are already far more complex. We've got delayed feedback, feedback that is added and subtracted and differentiated. We have edge and motion detecting feedback circuits. We have feedback that is integrated, often with nonlinear kernels, and for all we know there's FFTs being taken and all kinds of feedback circuits operating in transform space, too.
That's not to even mention the biggest elephant in the room, which is that we have no idea what the intermediate metrics of cognitive success actually are. Sure, optimizing survival of the related gene group is the ultimate metric of success -- but how does that translate to more immediate measures of optimal individual decision and action, the kinds of things that can be optimized by an individual brain in real time? The fact that the bookstores and libraries groan with pop-psych self-help books, and pro libraries under slightly more polysyllabic versions of same, is pretty clear evidence that we have at best a tiny handful of scattered clues.
I think this is a non sequitur. We know behaviors get reinforced, what are the other 79,999,999,998 neurons in behavioral parts of the brain for?
The answer is "categorizing things and figuring out which things caused which other things", which is really hard.
Meh, I agree with Carl here. Your model of the brain amounts to "it does reinforcement learning based on the outputs of some blackbox" and then dump 99.9999999% of neurons into the blackbox. But even this super-simplified model seems wrong, so you amend it to "only some parts of the brain do reinforcement learning, and the rest is total blackbox".
I don't think this is useful; I don't think it provides real insight rather than a fake feeling of having insight. What prediction about the brain can you make now that you know this theory that you couldn't make before?
I suspect you're factually wrong about that. I think you attribute way too much challenge to simple object segmentation of the world, which even very small brains in very primitive animals can do with ease, and way too little to the subtle agency decisions which human beings appear to spend 98% of their time and brainpower struggling to get right -- all of which involve exactly this complex web of feedback between remembered decisions and outcomes which lies at the heart of your issue.
I think it unlikely a priori that our vast painful existential struggles are the result of the last 0.000001% of the neurons struggling to finish the last 0.000001% of the general cognitive task. I rather suggest objectively taking the measure of the world through our senses is the *easy* task, occupying a very modest fraction of our brainpower, and deciding what we can and want to do with that model is the hard task. That is, I take the fact that most of us experience the bulk of our brain wattage being used to figure out what we should do next at face value -- that it's saying *that's* the hard task, the one that takes tens of billions of neurons.
Which suggest *whatever* the network that links our prior experience to our current decisions, it is something that requires billions of neurons to implement. So it's not something that can be plausibly modeled by a handful of logic gates or a flowchart that fits on one 11x18 page. Indeed, if I had to guess, I would guess an accurate model is extremely difficult to even understand -- since if it *were* less than nearly impossible to intuit, we would have done so long ago and be talking confidently and precisely about how to build intelligent awarenss from biochemical transistors as simple as a neuron (so long as you have enough of them), instead of waving our hands vaguely and incoherently about what "intelligence" and "awareness" and "intention" even mean.
Now that I think about it: another piece of strong empirical evidence is that we can *already* program computers with far simpler computing machinery than we have between our ears to "categorize things and figure out which things cause other things" (or at least are well-correlated). Doing exactly that -- to whom does this face belong? What is its age? If our user said X and Y just now, what is the next word Z going to be? -- are among the most salient modern triumphs of machine intelligence.
But having idiosyncratic self-generated goals and purposes, having intention, making decisions -- things that even rodents can do -- we have absolutely no idea how to duplicate in silicon. Unless one wants to invoke strange conspiracy theories or weirdly selective idiocy, that's pretty reasonable indirect evidence that having the experience of agency and successfully exercising it is the hard (programming) task, and just recognizing and categorizing patterns in reality is the easy one.
I disagree. If there is any lesson to be learned from machine learning, then it is that a large number of neurons is really helpful, both for supervised learning and reinforcement learning.
And for your point that a lot of tasks can be done by small brains, it also misses part of the truth. The number of neurons in animals is not just determined by intelligence, but also by body mass. An elephant has more neurons than a human. A lot of large whales have more than 10^10 neurons in their sensory areas. The statement "perception and classification can be done with few neurons" is not wrong, but evolution ignores this beautiful truth and uses lots of neurons on these tasks if the animal can afford it.
Well, in the first place, I don't think there are any important lessions to be learned from machine learning, because there's no evidence it has any relationship to real learning. There's not much you can learn about how to build high-performance airplanes by watching hawks, because the two devices solve similar problems in very different ways. So far as I can see, it remains to be proven that human brains and neural networks using TensorFlow have any useful relationship to each other.
Secondly, I suggest even computers argue against your point that more is always better. Which is better, RISC or CISC? It's not obvious. Which is better, more processors and parallel programming, or boosting scalar speed of fewer processors? Also not obvious. Which is more efficent among human organizations, small fast-moving groups that make a lot of small mistakes, or large complex groups with many layers of error-correcting management that only make great big mistakes? Et cetera. Basically, being large introduces problems of communication and organization that can overwhelm any gain from dividing the workload more finely.
And finally, we don't know what all the neurons in a whale do. Indeed, my impression is that the estimate of the social interactions and social intelligence of whales have steadily risen over the decades. They may need bigger brains because they have far more complex social interactions than, say, Diplodocus did (famous for having a huge body and a small brain).
I’ve always thought of that motivated reasoning as a way to economize compute power. Your brain says: Something bad may happen, I can’t do anything about it, my avoidance of detecting it doesn’t much change the odds of it happening, so I’m not going to think about it because if I’m going to have any hope I need to explore other stuff. The bit about not wanting to see the lion because it will give you a bad day I think only practically works if the lion future is competing with another future with a higher goal. “The tribe is starving. Janice remembered gathering some berries by the river a ways back. She took our remaining food to give her strength to run. There are lions there now and she may get eaten, but if she doesn’t gather the berries and remember exactly where they were to get back in time the tribe dies.” So the lion looker module gets downgraded against the berry finder module.
But at the edges where these things are fuzzy, as they usually are in real life, with lower stakes and overlapping error bars, this probably becomes procrastination/willful ignorance.
When I see a picture of a large spider, i get really creeped out, even though it is just a picture. Isn't this a lowering of my hedonic state? If I close the book with the spider picture, it gets removed from my visual cortex, thus causing my hedonic state to increase. The same is true if I saw an actual spider and freaked out and ran away. Running away removes it from my cortex, increasing my hedonic state. Why does this need to be reinforcement learning, isn't it just an evolved instinctual reaction to scary animals?
I usually prefer "reinforcement learning" to "evolved instinctual reaction" because instinctual reactions are really really hard and there aren't enough genes and brain structures to store all of them. My guess is evolution is under strong pressure to minimize the number of instincts and use reinforcement learning for whatever it can.
See eg https://slatestarcodex.com/2017/09/07/how-do-we-get-breasts-out-of-bayes-theorem/
i think there was also a post at some point about fear and reinforcement ( i think about dogs)... even if being scared of the lion is learned, doesnt running away still reinforce, because you are removing the fear. isnt your day *saved* and not ruined by running away? you no longer have to be in the presence of the lion/dog
As a neuroscientist studying reinforcement learning, I feel like I ought to have something for you here. As you know, we broadly think that reinforcement learning in the brain is mediated by dopamine. So one way of framing the question "which parts of the brain are / aren’t reinforceable" is "which parts of the brain do / do not receive dopamine input." Interestingly, the brain regions with the most dense dopamine innervation are the striatum (input nucleus of the basal ganglia, dealing a lot with movement and motivated behaviors) and to a lesser degree the prefrontal cortex (dealing with executive function / decision-making). Notably, sensory cortices do *not*, by and large, receive dopamine input. It strikes me that this relatively cartoonish diagram of the dopamine system maps on quite well to your intuition that "behavioral" regions should be reinforceable and "epistemic" regions should not, with executive regions somewhere in between.
"But suppose you see a lion, and your visual cortex processes the sensory signals and decides “Yup, that’s a lion”. Then you have to freak out and run away, and it ruins your whole day. That’s a lower-than-expected hedonic state! If your visual cortex was fundamentally a reinforcement learner, it would learn not to recognize lions (and then the lion would eat you). So the visual cortex (and presumably lots of other sensory regions) doesn’t do hedonic reinforcement learning in the same way."
I think this logic is wrong. Standard RL algorithms learn to maximize *long-term* summed reward, not just the reward from the immediate next state. (If they only learned to maximize the immediate next reward, they'd be awful at most tasks.) So a visual cortex that learned via RL would learn to recognize lions because, even though it leads to a worse state immediately, it avoids a huge punishment down the road. I agree with you that sensory cortex isn't implementing RL, but this is not a good argument for why not.
Rather than a purely "is reinforceable" vs "isn't reinforceable" distinction I suspect the difference has more to do with the relevant timescales for reinforcement. In the foot injury case, we'd have a very fast gait correction reinforcement loop trying to minimize essentially instantaneous pain. In the lion country case it sounds like something slightly longer timescale -- we make a plan to go to lion country and then learn maybe a few hours later that the plan went poorly so we shouldn't make such plans again. In the taxes case it's much longer term, it might take years before the IRS manages to garnish your wages, though you'll still eventually likely get real consequences. Politics on the other hand, often cost/reward is so diffuse and long-term that I suspect the only reason anyone is ever right about difficult policy issues is because the cognitive processes that correctly evaluate them happen to be useful for other reasons. The vision example I think is a mistake of timescale; a vision system which learned to not see something unpleasant would get a much worse net reward when you don't avoid the lion you could have seen and subsequently get mauled.
I'm coming at this from the ML side so I'm out of my depth biologically, but perhaps we have different relevant biological RL processes with different timescales? Eg, pain for ultra-short timescale reinforcement, dopamine for short-to-medium timescale reinforcement, and some higher-level cognitive processes for medium-to-long timescale reinforcement.
One observation: You can't avoid a lion trying to eat you by refusing to look at it. But you might be able to avoid another lecture from your neighbor Ug Bob about how you haven't made the proper blood sacrifices to Azathoth in a while, if you refuse to make eye contact with him and keep walking.
That is to say, huge parts of our brains developed in order to process social reality. And social reality, unlike physical reality, actually does change based on what information you have (and what information other people know you have, or what information you know other people know you have...). So controlling the timing of when you are seen by people around you to acquire certain information likely does have some degree of survival benefit. And the parts of our brains that learned how to do that are probably the ones that are involved in reading letters from the IRS today.
I think the major problem here is that from an individual perspective the "system" you are learning about is so frigging noisy and generally large/complex that "the answer" to any given scenario is only valid across a large population. Maybe we can slowly increase the percentage from say 60 to 70 to 80 to 90 to even 95 some day but 5% of all people is a catastrophically large number.
Perhaps the visual cortex is more of a reinforcement learner than you're giving it credit for. Children have to be taught to look both ways before they cross the street. I know that when I started working in a lab, highly-significant translucent specks in a vial of water didn't stand out to me the way they do now. In the lion example, as soon as the yellowish blob was seen - or even as soon as the hunter stepped out onto the Savannah where they knew lions existed - they'd be having worse anxiety not checking if the blob is a lion than they would if they looked directly at the blob.
It takes a lot of learning to carve out significant features from our visual field, build habits about what to pay attention to and what to ignore, and learn to coordinate our bodies in order to execute these sensory routines. That learning seems to happen via reward mechanisms, either self-administered or delivered by others.
In the lion case, it seems like anxiety is the reinforcement learning that forces people to look at the yellow blob. So how do people come to have that anxiety? I suspect it is socially reinforced. Children are punished for being careless and rewarded for being perceptive, and warned about specific dangers until their brain punishes them with more anxiety for carelessness than for caution.
I notice that with the taxes and email examples, these are often activities done in solitude, and where the direct dangers afflict only adults. It may be that there's not enough social reinforcement to instill sufficient anxiety for not doing one's taxes or checking one's email in order to overcome the natural unpleasantness of checking Quickbooks or gmail. Therefore, a whole class of problems becomes a source of akrasia.
Another explanation for these two examples in particular is that it might be that the threat of bankruptcy or losing one's job isn't actually as bad as it seems. Perhaps people are, in effect if not in conscious intention, checking to see if they can keep their job without checking their email, or somehow get out of paying their taxes (perhaps long enough to see a bump in their income).
So maybe we don't need to resort to a whole alternative brain architecture for special cases where reinforcement learning seems to predict some unrealistic behavior. Maybe instead, it's simpler to assume reinforcement learning, and figure out the simplest explanation for how reinforcement learning might produce the outcomes we observe. Then you could test this explanation empirically, and try testing several iterations of more complex explanations. If you try hard and fail to find an empirically verifiable reinforcement-based explanation for a behavior that seems contrary to the 100%-reinforcement hypothesis, then you start thinking that there must be some alternative brain architecture.
Humans don't have "decision-making algorithms." Humans are not computers. It's really embarrassing to read stuff written by people who don't understand that.
You should really read The Master and His Emissary. It does a lot of deep diving and presents many satisfying approaches to answering the question you have. The author is a psychiatrist and also has a great deal of expertise when it comes to studying the brain physically, both in humans and in monkeys. This makes him able to use hard science to approach the issue while still recognizing that humans are not computers. It's an incredible book.
Hi Scott. You might be interested in the works of Robert Trivers and Hugo Mercier on the hidden role of persuasion in shaping reasoning. My colleague Moshe Hoffman and I also address this in a chapter of our upcoming book. Lmk if it’d be helpful for me to send further details on anything.
Trivers was hugely influential in my own thinking about this topic. Perhaps people here would benefit from a little explanation of what he says. And what's this book?
I don't know which you're referring to with 'this' sorry, but here are links to all the ones I think are most relevant:
Trivers's book: https://www.amazon.com/Folly-Fools-Logic-Deceit-Self-Deception/dp/0465085970/
Mercier and Sperber's: https://www.amazon.com/Not-Born-Yesterday-Science-Believe/dp/0691208921/
Kurzban's book: https://www.amazon.com/Why-Everyone-Else-Hypocrite-Evolution/dp/0691154392/
Ours (which isn't out yet, but see, in particular, Chs. 8-9): https://www.amazon.com/Hidden-Games-Surprising-Irrational-Behavior/dp/1541619471/
For ours, I'm happy to provide a PDF of the galleys if you email me, but don't want to post a copy publicly.
Oh, and the TLDR: People are often motivated to persuade others of something, regardless of the truth. They're more convincing when they themselves believe that thing (for instance, because there's no risk of accidentally revealing the truth). This is the Trivers story, which the others touch or build upon. In our case, we add a bit of game theory to the mix.
Thanks very much for all this. By "what's this book?" I meant to refer to yours.
So to be clear, you're thinking of strategic overconfidence as the counterpoint to Scott's post above?
Gotcha. Appreciate you taking in an interest.
Uh, it's an additional point, rather than a counterpoint. There are various drivers of motivated reasoning. I think that persuasion is one very important one--probably the most important one--and wanted to add it to the discussion.
(FWIW, ATM, my guess is learning is probably not an important driver, though it was an intriguing suggestion, and I could easily imagine being convinced otherwise. I think I'd want to see more evidence that learning can explain features of motivated reasoning that persuasion and/or standard social psych stories cannot.)
This seems like a back-propagation error, or rather, back-propogation seems like the mechanism which decides how much to update “avoid looking for lions” vs “avoid lion-infested savanas” vs “carry a pointier stick”.
There was a paper posted to lw recently showing that predictive coding can implement back-prop, maybe this is a place where evolution's handy trick to simulate back-prop using local information breaks down.
It does not seem clear reinforcement learning is being "misapplied" as there might be no alternative and human reinforcement learning is simply far from optimal as it does not recurse the decision tree at all like reinforcement learning algorithms, but instead generalizes from memorized valent experiences with a patchwork of adaptive computational fragments like a babbling large language model. For accurately computing the utility of doing the taxes, you'd have to descend very deep in the decision tree until you arrive at the relatively narrow rewarding branch in the far future—a computation that is infeasible in our wetware. So a better title may have been "Motivated Reasoning As Suboptimal Reinforcement Learning Messing Up Our Epistemology".
As much as I wanted to believe this when I first read it (because it seems more parsimonious at first glance than my current model of this sort of thing) I'm going to roll to disbelieve because this model does *not* predict that people will feel better after admitting the thing they're avoiding isn't going to go away, and the ugh field will disappear.
ISTM if the issue were purely reinforcement learning, resistance shouldn't disappear upon admitting what you're avoiding admitting.
Now, is it something that affects AI design? Not going to argue one way or the other. But for humans this seems to be missing whatever element(s) predict the collapse of avoidance once you "admit" that what you're doing isn't working. ISTM the Litany of Gendlin wouldn't do anything useful if knowledge-avoidance were as simple as reinforcement learning.
I'm surprised you count this as a point against. I can't trivially make myself want to do unpleasant things (like check my tax status) by reciting the Litany of Gendlin. I always though of it as more aspirational.
The litany of Gendlin needs application, e.g. asking yourself if not checking the taxes is actually helping, or mentally predicting what will happen if you don't and then it turns out to be bad anyway.
More specifically, my observation both personally and of my clients is that the brain seems to be treating problems as if they can be avoided by avoiding the situation that reminds you of the potential problem. That is, it's treating the negative future as if it's conditional upon your perception or possibly *admission* of the problem as existing.
Or to put it another way, "if I don't admit it, I can pretend it's not true". Breaking this conditional prediction by admitting that if it's bad, it's already bad -- oh wait, crap, that's not the Litany of Gendlin, that's the other one... Tarski?
Anyway -- admitting that the problem is going to be there *anyway* seems to remove the option of "pretend it's not true if I don't look" from the planning subsystem, allowing less-favored but more useful plans to proceed.
I've been teaching something somewhat like this as a generalized anti-ugh-field technique, more specifically involving imagining that you've *already experienced* the ugh you're anticipating, so that avoiding it isn't perceived as an *option*.
Granted, it only works on ugh fields where the ugh is generated by the task itself: i.e. if doing taxes is unpleasant, then imagining it as having happened already (and unavoidable), will get one past it, IME. But if the ugh field is about something tied to completion of the task (or possible failure thereof), then imagining *doing* the thing isn't really going to help, nor will it address meta-issues to the whole process.
But for stuff like, "I don't really want to get up from this warm bed", it works great: just imagine you're *already* cold.
The type of ugh where one is avoiding *finding out* something because of an expected unpleasant answer seems to work the same way, i.e. imagining you already found out it's bad.
ISTM this would not *ever* work, though, if the underlying mechanism was some parts running on the wrong storage machinery, vs. just being simple prospect theory.
That is, the model I'm using here is that prospect theory means we compare plans of action by generating a diff between the expected future and the status quo, and so pretending the status quo already includes the negative update removes the negative ranking on the action producing the ugh.
(The "bug" in the brain here is just that the act of checking the maybe-negative thing is being treated as responsible for the projected negative future state, where in the present state the thing is not-bad-to-my-knowledge and the future state is probably-bad. And people seem to do this a lot.)
So, ISTM that just using known/understood concepts of how the brain rates actions based on predicted outcomes (and how it handles risk and probabilities) is sufficient to generate both ugh fields and avoiding checking on things that might be bad.
To be fair it doesn't rule out your hypothesis either. It just seems more parsimonious, especially since this model works without needing any prior reinforcement: just having information that *suggests* something will be bad will be enough to put an ugh on checking it, after all, even if every other time you checked it, it was good.
For example, checking your account balance regularly for years with no problems, then thinking you spent a lot of money and it might be low: you could generate an ugh field without any specific reinforcement having happened at all. (Unless you count what happens each time you think about it from then on -- but the first time it wouldn't have been conditioned as such.)
"why does the brain so often confuse what is true vs. what I want to be true?"
Are you applying this question to yourself or other people? I'm not able to relate to this question if I apply it to myself, but it seems plausible when I apply it to others.
Introspecting while keeping the intent of the question and applying it personally, I think I would rephrase the question as "why does my lizard-brain/base-animal cognition so often win against my metacognition?"
I think that rephrased question is answered by the basic fact that metacognitively I have no motivation. The reason I do anything is for base animal motivations. I mean, even the reason I use metacognition is from the fear of making a mistake or the gratification of being right.
Using the tax example (and assuming I didn't do my taxes), I think the reason for me not doing my taxes would be that my base-cognition would fear the act of doing taxes to the point of shutting down my metacognition in that domain. My motivation would be shifted to playing video games, data binging, or other distractions (to keep my metacognition from generating possible futures that my base-cognition fears).
I really don't buy the "Then you have to freak out and run away, and it ruins your whole day. That’s a lower-than-expected hedonic state!". You know what's an even lower-than-expected hedonic state? Getting mauled by a lion (or run over by a truck). It's so much lower that in many cases, it's permanently zero. I don't buy that a behaviour that involves *not* investigating potential existential threats to survive long enough. Especially one that is just as, if not even more relevant today than previously (how many lions did a primitive man encounter, and how many trucks could a modern man walk in front of?)
This post made me wonder, given the good logic of having a bad day after seeing a lion, why do we seem to get so much enjoyment from seeing lions (and large animals in general) up close? People love zoos and for a sizeable number the attraction seems sufficient to sink a lot of money into safaris etc. It feels like this is more than just enjoyment of nature's majesty a la an ocean view or a sunset?
Is this some kind of vestigial love of the hunt, at odds with (and perhaps made all the more thrilling by) the countervailing fear of a mauling?
Most people outright don't like spiders, I wonder if we simply haven't evolved the same excuse-making-fascination with them that we have with big cats?
Regarding "not turning the head", I see a big positive value in self-reflection and/or therapy. In a behavioural scenario, you want to avoid bad outcomes. But in an epistemic scenario, you want to say "seeing a lion is bad, but noticing it quickly is positive and I did a good job". Reinforcing this kind of thinking may help people to rescue some trapped priors.
I feel like you’re creating 2 cases out of a single algorithm. Let us say your brain is only optimizing sum of net future rewards (Return). Now if you miss a lion at the peripheral, there’s a chance you’ll die so your brain will focus on that. Once you see the lion, your hedonism will reduce for a short while, but compared to the chance of outright dying the reward is much higher when you spot the lion than when you don’t, so it makes sense that your brain chooses that option. On the other hand what is the net reward for knowing you are wrong and changing your prior. Your hedonism increases a bit because of your internal satisfaction of knowing you are right, but this can be smaller than the decrease for feeling like an idiot. In fact the lessee the chance that that piece of information affects you directly, the lessee the lifetime reward of correcting yourself of that piece of information. This way i don’t think the dichotomy you claim exists
This might be a better example than the lion:
https://en.m.wikipedia.org/wiki/The_Fox_and_the_Grapes
I think this is likely to come off as very naive, but I work in a very different field so please excuse me.
I find this kind of question- where we can think about thought in some way that combines decision theory, psychiatry, potentially artificial intelligence and different models of consciousness, to be like *the most important and interesting* thing- are there any groups of people that do research on this? Could I come at this kind of problem from being a practicing psychiatrist?
I think the most useful search term would be "computational psychiatry", if you have more specific questions feel free to email me at scott@slatestarcodex.com
Quick Google and...wow. I think this is maybe what I want to do with my life? Thanks for the pointer- have emailed you :)
“ This question - why does the brain so often confuse what is true vs. what I want to be true?”
I somewhat regularly find the need to talk myself out of something. If my dream jobs comes along and it falls through at the last minute, I comfort myself by saying, “That’s a sign that it would have sucked.” And that’s more comforting to believe than that it would have been everything I dreamed it could be. So I go with that.
But I know I’m doing it. Do most people not know “on the inside” that they are doing that?
When someone goes to Destin, FL and not St Barths or they buy a Camry and not a Lexus and say it’s just as good or say their super successful friend isn’t really living his best life - they know (at some level) that they are just telling themselves comforting lies, right?
I've thought for a while that something like this might happen around the "cognitive dissonance" feeling. That's an unpleasant feeling, and you get it more if you look for info that goes against current belief. Thus, this punishment would breed confirmation bias.
There is 2 things that make this more complex imho :
Immediate feedback is much more efficient at reinforcement than delayed one. So people not doing things to avoid pain is clearly happening, but much much more for immediate pain than future one. The more future and abstract the pain (or gain) is, the less reinforcing effect it will have. People will certainly avoid walking on an injured foot because it hurt, maybe not look in a dark corner that may hide a lion, but certainly not hesitate to do their tax in fear of a having to pay more than they could in 6 month.
On the other hand, there is a kind of meta reinforcement system active for abstract / delayed stuff : you directly enjoy (or hate) some mental activities, not directly linked to practical consequences. I constantly delay doing my taxes because I hate all administrative paperwork, not because I fear bad surprises. If anything, it's not doing them /rush them that had lead to bad surprises (and will probably do so in the future)... On the other hand, me enjoying math and logic games certainly helped getting my degrees easily... But i did it because i enjoyed this kind of mental activities, not because i saw that doing more of it helped with my studies...
Delayed reinforcement demand a conscious effort, and doesn't work so well (for me, but i usually have a relatively long time preference so i would be surprised if on average future reward /punishment have a lot of reinforcement effect )
Meta reward where mental activity itself activate the brain reward system seems a stronger effect. Then it remain to see why some mental activities are enjoyed more than others, and how and why it varies across people...
Me too!
Here is my take as to why.
In a post-rational world full of valence-charged facts, it is impossible to have a rational (or even civil) discussion. Being immersed in our post-rational era means we must acknowledge ‘lived experience’ without being able to opine on how it impacts our common goal of ‘equal opportunity for all’ or how we can work as a pluralistic society with differing definitions of “The Good.” Makes me wonder what will happen to the humanities like literature and theatre studies (founded on fiction and using empathy-like mechanisms for exploring differing definitions of The Good), but that’s for another time.
The greatest danger is believing only your opponents are infected with Humanistic Truth. While I hate to admit it, I am basically intuitive, not rational. Just like all people, practically all the time, I make hasty judgments. These are not based on reason. My false reason comes from the hidden operations of cognitive predispositions and a two-track brain. Instead of my mind acting as a judge weighing the facts, it acts as a press secretary seeking to justify my beliefs.
That’s right, “humans are not designed to practice reason. We are designed to make arguments that aim to support our preconceived conclusions, not yours,” The Righteous Mind: Why Good People are Divided by Politics and Religion.
Once a social value has been mapped onto the biological disgust mechanism in social interactions (e.g., valence-charged facts), then it is impossible to have a reasoned discussion about any topic (from racism, inequality, BREXIT, or the election of President Trump). We need a new narrative to allow us to deploy our critical thinking skills and address the growing inequity in our society. I find the best new narratives use allegory and metaphor deployed through humor that must be grasped and not explained.
Watch this video to see what I mean.
https://youtu.be/Ev373c7wSRg
I think Hansonian signalling/far mode is at work, as usual. People hear you say you believe X. Belief X indicates that you value Y. They value Y, so they socialize with you more. The system that caused you to say X is an adaptation to this. It really just doesn't enter into the question whether X is true.
Suppose someone says "I believe the solution to violent crime is that we need more creativity in public schools" or "I believe the solution to violent crime is that we need more religion in public schools". This does actually vaguely correspond to an empirically testable statement. But that's really far from the point - in a deep sense, nobody cares if it is true or not, they (the speaker and listeners) just care about what it indicates about the person who has said it. If it's false, well, at least this person is right-thinking and therefore someone they want to socialize with.
Does it really ruin your day to turn your head and see a lion? Or does it save the day to be warned against the lion so you can go on living? Does it ruin your day to see that you're not going to make the tax deadline unless you get down to working on your taxes, or are you glad to remember that you can avert a fine? Does it really hurt you to find out that you were wrong about politics, or does it please you to find out why you were wrong and why you can now confidently make a better choice? When I'm learning new things, it's painful, but afterward I'm so glad I went through the pain.
To me the difference between the lion case and the taxes case is something like - how quickly are you going to get feedback on your decision/beliefs? In the lion case, you can't actually avoid learning in short order if there is a lion, because it will probably eat you. In the taxes case, you can avoid it for a pretty long time! Short-term bias is a pretty normal factor in how humans make decisions and it seems pretty applicable here too.
Wait. I'm firmly in "the brain's architecture is complex and heterogeneous and it's probably a requirement for intelligence" camp, but there's no need to assume additional complexity and specialization to solve this particular problem.
Predictive coding framework, as I understand it, says the brain tries to fit expectation (top-down signals from its model of the world) to reality (bottom-up signals from its inputs), and ultimately, to minimize the mismatch between the two. This alone basically covers the "behavioral"/"epistemic" distinction already. Visual cortex tells you that your model is wrong (a yellow blob where you don't expect it), which triggers an action to resolve the mismatch (by checking more closely). That's the only mechanism you need. There's no distinct "hedonic reinforcement learning", there are only "hedonic inputs" from your body, which get processed the same way all inputs do. And there's no distinct "what is true" discovery in the brain, it's literally all about wanting its internal model of the world to be true - and to achieve that, it chooses from different, but potentially equally valid strategies like changing the model, acting so that the inputs change to a better match, or disregarding inputs and substituting its own.
The reason this procedure may cause problems is threefold:
1. That the inputs are imprecise. (Turning your head towards a yellow blob is the right action in clear daylight, but on cloudy night it will likely fail to penetrate the darkness and resolve the confusion, not to mention all the false alarms you'll be needing to process. In terms of detail/clarity/readability, stress, etc., signals from your body are more like the latter case.)
2. That overruling the inputs requires learned experience. (Children do in fact need to specifically learn that closing their eyes does not get rid of the distressing object they see. But we're practicing seeing all day, every day, from the day we are born, we've had plenty of time to find and learn correct heuristics. For taxes, not so much, and for many stressful situations in general, taking your mind off the source of distress and waiting until you calm down is in fact the correct action to take.)
3. That the brain optimizes for the inputs, and not directly for acquiring the best possible model of reality. (So, it will decide to check your taxes if and only if it concludes it will help you reduce your stress. The rational argument that resolving the source of your stress is the obviously correct action to take is merely one of the pieces of evidence to consider, actual practical experience with stress input is another. Say, I'm a depressed ADHD sufferer with a lot of experience in being late on payments, and my learned strategy of "wait for a reminder, hopefully get around to reading it right away, make a single money transfer for the right amount" does in fact reduce stress compared to "search for the bill, check your bank statement to see if you've paid it, manually count how much you're overdue, add penalty/interest, etc. etc.")
I would take issue with your argument at line 1, basically, where you assume -- and it is an assumption, one worth pondering -- that the metric to be optimized is congruence between physical measurement and interior model. That the brain is spending lots of effort trying to make its interior model of physical events match the outside real world.
In the first place, I think that problem was solved by rodents about 300 million years ago, and is something we can pretty much take for granted in the human brain. I doubt there are that many neurons devoted to that particular task. Doesn't mean they don't screw up, but they do it sufficiently rarely that there hasn't been any strong evolutionary pressure on it for the lifetime of our species. We're really, really good at interpreting sensory data and predicting dangerous events in the physical world. That's why even quite modest intelligence people can learn to drive fast on crowded highways or fly fighter jets in dogfights, and readily do as well or better than the best computers with the cleverest programs. It's not like chess at all.
What we've skipped over is tghe question of what the apex task of the brain is. Like all organs, it's selected for the fitness of the closely related gene group (which often but not all the time means survival of the individual). So the question is: what is the most critical and difficult task for the brain that touches on survival?
Your assumption seems to be "modeling the physical world to detect threats." I don't think that' s obvious. In fact, I'd say a much more plausible guess is "modeling the social world of other human beings to detect threats and opportunities." We are a highly social species, and the threats to our survival and prosperity almost always come to us from social interactions, not direct physical threats. The IRS is far more dangerous to us than lions, these days, and that has been true for at least 100,000 years, and perhaps since Australopithecus. (Even if we point out lions are a real problem for early hominids, it is equally true that the single best defense against lions in general is to be a member in good standing of a tribe of other humans with whom you can construct a common defense. You're far more likely to be die young from being rejected by the tribe than being eaten by a lion.)
But if Job #1 of the human brain is to optimize the response of the individual to social threat and opportunity, it is not at all obvious that this necessarily implies what we would typically call accuracy in modeling the physical (or even social) world. We already know there are plenty of functional delusions: each mother believes her own individual child is the best in the world -- the bravest, most handsome, most capable, sweetest tempered -- and this is clearly a very functional delusion. If parents evaluated the worth of their children, versus other children, in as objective a way as a 3rd party or computer, it would be a disaster. The same can readily be said about the perceptions spouses have of each other, about the perceptions of followers for a leader, perceptions of the citizens of the nation, et cetera. We can bring in religion: the feeling that God is watching you is clearly functional, most of the time. We can even adduce survival in highly stressful situations: those who have studied the survival of people under extreme circumstances (prisoners of war or gulags, lost at sea or in the wilderness) note that the survivors tend to share a highly functional delusion: that they *will* survive, even in the face of daunting odds, and that by planning and perservance they can overcome extreme bad luck. The belief in this unlikely possibility is *what* enables them to do what needs to be done to boost the probability of survival.
So if we *assume* most or all of the complex things the brain is doing are necessarily related to accurate prediction in the physical world, and fail to take into account that it might be (I would say almost always actually is) trying to optimize the social interactions of the individual, we are quite likely to go astray in our subsequent reasoning.
People often say things like "Thank God I noticed that lion in time!". I think the premise that noticing and escaping from a lion is a negative experience could use examination.
Expanding a little more, a reinforcement learning system can't be solely defined by the method. The reward schedule is a part of the system that can't be separated from it.
I get a particular kick out of having opinions that contrast to some sort of orthodoxy or are rare. The fitness motivation for rare beliefs is that they have a high potential to be unusually valuable. Everyone says there's a lion to the north but I am motivated and got up three hours early to find out there's five to the south and we should probably go east or west. I want to be that guy
But rare opinions are baseline low hedonic. Being or being percieved to be the only person who is wrong is more bad than being the only person who is right is good
What's my special kick then? I am unusually confident in my ability to be the only right person. Instead of fear of non-conformity I feel the appeal of trying to demonstrate that I have a unique value that could save us from five lions
Regardless of the quality of my actual opinions a key ability I have gained in this respect is to contrast my opinion against people not in my sub-tribe and also against people who are in my sub-tribe. I can direct the motivation to some degree as I choose
Not in my sub-tribe are people who still fear the north lion. But there's also people now in my sub-tribe who agree with me and say "this is why we should keep going north"
Here's where the hedonic motivation loses a lot of steam for most everyone. My desire to be specially correct can also be directed against the "go north" people who agree with me. I'm like "actually with five south lions and one north we should go north by northwest". Now I have something that is prospectively valuable to the whole tribe but is also prospectively agreeable to both sub-tribes. Here's a hedonic (and hopefully practical) north star - finding the rare belief that could be attractive to two opposed sub-tribes
It's notable that this general willingness to be of unusual value is perhaps unprecedentedly reinforced by western culture. We are the red nosed reindeer looking for foggy nights. The underlying hedonic motivator has been overlayed with a large dose of stories about how there is a very delayed and potentially more optimal social outcome
It's notable that Rudolph's situation is not hedonically ideal for him. You can only be so happy. If it's rare you should not typically be able to negotiate routine hedonic losses to reach the rare outcome, even if the group's sum of well being is improved on net. I think it takes some combination of social encouragement plus unconventional personal characteristics to get there as often as you'd like. In my case I have both a very high and low self regard. I'm very confident in my ability to be right in opposition to a chosen group. So I am on average less sensitive to near term low hedonic experiences related to being different. On the other hand I have something like low overall self regard or low expectation of group regard. So I have a lower expectation and ability to experience near term high hedonic group acceptance states that aren't exceptional. I have a lower barrier to the path to unusual prospectively high hedonic states and I have a stronger desire or need to achieve them to feel I have truly gained a secure status within the group
I'm no expert on cognitive science, but I happen to have written a longish blog post last year from a game theory perspective looking at similar ideas in the context of your post on trapped priors: https://ingx24.wordpress.com/2021/05/27/the-trapped-priors-model-is-incomplete-guys-its-time-for-some-game-theory/
I am not very attracted to the explanation that taxes and politics are novel problems compared to finding lions in the wild. Early humans were probably exposed to politics like situations that posed threats to them: "Grug thinks Bagug is spreading lies about Grug to tribe", "What if red-rock tribe is about to sack our village to take berries?", "What if bagug hunts bigger game than Grug and thus surpasses Grug in social capital within tribe?"(I guess this last one would probably be less eloquently put by our paleolithic ancestors). Not only a bunch of political situations would be present in their lives, given the current data on the levels of violence between tribes of early humans, I think it is fair to assume that those political threats were an even more immediate and lethal to them compared to our current political quarrels. Illustratively it is fair to assume that Grug is much less safe having and Ugh moment about the red-rock tribe coming to sack his village for berries than some college student having and Ugh moment about going out to vote for who he deems to be the most fiscally responsible candidate in his local senate election. For me the much more obvious explanation is that we feel like not paying our taxes in time or losing some election are not problems in the present but in the future. I want to iterate that I am agreeing with the main idea of motivated reasoning as mis-applied reinforcement learning, I just feel like the explanation of why some problems are put in the Ugh category and others are dealt with surges of adrenaline are easily explainable by time preference. I mean, when the SWAT breaks your door after 10 years of unpaid taxes you'll probably not have much time to have an Ugh moment, but those 10 years were really sweet 10 Ugh years.
Love is healthy confirmation bias. We choose a spouse, we have children, we love, protect, and nurture them. If our family life is a source of happiness, we assume the person we married is uniquely compatible as a spouse and co-parent. It's probably not true, but it can't be proven otherwise.
> Motivated reasoning is the tendency for people to believe comfortable lies, like “my wife isn’t cheating on me” or “I’m totally right about politics, the only reason my program failed was that wreckers from the other party sabotaged it”.
I think a large part of the reason people believe things like these is they are crony beliefs, that is, beliefs that one has because they are socially useful not useful in connection with the non-social physical world. <https://meltingasphalt.com/crony-beliefs/>
So if someone thinks their wife isn't cheating on them, they might come across as more confident, which will help them socially.
Of if they have the same political views as all their friends, they won't get ostracised for being a non-conformist.
"But suppose you see a lion, and your visual cortex processes the sensory signals and decides “Yup, that’s a lion”. Then you have to freak out and run away, and it ruins your whole day. That’s a lower-than-expected hedonic state! If your visual cortex was fundamentally a reinforcement learner, it would learn not to recognize lions (and then the lion would eat you)."
I find this sentence quite baffling. I think you are conflating reinforcement with goodness or pleasantness. But I don't see any reason why reinforcement on a neurological level needs to have any kind of emotional valence tied to it. Brain regions are reinforcement learners that are being reinforced to do something USEFUL, not necessarily something that feels hedonicly pleasurable. The visual cortex isn't reinforced to make you see butterflies and rainbows, it is reinforced to turn complex noisy data into compressed, regularized, USEFUL data that other brain regions can do useful things with. Pleasure / goodness arise as a meta-heuristic to direct behavior in useful directions like survival and reproduction. From a biological perspective pleasure is a tool, not an end.
A visual cortex that recognizes a lion and saves your life will be reinforced because that outcome was useful to your biological imperative for survival and reproduction. Evolution doesn't give a damn that it made you feel bad to perceive that lion, it cares that you located and evaded a threat and survived to reproduce.
The Elephant in the Brain¹ explains Motivated Reasoning quite differently, and more disturbingly:
Your brain has many parts that operates semi independently, and many of them are not observable to the "you" that's reading this.
One of them we can call your Moral Center. It makes you genuinely want to do The Right Thing, as defined by some rules evolution probably generated to solve eons of iterated prisoner's dilemmas.
Another part, which has no name, rationally, unsentimentally and subconsciously determines what would be in the best interest for you to do. Sometimes that is to do good things for other people, so they'll reward you. Other times it's to do terrible things to other people so you can take their stuff.
Things like the latter don't sit well with the Moral Center. So your brain has another module, sometimes called the Press Secretary, which comes up with a good enough sounding reason for doing the terrible thing that the Moral Center, which isn't very smart, will accept. So you do the terrible thing, reap the rewards, and feel good about your superior morals.
And that's how at least one kind of Motivated Reasoning works.
¹ https://amazon.com/Elephant-Brain-Hidden-Motives-Everyday/dp/0197551955/
You might be interest in a paper by Firestone and Scholl (2016). "Cognition does not affect perception: Evaluating the evidence for “top-down” effects". One of their arguments is similar to yours: perception is essentially epistemic, and so it would be really really Bad Design if cognition was able to influence it in any substantive way.
But the visual cortex doesn't learn to see lions, it learns to *see*. And learning to see -- that is, learning to see *everything*, trees, lions, refrigerators -- could, quite plausibly, be achieved by hedonic reinforcement. Which would come not from the emotional valence of the objects in the visual field, but from the satisfaction of building a good working model of the correspondence between the visual field and haptic space.
You might find this article interesting https://onlinelibrary.wiley.com/doi/10.1111/japp.12577
I think you might be understating the importance of social relations to this question. None of these calculations are made at the individual level. IOW I can't wonder "is the blur a lion?" without wondering, probably below the level of awareness, what other people think about the blur. If I yell "lion!" and it's not, will people make fun of me and cancel my invite to the reindeer games? Or maybe it's a lion but I know all self-respecting members of our party know that the Great Leader eliminated the lion threat last year. Is it worth yelling "lion!" if I then lose all my friends and allies? (Sure, if the lion is running toward me, but otherwise, maybe not.) This is the root idea of Dan Kahan's work on this, of course: That people value good relations with others in their group far more than they value epistemic accuracy. So there may be a good deal more simple reinforcement happening here, on the order of "say there is no lion"-->all my friends cheer and all my enemies scowl-->feel good.
You seem to be crediting Roko Mijic with the term “ugh field”, but I believe the term predates him.
Roko says in that article:
> Credit for this idea goes to Anna Salamon and Jennifer Rodriguez-Müller. Upvotes go to me, as I wrote the darn article
This also aligns with my having first heard the term at a CFAR workshop about a decade ago.