This post of Eli has many links and may be a good occasion for delving into some of the issues he has been talking about lately.
Overcoming Bias: Heading Toward Morality
Why? Because that is the conclusion he is drawing to:
Artificial Intelligence melts people’s brains. Metamorality melts people’s brains. Trying to think about AI and metamorality at the same time can cause people’s brains to spontaneously combust and burn for years, emitting toxic smoke – don’t laugh, I’ve seen it happen multiple times.
But the discipline imposed by Artificial Intelligence is this: you cannot escape into things that are “self-evident” or “obvious”. That doesn’t stop people from trying, but the programs don’t work. Every thought has to be computed somehow, by transistors made of mere quarks, and not by moral self-evidence to some ghost in the machine.
If what you care about is rescuing toddlers from burning orphanages, I don’t think you will find many moral surprises here; my metamorality adds up to moral normality, as it should.
snip
Yet there is also a good deal of needless despair and misguided fear of science, stemming from notions such as, “Science tells us the universe is empty of morality”. This is damage done by a confused metamorality that fails to add up to moral normality. For that I hope to write down a counterspell of understanding. Existential depression has always annoyed me; it is one of the world’s most pointless forms of suffering.
The last paragraph nicely illustrates the goal of my thesis. Much of resistance encountered to scientific thinking stems from confused thinking about existential questions, morality and the meaning of life etc. This pre-scientific – mostly magical and inconsistent – thinking can and should be dispelled.
On a side note:
As you will have noticed I often quote Eli’s posts – simply because he has eloquently written down what I only have been thinking about. Most of the times when I want to write something down I realize I can’t explain it in a few sentences or even a few paragraphs – NOT because it is confused or not coherent yet, but rather because after years of thinking on these subjects I have dissolved many mental categories, rewired others and created new ones. Then I fall into a kind of despair and write nothing (I really have to work on this).
The funny thing is that Eli Yudkowsky (and many others who also have not written yet, but are among the OB readers or elsewhere strewn over the internet) have undergone similar mental evolutions. So much for multiple discovery.
While many scientists and philosophers have “problems” with multiple discovery because they are concerned over priority, I think it is a happy thought that ideas are discovered simultaneously in many places when the time is ripe. It shows that intellectual progress is robust against individual chance events.
My main problem is how I will deal with this inferential gap issue in my thesis: while people working within traditional philosophical schools simply refer to their “founding fathers” – for instance Kant – and their advisors immediately know what they are talking about (indeed, they taught it to their students!), I can’t simply refer to Eli’s posts in my thesis – most people do not know him – not yet, anyway 
, maybe in 20 years time.
So I will either have to bite the bullet and write a huge expository chapter; or wait until Eli writes his book and ask my advisors to read it before I have my defensio 
(I guess that would be a quite cheeky approach – I’ll probably settle with the expository thing )
A last remark: why this large deviation from standard philosophy? Because standard philosophy misses some major insights, namely those gleaned from AI research of the last fifty years. Every philosopher should plow through this book (Russell & Norvig: Artificial Intelligence. A Modern Approach. 2nd Ed. 2002) before commenting on issues such as rationality or free will.
Funnily enough, Eastern Philosophy (Taoism, Buddhism) does not have this problem: that is because they stressed meditation, which is empirical/rational introspection versus the rather naive introspection performed in the West. Thus, the sages of Zen discovered many things about themselves which escaped the likes of Descartes, Kant etc.
