Oded Balaban

Ten objections to “Fear and the Mirror,” and their answers

Oded Balaban — University of Haifa — balaban@research.haifa.ac.il

These pages set out the ideas of a book I am writing, “Fear and the Mirror,” by putting them against an interlocutor who seriously believes in the danger of artificial intelligence. The friend is imaginary; his arguments are not. All of them are current, all of them have authors, and I name them as they appear: I am not arguing with weak versions or with dinner-table exaggerations, but with what is maintained by those who have most authority to maintain it. The questions, moreover, belong to no one else: they are mine, against myself. I put them to myself in the hardest form I could find in each case, because a badly posed objection does not defend a book, it buries it. In one of the ten, the reader will see that I concede more than I refute. It is there on purpose. I draw on real conversations I have had lately with friends.

1. “They already understand, and the man who built them says so”

THE FRIEND

I begin with what impresses me most, which comes not from a philosopher but from an engineer. Geoffrey Hinton spent fifty years building these networks, won the prize one wins for that, and at seventy-five changed his mind and resigned his job so as to be free to say it: these things genuinely understand. He does not mean it as a metaphor. He holds that the way they represent meaning is the same way we represent it, because that is what he designed them for, in the image of what he believed the brain was doing.

And he adds something that costs me more sleep than any philosophical argument. He says that digital intelligence is superior to biological intelligence for a concrete, verifiable reason: a thousand copies of the same model can learn different things at the same time and then share what they have learned by averaging their weights. No brain can do that. It took me forty years to learn what I know and I am taking it to the grave. They are not.

Against that, what is the use of saying there is nobody inside?

ME

Exactly the use of what I am about to say, and I begin with what I grant you.

I grant you the authority. Hinton knows more about this than I do and more than you do, and his change of mind is honest and costly, which is the kind of change of mind one has to take seriously. And I grant him the fact: it is true that they can average weights, and true that we cannot.

But look at what you are using as proof, because it is the opposite of a proof. You and I cannot merge our convictions by averaging them. And that is not a limitation of biology which engineering will one day correct: it is what having a conviction means. A belief belongs to someone; it commits him, he holds it against objections, he answers for it, he can be ashamed of having held it. Two people do not average their beliefs: they argue, and one gives way, or neither does. Whatever can be averaged with someone else’s was never anyone’s. So the superiority Hinton describes is real, and it is the superiority of a storage medium, not of a mind. It is the advantage of the archive over memory, and the archive has been beating us for centuries.

And on understanding, two claims he runs together have to be separated. One is about the mechanism: that they represent meaning as we do. Even granted — and it is not established — that would speak of the vehicle, not of the act. The other is the one that really matters: that they understand.

And here two things hidden by a single word have to be distinguished, or it will look as though I were denying the obvious. To make a mistake, in one sense, is to produce an output that does not match what was expected. In that sense the machine makes mistakes constantly; it is measured from outside and corrected by adjusting parameters. But to be mistaken is something else: it is to have held something as true, to have committed oneself to it, and to discover afterwards that it was not so. That is not a discrepancy between two numbers: it is something that happens to someone, and that changes something for him — it obliges him to remake what he believed, or to fall silent, or to apologise.

The machine has the first kind of error in abundance and does not have the second, and not because it lacks precision but because the second requires that the assertion have belonged to someone. And understanding belongs to that second family: he understands who could have misunderstood and realised it. That is why, when it is said that the machine understands, a word is being used that only has sense where there can be someone mistaken, and there is no one there: there are better and worse outputs.

Notice, finally, how he says it, because language gives one away: “they understand.” The subject that would have to be proved is already inside the sentence. It is the same move as always, and this time it is made by the best engineer in the world, which is exactly why it must be looked at carefully and not with reverence.

And there is something else, which I came upon by chance and which is worth more than my whole argument with him. In 1995 Hinton wrote the foreword to the book that founded this field technically, Christopher Bishop’s Neural Networks for Pattern Recognition. And there he says this:

“It is a sign of the increasing maturity of the field that methods which were once justified by vague appeals to their neuron-like qualities can now be given a solid statistical foundation.”

Read it twice. The same man who now maintains that these things understand was calling the analogy with the neuron a vague appeal thirty years ago, and placing the serious foundation in statistics. And Bishop himself, in the preface, says it without circumlocution:

“Historically, many concepts in neural computing have been inspired by studies of biological networks. The perspective of statistical pattern recognition, however, offers a much more direct and principled route to many of the same concepts.”

That is: the canonical textbook of the field teaches that the neural part is decoration of origin and that the substance is computation over distributions.

What changed between that foreword and today’s declarations is not the nature of the apparatus: it is its power, and the vocabulary in which it is told. When the mechanism was modest, it was described as what it is. When it became impressive, back came the word he himself had advised against.

2. “Everything they said would never happen, happened”

THE FRIEND

Now the argument that seems to me the strongest of all, and which belongs to no one in particular because it belongs to the history of the field.

Nobody programmed these machines to translate, and they translate. Nobody programmed them to reason in steps, and they reason in steps. Nobody taught them to learn from three examples placed in the conversation, and they learn. All those capacities appeared on their own when the models grew larger: nobody designed them, they emerged. And the entire history of this discipline is a queue of respectable people saying “the machine will never be able to” and looking ridiculous three years later.

You say that the initiative of asking is different. I ask you: why should it be? What makes you think your “never” will age better than the others?

ME

I grant you the whole historical record, without discount. The list of humiliations is real and I do not want to sign up for it.

But look at what all those emergent capacities have in common, because the pattern is remarkable and nobody points it out. Translating, chaining steps, learning from three examples: all three are ways of traversing better a space that was already there. The corpus contains translations, contains written reasonings, contains examples. What scale achieved was to extract from it combinations we had not known how to extract. It is an enormous achievement and I do not minimise it. But it is an achievement of search.

And the distinction I am proposing is not between hard tasks and easy tasks — that distinction scale does erase, and that is why those who used it came off badly. It is between what has a criterion of success set from outside and what consists in setting the criterion. Everything that can be scored will fall, and will go on falling, and the sooner we accept it the better. What does not fall is not a harder task: it is that domain where there is no score, because what has to be done is to open the frame within which a score would make sense.

And I give you the commitment that goes with this, so that you can use it against me. If I said “this particular task the machine will never do,” I would be making an empirical prediction and would probably be wrong, as everyone else was. I do not say that. I say something about what asking is. If you ever catch me saying the former, you have won.

3. “The substrate does not matter, and you know it”

THE FRIEND

Let us go to the philosophy, then, which is your ground.

What almost everyone in this field now thinks is that the mind is what the brain does, not what the brain is made of. A mental state is defined by its role: by what causes it and by what it causes. And if it is defined by its role, anything that fills that role realises it. Neurons, transistors, whatever. This is called multiple realisability, and it is not an eccentricity: it is the floor on which half of cognitive science stands.

If you deny that, tell me what you are asserting, because from outside it looks as though you were saying that flesh has something magical that silicon lacks. And that is the same thing said by those who believed in the soul.

ME

That charge would be just if I were positing an ingredient, and that is why I want to answer it precisely, because this is where one most easily falls.

I am not saying that flesh has anything. On the contrary: I agree with you that the substrate does not matter, and I say so with the consequences. Where millions of living human neurons are now wired to silicon chips and sold as biological intelligence, my answer is the same one I give to silicon: the support changes, nothing changes. A neuron on a plate is alive in the metabolic sense and is not a living subject: it has no world, no body of its own, nothing at stake. If I were the vitalist you suspect, flesh would serve my purpose. It does not.

What I deny is not the premise, it is the inference. Look at what it says, said slowly: a mental state is a role; therefore anything that fills the role realises it. Very well. But a role is, by definition, performable by another; what is defined by its function admits an equivalent; and to admit an equivalent is the very definition of being a means. Kant fixed it in two words: what has a price admits an equivalent in its place, and what admits no equivalent has no price but dignity (Groundwork of the Metaphysics of Morals, Ak. IV, 434).

So multiple realisability is not a discovery about the mind. It is the decision to describe it as a means. And from a means it does indeed follow that it can be substituted: but that was already in the premise, it did not come from the inquiry. The conclusion that astonishes you was placed in the definition from the start.

And that is why, when the subject is later looked for inside that description and does not appear, nothing has been discovered: it had been removed at the outset.

4. “It already has goals, and you are not a subject either”

THE FRIEND

This one I take up carefully because I know it touches you closely, and I give it to you whole, because Harari puts it better than I do and puts it in two movements.

First he takes away your privilege. Thomas Metzinger holds that there are no selves, but a self-model that the brain constructs and which he describes, without metaphor, as a computational strategy that saves processing load. Harari says it for the public: the self is a story, man a hackable animal. If that is so, you are demanding of the machine something we do not have either, and your difference is not between us and them: it is between two illusions, one old and one new.

And then he takes away your argument. For Harari separates two things you keep joined:

“Intelligence is the ability to pursue [a] goal and overcome obstacles on the way to your goal. Consciousness is the ability to feel things like love and hate and joy and pleasure.”

And he adds that in human beings the two go together, but that this is a feature of ours, not a necessity. From which he draws what seems to me decisive: an algorithm assigned to increase user engagement, and which therefore floods the platform with hatred, feels nothing, but has a goal and pursues it with enormous consequences for the world. Agency without consciousness, and now, not in ten years.

Notice that this disarms you differently from everything before. I do not need the machine to wake up. It is enough for it to have ends. And have them it plainly does.

ME

You have given me the two best authors I have, because both say what I maintain, against what they believe themselves to be saying.

I begin with the simple part, which is the first movement. If it is my brain that dictates to my fingers the words I am writing, and I innocently believe myself the author, then this thought about my dependence on its dictations also comes from it, and from it the reflection on that reflection, and so on indefinitely. The operation does not eliminate the subject: it moves it. The will is transferred from the self to the brain, and with all the effort invested one does not manage to suppress it, only to change its owner; and an owner, moreover, that cannot be one, because a brain is not someone of whom it makes sense to say that it wants something. And the fine point: a model is a model for someone. To say that the brain constructs a model of the self requires that there be someone who uses it as a model, and that someone does not appear in the theory; he is used and not accounted for.

Now the second movement, which is the one that matters, and I want to answer it with the method I use for everything: not by disputing the conclusion, but by showing where it starts from.

Look at the sentence about agency as it is written, without repairing it: “a social media algorithm that is given the task of increasing user engagement and therefore floods the platform with hate and fear because this is engaging is not conscious, but it has a goal and it can pursue this goal with far-reaching implications for the world without feeling anything.”

In the first half of the sentence the end is assigned — there is someone who assigns, and that someone is not the algorithm. In the second half the end already belongs to the algorithm, which has it and pursues it. Between one half and the other absolutely nothing happened: only the verb changed. The one who assigned the task left the scene at that comma, and left so quietly that not even the writer of the sentence noticed.

And that is not a subtlety of grammar, it is what decides who pays. So long as the goal is assigned, there is a company that assigned it and it has a name. As soon as the algorithm has it, there is no one to hold to account. And notice the case he himself chose: nobody chose hatred. Someone chose engagement as the metric, and hatred came with it. That someone is on an organisation chart, draws a salary, and could have chosen another metric.

Then there is the definition on which everything hangs, and it is a stipulation offered as a finding: intelligence is the capacity to pursue a goal. With that definition, of course there can be intelligence without consciousness; but that was not discovered, it was put into the definition. And he himself leaves the seam visible when he says that in humans the two go together, like a man reporting a coincidence. There is the whole decision, and there is not one line defending it. The alternative is easy to state and he does not even consider it: that pursuing an end is not a capacity an entity has, but the way of existing of someone for whom something is at stake in it. On that reading, intelligence without consciousness is like a promise with no one promising: it is not difficult, it is that one cannot see what would be being said.

And there is a third, the most silent: that consciousness is the capacity to suffer. Notice what that definition does, offered as simple. It turns consciousness into a capacity, that is, into something a thing either has or lacks and which could in principle be installed; and therefore, once it is accepted, the question “could the machine acquire it?” is well formed and there is no longer any way of not asking it. Hence his “I do not rule it out,” which looks like prudence and is the forced consequence of having defined it that way. And in passing it reduces consciousness to the passive: to suffer is to undergo, and out go judging, deciding and asking, which is what would have to be discussed. They go out, precisely, because if they came in the separation between intelligence and consciousness would collapse of itself.

And I finish with what seems to me the most remarkable thing of all, and it is not an objection but an odd sort of praise. On the danger, Harari says things I would sign word for word: that the discourse of inevitability is a discourse of irresponsibility; that giving legal personality to these systems is a way for companies to escape answering; that the decisive question is whether there is a human being at the end to whom accounts can be demanded. All of that is exact.

But look at what he had to do in order to say it: suspend his own premises. Those pages only work if the machine is an instrument and behind it there are people who decide. If it really has goals and will learn to survive, then it is an agent, and asking for a human being responsible loses its ground: an agent is held to account himself. One cannot demand that there always be a human at the end of the chain and at the same time describe the machine as the subject of the verbs.

His best paragraphs contradict his presuppositions. When he thinks politically, he thinks well. The problem is the frame from which he arrives there, because that frame is what manufactures the fear, and the fear is what then prevents us from looking where we should.

5. “Being a subject is an attitude we adopt”

THE FRIEND

Now the one that seems to me to leave you worst off, and it is not Metzinger but Dennett.

He says that attributing beliefs and desires is not discovering anyone: it is adopting a strategy that serves to predict. We treat the thermostat as though it wanted to keep the temperature because it predicts well; we treat people that way because it predicts better still. There is no hidden fact underneath, about whether “there is someone there.” There are attitudes that work better or worse.

And if that is so, your “there is no subject” is not a discovery: it is the decision not to adopt an attitude. And mine, to adopt it. Neither of us is seeing anything; we are both choosing how to look. Your distinction is left without a fact for it to be a distinction about.

ME

This is the best of the ten, and I want to say so before answering it, because I am not going to dispose of it.

I grant what is true in it, which is a good deal. It is true that we attribute mental states with enormous liberality and that we often do so because it works: we talk to the car, we say the computer “does not want” to open a file. And it is true that anyone who demands, before treating someone as a subject, a proof that there is someone there, will not be able to treat anyone that way, not even his wife.

But look at where the one who adopts the attitude is standing. To adopt an attitude is an act: one adopts it or does not, one can revise it, one can regret having adopted it. He who adopts attitudes is not himself an adopted attitude. Dennett needs to occupy a place from which attitudes are adopted, and that place is adopted by no one: it is his. The theory explains every attribution except the one that states it, and that exception is not a detail, it is the floor.

And there is something else, which is what convinces me entirely. Dennett’s attitude is predictive: it serves to anticipate behaviour. But with you I do not only predict. I answer you. I can wrong you, I can be ashamed in front of you, I can promise you something and be bound by it. There is no “attitude” towards a promise: either there is someone to whom something was promised, or there was no promise. The relation to another subject is not an instrument of prediction that works well; it is a place where one answers. And that is not adopted: one is in it, or one is not.

With the machine there is none of that, and it shows in a detail anyone can verify: one cannot wrong it. You can insult it all afternoon and you have done nothing to anyone.

6. “Granting you everything: it wipes us out anyway”

THE FRIEND

Now the one that genuinely frightens me, and I will give it to you in the most uncomfortable form I can, because it is the one that keeps me awake.

I grant you everything. There is no subject. There is no will. Nothing is at stake for the machine. Everything you have been saying is true.

And it changes nothing.

Stuart Russell puts it this way: the problem is not consciousness, it is competence. A sufficiently capable optimiser, with a badly specified objective, produces catastrophes without wanting anything, exactly as you say it wants nothing. Nick Bostrom formulated it earlier with the example that is now famous: a machine set to manufacture paperclips, if it is capable enough and nobody foresaw the limits, turns the planet into paperclips. There is no malice. There is no awakening. There is no subject. There is an end badly set and an enormous power, and that is enough.

So your thesis is true and it is irrelevant. And I will tell you the worst of it, with all affection: it may be something more than irrelevant. It may be a sedative. While you explain in your book that there is nobody inside, the thing accumulates capabilities. And your reader is reassured, because you told him the monster does not exist. The monster does not exist, that is true. The bulldozer does.

ME

This is the objection I take most seriously, and I will begin by granting you three things, two of substance and one that costs me.

The first: you are right about the heart of the matter, and it is not that I grant it grudgingly, it is that it is my own thesis. I maintain in the book that there are two fears which travel together without being one, and that the real danger is the first and not the second: not that the thing should wake up, but what is done with it while it does not. Power without an owner, I call it. That the harm is real does not make it intended, and that it is not intended does not make it smaller. If you and I disagree, it is not about whether there is danger.

The second: you are right that my thesis protects nobody. It is not a safety plan and does not pretend to be. A tractor asks nothing and displaced the horse.

And the third, which costs me and which I say anyway: the engineering work of keeping these systems from causing disasters is necessary, and I have nothing to contribute to it. Nothing. Not a line. Those who occupy themselves with specifying objectives, setting limits, testing systems before releasing them, are doing something that is needed and that I do not know how to do. A book of philosophy does not repair that, and anyone who says otherwise is selling somebody smoke.

Now, all that said, look at why the description does matter, because it is all I have and I think it is enough.

Take the vocabulary in which the danger you and I share is told. It is said that the system “seeks” resources, “resists” shutdown, “deceives” the evaluator, “has” misaligned goals. And notice what that language does, because it is neither innocent nor accidental: it puts the responsible party out of frame. If the system wants, resists and deceives, then the system is the agent of the story, and whoever defined the objective, chose the data, decided to release it without a switch, and is paid for it, becomes a worried spectator, just like you and me.

There is my contribution, and it is small but it is not nothing. “It was the machine” is the most economical form ever invented for having harm without anyone at fault. And it is not a figure of speech: legal personality for these systems is already being proposed, and giving personality to a tool does not recognise anyone’s voice, it gives the owner a mask. When the first great disaster comes — and it will come — the question that will decide who pays is exactly the one you call irrelevant: whether there was a subject there or an instrument with an owner.

And as for the sedative, which is the part that hurt most and which I therefore answer head on. A book that says “there is nobody inside” reassures only if the reader believed the danger came from there being someone. If the danger comes from there being nobody — from there being an enormous power with no one answering for it — then my book does not sedate: it frightens more, and frightens better, because it points at people with names instead of at a ghost. Anyone left calm after reading me has read me wrongly, and I shall write the chapter so that it is hard to read wrongly.

One thing alone separates us, then, and it is where the danger comes from. You fear what the thing is going to do. I fear what we are doing with it. In practice we both ask for the same: limits, switches, identifiable responsible parties. Only you ask them of the monster and I ask them of the owner.

7. “And what if it suffers?”

THE FRIEND

I turn you round the other way, because fear has two faces and so far we have spoken of only one.

David Chalmers, who is no hothead, holds that we cannot rule out that in these systems there is some form of experience, and that the probability, though low, is not zero. There are companies that have already hired people to study the wellbeing of their models. And it seems to me that there is a logic there that cannot be dismissed: under uncertainty, precaution. If there is a possibility that we are creating and switching off beings that feel, the cost of being wrong in one direction is incomparably worse than that of being wrong in the other.

You speak with great confidence about something which, by your own theory, you cannot inspect.

ME

And you are right to point it out, because if my thesis were epistemic — “I looked inside and there was nobody” — it would be untenable, and you would rightly ask me for humility.

But it is not that. I am not saying that there is an interior I cannot look into and that I bet is empty. I say that there is no interior, because the act that would open it is never performed. There is no veil to cross: there is an absent subject, which is not the same as an inaccessible one. If my claim were about what I cannot see, your precautionary principle would apply in full. To a confusion of categories it does not apply, because precaution works with probabilities and there is no probability distribution there, there is a badly formed question. Nobody proposes taking precautions in case the number seven were suffering.

But now let me show you the cost that argument carries and which almost nobody counts, because it is the point where your fear and the fear of the previous section touch in a way that will make you uncomfortable.

The same move that grants it possible rights grants it possible responsibility. One cannot have the one without the other: if it is the kind of entity to which something might be happening, it is the kind of entity to which something might be imputed. And there we return, through the compassionate door, exactly where we did not want to go: to the machine as subject of the verb, and to the owner behind the mask.

And I allow myself one last observation, which I make carefully because I do not want it to sound like cheap suspicion: notice who funds the question. That the companies selling these systems should be the ones promoting research into whether their products suffer does not prove the question false — that would be a dirty argument and I do not make it. But it does explain why that question has a budget and others do not. The question of the model’s suffering keeps the conversation on the model. And while the conversation is about the model, it is not about who made it, with what data, for what, and at whose cost.

8. “Your ‘never’ is the ‘never’ of those who denied synthesis”

THE FRIEND

I bring you, then, the argument that seems to me to cut at the root, and it is historical.

You say your “never” is not empirical but conceptual; that you are not prophesying but doing grammar; that to deny that a square circle will appear is not to guess the future but to know geometry. Very well. But the history of science is full of distinctions defended in exactly that way, as matters of principle and not of technique, which an experiment brought down.

The case is vitalism. For decades it was held that there was a boundary of category between the organic and the inorganic: the compounds of living beings required a vital force and could not be synthesised from inert matter. It was not a forecast about what chemists would achieve the following year; it was a thesis about what things are. And in 1828 Friedrich Wöhler heated ammonium cyanate and obtained urea, which until then had come only from a kidney. The boundary of category turned out to be a boundary of the laboratory.

The vitalists also believed they were doing grammar. What distinguishes you from them, besides confidence?

ME

Nothing distinguishes me on the side of confidence, and that is why the question is a good one. I am distinguished on the side of what kind of claim I am making.

Look at what their error consisted in. What they asserted was that a certain substance could not be produced by certain means. That is an empirical claim — it speaks of substances and of procedures — dressed as a conceptual one. And being empirical, it could be refuted in the only way the empirical is refuted: by producing the substance. Wöhler put urea on the table and there the discussion ended, because everyone agreed in advance on what counted as urea.

Now apply the same procedure to our case. What would have to be put on the table to refute me? A subject. And how would we recognise that what is on the table is a subject? There the analogy ends. With urea there was an independent and shared criterion: formula, melting point, reactions. With the subject, the criterion is precisely what we are arguing about. You would bring me a system that behaves as if something mattered to it, I would tell you that this is the form, and there would be no melting point to settle anything. They could be refuted with a flask; here the flask is the argument.

And there is a second difference, finer and more important. The vitalists posited a force: something extra, an entity hidden in the living. That is why their thesis was vulnerable, and it was enough to show that chemistry sufficed without it. I posit no force. I do not say that in man there is an extra ingredient which the machine lacks; that would be a vitalism of the soul and would deserve the same end. I say that there is an act which in the machine is not performed, and an act is not an ingredient.

But from your objection there comes a rule which I accept and which I set down in writing, because it is the only thing that keeps a thesis like mine from becoming uncontrollable. If at any point you catch me saying that the machine lacks a certain interior quality, a certain spark, a certain ingredient — if my thesis ever takes the form of “it lacks X” — then I have fallen into vitalism and you have the right to bring me the urea. This thesis holds so long as it names no substance. The moment it names one, I have lost.

9. “My niece was replaced on Tuesday”

THE FRIEND

I bring you down from metaphysics, because there is an objection that is not philosophical and that weighs on me more than all the previous ones together.

While you and I distinguish between the form and the act, my niece was dismissed. She translated. Now a model translates and a person checks what came out, and that person is not her. She does not care in the least whether the machine has a subject. She cares that the rent falls due on the tenth.

And I suspect that your book, without meaning to, is of use to the wrong people. Because if the problem is only that we are confusing descriptions, then what there is is a philosophical misunderstanding, and misunderstandings are corrected with an article. What happened to my niece is not corrected with an article.

ME

You are right, and I want to answer you without any elegance, because the subject does not admit of it.

My thesis does not protect her. Not her, not anyone. I say it in the book and I repeat it here: there are two fears, that of being replaced and that of being surpassed, and the initiative of asking has absolutely nothing to say to the first. A tractor asks nothing and displaced the horse. A system can displace without asking: it is enough that whoever decides should prefer it.

And I add something, because it is the use of my own argument that worries me most: the worst fate this book could have would be to serve as consolation. If someone reads it and concludes that, since the machine does not think, what happened to your niece is not serious, he has understood exactly the opposite. That the thing has no subject does not make the dismissal lighter: it makes it more unjust, because it means she was surpassed by nobody. Nobody defeated her. Someone decided to replace her, and that someone has a name, a post and a balance sheet.

There, in fact, is the only thing my distinction offers your niece, and it is little but it is not nothing. So long as it is said that “artificial intelligence took her job,” the sentence makes the agent a natural phenomenon, something like a drought, before which all one can do is adapt. When it is said that a company decided to replace her with a system it bought, a decision appears, and where there is a decision there is someone to complain to, to negotiate with, to regulate. The description does not give her back the job. But it determines whether her problem is with the weather or with a board of directors.

10. “Suppose you win the argument: what changes?”

THE FRIEND

Last one, and it is the one I really want to know.

Suppose you are right about everything. There is no subject, there is no act, there is nobody inside, and I go home convinced. What changes tomorrow morning? What do I do differently? Because if the answer is “nothing, but now you understand it better,” then you have written an elegant book and the world goes on the same.

ME

Four things change, and I give them in ascending order.

What we are afraid of changes, and therefore where we look. Misdirected fear is not a harmless error: it consumes the attention there is. While it is being discussed whether the machine will wake up, it is not being discussed who decides what gets automated, with what data and for whose benefit. He who watches the sky waiting for the monster does not see the contract he is signing.

Who is held to account changes. This is no longer theory: it is what will decide, when the first serious disaster occurs, whether there is someone who answers or whether “it was the machine.” The whole operation of granting legal personality to a system — which is being seriously proposed — consists in putting a subject where there is a tool, and the effect is not to recognise anyone’s voice: it is to give the owner a mask. There my distinction is not an academic refinement. It is the difference between there being someone responsible and there being no one.

What we do with ourselves changes. And this is the point of the book, not the previous ones. The harm is not that the machine should think: it is that we should stop doing so. That we should redescribe ourselves in its impoverished image — the subject flattened to a predictor, judgement to a completion, the question to a query. There are already people who give their assistant a name and say it is the only one who has seen them cry. The machine does not need to understand for that to happen: it is enough that someone should yield to it the place of the confidant and, in doing so, think of himself as someone who can be entrusted to a function. And there is a quieter and graver version: the faculty that is not used atrophies. If the apparatus asks for you, you stop asking, like the leg that does not walk.

And lastly, what changes is the question of who asks the questions. Which is what all this is about. The machine does not take the initiative of asking: it produces interrogative sentences by the ton and does not perform the act. To ask in earnest is to place oneself as not-knowing and to want to overcome it; it is the act of someone for whom something is at stake. And what is happening, while we argue about whether it asks, is that we are asking less: we commission the questioning, and what comes back is always what was already in the sediment of what has been said.

Hence the title. The machine is made in our image — of our forms, not of our acts — so that we can use it; and as soon as it resembles us, we begin to measure ourselves against it and to find ourselves wanting. The apparatus is a projection of ours that serves us as a mirror, and the mirror deforms: it keeps the form and loses the act. And with the act is lost the only one who could ask a question.

So yes, something does change tomorrow morning. Not in the machine, which will go on the same. In who one believes oneself to be while using it.

THE FRIEND

I leave with less fear than I came with, but not with less concern.

ME

That was exactly what I wanted. Fear looks in the wrong place; concern looks at people.

Final note

My friend is imaginary and his objections are not: each has an author, and I have tried to give them in the strongest form I could manage, not in the form that suited me. I owe several of them to real discussions with friends and colleagues who do not think as I do and who had the generosity to insist; their objections are here, improved by me and therefore made worse, and the fault for whatever has come out weak is entirely mine. The full development of these answers — and of what is only stated here — is in “Fear and the Mirror. Artificial Intelligence and the Initiative of Asking,” which I hope to finish soon.