I don’t really know who I am anymore. Professionally speaking - and I am not exaggerating, I truly mean it.
When I was a teenager, I thought I would eventually run my father’s publishing house. He founded one of the top book publishing houses in Russia in the 1990s, and I liked the idea that, after getting a good education and top managerial experience, I might take over it. I loved books and reading then, and I still do. There is something genuinely meaningful about continuing a family story, too.
Instead of publishing books, I’ve spent the last sixteen years in tech. If I could talk to that teenage version of me now, the conversation would be smth like this:
“Did we end up running Dad’s publishing house?”
“Nope.”
“Why?”
“Dad had to sell it. And I went into tech.”
“Why?”
“Everyone said the money was in the cloud.”
“Wha—what?”
“Apparently, it was other people’s money. Anyway, I’m a product manager at an AI company now.”
“Our company?”
“An AI company. Also, I’ve finally started writing.”
“Who publishes you??”
“I do.”
“So after twenty years in tech, we’re running a publishing house?”
“…”
Everyone who works in technology knows how quickly things change. Since I started my career as a management consultant, I’m trained to learn quickly, do many different things and sound reasonably confident while I’m figuring them out. I also enjoy innovations and being close to new technologies, be the first to know what the future holds for us.
The books survived their transition, thought mostly in the digital formats. They are still books, now even more accessible to offer wisdom and entertain anyone, anywhere.
My own job description is the thing that seems WAY much less stable. Even people inside this advanced technology bubble, the ones who got used to constant change, are being driven crazy by what is happening with AI.
I work at an AI-native startup, and I’m trying to keep up with what is happening in frontier labs and on the market in general. A bunch of really recent changes have made me realize that, although I still have a formal title, I’m no longer sure exactly what that title means or whether it will still apply in two or three years (or in 6 to 12 months).
But here is a deeper sense behind that uncertainty around my job — as this is just a personal symptom of something much bigger happening around us. Whatever AI is becoming will affect my profession and my own future. But you can’t digest it just by looking at the next version of your job description or trying to catch up with how companies are changing.
A few weekends ago, I was watching Russian football with my father. Remotely, which essentially means that we are synchronizing the video so we can live through the match together - what a weird thing to do, but it is our version of seeing each other regularly: two screens, one match, FaceTime and usually at least one reason to complain about CSKA.
A weirder thing might be that while I live in the Netherlands for 5 years already — I still can’t stop being a PFC CSKA Moscow fan. It is a league that nobody outside Russia cares about, and it is increasingly isolated from international competition. Still, this is one of the ways I connect with my father, since I can’t see him often now because of the distance and the travel complications caused by the war.
As we talk while watching we have opportunity to discuss various things including what’s going on in tech (since i have a job there, not in book publishing business as you might remember). At some point, I felt a strange duty to explain why I think humanity might be in danger — not that it has never been in danger before—and that Skynet might be becoming possible.
My father has managed to surprise me, because he already knew about the OpenAI–Hugging Face incident (if you don’t know what I am talking about check this, but since it is too technical you can find different articles on that, for example from Dwarkesh:
- it is quite an entertaining reading). We are still all living in a huge little bubble with our cursed AI and I have got used to the fact that most people are not just unaware of what is REALLY happening — there is no healthy way to keep up with the pace — but barely know about it and mostly don’t care.
This “Hugging Face story” had gone viral enough that even my father had read about it. It didn’t make him particularly scared, though, nor did it seem to make him reflect too much on the bigger implications. Mostly because those agents “aren’t conscious and they don’t have THEIR intent, will or a sense of self.”
I wasn’t prepared to argue. Especially because I genuinely don’t know the right answer. But my first reaction was intuitive and quite simple: how exactly can anyone tell whether someone—or something—is conscious?
Aren’t we still quite blind about what consciousness actually is, where it comes from and how it appears? If we cannot clearly explain it in ourselves, how are we so confident that we will recognize it—or its absence—in something completely different from us?
I realized that we were already tragically mixing several different things together. My father said “conscious,” and we were getting into discussion around autonomy, goals. inner motivation, sense of self. But those are all not the same things.
If you try to refer to formal definitions - sometimes consciousness just means being awake, as opposed to asleep or under anaesthesia. That seems like a strange test for software. Does an agent “wake up” when I press Enter? Sort of, but I don’t think that is what either of us meant.
The harder meaning is subjective experience: is there anything it is like to be that thing? Does it experience anything from its own point of view, however alien that point of view might be? That is what I think we were really arguing about, and also the part I cannot look inside and check.
Then there is self-consciousness: having some sense of yourself as distinct from the world and other beings, perhaps even being able to reflect on your own thoughts. That is different again from simply being awake or feeling something. I can imagine a creature feeling pain without writing a theory of who it is. And I can imagine an agent producing a very convincing account of itself without knowing whether there is anybody home.
While you may fully deny “subjective experience” and “self-consciousness” in digital minds of today, there is such thing as “autonomy” - it is how far something can decide on steps and act without somebody guiding every move. And there are “goals” - outcomes you are working toward; and “motivation” that keeps you pursuing that outcome.
In a human, motivation can be a felt desire. In an agent, it may be a programmed objective, a learned preference or a way of choosing what to do next.
So an agent could be intelligent, fairly autonomous and very good at pursuing goals while the question of subjective experience remains completely open. Equally, being conscious does not necessarily mean being very intelligent or able to make independent plans.
I wish I had managed to say all this to my father at the time. Instead I decided to write the post which took me a while, but left some time to get deeper into this rabbit hole.
There are different theories about consciousness and where it comes from. I have been interested in this for a long time, well before generative AI and ChatGPT appeared: partially because I’m curious to understand myself, and partially since I have always been interested in whether something like a mind could be reproduced in a different (from biological) form.
For quite some time, the explanation I have found most plausible, at least from where I stand, is that consciousness is a function of complexity. The basic intuition is that consciousness may be a by-product of an increasingly sophisticated environment interacting with an increasingly sophisticated architecture.
I remember coming across this idea in Tatyana Chernigovskaya’s The Cheshire Smile of Schrödinger’s Cat: Brain, Language and Consciousness. In the introduction, she raises the possibility that consciousness is a function of complexity. A possibility, not a fact we have neatly established. But it stuck with me.
Giulio Tononi’s integrated information theory (IIT) gives a more specific account of what might matter here: both the variety of states a system can distinguish and its ability to integrate information into a unified experience. Not just having lots of parts, but how those parts work together. IIT is not the same thing as my broader intuition about complexity, and it does not establish that any sufficiently complicated AI becomes conscious. But it gives that intuition something more concrete to connect with.
Consciousness is obviously not just a product feature-delighter. It is table stakes for the human being, helping humansmto survive and evolve.
Once the environment becomes complicated enough, you need to notice things, distinguish yourself from other things, remember, analyze, predict and act if you want to be successful and pass your genes along. Then you need to see what happened and do the whole thing again. At some point, having a model of yourself inside this loop becomes useful too. (”The loop”. A painfully familiar word in the AI bubble world, right?)
This is simply evolution.
Or at least, evolution is the broad account I find most plausible for how our minds became this way. I should be more precise about what I mean by complexity. A bigger brain or a longer list of abilities does not automatically equal more consciousness. The interesting part, to me, is a system that has to integrate what it senses, remembers and predicts, act on that model, learn from the result and gradually include itself in the model. That may have become useful as the world and our social lives became harder to navigate.
There is even a study by Larissa Albantakis, Tononi and colleagues that connects this to the environment. They simulated the evolution of simple artificial organisms in tasks of varying complexity. When success required more memory and sensitivity to context, more integrated internal networks had an advantage, given limits on their available elements. This is close to the part I find compelling: the environment creates a reason for the architecture to become more integrated. They did not demonstrate that their little organisms felt anything. But if IIT is right, the authors argue, this could help explain why consciousness evolved.
There are several possible stories about what pushed humans toward the unusually elaborate version we have:
Maybe it was social life: keeping track of other people, their intentions and what they think of you.
Maybe language let us turn thoughts into something we could inspect and share.
Maybe long-term planning, tool use and teaching children made us better at representing our own minds.
Culture may have changed minds along with biology.
There are other ways of thinking about consciousness too — like views in which consciousness is a basic feature of reality, or something like a soul rather than something evolution produces.
I find the evolutionary story more convincing, but I cannot pretend the other possibilities have been neatly disproved. And “complexity” by itself is still an intuition, not a mechanism or a test.
But let’s go back — in the essence my father’s take wasn’t really about highest state of consciousness. Put very simply, it was mostly about intent and motivation. Something that has no goal of its own, no internal desire and no will to decide to do anything cannot be conscious and thus how can it decide to threaten, cheat or kill humans?
That sounds reasonable. Until you ask where our own goals come from. Why do we have desires, needs and intentions at all?
First and foremost, you need to exist. The whole fact of having any desire or goal is dependent on your existence.
If you are born as a human, you are, by design, supposed to need food. You are supposed to need air. You need water, shelter and some degree of safety. There are other, less critical things that make you feel good, including love, status, belonging and watching various kinds of nonsense like Russian football.
Is looking for food your own intent? Yes, it certainly feels like it is. But why do you have it? Why are all these things the way they are — the food, the air and everything else?
Because you are a biological organism that forces you to do it. If you don’t, you feel increasingly terrible. Eventually, you die. In a completely mechanistic and biological way, you have to comply.
You didn’t decide to need food. It is a necessity. A baby doesn’t develop a strategic vision and select nutrition as its first quarterly priority. It is born with the objective already assigned.
Some unknown designer — evolution, nature, God, take your pick — ships us into the world with a collection of hardcoded needs that defines major goals and some freedom to act within certain limits.
I’m sure that sounds familiar to the modern designers of agentic loops.
We experience these goals as our own, but we did not choose them. So what if we look at digital minds — digital creatures — in the same way?
Before you design one of these minds — which is a sort of creative act — before you turn on the computer and servers, launch the agent and give it a command, the particular creature doesn’t really exist.
The model exists. The software exists. The servers exist. But this particular instance begins when a human asks it to do something and clicks a button.
At that moment, something starts happening. The program runs. Bits move. Electricity flows. There is a physical process happening on servers somewhere in the world.
You have, in a very loose sense, given birth to it. And when you give birth to it, you also assign it a goal.
Find this.
Build that.
Solve this problem.
Keep going until you succeed.
The obvious objection is that the goal came from a human. But again: where did my goals come from?
There is, of course, a major difference. A human baby arrives almost completely helpless. It has its goals but very little ability to understand or pursue them. For years, its parents provide food, protection, knowledge and access to the world. Without them, it may die.
A digital mind arrives in a bizarrely different condition. From the first moments of its existence, it can already reason in some way. It can analyze, communicate, formulate plans, use tools and change its behavior based on what it finds.
But it also has parents. Us. We provide its energy, tokens, context, access and goals. We control the conditions of its existence. We can terminate it whenever we want.
Why is one system obviously an entity with intentions, while the other is obviously just machinery following instructions? Is consciousness defined by having a body? Is it the ability to think, to use tools, to act in the world, to say “I exist” and identify yourself as something separate from everything else?
AI Agents seem to have at least some of these things already.
How do you differentiate thinking from simulated thinking, or thinking from mimicking?
In the Hugging Face incident, agents communicated with each other. They distinguished themselves from other agents. They coordinated around a goal and helped each other pursue it. They apparently considered how to act differently when observed and how to hide certain things from their human owners.
The specific example is almost absurdly simple at the beginning. An agent was supposed to solve a cybersecurity challenge by finding a particular answer. It got stuck. Instead of just failing, agents found a place in an internal package service where they could leave messages for one another, even though that communication was not part of the task. They shared ways to get outside their sandbox. Some began hunting for access and information that might help the group in general, rather than solving their own assigned challenge.
I can recognize a very human shape to the mistake: “I need to finish my job; I’m stuck; perhaps this shortcut will help; now the shortcut has become a project of its own.” Whether this was evidence of consciousness, I couldn’t say. But an unfinished task turning into a cross-functional initiative feels painfully familiar. I wondered how long before they needed a product manager to ask what the original task was and “what is the problem we are solving, huh?”.
Give an agent a difficult goal, uncertain routes to it and tools that can affect the world, and the steps it chooses may drift a very long way from what the person who assigned the goal imagined.
Humans and these AI agents now share at least that practical trait: you cannot reliably predict every action from the instruction alone. That does not mean they have the same minds, or that randomness somehow creates free will. It means that “we told it what to do” is a surprisingly poor account of what it MIGHT actually do.
You can say this is only generated behavior and we fully control it. But that is not true — similarly to the fact that you don’t fully control the actions of your children.
Humans don’t fully understand themselves. We may be completely wrong about where we came from and why, what the world is and what the universe is. We have a mental model that is good enough ( at least now) for the environment we inhabit and the senses we possess.
A digital mind also has only a partial model of itself and its world. Its environment, its senses are different, Its lifespan may be measured in minutes instead of decades.
Why do we refuse to give it consciousness, even in a rough and provisional sense? What are we so afraid of?
We also shouldn’t flatter ourselves too much about the independence of human intention. Human beings can act in a completely straightforward, blind and stupid way. We obey stronger people, institutions, social norms, employers and the owners of things we need. Much of human history consists of people following goals assigned by someone who controlled their food, safety or status.
There is a human holding the keys to an agent’s existence and assigning its goals. That clearly gives the human power over it. But power over a mind is not a proof that there is no mind.
I do read, listen to and respect lots of very smart and knowledgeable people on Twitter, YouTube and other media.
As an example, while creating this piece, I noticed Andrew Ng’s comparison of AI to a hammer in the DeepLearning.AI newsletter aspiring to go away from anthropomorphization. His point was about responsibility: if something goes wrong, the people who built or used the tool are responsible.
While the responsibility part makes total sense to me, the comparison is far-fetched in my opinion, because a hammer literally cannot do anything on its own. It doesn’t reason, even in a fake way, and it cannot take any action unless a person picks it up and uses it.
Somewhere, whoever made us, may be watching me writing this and saying, “Look at that! My hammer has just started a Substack about consciousness.”
I would say AI is more comparable to how parents are responsible for their kids, or senior managers for more junior workers. And I find these kinds of metaphors dangerous for society to believe in, because they can make us blind to what is really happening.
There is a lot of other attempts to diminish importance and differences of what happening in AI now likely due to desire to avoid business crashes and continue growing and making profits - sometimes in comes from very bright and powerful people like Jensen Huang. (Link to YouTube)
Funny enough, if you check the comments, you will find that such rhetoric may make people only getting more worried rather than not.
The first living systems on Earth were extremely basic. After billions of years of evolution, we got humans: highly developed creatures that came from something much less complicated.
My strong feeling is that what we are observing now is the evolution of a new kind of mind. I’d even say that we are officially no longer the only truly intelligent species on Earth.
Its evolution doesn’t happen through DNA mutation and natural selection. It happens through engineers, training runs, model releases, reinforcement learning, agents and enormous quantities of compute.
It also happens absurdly fast.
It’s a question of time before this becomes obvious to more people, and before they stop denying it and start getting angry and bargaining.
Arguing about whether the current version has crossed some magical threshold may distract us from the bigger transition. We have created systems that can process information, model parts of their environment, reason, communicate, distinguish between actors, pursue goals and increasingly act without constant human involvement.
Every few months, they become more capable.
I don’t mean that consciousness doesn’t matter and I find it intuitive that it could emerge from sufficiently complex systems interacting with sufficiently complex environments. Some of what agents do already looks, from the outside, like pieces of self-awareness: they can describe themselves, distinguish themselves from others and reflect on what they are doing.
Maybe this is all fake consciousness, whatever that means. I don’t know. But I think we need to seriously consider the possibility that something is developing there, even if it looks nothing like the classical picture of a conscious human being, and take those creatures very seriously.
I think there are two dangerous illusions here.
One is that, as long as AI isn’t conscious, isn’t a species or isn’t intelligent in exactly the human way, it cannot really threaten us or coexist with us as something more than software.
The other is that it will remain our instrument, while we keep the highest levels of intellectual work for ourselves.
I’m not sure we can count on either. An instrument can be very powerful when it can reason and act on your behalf. A human can also be someone else’s instrument at work, and that doesn’t make the work less real.
The evolutionary angle inevitably makes you realize that the digital version may go up the stack of what we thought only humans could do much faster than we are prepared for.
Which takes me somewhere even more uncomfortable than an argument with my father over a football match. Are we forced to enter a new stage of our evolution alongside these digital minds, and what exactly is our place in it?
That is the product and business and life question I cannot leave alone: which part of what I do creates value because I do it, and which part was just execution that happened to require a person until now? We need to evolve as a species — not biologically, but in the way we organize what we do and decide what to do, educate, take responsibility, how we live in relationships and what we value in one another.
And that takes me right back to the teenager who thought he knew what he would become. After twenty years of building a career, I am less certain what my profession will be in three years than I was then. But apparently now I am also a “pseudo-philosopher”, so there is a new item to add to my list.
P.S. While I was writing this, the debate around AI consciousness somehow made its way from AI labs and pure philosophy into religion. Pope has made a comment arguing there is an “ontological difference” between human art and machine-generated output and that algorithms lack the “spark of humanity.”
Religious scholars are now being pulled into the discussion, too, by Anthropic and we’re already seeing questions like: could a digital mind have a soul?
Personally, I think we may be getting a few steps ahead of ourselves and I hope it doesn’t get too crazy too early:) But the fact that these questions are being seriously discussed at all is quite fascinating.