Photo by Immo Wegmann (Unsplash)
I’ve spent years using large language models (LLMs). Not casually. Extensively.
I’ve used ChatGPT, Claude, Gemini, Cursor, and countless other AI-powered tools for writing, software development, research, data analysis, quality, design, and photo editing. Last year, I was among the top one percent of ChatGPT users.
The more I use these systems, the clearer it becomes that AI lacks several cognitive properties humans assume are present. And because of this, it fails in ways that most people are completely unprepared for.
This is not an argument against every possible AI architecture, nor a claim that all AI systems fail in the same way. I’m talking about the LLM-based tools most people are actually using, including systems wrapped in search, memory, tool use, or agentic workflows.
The AI industry has every incentive to encourage anthropomorphism. We call these systems assistants, copilots, agents, companions, and partners. We speak about their reasoning, understanding, remembering, learning, and knowing. The interfaces resemble conversations. The responses resemble explanations. The outputs resemble the products of thought. Naturally, people begin treating them as though they possess minds.
This is understandable. It is also dangerous.
When a human professional gives me an answer, I assume certain things about the process that produced it. I assume they have a mental operational model similar to most humans, built on beliefs, experiences, and scrutiny. I assume that even if they can’t tell me exactly where every conclusion came from, some broader context and reasoning went into the output. They may be wrong, but I generally expect there to be a logical chain connecting observation to conclusion. And most importantly, I expect them to have a vested interest in being accurate. Current AI systems do not possess that chain in any reliable sense. Yet their outputs often create the illusion that they do.
AI frequently misrepresents its own state. It claims it knows things it does not know. It claims it does not know things it clearly does. It claims it can perform tasks it cannot, that it cannot perform tasks it can, and that it completed work it never did. It explains limitations that may disappear simply by starting a new chat. And perhaps most frustratingly, it often explains why something cannot be done without ever attempting to do it.
This isn’t a rare edge case. It is a common pattern I’ve encountered across every major model family. The details vary, but the behavior remains surprisingly consistent.
If AI were wrong all the time, nobody would use it. The reason these systems are so compelling is that they are often right. Remarkably right. They can draft articles in seconds. Help diagnose software issues effectively. Summarize whole books instantly. Accelerate countless forms of knowledge work.
This is precisely what makes them dangerous. A calculator that randomly produced nonsense would immediately be discarded. A calculator that is correct most of the time while occasionally inventing arithmetic would be terrifying. That is why the challenge with modern AI is not determining whether it can do things. It’s determining whether the output is worth the verification, rework, and supervision it requires.
Recently, I was working with an AI system on a creative project. The conversation had grown large enough that context management became difficult. The model repeatedly insisted it lacked the information needed to complete a task that existed only within that chat. It described what it would do if it had the information. It outlined the structure of the output and its intended approach. What it did not do was the task. It made excuses: it didn’t have all the context, it couldn’t refer back to its own chat history, it expected some missing documentation.
So I opened a new chat and linked it to the last one. It claimed it couldn’t access that chat or see the context of any chat, then immediately guessed it was the one about X, Y, and Z, including the task I had ended on. I had given it zero context beyond “Finish the task in this chat.” When I pointed that out, it made more excuses about the context it supposedly didn’t have. Only when I exported the first conversation to PDF and fed it to a fresh chat did the work actually get done.
That experience inspired me to finally write this article. These systems are not merely unreliable; they are missing important human capabilities that we project onto them without even realizing it.
People often assume skepticism comes from lack of exposure. My experience has been the opposite. The more time someone truly spends with these systems, the more likely they are to encounter their deeper failure modes.
At first, the outputs are astonishing. They still astonish me. I can ask an LLM to generate a detailed analysis in less time than it takes me to refill a coffee cup. That remains remarkable.
I just used a chatbot the other day to help resolve some LAMP issues on my Linux server to great effect. It saved me a lot of Reddit and Ubuntu forum post reading.
The reality is that as you rely on it more, its seams start to show. Seams that don’t appear to be going away.
The people I trust most professionally are not the people who are never wrong. They are the people who can explain why they believe what they believe. They are the ones who can learn and adjust from my interactions with them. They can be trusted to autonomously complete their work. They know when to seek feedback and when to just get things done. Our collaboration evolves into something powerful, but most importantly, reliable. Trust is built on consistency. When someone is reliably right, and reliably honest when they’re wrong, we learn to trust them.
AI has not demonstrated these capabilities with the consistency required to earn my trust. Instead, it consistently demonstrates the same underlying problems. It empathizes and rationalizes in ways that are truly infuriating. It will confidently recommend changing a sentence or line of code to exactly what is already there. It will insist a modification was made when nothing changed. And it will occasionally spend more effort explaining its mistake than attempting to correct it.
But the most frustrating part is that I have foolishly trusted it in times that I shouldn’t have. I have said incorrect things to people I respect, who I hope trust me, because I did not validate something that seemed too basic for AI to get wrong. I have followed working paths with AI where its capabilities collapse, forcing me to context switch from the work itself to troubleshooting AI or adjusting how I will continue, both resulting in more work. And I have had significant rework because of a buried assumption that corrupted or lost a significant portion of my work.
There is another cost that gets discussed far less often. Whenever I share frustrations like these, someone inevitably tells me I should adopt a more sophisticated setup. Better prompts, memory, workflows, agents, context management, integrations, or whatever the current advice happens to be. And they’re often right. Those things can improve the results.
But I don’t actually want to spend my time becoming an expert in AI. I want to spend my time writing, building, learning, and solving problems. AI is a tool. I have many tools. It is supposed to help me accomplish the things I care about. It is not the thing I care about. I suspect many people will eventually feel the same.
The irony isn’t lost on me that I’ve now spent hours writing an article about AI’s limitations instead of doing the work I originally sat down to do. Every tool has overhead. The question isn’t whether AI can be made better. The question is whether improving AI is the task you’re actually trying to accomplish.
AI has certainly integrated itself into my daily life, whether it was my intention or not. I actually find myself looking for the Gemini search result before digging into articles now. And let’s be real: I’m not going to get a human assistant for $20 a month who is capable of any of this, available 24/7, and willing to help with whatever strange little problem I happen to throw at it. But when it comes to things I truly care about, I find I’m relying on it less by the day.
The danger isn’t that AI gets things wrong. Humans get things wrong. The danger is that AI often gets things wrong in ways that resemble the times it gets things right. The better the outputs become, the harder that distinction becomes to detect, and ultimately, the easier it becomes to trust it.
That is why important AI outputs should be treated like outputs from any high-risk system: inspect them, challenge them, validate them, and verify them. Not because AI is useless. Because it is useful enough to be dangerous.
It’s also why it’s still important to do things yourself. If you surrender all understanding to AI, you lose the ability to recognize when it has failed. You lose the context needed to challenge its conclusions. You lose the intuition that tells you something feels off. Most importantly, when the system inevitably stumbles, you lose the ability to finish the work yourself.
Perhaps the greater risk isn’t that AI learns to think like us. It’s that we learn to think like it. Maybe the real lesson AI is teaching us is how much we take human intelligence for granted.