leetcode vs AI assisted coding

Background

Erdos Research is primarily focused on reskilling non developers to AI assisted software development.

In this objective, systems like leetcode act as a limitation because they need to be adapted to an AI assisted coding world.

LeetCode measures how well you write code; AI has shifted the real question to how well you decide what code should exist at all.

When code is primarily AI-generated, systems like LeetCode, HackerRank, Codeforces, and similar platforms begin to show structural limitations. These platforms were designed for a human-coding era, not an AI-co-creation era.

These systems operate under the belief that good engineers are fast problem solvers who write correct code under time pressure. With AI assisted coding, this changes because AI writes syntactically correct code rapidly and consequently, humans increasingly act as specifiers, reviewers, and system designers.

In a world where AI assisted coding plays a significant part, we need to test more than the manual implementation of the code. We need to test for judgement, constraints, tradeoffs, and intent

By implication, that means we also need to test for:

• Problem framing

• Choosing the right abstraction

• Rejecting incorrect AI outputs

• Knowing when not to code

The Specification Layer is the new bottleneck

AI assisted code shifts the level of abstraction. In AI-assisted development, the hardest part is no longer the algorithm—it’s the spec. As we are all painfully aware, AI does not eliminate bugs—it changes their nature by introducing new modes of failure such as:

• Subtle logical errors

• Hallucinated APIs

• Silent assumption violations

• Correct code for the wrong problem

• Security considerations are missing in code.

Consequently, we need to test:

• Reviewing AI output critically

• Stress-testing assumptions

• Designing adversarial test

AI Engineering also needs to test aspects that do not appear in current testing such as Data pipelines, APIs, Distributed systems , Observability Cost, latency, reliability

Code Is no longer the scarce resource

Historically, code was expensive - thinking about code was not measured. Now, with AI assistance, code is no longer the scarce resource - so (human) thinking and judgement becomes the scarce resource - but that’s still not (yet) measured.

Another important characteristic thats not measured is human-AI collaboration.

In practice, engineers already now:

• Co-create with AI

• Iterate via prompt refinement

• Maintain mental models across AI revisions

This needs some hard to articulate skills:

• Prompting as specification

• Steering AI without overfitting

• Detecting when AI confidence exceeds correctness

Finally, they currently optimise for individual performance - not team collaboration - which becomes an important metric.

Implications

Systems like leetcode enforce the wrong hiring signal - especially if we acknowledge that AI writes most of the code.

LeetCode selects for

• algorithm recall

• short-term memory

• speed under pressure

But industry now needs a different type of person

• systems thinkers

• product-aware engineers

• AI supervisors

• forward-deployed engineers

• AI product managers

This creates a false negative problem because great real-world engineers and problem solvers fail interviews,

Finally, a personal perspective - AI is evolving asymmetrically (not everyone will agree I know) - but evaluation assumes symmetric collaboration.

Thus, current systems test for a world which is rapidly evolving because they are testing for the wrong bottlenecks

Ultimately, we need to assess new primitives

• Spec-driven tasks

• AI-review challenges

• System design with AI in the loop

• Evaluation of tradeoffs, not outputs

• Ability to say “this solution is wrong—even if it runs”

Erdos Research is primarily focused on reskilling non developers to AI assisted software development. Please contact us at Erdos Research if you want to work with us