🤖 LLMs for Serious Software Engineering

LLMs are moving targets

Rankings change quickly because models, context windows, coding agents, IDE integrations, pricing and usage policies evolve constantly. A leaderboard is useful, but it is not enough for production decisions.

The ranking changes with the workflow

• A chat model may be excellent for architecture but weak at applying diffs.
• An IDE agent may be excellent for implementation but weak for deep design.
• A repository agent may be excellent at refactoring but too expensive for routine tasks.
• A fast model may be perfect for drafts but unsafe as final reviewer.

The strongest professional setup

For huge projects, I would not choose only one model. I would combine GPT-5.5 for architecture and reasoning with Claude Opus for repository-level review and sustained code work. Then I would use Cursor or another IDE agent to apply and inspect changes.

This pairing works because models have different instincts. One can propose the plan. The other can challenge it. One can write the patch. The other can review the diff.

Two-model review loop:

1. Ask Model A for architecture and patch plan.
2. Ask Model B to criticize the plan.
3. Apply only the corrected plan.
4. Run tests.
5. Ask Model B to review the diff.
6. Ask Model A to write release notes and rollback notes.
7. Human approves final merge.

Small projects lie

Small projects make models look better than they really are. Huge projects reveal whether a model can reason across distance, history and constraints.

Scale dimensions

• Number of files and modules.
• Number of languages and runtimes.
• Database coupling and migration complexity.
• Test suite size and execution time.
• Deployment and rollback constraints.
• Legacy code and undocumented behavior.

Dangerous AI behavior

• Invents files, APIs or configuration keys.
• Changes unrelated code to make a patch look complete.
• Deletes safeguards because they complicate the solution.
• Ignores database migration risks.
• Does not explain what must be tested.
• Confuses a quick fix with a durable design.
• Claims certainty without evidence.

Good signs

• States assumptions explicitly.
• Searches relevant files.
• Produces complete function-level patches when requested.
• Preserves naming and project style.
• Separates diagnosis, patch and validation.
• Proposes tests and rollback steps.
• Warns when a change can affect production data.

Good prompt structure:

Context:
- Project purpose
- Current files
- Current failure
- Constraints

Goal:
- Desired behavior
- Non-goals

Patch rules:
- Do not refactor unrelated code
- Provide complete function definitions
- Keep code English-only
- Add validation steps

Output:
1. Diagnosis
2. Patch plan
3. File-by-file changes
4. Complete functions
5. Tests
6. Rollback notes

Bad:
Fix this code.

Better:
This Django management command is slow in --migrate-status mode.
Analyze the uploaded service and command file.
Find the hot path.
Propose one optimization patch that keeps behavior stable.
Return complete function definitions only for modified functions.
Add a short test checklist.

Where GPT can still fail

• It may over-architect if the request is broad.
• It may propose a clean design too large for the current patch.
• It may need strong constraints to avoid multi-step confusion.
• It can miss repository conventions if files are incomplete.
• It can produce plausible code if tests are unavailable.

Guardrails

• Provide exact files and current error output.
• Ask for minimal patches.
• Ask for complete functions, not vague insertions.
• Run tests after every patch.
• Use Claude or another model for review.
• Keep a project memory or architecture document.

Recommended Claude Code task:

Goal:
Optimize --migrate-status execution time.

Scope:
Only migration_status_service.py and migrate_safe_v2.py.

Rules:
Do not change public CLI behavior.
Do not add database indexes.
Do not rename existing models.
Preserve output format.
Run existing tests.
Report changed functions and risk.

Deliverable:
Small diff + test result + explanation.

Review workflow:

1. Generate patch with GPT or human.
2. Give diff to Claude.
3. Ask for failure modes.
4. Ask for missing tests.
5. Ask for simpler alternative.
6. Apply only verified improvements.
7. Run tests again.
8. Record final decision.

Risks

• Agentic sessions can become expensive on very long contexts.
• Large diffs can be hard to review if the scope is too broad.
• It may still make assumptions about runtime behavior.
• It needs explicit limits around database, security and production operations.

Guardrails

• Use small bounded tasks.
• Require a diff summary.
• Require tests and failure notes.
• Ask it to preserve public APIs.
• Use separate review sessions for important patches.
• Do not let it rewrite architecture casually.

IDE-agent risks

• Easy to accept a diff too quickly.
• Agent may change neighboring files unnecessarily.
• Large diffs can hide subtle regressions.
• Generated UI can look good while data flow is wrong.
• Fast iteration can bypass architecture thinking.

Rules

• Keep one goal per edit session.
• Reject broad diffs unless explicitly intended.
• Run type checks, tests and lint.
• Ask for explanation of each changed file.
• Review database and security changes manually.
• Commit small batches.

Gemini workflow:

1. Use it for fast exploration.
2. Use it for UI and multimodal analysis.
3. Use it for Google-specific development.
4. Ask GPT or Claude to review risky architecture.
5. Run tests and validate output.
6. Keep critical DB/security changes gated.

Where to be cautious

• Schema migrations.
• Security-sensitive logic.
• Long-running backend refactors.
• Complex legacy code with hidden dependencies.
• Tasks requiring exact project memory.

C++ and systems work

C++ punishes hallucination. The model must reason about ownership, lifetimes, memory, build systems, ABI compatibility, compiler errors, undefined behavior and performance.

Recommended order

1. GPT-5.5: deep reasoning, invariants, debugging complex failures.
2. Claude Opus: careful review and broad codebase understanding.
3. Cursor: practical editing and compilation loop.
4. Specialized/local models: useful for search, boilerplate and comments.

Minimum validation:

- build
- unit tests
- sanitizer run
- benchmark if performance-sensitive
- code review for lifetime and thread safety

Java and enterprise systems

Java projects are often large, layered and convention-heavy. Success depends on respecting architecture: domain, services, repositories, transactions, security, tests and deployment patterns.

Recommended order

1. Claude Opus: strong on large repository navigation and consistency.
2. GPT-5.5: strong on architecture, debugging and explanation.
3. Copilot Enterprise: useful for organization-wide adoption.
4. Cursor: useful if the team accepts an AI-first IDE workflow.

Java checklist

• Transaction boundaries.
• Spring configuration.
• Dependency injection style.
• DTO/entity separation.
• Security annotations.
• Backward compatibility.
• Integration tests.

JavaScript and TypeScript

Frontend work benefits enormously from IDE agents because the loop is visual, iterative and multi-file. The main risk is producing attractive UI with incorrect data flow.

Recommended order

1. Cursor with Claude/GPT/Composer: best practical frontend cockpit.
2. GPT-5.5: design systems, architecture and debugging.
3. Claude Opus: refactor and review.
4. Gemini: fast UI/multimodal prototyping.
5. Copilot: inline speed.

Frontend validation:

1. TypeScript compile
2. Lint
3. Component state flow
4. Error states
5. Loading states
6. Accessibility
7. Responsive behavior
8. API contract
9. Security and escaping
10. Visual regression if available

Company-specific evals are the gold standard

The best benchmark is a curated set of tasks from your own history: old bugs, failed migrations, refactors, performance regressions and support incidents.

Internal eval set

• 10 real bugs.
• 5 multi-file refactors.
• 5 failed migrations.
• 5 performance issues.
• 5 security-sensitive changes.
• 5 documentation/runbook tasks.

For each AI output, score:

Correctness:       0-5
Minimality:        0-5
Test quality:      0-5
Risk awareness:    0-5
Maintainability:   0-5
Review effort:     0-5

Reject if:
- unsafe database change
- invented API
- missing rollback for risky change
- breaks public behavior
- cannot be tested

Workflow script:

Architect prompt:
Design the minimal safe patch.

Reviewer prompt:
Find what can break.

Implementer prompt:
Apply only the approved patch.

Tester prompt:
Explain the failing test output.

Release prompt:
Write validation and rollback notes.

Useful prompts:

Architecture review:
Review this design for hidden coupling, operational risk,
rollback difficulty and test coverage.

Patch critic:
Find the most likely regressions in this diff.
Be severe and specific.

Migration safety:
Explain the DDL impact, lock risk, data-loss risk,
rollback path and validation plan.

Test design:
Design tests that would fail if this patch is wrong.

Release summary:
Write a concise production release note with validation,
monitoring and rollback steps.

Prompt anti-patterns:

Avoid:

- Fix everything.
- Improve this code.
- Make it better.
- Refactor the module.
- Optimize performance.

Prefer:

- Fix this specific failure.
- Preserve this public behavior.
- Modify only these files.
- Return complete functions only.
- Explain validation steps.