I Had AI Audit a Video Generation Platform: A 281-Line Report That Kept My Delivery Consistent

A development log about turning "vague requirements" into "executable criteria" in one pass

This is a real experience from my own product. What changed this round wasn't a new feature, but delivery consistency in video generation tasks: how shots are split, how credits are deducted, how failures are compensated, and what counts as a complete final cut.

The thing itself isn't complicated. The director console splits the user's script into N shots, calls a cloud video API once per shot, and stitches them into a final cut. A 4-shot task deducts credits for 4 shots. The problem lies in failure handling: when 3 of 4 shots succeed and 1 fails, the system "cleverly" stitches the 3 shots into a version of the final cut and delivers it. But the user wanted 4 shots and got an incomplete cut.

Harder than the technical side is the definition: does a missing shot count as complete? How are credits refunded? How should the re-run entry point be designed? More troublesome still, there's no single place in the code that can clearly answer "is this final cut actually complete or not."

Before fixing anything, I made one decision that changed the quality of the entire round of conversation.

1. Don't let the AI "design a solution" — let it "gather evidence"

My instruction to the AI went roughly like this: scan the full chains of task initiation, credit deduction, queuing, cancellation, completeness checks, and settlement, based on the actual logic in the code, not from memory or solution design, and give me a factual audit report.

The phrase that actually did the work was "not from memory or solution design."

The reason is simple: the AI's default mode is to give you a plausible-looking solution. Drawing on "typical architectures" in its training data, it can write something that looks quite convincing. But real bugs don't live in the architecture — they live in implementation details: who writes to memory first and who persists to the database second, which branch fails to recompute state, which retry processes something twice. If you don't force it to read the code, it answers from memory. And on a problem like delivery consistency, "from memory" is dangerous.

It came back with a 281-line report. A few characteristics built my trust: every conclusion came with a code location I could verify line by line; it explicitly noted "what wasn't done"; it dared to say "no infinite loops found"; and it dared to say that something "isn't a bug, but a business-acceptable cost."

My takeaway: having AI do verifiable tasks — reading code, listing facts, providing evidence, flagging uncertainty — produces output a tier above having it do open-ended design. Because verifiable means refutable, and refutability is where quality comes from.

2. The 12 issues it surfaced

The report was tiered by severity. The ones directly related to delivery and entitlements fell mainly into three kinds of inconsistency:

First, concurrent batch creation lacked validation, so in extreme cases task state and credit records could mismatch. Second, certain branches where a switch was off or a rule was disabled would make the refund flow return success early — the caller thought it was handled, but it would never be retried. Third, deduction and refund used rules from different points in time, and the time gap could cause amount mismatches.

There were also: tasks permanently stuck in "compositing" after a compositing failure; appending to an already-cancelled batch, causing tasks to never execute; network requests with no timeout; queuing that waits indefinitely by default; and so on.

I especially want to highlight the form of this report: it wasn't "I think this could be optimized here," but "line 437, when credits are disabled it directly returns true, and the caller marks it as settled based on that, so this refund will never be retried." Conclusions that land at this granularity are conclusions you can start working from directly.

Of course, if these issues aren't addressed, what's affected is users' trust in delivery completeness. So my goal wasn't to wait for feedback, but to proactively tighten every boundary before release.

3. Two sentences set the definition, and it made all the trade-offs for me

After reviewing the fix order, my instruction was plain:

Delay is fine, the core is fairness and rigor... not a single error is allowed.

And the most critical product definition:

1. Refund based on the actual deduction records.
2. When shots are missing, don't produce a final cut — only provide a "N shots missing + re-run" entry point.

Those few dozen characters were equivalent to compiling "product values" into technical criteria. After that, almost none of the branch decisions needed to be asked of me again: refund exactly what was deducted, so the actual deducted amount must be recorded rather than recomputed at refund time; if there are usable clips, don't refund, and only refund fully when not a single shot succeeded; when shots are missing, rather than produce a final cut, only provide the re-run entry point; tasks already submitted to the cloud can't be cancelled, because the external service may have already started processing.

This is the one point I most want you to remember from this whole piece: AI doesn't lack capability, it lacks criteria. If you just say "optimize the failure handling," it'll give you three solutions that are all not wrong, and you'll spend three rounds picking one. If you write clearly what counts as fair, it can land the definition on every branch in one round. "Not a single error is allowed" sounds like emotion, but it's actually the most efficient requirement description — it turns on the default behavior of "must verify branch by branch."

4. A few judgments I most want people to steal

First, systems involving settlement and entitlements must distinguish "failure" from "unknown." The cloud API processes on submission, and there's no query and no cancellation. So after a submission request times out, you don't know whether the other side received it: it might not have, in which case retrying is safe; or it might have, in which case retrying would process it twice. Generic logic lumps timeout into "transient errors" and retries automatically — that's "creating inconsistency in the name of robustness." The right approach is to model it separately as "submission result unknown": no automatic retry, no automatic re-run, just log an alert and leave the entry point to a human.

Second, a constraint must be blocked at every exit, not just the main path. When implementing "no final cut when shots are missing," I required going through every path that can produce a final cut: server-side automatic compositing, server-side takeover of failed compositing, client-side upload, and user manual re-stitching. Blocking only the main path is as good as not blocking at all — the user clicks "re-stitch" and the fragments still come out.

Third, the first step of performance optimization is classification, not caching. One issue was that every progress change wrote the entire batch state to the database. The fix isn't to add a cache, but to classify: progress-type data, like percentage and queue position, gets batched writes, persisting the last one after a few hundred milliseconds; entitlement and terminal-state data, like deductions, refunds, settlement, and success/failure, is always written immediately. Losing undeletable state to save a few writes is the most expensive optimization there is.

5. Leave the order to it, keep the authority to issue commands for myself

Its proposed fix order wasn't by code location, but by "risk ÷ change size." After reading it, I replied with just two words: "Start fixing."

After that, it did one logic change per commit, and 22 of the 32 commits landed directly on the task and entitlement chains. That means if any one change was wrong, I could revert just that one. Rollback granularity is risk-control granularity — that's why I insist on splitting commits. The verification gate was also plain: type checks and lint all pass, write one-off verification scripts for critical logic, and delete them once they pass.

I also specifically followed up on one thing: for these changes, does the desktop client need to change too? It's easy to answer "probably yes" on a hunch, but what I wanted was a point-by-point assertion check. The conclusion was that the client doesn't need to change — the server-side fallback already blocks old clients — and in the end I only added a very small notice on the desktop side.

First prove "no changes needed," then change the one thing that truly should change—far cheaper than "just change everything while you're at it."

6. Three red lines, and one lesson about the boundaries of automation

This round I drew three red lines: the production database is read-only—query real transaction records, but never modify them; commit only, never push—any externally visible action must be initiated by me; building is not releasing—builds are reversible, releases are not, and in between I said "stop, don't ship yet."

There's a subtler one too: desktop notifications appear only when a task has made no progress for over 45 minutes, but they only alert—they never auto-cancel a task. Because canceling is a decision to "abandon a committed task," and that must be made by a human. The system's job is to keep people informed, not to decide for them.

7. Release: don't trust the tool's own "success"

The release script prints "✅ Release process complete" at the end. But I insisted on independent verification—making requests myself from the client's perspective: can the update endpoint return the correct version to old clients, does the downloaded full package's sha256 match the local copy byte for byte, is the public copy consistent with the local version.

During verification I hit a false alarm: the script read the "artifact details" using a guessed field name and printed "0 artifacts," which looked like the package hadn't been uploaded. In reality, that endpoint simply doesn't return artifact details—whether the package is actually there is proven by "the endpoint can provide a download URL" and "the downloaded bytes' sha256 matches exactly."

Lesson: judge by the chain of evidence, not by the dashboard. Especially when using AI—it will give you a beautiful summary table, but what determines success or failure are those few pieces of independently reproducible evidence.

Conclusion: when execution becomes cheap, judgment is what's expensive

This round it read a 2,400-line interface file, mapped the complete call chain, wrote a 281-line report, fixed 12 items ordered by risk, wrote verification scripts, documentation, and user-facing copy, built and signed the desktop client, and performed 4 independent verifications after release. It doesn't get tired, annoyed, or miss things—what used to be the most expensive part of "execution" is now cheap.

And almost everything I did this round was judgment: what counts as fair, what counts as rigorous, what costs are acceptable, when to release, what words shouldn't be shown to users, and the bottom line of "not a single error allowed."

So my conclusion is: don't use AI as a "faster programmer." Use it as a "tireless auditor + extremely obedient executor"—it will amplify your vague intentions into code, fast and accurate, including accurately wrong. What it truly needs as input is for you to first write "what is correct" in a form it can execute.

AI doesn't lack capability; it lacks criteria. The criteria are yours.