Hi Greg, I'm excited to see a question like this come up in the Cafe. Kudos for a well thought out process before you start building. I also think your human gates are in the right places.
Now for my thoughts on your plan:
- Accuracy: The 'incorporated documents you never received' part is the one I would be mindful of. AI will review a subcontract that references Exhibit X and the prime, and fill those gaps with what documents like that usually say, and give you a response that sounds finished and confident. So the way to get around that is to make the absence a required output. Have it list every document the subcontract incorporates by reference in a table, mark each one provided or missing, and say plainly that the review is partial. If it has not seen the prime, have it label the flow-down section explicitly as a guess, so that you know what's factual and what isn't.
- On indemnity, if a provision can be read 2 ways, ask it to give you both readings, and to rate the riskier one. You want it to be reading it against you, not for you.
- Redlines: Your own standard positions are what drive the output quality on the redline. You have to give it those positions in order to anchor to something, so that it does a comparison, which is the thing it is genuinely good at. I would go a step further than your plan and stop it drafting where you haven't taken a position. That means ask it to say the outcome is not on file and leave the clause alone. The negotiating goal is yours to set, so don't allow it to default to anything else. I would also set a rule to have it quote each clause word for word to avoid having it make things up, so the markup tracks the original subcontract wording as it was received.
- Time savings: Honestly, I cannot give you a real number. I would simply recommend that you track two things: hours from subcontract receipt to counsel handoff, and the share of HIGH flags counsel confirms. The confirm rate is the most important one in my view. A bunch of unnecessary flags is just noise and it will probably take some time to fine tune. The way I would anchor it is to run it over 5 or 10 subcontracts your counsel has already redlined, and count how many issues AI catches that the counsel raised. That gives you a miss rate off work you have already done. I would want zero misses on the clauses that can really hurt you, so be prepared to put up with plenty of noise and iterations to get there.
- Sign-off: If a summary can be read on its own, somebody will read it on its own. So the design needs to force the person to read before the AI gives an analysis. And now I'm actually rethinking the design as I'm writing this. The way I would build it is to have the AI produce 2 separate outputs and release them in order. 1) A reading list per individual signer, including every article it examined, by number, without any recommendation. The person signing works through that personalized reading list in the subcontract itself and writes their own answers to a short fixed set of questions. What is the notice period. Is the indemnity mutual. Is there a cap on damages. Whatever your standard positions already cover. That means they have to open the document to answer anything at all. Only then the second part, which is blocked until part 1 is completed. 2) The full analysis with the ratings and the reasoning done by the AI, compared and reconciled against the output of Part 1. Every place the signer's answer and the AI's answer differ gets written down and resolved, and the sign-off goes on that reconciliation. This gives you 3 things: a) The signer cannot skim, because they produce their answers before they see any. b) The analysis still does its job, only second. And c) you get the confirm rate I mentioned above for free, since every disagreement is already recorded. It will cost you more time on the first few subcontracts. I would still do it that way, because what you end up with is a record that the person who signed actually read it.
- There is an added audit advantage of doing it this way. A reconciliation gives you a dated record of what was examined, what your person concluded, and where the AI disagreed. It also shows a person formed the view first and the AI checked it afterwards, which is the order you want on the record if anyone ever asks how a subcontract came to be signed. This is also how you work out how much confidence to put in it. Every reconciliation adds to that picture, so after a handful of subcontracts you know which parts of its review you can lean on, and that evidence is what should decide how much you eventually let it do on its own.
- Tools: The only thing I would insist on for this type of work is that this runs on a company account and not a personal one. On which one (Claude or ChatGPT), both have their merits. Both flagship models now sit at around a million tokens of context, which is far more than a subcontract, its exhibits and a prime contract will ever take up. So capacity is not an issue. I use Claude for this kind of work, and I should say plainly that I also teach on it, so weigh my preference accordingly. What I like about it for contracts is how it holds a long document together and how readily it tells me when it is unsure. OpenAI's flagship is aimed at the same complex reasoning work and plenty of people get good results from it. Anthropic's own page lists long-context handling and honesty among its strengths, and positions Fable 5.1 for demanding reasoning. OpenAI positions Astra for the hardest end-to-end work and complex reasoning. Both vendors claim the same ground, so the honest answer is just pick one, or try both and get your counsel to tell you which responded better.
- There are purpose-built contract review tools out there as well, but I don't have first hand experience using any, so I cannot make a recommendation.
-
Human review design: On your first option, I would not recommend holding sections back from the AI. A partial review is a weaker review honestly. The second option is the one I would build, which is reflected in the design above in the Sign-off bullet. What makes it work for PMs is the personalized reading list. Each person gets the articles that belong to them, not everything. But, everything does get read by someone. The split should be defined by who owns the answer, commercial or operations. Then call out an owner for every section in your own standard positions document. A PM will read something carefully when they're named as the owner of a response. As I said above, this design gives you evidence at the end, so you know whether the contract was read, and by whom.
One thing I would add that is not in your plan yet:
Before any review happens, get it to map the subcontract against your standard positions document, and then stop there. The map should give you a table: each of your positions, the article in the subcontract that corresponds to it, the owner who will review it, and a plain statement of anything it could not place. That last column is the valuable one. A position of yours with no matching article means a protection you think you have is missing from this contract. An article with no matching position means something is in there that your playbook may not have considered. Both of those are good findings to have before any analysis starts. Ask it to stop and wait for a person to confirm the map. Not the findings, just the map. The reason I would build that gate in is that everything downstream inherits it, and catching a mismatched article to a position early will save you hours later. It also settles ownership before the reading lists go out, which is the part that makes the personalized lists work at all. So you end up with a check at each end. A map confirmed before any analysis, and a reconciliation after it. I use this same approach in every workflow I build. Agree the structure before you trust the judgment, then check the judgment against something independent afterwards.
For transparency, I'm an Associate Member. I teach construction finance and office teams to build workflows like this one. Happy to compare checklist structures if that would help, and I would really like to hear about your learnings when you have them.
------------------------------
Adileh Mountain
Founder & Principal Advisor
Strat Scale Advisory
Murrieta CA
(832) 405-0120
------------------------------