Question
How does generated content improve with use?
Work
Product design, review analytics and feedback workflows for content generation.
When content is produced at scale, a small recurring problem becomes a lot of repeated work. Someone corrects a fact, adjusts the tone or rewrites a message for its audience. The next draft arrives with the same problem.
Reviewers develop a detailed understanding of what good work requires. The design challenge is making that understanding available to the system producing the work.
I’m extending Spark Studio to explore how edits, review decisions and their reasons can improve the next round of generated content.
Find the patterns in review
I added analytics to show which emails are revised, which parts change most and where work goes through repeated review.
The team can look across message types, audiences and properties to investigate recurring problems. A concentration of revisions gives them a place to look more closely.
These views describe review activity. Establishing whether the content is accurate still requires examining the work itself.
Turn repeated corrections into findings
I designed the app to suggest findings using fixed counting rules. It looks for repeated reasons for requesting changes, repeated changes to the same part of an email and repeated comment types. AI does not decide whether these patterns exist or which evidence supports them.
A review pattern must involve at least three separate emails and two different people within 90 days. Each suggestion applies to a specific resort, audience or message type. Personal preference reasons, revoked decisions and archived emails are excluded, and a person’s repeated actions on the same email are not counted twice.
The wording follows a template that states what was recorded: which change occurred, across how many emails and involving how many people. Every suggestion links to its supporting evidence.
This gives reviewers something specific to investigate. Repeated subject-line changes may point to weak instructions, a changing brief or something else. The counts establish the pattern; examining the evidence helps explain it.
Decide what the system should carry forward
A person accepts or rejects each suggested finding. Only accepted findings are saved in Sanity for the writing agent. Suggestions that fall below the threshold before review are withdrawn.
That creates a deliberate connection between review activity and future generation. The writing agent can draw on observations the team has examined, with their scope and evidence attached.
Accepting an observation is separate from approving a new writing rule. The next design question is how each finding should influence the work. Some may justify correcting a source or clarifying an instruction. Others may call for further investigation.
This exploration focuses on refining the context and instructions around the model. The resulting changes still need to be tested against new drafts.
Test whether the work improves
The learning loop needs a way to assess its own changes. The next phase is to compare drafts produced from representative briefs using the previous and revised guidance.
Those comparisons should examine factual correctness, audience fit and brand expression, alongside the review required. Fewer edits alone would not establish that the content is better.
The ambition is for the team’s expertise to accumulate. As the system becomes better informed, people should have more room to develop the communication and fewer recurring problems to repair.