The short version
I run product at a telehealth company. It is a regulated health product, the numbers get questioned hard, and a wrong claim in a founder update is expensive. For the last five months I have worked with Claude connected directly to the systems the work lives in: analytics, the ticket tracker, the code repo, Slack, email, the calendar and our design files.
To write this, I had the AI go back through its own record: the full list of sessions, the roughly 3,000 prompts I typed, the scheduled routines and their run history, and the connector setup. Everything below comes from that record, including the parts that broke. The split surprised me. Most of my AI sessions are not chats. They are routines that run on a schedule, check the systems, and stay quiet unless something needs me. The few that speak for me hand me drafts.
The stack
The AI is only as useful as what it can reach. Most of these are connectors (MCP servers) attached to Claude. Where a tool has no connector, the AI uses its API or command line.
| Tool | What it does for me |
|---|---|
| Mixpanel | My most-used connector by far, with several thousand queries. Funnels, experiment readouts, dashboards, property audits, anomaly checks and tracking loss. |
| Marketing data warehouse | Read-only, and built outside my team. Joins the product funnel to ad spend and attribution. A standing rule says data questions get checked here and in Mixpanel. |
| Slack | Searching decision history, reading release channels, and writing drafts for me to send. Custom bots handle design sign-off, ticket lookup and feature-request intake. |
| ClickUp | No managed connector, so the AI uses the API. Tickets, sprint health, prototype-to-backlog, a front-door intake flow and a space reorganization. |
| GitHub | Command line, read-only. What actually shipped, and pull request diffs read against the ticket's scope. |
| Gmail and Calendar | AI meeting notes in my inbox are the source of truth for decisions. The calendar shows who was in the room before an action is routed. |
| Google Drive | PRDs, SOPs, cost sheets and template decks. |
| Figma and FigJam | Prototype references, roadmap Gantt charts, event diagrams and kickoff briefs. |
| Playwright | The QA-repo browser tests behind the tracking checks. |
| Built-in browser | QA of per-pull-request preview environments, and setup steps that only exist in a web UI. |
| Claude Docs and Artifacts | Decision memos for the founders, compliance registers and shareable team pages. |
| Session analytics and Vercel | A 72-hour UX pulse (rage and dead clicks) inside the weekly update, and password-protected prototypes for demos. |
| Scheduled tasks | The routines. Most run locally in the desktop app; the morning brief runs in the cloud. |
The weekly rhythm
My pattern is to do a job once, live, with the AI, and only then schedule it. The weekly product snapshot took about ten prompts to get right, and ended with me typing "love the tier system… let's schedule this weekly".
| Routine | When | What it produces |
|---|---|---|
| Morning brief | Weekdays 7:00 | Meant to be one page: decisions needed, what is blocked on me, meeting prep. In practice it stalls on a permission prompt (see What broke) |
| Dev review | Daily 7:00 | A review comment per ticket, checking the pull request against acceptance criteria and analytics events. Stale since mid-September (see What broke) |
| Anomaly monitor | Daily 8:00 | Eight core metrics and two tracking-loss ratios vs the same weekday last week; a push only if something is off |
| Tracking parity check | Daily 9:13 | Tracking tests against production and QA, posted to a tracking channel |
| Release parity check | Every 2 hours, 6:15 to 18:15 | The same test against production, only when a new release appears |
| Experiment monitor | Mon and Thu | One verdict per experiment; a push only when a decision is needed |
| Paid-channel funnel | Mondays | Funnel by acquisition channel, as a Slack draft for marketing |
| Product snapshot | Fridays | A three-tier weekly update, as a Slack draft |
The weekly product snapshot
This replaced my status update, which meant reconciling six or more systems by hand. It reads closed tickets, the week's calendar, the week's AI meeting notes (1:1s excluded), merged pull requests and deploys, two Slack channels for blockers, the Mixpanel onboarding funnel and the last 72 hours of session analytics. Then it writes three tiers:
- A public draft. 20 to 28 lines, story first, no ticket IDs.
- An appendix. Decisions this week, actions by owner, and a meeting index. Actions owned by "the group" go last, flagged as a risk.
- Private PM notes. The sharp diagnostics, split into per-owner briefs. Checking that routing against calendar invites caught four misrouted actions in one week.
Claims from auto-generated notes are cross-checked against tickets and GitHub. One week the private notes caught a release note that the data did not back up, and an A/B split running well off its 50/50 target. I edited the draft and sent it myself.
The anomaly monitor
Its rules, summarized from the spec:
{
"source": "server-side events (browser events only for the loss ratios)",
"compare": "yesterday vs the same weekday last week",
"metrics": "10: eight core metrics from leads to cancellations, plus two tracking-loss ratios",
"flag_when": {
"any_core_metric_moves": "more than 35% either way",
"failures_or_cancels_rise": "more than 50%",
"tracking_loss_exceeds": "15%",
"core_event_is_zero": "urgent outage, listed first"
},
"otherwise": "one line: All normal for <date>",
"if_connector_fails": "report it and push; never fail silently"
}
When it fires, it does the first pass of triage. One morning it flagged a sharp move across funnel and payment metrics. Unprompted, it pulled a 14-day trend, found the move had started days earlier, and confirmed browser and server events agreed: a real acquisition change, not a tracking fault. The catch: that push never reached me, because the terminal was active at the time. More on that under What broke.
Tracking parity checks
These exist because a rebuilt page once shipped with part of its tracking missing, and it took too long to notice. Data you never collected cannot be backfilled.
The AI helped write the blameless post-mortem, then a browser test (Playwright) that confirms our tracking setup survives every release. It runs daily against production and QA, and again within about two hours of a daytime production release, instead of waiting up to 19 hours for the next daily run. It tags people only when production fails, so the team does not learn to ignore the channel. Every recorded run since has been clean. The background on why browser events go missing is in building click-funnel events you can trust.
The experiment monitor
Twice a week, each running experiment gets one verdict: keep running, ready to call, guardrail regression, health check failing, stalled, or inconclusive. It never concludes or edits an experiment, and ignores variants under 30 exposures. Its most useful flag was a false alarm: a payment guardrail down 48%. Following up in the same session, the AI found the metric counted payment events, not people, so it mostly picked up recurring payments. Per person the variant was flat or ahead. The call was to keep the test running and fix the metric definition first.
Dated decision checkpoints
This is the pattern I would steal first. At launch, I write the verdict criteria into a one-time scheduled task, so the data is judged against rules set before anyone sees it:
{
"question": "Did the portal relaunch cause the rebill gain?",
"runs_on": "week 7 after launch",
"baseline": "frozen pre-launch range",
"checks": 8,
"example_rule": "non-rebill below the bottom of the frozen baseline range supports a real gain; at or above its midpoint erases it",
"output": "verdict, every check in a table, next steps"
}
A portal relaunch got three: week one, week seven and week ten. The week-seven verdict raised confidence that the relaunch caused the gain from about 50 to 60% to about 65 to 70%. The checkpoints also surfaced things nobody asked about, including an identity regression where, after launch, most dashboard views had lost their user ID. Ten weeks in, none of the three fixes it flagged had shipped and the identity gap had grown further. A follow-up read it proposed was never scheduled.
Working sessions
The rest is live work. I keep long-running chats by topic (repo questions, Mixpanel, weekly updates, the backlog, launches), each with roughly 190 to 270 prompts, so context builds up instead of restarting.
| Kind of work | Share |
|---|---|
| Product analytics | ~24% |
| Prototyping and design review | ~17% |
| Stakeholder writing | ~13% |
| Codebase and release questions | ~12% |
| Backlog and operating system | ~11% |
| QA and tracking integrity | ~7% |
| Strategy documents | ~5% |
Analytics I can defend
- Removing an option without a re-test. I asked whether an option that let users defer a choice was hurting conversion: "Can you be super certain about those 2 gaps? I want to communicate this to founding team." About 1 in 6 users chose it, and their checkouts turned into payments at a small fraction of everyone else's rate. The verification pass caught one of the AI's own two claims as wrong before it reached the founders. Because a re-test would take about three months, we removed it outright.
- Reframing a headline number. The team doubted a reported 30% drop in cancellations. The drop was real, but the rebill rate had barely moved: a small fraction of what a genuine one-third cut in cancellations would produce. Cancels had moved from self-serve to support, and a marketing change landed the same day. It was mostly a channel shift.
- Sizing cannibalization. Before answering how much a newly launched product was eating into an existing one, the AI flagged a trap: list properties match on any item in an order, so filters for two products can match the same order. It verified, and I gave the launch update "with higher confidence".
I also put server and browser events side by side on the same Mixpanel board on purpose (never in the same funnel), "because we get ad-blocked on client side". The gap became the tracking-loss metric the anomaly monitor watches.
The codebase as the source of truth
One chat, about 200 prompts deep, exists only to ask the code and release history how the product behaves. Doctors, patient services and marketing send me questions, such as whether cancelling one plan cancels the others or which states have different intake requirements, and I forward them word for word. Release questions go the same way. As I put it in one prompt about tracking for a new product: "never assume when it comes to onboarding".
Prototypes, writing and the backlog
- Prototypes to tickets. An onboarding prototype kept the "same event steps, same questions" as production with better UX. An end-to-end prototype became a PRD, a FigJam brief and a 24-ticket backlog with annotated screenshots. A portal redesign audit across 8 test accounts produced 41 prioritized tickets.
- Decision memos. For a multi-product launch, a competitor scan led to a decision memo for the founders. Checking it against the launch ticket, the AI found and fixed four errors in its own work.
- Compliance sweeps. Every live onboarding version checked against five copy rules, with where each issue lives, plus an imagery audit and a language search across the whole site. Customer-facing projects get a compliance step at intake.
- Writing for the reader. I paste a colleague's question as asked, get the answer, then ask for a short reply. I name the reader (marketing "only cares from starting at account created"), and I type "make it way way shorter" a lot.
- Backlog forensics. For one billing feature, the AI rebuilt a 38-entry decision log from Slack, email, Drive, the tracker and the calendar. More than a third of the ticket's life was spent in design with nothing blocking, and the open question took two minutes to answer once asked. After launch, short-window payment recovery rose by close to a fifth.
How I brief it
- Context first. Goal and audience, a linked skill file, the data source and where to focus, known facts (for example, the top cancel reason from exit surveys), and a demand that every UI change be justified by the data.
- Frameworks as skills. A PM skills library for user stories with Gherkin criteria, plus CRO and UI/UX skills.
- Corrections become memory. Query both analytics sources, the ticket convention, the shape of the private notes.
- Make it review itself. Before anything important goes out, I have it check its own work from several angles, sometimes with several agents working in parallel. A multi-angle review of one email set made 91 edits. The same habit caught the four memo errors and the wrong founder claim above.
- Ask for an opinion. "Is v6 a winner?" It called it "winning, not yet won" and held off until repeat-purchase data showed buyer quality. That caution paid off: a closer read later showed purchases had not moved. The full story is in the onboarding case study.
Guardrails
Approval pop-ups are not where the safety lives. The guardrails that matter are written into prompts, routine specs, tools and repo instructions. Unattended routines run on a fixed allowlist: read tools, Slack drafts and one alert channel.
- Fresh data or nothing. After three wrong claims in one session (a spec naming the wrong event, a metric lifted from a meeting note, a bad baseline), my rule became: "DO NOT provide me anything you don't have backed up with newly refreshed data, don't trust what's said in chats." Every number comes from a query run that session, with its comparison window stated. Meeting notes count for decisions, never metrics. It re-checks the release list before saying anything is live, checks that a change in a metric does not start before its supposed cause, and labels anything it could not verify.
- Question every number. "30% compared to what, and what's the time period?"
- Drafts, not sends. Anything in my voice is a Slack draft I read and send. Only machine status and deduplicated review comments post directly.
- Dry run by default. Tracker tools write only with an explicit flag, and a reorganization ran "with gates of my approval" and rollback snapshots.
- Read-only on business systems. Routines never conclude experiments or change ticket status, and migrations are printed for a human to run.
- No patient lists. The AI never compiles or exports lists of patients. Outreach goes through a cohort synced to the CRM.
- Analytics hygiene. Never mix browser (Title Case) and server (camelCase) events in one funnel. List properties use "contains". Exclude test traffic.
- Percentages and plain language. No absolutes in anything that leaves the building. No emoji or em dashes, and review comments are short, specific and in plain language.
What broke
- Routines rot quietly. For weeks, from at least mid-September, the daily dev review reported "0 tickets" while its config pointed at a closed sprint and a status that no longer exists, and it had lost repo access. It only came to light in mid-October, when one run was asked to work out why. A clean report looked exactly like a broken one.
- Local routines need the machine awake. I had a ten-day gap with no runs, and checkpoints landed one and seven days late. The Slack bots moved to an always-on host because laptop sleep kept killing them.
- Alerts do not always arrive. "Only push when something is wrong" drops exactly the alert that matters when the push fails. Runs log "Remote Control isn't connected" or "terminal was active", and the push is simply skipped.
- Unattended runs stall on permissions. The cloud morning brief has stalled on nearly every weekday run since it started in October (one of about 20 finished), waiting on an approval prompt for a Mixpanel read tool nobody pre-allowed.
- Findings do not ship themselves. A checkpoint can flag a problem three times in a row. Unless the finding becomes a ticket with an owner, nothing changes.
What it changed
I cannot give you an honest hours-saved number, so I will not invent one. The record shows decisions made on firmer ground, a tracking incident turned into permanent monitoring, and first passes that now run without anyone doing them by hand: the status update, the morning metrics check, experiment hygiene, the channel report and tracking checks after deploys. Some are young: the experiment monitor and channel report have only run a handful of times. It also clarified what should not be mine. A QA coverage audit ended with me concluding that test automation belonged with engineering.
If you are setting this up
- Connect the systems first. Without them you are asking a smart stranger to guess.
- Do the job once live, then schedule it. The first session is the spec.
- Write "why this exists" into every routine. Name the incident behind it.
- Start draft-only. Post directly only for machine status.
- Set decision rules before the data arrives.
- Turn every finding into a ticket with an owner. A flag nobody owns is just a log line.
- Distrust a quiet routine, and pre-allow what unattended runs need.
- Fix your tracking first. AI on broken events gives wrong answers faster.
The takeaway
The useful version of AI in product management is not a smarter chat window. It is small, well-specified routines connected to the real systems, draft-only by default, plus the habit of making the AI prove its numbers before you repeat them.
The routines do the gathering. I still make the calls, and anything in my voice still waits for me to press send.