The Tool Couldn't Tell Me If It Was Working

A system that can't answer "is this working?" isn't finished. It's just running.

In June I built ai-stock-watch, a scheduled Lambda that scores a watchlist of stocks from price history and emails me a ranked report each evening. EventBridge triggers it, SES sends the email, and S3 keeps an archive. I did the platform work properly: a pure scoring core, secrets read from SSM at runtime, a Docker-free build, and CI.

Every evening the email arrived and looked reasonable. Three months in, I asked whether it was actually useful, and nothing I had shipped could tell me. Answering that question turned up five failures. None of them has anything to do with markets. You can find every one of them in production systems.

It isn't financial advice, and as you'll see, the tool's own numbers argue against treating it as any.

Notice where the markers sit: at the boundaries between parts, not inside the scoring code.

1. The deploy asked for one thing, and the package did another

The archive was the obvious place to start, so I opened the bucket. It was empty.

The stack created the bucket, passed its name to the function, and granted write access. But the code checked report.save_to_s3 before writing, and that value only existed in the packaged config.example.yaml, where it was false. No environment variable could override it. The email went out every night and left nothing behind.

That was the third time this repo had the same bug. EMAIL_ENABLED had it on deploy day, and the v2 feature flags had it later that month. Each time, the code was correct, the infrastructure was correct, and the tests passed. The bug lived in the gap between the layers, where unit tests never look.

Lesson: test the effective config after deploy, not the config you think you deployed. A one-line smoke check (is archiving on? are the flags I set actually set?) would have caught all three.

2. The retention rule would have deleted the record

Fixing the override was mechanical. Two more problems sat behind it.

First, the archive held rendered HTML emails, which you can't analyze later. Each run now also writes snapshots/YYYY-MM-DD.json: every ticker's rank, score, and inputs, plus the config that produced them. A score means nothing a year from now unless you know which weights produced it.

Second, the bucket had a 90-day expiry rule. As soon as the archive started working, it would have begun deleting the record it existed to keep. I scoped the rule to the rendered reports/ prefix and let snapshots accumulate.

Lesson: archive data, not presentation, and check that retention policy matches what the data is for.

3. The job that fails by not running

I added two CloudWatch alarms: one for a run that errored, and one for a run that never happened. The second is the one that matters. A failing job is loud. A job that quietly stops running looks exactly like a quiet day.

While I was in the deploy config, I found that sam deploy --guided had written samconfig.toml with stack_name = "sam-app" and an empty recipient. A plain sam deploy would have created a second stack that emailed nobody, and nothing would have flagged it.

Lesson: alert on absence. "It failed" and "it didn't run" are different failures, and only one of them gets noticed without help.

4. Every number needs to say where it came from

An archive that starts today needs a year to say anything. But the score is a pure function of its trailing price window, so I could recompute any past day exactly. A backfill rebuilt about 1,200 sessions back to 2022 in about 25 seconds.

That's also where a project like this starts lying to itself. A reconstructed row says what today's rules would have done, applied to a watchlist I picked knowing how those names turned out. So every snapshot is stamped source: "backfill" or source: "live", and the two are never pooled. Inputs that depend on what was knowable on a given day, like headlines and earnings calendars, can't be reconstructed at all. The backfill refuses to run while any of them is enabled.

The reconstruction also guards against a calendar bug from an earlier backtest. Building sessions from the union of every series' dates included market holidays when futures traded but stocks didn't, and four of those holidays produced phantom sell-offs. Sessions now come from a single equity calendar, and a test covers it.

Lesson: provenance is a field, not a memory. Backfilled and live data look identical in a JSON file.

5. Measure against the baseline that can embarrass you

With history in place, I could finally measure the output. The first backtest of the original rules was unflattering: a 35% hit rate, a median trade of −3.0%, and about 5,500% annualized turnover. Switching from fixed thresholds to ranking and smoothing the score over 40 days fixed the worst of the churn.

The bigger question was which baseline to compare against. Beating SPY feels like success, but my watchlist was full of names I chose already knowing how they had done. The real test is whether the scoring beats holding the whole watchlist and doing nothing. Against that baseline, the information ratio was about 0.03. The list was doing the work, not the logic. I wrote that into the commit message so next year's version of me can't forget it.

The same trap shows up everywhere. A deploy-risk model that beats random might not beat "flag every Friday deploy." The baseline that tells you something is the dumbest thing you could have done instead with the same inputs.

Then the evaluation became shipped code instead of a throwaway script. src/evaluate.py reports rank correlation against forward returns, returns by tier, and a portfolio simulation against both baselines. I used it right away. Momentum carried almost no signal (standalone rank IC +0.019), so I dropped it, and that raised the 21-session IC from +0.091 to +0.103. A cliff in the "broken setup" rule had scored PLTR a flat 0.0 on June 26, after which it rose 62%, so I turned the cliff into a slope. Together the changes cut turnover from about 970% to 735%. I treated the headline return change as noise, and the harness let me make that call from data.

Close the loop

Then those tuned weights sat undeployed for three weeks, until they went out with an unrelated change. The live record kept accumulating under the old rules the whole time. A measurement harness is only half a loop. Shipping what it tells you is the other half.

What I'd do differently is simple: ask "how would I know this is wrong?" on day one, not month four. I built idempotent infrastructure and safe secret handling before I built any way to know whether the output was worth reading. Platform engineers are especially prone to that order, because the infrastructure is the part we enjoy.

The snapshots are now piling up as real live data. In a year I'll have evidence the tool didn't help write. If the information ratio is still about 0.03, the right move is to hold the list and keep the email as a reading aid. Either way, the tool can now tell me that itself.