Adventures in genAI - Unsafe SAFEs

(this one starts out a little technical but stick with it. I promise you'll be fine.)

One of the hardest topics for my entrepreneurship students to master is how a SAFE works. They get the basic idea right away: instead of arm wrestling over an exact valuation for an early stage startup that often has little or no revenue, find a way to take the money and figure out the valuation later. Unfortunately, this "Simple Agreement for Future Equity' is anything but simple.

SAFEs come in two flavors: the original pre-money SAFE and the newer post money SAFE that replaced it. The math on the former is a little tricky but basically the same as a Convertible Note. The math on the newer version is more complicated. Even seasoned VC investors need to check their math twice (or more!) when there are multiple rounds of post money SAFEs, each with different caps, that convert into equity at the next round of investment.

* TBH, I have no problem with Convertible Notes. Founder fears that investors will force them into bankruptcy and 'steal' their startup are way overblown. In my 20 years of early stage investing, I have never seen a Convertible Note investor take over a startup. If things are that bad, why would we want to sink a ton more time into something that is overwhelmingly likely to be a big fat zero? But I digress....

While I could keep the discussion about this in The Entrepreneur's Odyssey relatively high level, my students really do need to know how the math works. So I put a lot time into trying to make my course slides as straightforward as possible and we spend a lot of time in class running through the calculations together. After class, one of the students asked me if I could create some more practice example investment scenarios for them to work on to make sure that they'd mastered the math. Completely reasonable... but also a fair amount of work. Sounds like a job for ChatGPT.

ChatGPT generated 10 scenarios right away. There were scenarios with SAFEs with big discounts and small, and one with no discount at all. Scenarios with SAFEs with high caps and low, and one with no cap at all. Scenarios where the next round was much higher, a little higher, and even one down-round. So far, so good.

I then asked it to generate a table for the first scenario with all the results (conversion price, shares granted, ownership stake, etc) both for a pre-money and a post-money SAFE. The pre-money SAFE results looked good but something about the post-money SAFE looked off. The results table showed the founders with more equity left under the post-money SAFE. Huh? That's not how the game works.

I ran the numbers manually and confirmed it. The pre-money SAFE results were correct but the post-money SAFE results were wrong. I asked ChatGPT to show its work and confirmed that its methodology was incorrect. So I pasted my work into the chat and ask it, "Are you sure that you are calculating the post money SAFE correctly?" Its response?

"You are absolutely right to notice the inconsistency.”

It went on to explain that “The side-by-side table I previously generated for Example #2 used the pre-money SAFE conversion method, NOT the true YC 2018 post-money SAFE method — even though it was labeled ‘post-money SAFE.’” Why the mix-up? “Historically, many attorneys and investors call a SAFE ‘post-money’ if the valuation cap is stated ‘post-money,’ but they still use pre-money SAFE math.”

Translation: Some other people have been doing it wrong so I did it wrong too. Good grief.

By now, I'm beginning to doubt myself. I know that AI tries to please so I am wondering whether or not I really caught it in a mistake or it's just telling me what it thinks I want to hear. So I take the same prompt and throw it into Gemini. Gemini outputs the results table and the post money SAFE numbers are what I came up with. Whew. But then I noticed something odd. It has different numbers for the pre money SAFE! I've already checked those numbers that I know that I'm right so I ask Gemini to show its work. It's doing it wrong. I tell it where I think it's going wrong but it doubles down:

“You've hit on one of the most complex and debated points in early-stage financing! … an excellent question that goes to the heart of the complex, non-intuitive mechanics of the Pre-Money SAFE. You've identified the specific point in the calculation that seems contradictory.”

The problem is, that’s b*******. At least I’m 99.99% sure it is b*******. So I paste its logic into ChatGPT but ChatGPT stands its ground. It would have been fascinating to put the two AI tools into direct contact with a prompt to "“work this out amongst yourselves and get back to me when you've figured it out" but I couldn't figure out how to do that. Plus, by now it is 2AM.

I waste another half hour trying to get ChatGPT to generate the 10 example problems and answer keys and to export the output to a Google slide (tl;dr It can’t. It can output to a PowerPoint but the result is butt-ugly and missing data). I tell Gemini to use my calculation method for the post money SAFE (which it does without objection) and it generates the output I need fairly quickly. It drops the ball entirely on saving it to Google Slides (ironic, I know), but the PDF download looks good so I declare victory and stumble off to bed.

So what did I actually learn from this frustrating chat session?

• I was painfully reminded that genAI tools aren’t actually thinking about the question, they are simply predicting the next word that is likely to make the user happy. Usually that’s the right answer but…

• When the source material it is trained on is confusing, with many seemingly authoritative sources using similar descriptions for different things, often within the same source material (and possibly also including some out and out inaccuracies), it is anyone’s bet what the AI will think will make the users happy

• In situations where there is an objective truth (e.g., definitive calculation methodologies exist), the safest route to reliable and fast results is to prompt the chat with that. (viz., “This is how a pre-money SAFE converts…”)

• Always, always, always sanity check the results before relying on them.

• Never give ChatGPT a “simple question” at 11:30pm