Is the SaaSpocalypse Over?

[00:00:00] Speaker A: Well, I think we can say the SaaS apocalypse has passed and we're all still here for now. I guess you sit at an interesting place as you're an AI tool which has a strong vibe, coding, replacement risk, but obviously doing really well. We're customers. It does a better job than I think I could do by myself and I'm reasonably technical. But how, how are you feeling today about like the business and, and the viability, I guess. [00:00:39] Speaker B: Yeah, good questions. I feel good actually. That's the numbers. If you want me to share, I can share a sure slide and that might actually be very visual and cool. Um, so this is like actually going in deep. I shared this on another podcast the other day, just kind of looking at our stats over the last 12 months to kind of see what's been going on, you know, and that really ties into kind of the patterns, how we're seeing AI affecting SaaS. So we're still growing, but it's a lot slower than it used to be. So if you look at like our all time kind of MRR graph and you can see it's pretty steady, somewhat steep, and then over the last 12 months it's somewhat leveled out and I've heard similar stories from a lot of other founders as well. [00:01:37] Speaker A: Our graph looks very similar. Just rewind of years and the flat part is longer. [00:01:42] Speaker B: Okay. [00:01:43] Speaker A: And just for folks who are listening only, like Aaron showed the first graph here was like MRR up 5% but ARPU and LTV up quite a bit. So, you know, not getting more customers, but customers paying more. And so that, that's like I always say, one of the best levers in SaaS is expansion revenue. And so you have that, you have that nailed in somehow. [00:02:08] Speaker B: Yeah, exactly. Yeah. So what we've really done, like, like if I jump ahead and you can see like really acquisition, I mean that's been the biggest challenge for many different reasons, but you can see year over year that's a pretty big drop. And so what we put a whole lot of focus on this year is getting the right customers and better customers, you know, really narrowing down like on our ICP and our targeting and what we spend on marketing and all those kinds of things, landing pages, all that, even restructuring our plans a bit to target like our ideal customer profile. And so what that's led to is good. We have more revenue per customer, you know, and that's up 20% from last year. And lifetime value especially is up a lot higher because with this we've also been able to decrease churn A bit. So that's kind of been our focus and that's kept us afloat. [00:03:07] Speaker A: Yeah. Are you all kind of a high or low churn business? You don't have to give exact numbers but like, yeah, good. [00:03:14] Speaker B: Or I got a slide for that so you can tell me if it's high or low. There's a subscriber turn rate, so we have dropped that 27% this year. [00:03:24] Speaker A: Yep. [00:03:24] Speaker B: So that's good. [00:03:25] Speaker A: So from like 8% to 4 or 5%. [00:03:29] Speaker B: Yeah, yeah, yeah. 8 to 10 to, to that. So we're getting those better fit customers. And probably more importantly is revenue churn. So that's factored in, you know, upgrades and plan changes and pricing changes and all that kind of stuff. So that's down as well a bit. So that's been helpful in offsetting kind of like the decrease on the front end of new people coming in. So we're getting more of the better people, which is helping a lot. And I think the other big thing for us for growth is we did do a price increase or more of a restructuring. But probably our most popular targeted plan, like the one that says recommended on the pricing page, you know, or most popular, that one, we did have a substantial price increase to bring us kind of in line with competitors. And alongside that was when we released a whole lot of new like agentic kind of features and stuff which increased a lot of costs on the back end as well, you know. So there was some cost increase but mainly it's just like trying to get figure out okay, what's the best price range for the average best ICP person, you know. And so that's really helped us as well, like keeping up with growth slightly and not being in the negative. [00:04:48] Speaker A: Yeah, yeah. Do, do you think that like the focus on certain ICPs is, is like a product thing or like a marketing thing because like customer acquisition is, is gone down. So I think you could argue that like, oh, this focus on this particular icp. [00:05:06] Speaker B: Yeah. [00:05:07] Speaker A: Isn't successful if you have less trial, like relatively less trials, but you're getting higher paying customers. So like do you call it a success? [00:05:15] Speaker B: Yeah, I mean it is both, obviously. Like, I mean probably the biggest cheat code is talk to your customers, right? Yeah, just talk to them and figure out what they want. So that may be surveys that you're collecting during signup or during churn cancellation, you know, to figure out what's actually going on their head. Most importantly, just get on calls with them, even if that's not necessarily like scalable to have Like a whole sales process. But the, the insights that it reveals are huge. Like on, on. And then figuring out, like, I just sent. So I read the Hidden Multipliers, new book by Jason Cohen. I know you're a friend of his. And so I just sent, you know, our agents that all. All our data, you know, our whole data lake for the whole product and stripe and everything say, all right, what is our best icp, which in that book they call the Carol, you know, essentially. So it's narrowing down much more than just like, oh, it's women over 35 or something, you know, it's narrowing down to like, what's important to that person, you know, and where the money and budget comes from for that person, etc. You know. [00:06:24] Speaker A: Yeah. [00:06:26] Speaker B: So really narrowing down to that is super helpful. And we do that based not by just. Not just like the money that they bring in because we have, like, probably our biggest customer that pays us, like a big enterprise deal. They use the product in a completely different way than almost everyone else, you know, So I can't just like decide by revenue alone because that's, that's really not. I mean, that's great that we have that revenue from that customer or a handful of customers that use it in a strange way. But that's not our ideal profile, you know. So essentially we just narrowed it down, looking at every single customer who's using it, how they're using it, what they're using it for, what they've trained their AI agents for, on and all that kind of stuff, you know, to kind of get a much better general ICP that's not tied to a specific industry or anything like that, but really tied to what is important to that person. [00:07:24] Speaker A: Yeah, yeah. How many, like, orders of magnitude your hunt? Hundreds or thousands of customers? [00:07:32] Speaker B: Yeah, I think it's somewhere between five hundred and a thousand. Okay. [00:07:39] Speaker A: I just, I think one of the biggest, like, risks for folks early on, and you're not at the early on stage, but like, for folks early on is like, I call it signal density. It's like they don't have enough people saying the same. To draw the conclusions that you have, like, you need a reason. Like, we have, we have a lot of customers, so like three or four thousand customers that all pay us a little bit of money. But it's easier for us to say, to draw those conclusions. We have a lot more volume. You have enough volume to be able to say, like, oh, we have five or five hundred or a thousand customers. There are these bands of customers that are that are really good. But if you have 20 customers, I think the likelihood that all 20 of them or enough of those 20 are similar for you to be able to draw conclusions is like really hard. [00:08:29] Speaker B: Yeah. [00:08:29] Speaker A: So it's kind of a chicken or the egg thing. [00:08:31] Speaker B: No, that's very true. I feel like our first. So we're almost four years old now, so we're one of the first in the space for AI customer support agents, you know, using RAG and technology. And we, we actually released like the same week that chat GPT APIs came out, you know, so we're one of the first there, which is great building brand and SEO and things like that. But it was completely new industry, completely new everything, you know, so we had no idea like what the best use case was, what our ideal customer was, you know, and we're really relying a lot for our initial growth on kind of just like the whole viral like, oh, AI is like amazing. We need to try it, you know. [00:09:13] Speaker A: Yeah. [00:09:13] Speaker B: Every, every company out there figuring out how they can use AI and how are you. [00:09:18] Speaker A: How are you acquiring customers now? [00:09:21] Speaker B: Yeah. So I wish I knew. No, still, I mean, why. [00:09:29] Speaker A: Let me, let me frame that. I talk to a lot of founders either in like my work with Tiny Seed or I do some coaching and like this podcast. Get. Get to chat with a lot of smart folks. Not a lot of people have a great answer for that these days. I'll say they know what marketing they do. They have less of an idea of what works well and even less what they should be doing more of. [00:09:53] Speaker B: Yeah. So I can share kind of what I have too here. This is just something I post on X recently. This is purely going just on what data we have. You know, as you can see, there's like a huge chunk that's direct, which basically just means we have no idea. [00:10:10] Speaker A: Yeah, yeah. [00:10:11] Speaker B: Where they're coming from. But search was pretty big and that has dropped this year, definitely. But not massively so. Organic still is our biggest source. And even like last year, like AI, like referrals and stuff from ChatGPT and perplexity and things like that. Like that definitely we get, when we do like sales calls, like for more kind of like higher tier potential customers, we do a lot of like demo calls and things like that with them. And I always ask them, you know, how they found out about us. So AI search or AEO or whatever you may call it, that's definitely been a pretty decent source. And then of course we have just referrals, you know, word of mouth, what we've really been pushing into this year. A whole lot is events and in person stuff. So gone to a handful of conferences and just really not necessarily like did the whole booth thing, but just hardcore like hit the floor. The whole team talking to every single person out there, you know, like giving them demos just on the fly or whatever. Trying to be pre, pre booked meetings however we could. And we found that that is a huge leverage. I mean it's a lot more work, it's slower. Well, kind of differences. Yeah. So we went to work Camp Europe and US Press conf and we went to one that's basically for agency owners that was actually great. So that's another push that we've been doing this year is really building out like an agency partner program and let them sell our product to small businesses, you know, small medium businesses, their own clients and help them win in the process, you know. [00:12:06] Speaker A: Yeah, that's always the like gold mine. Right. Of every founder's dream is like let me get channel partners in place to sell this stuff for me and I can sell this one agency and then they go sell 10 clients a month on it or something like that. And my experience, that's like a lot easier to say than to do. So if you guys are cracking that, like that's great. That must have been a significant amount of work. [00:12:34] Speaker B: Yeah, I feel like we're just starting to like we have this huge list of leads and people are talking to and then you're trying to get them. Okay. Figure out, identify one client that might be able to use this, you know, to get kind of get that going. So we've definitely had a handful of growth and we haven't seen like an exponential lift from that. Yeah, but that is the idea. I mean you, you do the work once and then they do the work for you. And yeah, I think as we're talking just more general about like AI and SAS as you're talking. I think that's the pattern that I'm seeing and hearing. Especially if you're. If it's B2B, this is like where it ties in is yes, we have like all these new AI tools and technology and probably competing products and things like that and even like people vibe coding stuff. But for the most part like your normal like SMB is not gonna like vibe code their own solution or they wouldn't even, they don't even know how to like identify and purchase software like for a certain use case, you know. And so that's kind of where when I talk to these agencies, that's kind of the same thing that they're telling me. You know, it's like they need someone to look, to go into the business, figure out, okay, what are the workflows they're doing, what are the pain points they're experiencing, where are the inefficiencies, and interpret that into a solution that AI may or may not help with, you know, and that really hasn't changed since it was just SaaS and not AI. You know, it's not really any different. It's maybe even more complex because AI feels even more like, overwhelming to people because of the speed that it's moving and all that kind of thing. [00:14:15] Speaker A: Yeah, yeah, yeah, yeah. I mean, I do a bit of consulting for companies around like, AI, like strategy and implementation. I think one thing that has been illuminating for me is like the pace of change with AI is so rapid, but the ability for even moderate sized companies, you know, kind of doing a half billion dollars a year up is like really slow. [00:14:41] Speaker B: Yeah. [00:14:42] Speaker A: You know, I, I have direct line to vice president of marketing at a big old, huge company and they're like, it's going to be six months before we can do that. [00:14:51] Speaker B: Right. [00:14:51] Speaker A: I'm like, you're shitting me. Like, like, who knows what this is going to look like in six months. And so I think, I think like, with respect to the SaaS apocalypse or people vibe coding their stuff, like, if you're selling at big companies, the ability for them to change is very, very, very small. If you're selling to you and me and like, I could just pick up, you know, Help Scouts Tool tomorrow or Doc Spot or, you know, ATO or HubSpot or whatever. And no one's going to tell me what to do. I think that's where the risk comes in. And so like, probably what you found is like, oh, our best customers are sticky because they have this like, degree of enterprise ness where, oh, getting them over the hill is hard, but once you do, they're going to stay for a long time because just their ability to change is much less. [00:15:39] Speaker B: Yeah, I would say probably around about at least five employees. Like once you're talking to an employee and like not the boss who does everything, you know, [00:15:50] Speaker A: that's a good litmus test. [00:15:51] Speaker B: Yeah, yeah. Who are assigned to like, have a role, you know, and this is what they do and they have to get approval, you know. Yeah, it's harder. Like we have, we have people that we're like doing pilots with and I'm like, oh, this contract is not even that big and it's taken them months, you know, and they're saying, how many months will this take to deploy? And it's like, honestly, like, we've built it. Like, there's literally an agent built into the platform. You just say, build it for me, and it does it, you know. [00:16:16] Speaker A: Yeah, yeah. [00:16:17] Speaker B: And it tests everything for you and trains your bot and sets up all the escalation rules and everything. You know, like, we can get this working for you in a day. Like, oh, we have to go through this process approvals. The whole support team has to test the bot for a year, you know, before they feel okay using it. [00:16:37] Speaker A: You know, I mean, that was like, so we're customers. That was our, that was our process, I think, is we graduated into it. We had it running for a while and, and we evaluated its responses or we evaluated. We tested it internally, I think, for a while. [00:16:51] Speaker B: Yeah. [00:16:52] Speaker A: And then we turned it on. And it always has had like a good escape hatch. Like people can talk to a customer and then we load the Help Scout widget if they need to send us a message. Yeah, yeah. I think we saw initially like a pretty good drop, like 30 plus percent reduction in tickets, which is good. I think the most, interestingly, the most helpful thing for us has been we now have like, in our own internal process where we have an agent that monitors all conversations in Help Scout and has us go updating our documentation every week. So, like, hey, someone asked this question. There's not a doc about this thing which is probably just making your tool better because you're trained on like our website and on our help docs. So like, I think that's, that's like a little microcosm of, I think overall what's happening with AI and like the, the change in dynamic of the need that people have. Yeah, because that's really what's happening is like nobody's buying a help desk solution anymore. You know, like the way. The way they used to. They're buying a different thing. And that's probably like what we all can't get our head around because it's changing maybe, and it's changing for each customer. Like kind of where they are in this, in this journey is like, how, how. How much other stuff do they have going on that's helping their business? And how. How can you fit in on that is a moving target. And you're probably like right in the, like, we're not in that really at all as podcast hosting software, but you're probably kind of right in the middle of that because this is A a place where people like, rightfully I think, say, hey, we can use AI to make this a lot better and faster and cheaper. But you're also not the, you're not the only solution they're deploying around this. There's a fair amount of people involved, probably for all of your customers. [00:18:46] Speaker B: Yeah. And I think one thing also would be kind of like the whole. I could build it myself now, you know, and in the last year since kind of agentic coding, you know, Claude code and codecs have become, become like much more accessible to normies, I guess. And they're trying to actually build things themselves and play around with it and stuff. I know a lot of people have been saying that's the SAS apocalypse where anyone can build their own software. So I actually dug into kind of our churn to kind of get an idea what that looks like. This is looking at like when. So we kind of do like a. We do force a little mini survey when they're canceling, which is super helpful. And you can see kind of like how that has increased to where people mention like either using AI or just creating their own custom solutions. So that's definitely been a big increase, but it's still not like a massive amount of the churn reasons like it's 13% in the last quarter and this isn't like a massive amount of data statistically. But I thought it was kind of interesting to see how that is changing and how that is affecting. What is cool though is when I dig into these people like they were not a best fit customer, you know, so. [00:20:14] Speaker A: Right. [00:20:15] Speaker B: So either they're too price sensitive or they just like it's a tiny team that was able to do a solution, you know, or they just didn't have enough support tickets or something like that. You know, it's not stockspot. Even without making their own alternative was. Was not enough value for them, you know, to pay our monthly rate. So it's not like a massive loss in that regard. Even though it's scary when you see those like alerts come in that says, oh, I created my own alternative. Yeah, yeah, yeah. [00:20:45] Speaker A: I mean, yeah, I would, I would look at this graph and just for folks who are just listening, kind of the initial cohort here is this like fourth quarter last year, 1.6% of churned customers cited using like DIY or custom alternative increased pretty progressively to now it's 13% of churn customers are rolling their own solution. Yeah, yeah, this feels right. This feels right. I'm sure the biggest one is I'M not using it enough and it's too expensive. Which. Those are kind of the same thing, you know, those are flavors of the same thing. [00:21:20] Speaker B: Yeah, exactly. And they both kind of identify like, not a good fit. Um, yeah. So in our case. [00:21:25] Speaker A: Right. [00:21:26] Speaker B: In our case, we did have a price increase last year. [00:21:29] Speaker A: Okay. [00:21:29] Speaker B: So you can kind of see how that's gone, you know, from 18 up to 27. That's like a 10 points kind of increase, which makes sense. And that's fine. We expected, you know, that. And that's still. Overall, it increased our, our revenue and profit, you know, and we just filtered out kind of the, the less of a good fit users and attracted more of the good fit users. [00:21:52] Speaker A: Yeah, yeah. I want to ask you about the technical side of things because I think one thing that is glossed over a lot and not with you, I'm sure, because you're an AI tool company. But we all think of like, oh, SaaS has great margins and it's kind of a lot of fixed costs and then everything's gravy on top of that. But you're paying for tokens for every customer. How do you. I guess, first question is like, how do you think about that in terms of how you price things? How do you think about. Oh, we expect our token cost to be this. And so we need to bake this in to have the kind of margin we're looking for. Maybe there's something else you do. [00:22:36] Speaker B: Yeah, it is an interesting world. So when we first launched, it was GPT 3.5 and GPT 4. And since then, the cost of intelligence, like for the same amount of intelligence as that has dropped. I think it's something like a thousand extras. It's high. You know, it's in the hundreds of how much like smarts you get for the same amount of money. Right. [00:23:01] Speaker A: Okay. [00:23:02] Speaker B: And so that's been great for us. So first of all, when that launched, I mean, it was expensive, like GPT4. It would be like, it's like $0.05 per, like customer support question that would come in. Right. And so I was fully bootstrapped, had no cash to like put into this. You know, it wasn't trying to run like an unprofitable company. So we actually started bring your own key. So we never actually paid the AI cost ourselves for the first two years or so of, of the product. And so businesses, they sign up, they had the software and they had a place to paste in their OpenAI key, and we just used that for all the inference. So that was great. For us. And surprisingly it didn't. Like people didn't complain. Obviously it added friction to the sign up and onboarding process. But at that point in time, AI was still so exciting and fresh that no one cares. They do that no problem. Create their own OpenAI account and add that key. So that was really helpful for us. Once we started getting like faster, better, cheaper models like GPT4O I think is when we started doing this is where we started building in like a certain amount of usage, you know, and we would charge. We kind of just based on the plans, we say, okay, you get this many user messages or whatever, right. Or this many support questions that's tied to the plan level. And that worked for a while. And then like in the last year as everything started getting more agentic and more complex, we were able to use like cheaper, smarter models still like when the five family came out and stuff and reasoning started being released. But at the same time like you're starting to add tool calls and it's like a full agentic chain where it's calling this tool and coming back and calling another tool coming back, you know, and all that keeps adding to the token cost. So we kind of had to reformat our thing and instead of tying like those limits to like per user question, because user questions could vary like in the cost substantially depending on everything from the tools they have enabled to the training data, to the type of question, et cetera, to a more of an AI credit based. We just renamed it AI Credits and we said okay, before one question was equal to one credit. Right. And then we just looked at all our customers on average, how many tokens or how much token cost do they use per user message or whatever. And then we just kind of set that as like the baseline so that when we made that transition for most users, or at least on average across all users, like the message credits and AI credits were kind of similar. The same unless they added new features or change the model or something like that. Yeah, that's kind of how we approach that. [00:25:59] Speaker A: Yeah, yeah. Do you. One of the things like I'm looking at right now with like just my own internal work and we have a, we have a bit of AI both in terms of like our own internal process, like how we do development and how we do, you know, PR reviews and stuff like that. But then, but like so we have AI in the product and we have our own stuff is how we're evaluating new models. That must be a massive thing for you when you change your own internal models. We Have a pretty shit set of evals. It's basically like, oh, Opus 5.5 is awesome, let's just start using that. That's not smart. As you get more entrenched in agentic systems and your product and your internal processes are relying on this stuff more and more, how do you look at evaluating, I guess two questions. How do you look at evaluating new models and how do you think about when you should look at new models? If the other one ain't broke, don't fix it maybe. Or is that wrong? [00:27:12] Speaker B: Yeah, that's the other tough side of doing AI agents. So it's kind of been a mixed bag. What kind of happened is users would be able to choose which model and so new one would come out, we'd add it. We may make some, just do some testing with a new bot in our default prompt templates. One of the downsides with evals and stuff is we let users change a large amount of like their system prompt essentially. So that gives them a lot of power. But it means that changing anything like a model can completely change the behavior of their bot. That's a bit of a nightmare because it's fully open ended. So there are like internals have some [00:27:57] Speaker A: like fixed evals with like this is the system prompt, this is the input, this is the output. And you can compare different models? [00:28:04] Speaker B: Yeah, yeah, so we can compare. So like for example, like we may need to adjust like our internal like tool descriptions or things like that, classifiers, stuff like that, that the customers don't see. So we'll run evals against that and testing. But yeah, it's hard and sometimes you have like regressions. So for example we have like an internal classifier that classifies if the bot was able to answer the user or not. And that kind of goes into public or into stats that they see if the AI is able to respond or not. And that had a fully different meeting in the GPT4 family. And then the GPT like once GPT5.4 came out, it just took those instructions a lot more literally. And we had about probably like a quarter of our customers where suddenly it just looks like when they upgrade that model it just looks like their bot is performing horribly. You know, when the answers really haven't changed. In fact the answer quality probably went up. It's just classifying it differently more so stuff like that can be a nightmare to deal with. And a lot of times you're just kind of dealing with it when it's discovered like it doesn't evals and stuff don't show that necessarily. [00:29:22] Speaker A: Right, right. What is your, like, your current, like, personal stack for, like, I guess like development work, but also like, what, what kinds of internal, I'll say agents. That's such a bad term. But like, what kind of AI systems do, do y' all run? [00:29:40] Speaker B: Yeah. So as far as models, like, I kind of love this topic because it makes me mad a little bit. Like, everyone on X and stuff, they're excited about the new release and they gotta use Astra for everything. Or Opus 5.5 just came out and everyone's going nuts about that. Right. But when you're actually like, building a product that like, real customers are using, it's not, you don't just always just throw the latest model at it. Like, there's a, there's a balance between like, costs, performance for specific tasks. Speed is a huge one. Like, we're doing like AI agents. Like, you have to respond fast to users so you can't turn on reasoning max or whatever. And it's going to take five minutes behind the scenes before they ever get a response. So while all these benchmarks maybe are tied to whatever the max reasoning setting is for that model, and that's what people are looking at internally, we're trying to figure out, okay, what are the actual our internal benchmarks on? None or low reasoning or medium, Both speed and answer quality. So we're balancing cost, speed, quality, all those things kind of in one thing. So it's tough, it's very different. And the models that get me excited was the GPT6, Luna and Sol release the other day. [00:31:02] Speaker A: Yeah. [00:31:03] Speaker B: And meanwhile, everyone's like talking about whatever the next biggest one, and that's what they're trying in their coding agent. But for me, I'm like, Luna was already, that was like the miraculous model [00:31:14] Speaker A: for us, like, because it was fast and cheap. [00:31:18] Speaker B: Yeah. So fast, so cheap. They had just dropped the price 80%, I think, last month. And we actually passed that on to our customers, which is amazing that we could do that. So their bots all got smarter and using the same amount of credits or whatever. And then just the other day they did Luna, you know, distillation of astra, essentially. So GPT6, Luna, it's substantially better and it's half the price again, you know, so that's really our workhorse. So we're defaulting everyone to that, even internally making some decisions where we just secretly route things through that, you know, or force upgrade people from more expensive poor models. [00:32:00] Speaker A: Interesting. Interesting. Yeah, I, I, you know, I, I think we all for myself at least try to be pretty agnostic. And the tools that I use, the harness and the model, I think it's important because even though I don't do a ton of development, I do a bit, but I do mostly marketing and knowledge work and stuff. It changes dramatically. I was a huge Opus fan for a long time up until three, four, five months ago. And I was just like, I just like I can't, I can't get what I want out of this. This is crazy for me. A long time. Codex was really, really good. I still think Codex is really good. The newest model is. The newest Opus model is very, very good. But it's crazy that it's just such a moving target. Like you just can't stay married to it. Even like as a tool I use when I'm sitting at the computer to do stuff, like I just can't, I can't, I can't be married to one tool because like tomorrow it could be, it could be very different. And I do think there's this like degradation over time. Like it comes out and it's amazing. And then it gets worse as they're I think ramping up to release a new model and that. It happens for both, both companies, I think. [00:33:20] Speaker B: Yeah, it could be that. And sometimes older ones like you mentioned are, are better. So internally we still use like some of our kind of bulk things. Behind the scenes we're using Gemini like 2.5 flashlight, you know, because still like the quality for what we're doing with it like extraction and the price, like there isn't anything comparable in newer model families, you know. Yeah, they just have gone up in cost and power. But there's not those super dirt cheap like ones that are great for those kind of bulk deals, you know. So even like we have some that are still using I think GPT5 nano, you know. [00:34:00] Speaker A: Are you guys using any local models? [00:34:03] Speaker B: No, I mean I toy around with them on my Mac. [00:34:06] Speaker A: Yeah. [00:34:07] Speaker B: But we haven't found any like practical use case. Yeah, we do use some specialized stuff. So we're doing RAG essentially which is retrieval augmented generations. So users upload their docs and then we're extracting that, like finding the best way to extract that, insert it into vector databases, generate like sparse and deep embeddings, figure out all the search logic and things like that. So there, there's surprisingly been very little progress in embedding models over the last few years. Yeah, some of them are actually quite old now like in AI terms. So that's kind of Interesting that there hasn't been improvements there there. We do use a various suite of embedding models. So for example, if a bot is like foreign language then we'll use like a different embedding model than like OpenAI. We'll use like Cohere because that has like better for other languages and things like that. So we're trying a lot of different things and even like extractions. So we recently added obviously your bot learning from your documents is a huge part of our product. So PDFs was always a struggle. So we've always had our own internal extractor but that would fail. Images or complex PDFs are the absolute worst data source for AI there is because that's fully visual. Fully visual layout dependent. You have tables, you have weird things. You know, it's, it can be really tough. So we're using Mistral for that now. [00:35:50] Speaker A: Oh, interesting. Okay. [00:35:51] Speaker B: Yeah, because they have like the best and they have their own small model that's trained purely on document extraction. You know, it's not cheap, it's, it's pretty expensive. You know you're paying per page of PDF. So we pass that on as like higher as using their AI credits, you know, if people choose to that they need that for like more advanced PDFs and spec sheets and things like that. Yeah, yeah, definitely use everything out there. [00:36:19] Speaker A: Yeah, that's. I mean that. No, I think, I think that's smart. I mean like we were talking about this in our little team meeting. Like we have a huddle on Thursdays today is like I play around with local models a little bit and I was like, hey, yeah, like Quinn, Quinn 3.8 is really good for like a lot of stuff that we all probably do. Like it's probably plenty good. I was like, I think there's probably a world pretty soon where there'll be like much smaller, much more specialized models for, for marketing for hey, we write PHP code. Like there's. Is there a model just for php? Because like it doesn't need to know Rust and it doesn't need to know who the Queen of England was in 1935. And you know, it doesn't need to know all that. It just needs to know how to write really good PHP code. And I think that's probably where local will like really take hold is like what if it can run on like, you know, my 8 year old M1 MacBook Air, like do everything I want it to do. That's probably what's going to happen. [00:37:16] Speaker B: Yeah. Our problem, like experiment to that like Obviously local means it's running local. That doesn't really work like for software that's being consumed by other users, you know. [00:37:26] Speaker A: Yeah, right. [00:37:28] Speaker B: So that's like a challenge. Yes. If it's an open model, that means that it could be hosted in the cloud somewhere. But from all our research, like in general, like where those models, those open models are hosted for the most part, like they're not as reliable. Their costs often end up still being more than just using like OpenAI or Cloud or something like that. Just because they don't have the same economies of scale and because those models are changing so often as well. [00:37:56] Speaker A: So interesting. [00:37:57] Speaker B: I mean, there's going to be Quinn 3.9 soon enough. Right. And they're going to stop serving 3.8. [00:38:04] Speaker A: Yeah. [00:38:05] Speaker B: So that's kind of a challenge there, I think, where, for like a B2B, like SAS, where that comes and useful is if you have like security. Privacy conscious, like companies are like, oh, we want to host our own inference, you know, and we haven't figured out how to do that and let them connect DocSpot to that. It's just no one's been at that complexity yet of our customers. A lot of them ask for it if that's an option. It's like, well, do you really Want to pay 20 grand to rent your own private GPU cluster or whatever? I mean, sure, you can do that. But [00:38:45] Speaker A: yeah, I think those barriers will continue to come down for the regulated industries. They'll just. People will accept, it's just like email and there'll be a good secure way to do it. [00:38:54] Speaker B: Yeah. [00:38:55] Speaker A: Over time, I think. [00:38:56] Speaker B: Yeah. Speaking of small models too, we do use one. This is kind of cool use case. So for privacy, there's one that's open one released I think by the US Government actually, or some division or something. And it's basically a tiny model that's trained on PII redaction. So we actually have that in our chat widget because a common worry is like privacy, like how do you control what users enter in to the chat widget? And so we actually have that embedded in our chat widget and if you turn that feature on, it loads that tiny model in the user's browser and that tiny model looks at whatever the user types in and if it finds like a name or a phone number or a Social Security number or credit [00:39:44] Speaker A: card number, it sends it. [00:39:47] Speaker B: Yeah. Before it even leaves their browser, which is really. [00:39:51] Speaker A: That's cool. [00:39:52] Speaker B: Yeah. And we do that transparently. So it sends like a placeholder, like credit Card number one. And that's all you see on our side. But the AI is smart enough to know, like, if they ask, oh, what is my credit card number? It'll reply with that placeholder, which gets dynamically replaced in their browser. So they don't even. They don't even see that it was redacted. You know, that's cool. [00:40:13] Speaker A: That's cool. Last question. Like, what. What are you excited about? Like, I think we're all in uncertain times, right? There's, like, a lot with, like, you know, customer acquisition and, you know, demand for software with AI and all this kind of stuff. What are you looking at? And you're like, whoa, that's awesome. I can't wait for this thing to come to fruition. [00:40:35] Speaker B: Oh, gosh, this. I feel like we're always behind, right? We're cutting edge probably compared to most of the population, but there's so many things I want to try. So, yeah, like, I know JEV was all the rage. The new Jev. [00:40:48] Speaker A: Yeah. [00:40:49] Speaker B: Classifier model. Last week, so I finally ran some benchmarks. Like, in our use case, the best idea that I have for it was using it for re ranking. So in retrieval augmented generation, essentially, we're doing a search based on the user question. Or in our case, we do anywhere from two to five parallel searches of different phrasings, and then we combine all those results and then we try to figure out what's most relevant and we cut it off. And then we only return, like, the top six chunks to the AI to answer from. Right. And that can be hard when you have, like, really complex questions that go across multiple chunks. So JEV could be super useful to where we search when we pull in the top 100 results. And then Jeff looks at them all and says, okay, it gives them each a score. Which one is the most relevant to this query, and then that way we can return only the most relevant things and hopefully get better results. So I was just benchmarking that last night, and there's been the RE ranking models before. Like, Cohere has one, but it's super expensive. Like, it would add like a few cents per per request to us, which hasn't been enough of a quality improvement to, like, try to pass that cost on to users. But this is at least a 10x factor cheaper. In fact, it may save money in that because you're returning less context to the model, so you're using less tokens on that side. [00:42:18] Speaker A: Yeah, yeah, yeah. I think JEV was the first time in a long time there's been like, a fundamental paradigm shift in the models. It's not like, oh, it's smarter or better, faster. It's like, whoa. It's like, super different in just, like, what it works and how it returns. Yeah. So I think, yeah, that kind of thing will be. Will be interesting. Different paradigm shifts like that. And. And I think we just. We. There will be people who are able to think outside the box like that going forward, and probably not the frontier model companies. Like, probably. It. It has to be somebody from kind of outside the box, I think. [00:42:54] Speaker B: Yeah, yeah, yeah. So that's one thing I was excited about. And then I mentioned Luna and Saul 6. [00:43:02] Speaker A: Yeah. [00:43:03] Speaker B: I feel nerdy being excited about these things. I know Jev went viral, but I still feel like no one knows, like, what it's actually for, you know, it's for very specific use cases, you know? [00:43:13] Speaker A: Right. [00:43:14] Speaker B: And it's a great paradigm shift for those use cases, but yeah. Yeah. [00:43:19] Speaker A: Cool, buddy. Well, it's good to catch up. Really, like, really love the tool. Thank you. Thank you for building it. We'll have a link to Dogspot in the description below and on Twitter. Your. Your Twitter handle is. [00:43:31] Speaker B: Is at Ugly robot. So. [00:43:33] Speaker A: Okay. Love it. All one word. [00:43:35] Speaker B: Ugly robot. Yeah. [00:43:36] Speaker A: Okay. Awesome. Aaron, thanks so much for joining. Appreciate it. [00:43:40] Speaker B: Yes, thank you.