Slightly Caffeinated

TJ (00:00)
Welcome back to the Slightly Caffeinated Podcast. I'm TJ Miller.

Chris Gmyr (00:04)
I'm Chris Gmyr

TJ (00:04)
So Chris, what is new in your world, man?

Chris Gmyr (00:06)
It's been a week. I feel like I've just been just generally tired and ⁓ we have a loop bands so kind of similar to Apple Watches and it gives you like a rating of like your recovery and strain and sleep scores and the whole bunch of other data, which is really cool. Like my recovery has been terrible this week for whatever reason. I don't know if I'm just not sleeping well or. Food stuff cropping up or just feeling.

stressed when I don't really realize it or something. yeah, just been dragging in the recovery scores. I haven't been very good. Also doesn't help that I've been consistently waking up at 5 in the morning to go to the gym, so cutting into some sleep time there. So luckily, we're getting into the weekend and hopefully catch up on some sleep and rest and try and reset a little bit.

Yeah, but besides that, like generally things are going pretty well. ⁓ I talked about the go live stuff last week. We got a few people into production, but unfortunately we caught a snag on basically the third party side. We were banking on a feature that they had and then turns out it doesn't fit our needs. So we had to pivot a little bit.

TJ (01:17)
Thank you.

Chris Gmyr (01:19)
and do a quick dev run on implementing their data in our way. So we have to do a little bit of work to pull in their data on our side so we can do some like better reporting on our side. So we haven't been able to ramp up prod traffic and usage as much as we wanted to, but we should be getting back to that in the next week or two. So kind of unfortunate, but we at least have some.

traffic going through that we can, there's been like a few little like tweaks and changes, little bug fixes, error handling, logging and things like that that we can clear up while we still have like pretty low traffic. So that's been nice, but it would be nice to get, you know, a lot more traffic going in here to see, you know, how everything actually responds and how the service handles all that stuff.

TJ (02:03)
Yeah.

Chris Gmyr (02:09)
Yeah, and then outside of work, Scouts started up this week. So we're back into that most weeks or occasional weekends. And then my son decided to sign up for Science Olympiad. So I know we talked a lot about that on your side ⁓ last school year. It seems like we don't have to do like as much as you were involved with. And maybe that's just because they're

TJ (02:17)
yeah.

Nice!

Chris Gmyr (02:36)
⁓ younger and they just might have more helpers and people available like at the school to help they Are taking volunteers, but it doesn't seem like a requirement So we're gonna help like here and there but it doesn't have to be like a weekly thing or pick ⁓ an activity or something like that So yeah, I'm like are you sure you want to join because we got like this and that and like are you sure we don't have to be like required to be there because

TJ (03:01)
Right, right.

Chris Gmyr (03:02)
TJ said like he was a ball and told to do that. Yep. So luckily, at least so far, we're still getting trickles of information because it hasn't kicked off yet. But he signed up and ready to go and we'll see what happens next. But yeah, got plenty of other fallen told activities going on at the moment. Yeah, that's that's roughly about it for me. What's new in your world?

TJ (03:04)
I got, yeah, I was voluntold. Yep.

Yeah.

Yeah. Jeez. Yeah.

man. Well, it's been, it's been a bumpy road, mental health wise, but, Tackling it, doing the re reaching for my tools, doing, doing the things. So, you know, that's, Hopefully we'll be on an upswing here pretty soon. I've been deep, deep in prison work. just slogging through this like big streaming output refactor.

because we might get into it later, but like, I have to go back and touch every streaming provider. So we have like nine providers where we support streaming and I have to go through all of them and basically like rewrite streaming with all the new events so that all the new methods work and everything. So it's just like.

Chris Gmyr (03:57)
Mm-hmm.

TJ (04:12)
I guess it didn't really like, it didn't really dawn on me like how much work it was actually going to be. And it is, it is a lot of work.

Chris Gmyr (04:23)
Yeah, especially for so many providers. I guess I didn't realize that you had that many providers in there already.

TJ (04:30)
Yeah, no, it's crazy. have the count.

Yeah, we support 11 providers and nine of them support streaming.

Chris Gmyr (04:37)
Yeah, that's a lot.

TJ (04:38)
Yeah, that's crazy.

That's a lot of providers. so there's, and, Claude has been, I don't know, we can talk about that later. Uh, so no, I've just been like deep in that. and you know, we started middle school and we just had like meet the teacher last night. So it's been lots of like still settling into like new routines early, super early mornings. Cause I dropped him off at like seven 30.

So it's like super early mornings doing that. I'm trying to get into like a new routine where like I drop them off and go to the gym because if I'm up and already in the car might as well like stop by the gym on the way home, you know, and still probably get back before my wife is up. Um, so yeah, I think, I think that's, that's what I'd like to settle into. I haven't done it yet, but we'll see how next week goes. Um, there's just a lot of, a lot of settling in with all of the.

Chris Gmyr (05:15)
Yeah, exactly.

TJ (05:32)
the new routines and everything still, but, he's enjoying it, which is the, big, the big win. Elementary school was tough for him. And, so far he's loving middle school. So yeah, he's trying to start a, he's trying to start a club. he's trying to start a cryptid, a cryptid club, because he's my kid and I love that.

Chris Gmyr (05:42)
Nice. That's awesome.

That's cool. I feel like that's half the battle right there is just like having them enjoy school and going to school and not only like just seeing their friends and like some fun activities, like hopefully enjoying at least some of the education that they're getting.

TJ (06:05)
⁓ he's like he loves his science teacher. He talks about it all the time, which is so cool. We met we met all of his teachers last night and I totally see why. Like he's got a bunch of really cool teachers. Found out he's in accelerated math. Had no idea. So we'll see how that goes. ⁓

Chris Gmyr (06:23)
Yeah,

that's awesome. Science and math, go get it.

TJ (06:26)
Yeah,

yeah, so that's cool, man. You want to talk some coffee?

Chris Gmyr (06:31)
Yeah, let's talk some coffee. I finally got some new beans. It's been a while just on kind of the regular purity coffee that I get. But I was perusing their site last week or the week before because my subscription was coming up for renewal and saw that they had like a good handful of other beans that I didn't see before. So a couple that I got was from this ⁓ Secret Cups.

a roaster which kind of works through purity and it's pretty cool. It's like a women owned brand. So they run it. It's like responsibly grown and all that. So I'll put some links in the show notes if everyone wants to check it out. But this one is from Honduras and this is my second or third day of it. And yeah, it's tasty, really good.

TJ (07:25)
Hmm.

Chris Gmyr (07:25)
And yeah, it doesn't have all the other stuff in there that I'm trying not to drink, like the pesticides and all that stuff. So yeah, benefit. That's a little bit more expensive than just the regular period-y stuff. But I figure, yeah, just try it out. And it's basically what I was paying for for other single origin, because this is a single origin, too. So it's nice that it can get the cleaner cup of coffee for a similar price.

TJ (07:32)
Yeah.

Chris Gmyr (07:50)
and goes to some benefiting some good companies out there.

TJ (07:54)
Yeah, dude, this looks great.

I might have to snag a bag, try it out.

Chris Gmyr (07:58)
Yeah, do it.

TJ (07:59)
yeah, this is cool. Just scrolling through it now. I dig it, man. Yeah. I've been on some like not, not new stuff, but switched it up. another one we've talked about in the past, counterculture ended up with a couple. Yeah. Ended up with a couple bags. ⁓ the local grocery store, had them on like clearance. ⁓ and so I picked up a couple of bags of hologram. We've been going through that and that's been.

Chris Gmyr (08:11)
of conic culture.

Mm-hmm.

TJ (08:25)
That's been a good set. I like it.

Chris Gmyr (08:26)
Yeah,

I like hologram. That's what I do mostly for cold brew in the summer. And that makes a really good cold brew for sure.

TJ (08:35)
⁓

yeah, I bet that. Yeah, I've not made it with it before, but just thinking about it like that, that would make a really nice cold brew.

Chris Gmyr (08:44)
Yeah, I love counterculture and I'll have to circle back to them in the future. But what's cool is, one of their headquarters is in Durham, North Carolina, which is about 30 to 40 minutes away from me. So basically like right down the road. And when I was ordering from them, because they roast it the next day after you order it through their website. And then I'd basically get it the day after that.

TJ (08:58)
⁓

Chris Gmyr (09:13)
So I'd be getting like fresh roasted coffee from a high quality roaster kind of culture two days max after it was roasted and just like fantastic. Um, so yeah, highly recommend kind of culture. It's great.

TJ (09:22)
Wow.

Yeah, no, I'm, I'm thoroughly enjoying it. and it was cool. We had a, ⁓ a listener reach out to, and recommend some coffee. Yeah, this is, this is great. more of you should reach out with your coffee recommendations because, that's super rad. recommended some Carver coffee. we can put the link in the show notes. and they, so they really liked the Vietnamese cinnamon, which sounds super interesting. So.

Chris Gmyr (09:35)
Awesome.

TJ (09:51)
They've got looks like they've got like a whole bunch of offerings like loose leaf tea and ⁓ other stuff too. So something worth checking out.

Chris Gmyr (09:59)
Yeah, that's really cool. I've not heard of this company, but yeah, the website looks super cool. Lots of offerings. So yeah, I'd definitely be up for trying it. And super awesome that we got some listener recommendations. I'll send those in.

TJ (10:17)
Yeah. Yeah. Yeah, I dig it. I appreciate that for sure.

Chris Gmyr (10:19)
appreciate it.

TJ (10:21)
sick, man. So moving on from coffee, return of Clickhouse. Yeah. Yeah. So where we left off is we were doing a part of our migration strategy. I guess to like refresh the listeners on the project, at GeoCodeo is like, do essentially metered based billing. and.

Chris Gmyr (10:26)
It's been a little bit. We haven't heard about ClickHouse in a while.

TJ (10:44)
we need to track all the API requests that come through our system because we use that to deduct credits or monthly we roll up your usage and charge you. We have different ways that you can interact with us as far as payment goes. So either way, it's like metered-based billing based on number of requests and number of additional data pieces that you add on to

Chris Gmyr (11:02)
Mm-hmm.

TJ (11:08)
your requests. So we track every single request. And so right now we're storing them in a MariaDB database with the TokuDB engine powering it. And that has been sustainable for a while, but we're getting close to like being at like a billion requests, like a billion log requests per month. ⁓ Yeah. So, and that's,

Chris Gmyr (11:28)
And that's a ton of requests.

TJ (11:33)
That's not necessarily equating to HTTP requests. I think we're probably closer to 700,000 HTTP requests. But just the way we total the additional things that make up a request and data appends and spreadsheet uploads and stuff like that, it comes out to a billion logged units.

Chris Gmyr (11:50)
That makes sense.

TJ (11:54)
But I think it's probably close to like 600, 700,000 actual HTTP requests a month, which is still a lot. And so we end up with, I guess that's not necessarily true. We end up with pretty close to a billion records in the database anyways.

it gets a lot of traffic. And so we're really like at the limits of what we can do with MariaDB and tracking these requests. So we decided to take a step back, look at the system and decided to go with ClickHouse as the storage engine for requests. And we've been on this like big project to make all of this happen and like migrate from MariaDB over to ClickHouse.

Part of the project is, I'm trying to think about the next best thing to explain here. part of this migration process is what we wanted to do is set up dual entry. So we're still having all of the reads come from the MariaDB database, but each incoming request we log to both places. It goes into MariaDB and goes into ClickHouse.

And then we can like audit the two systems that way, make sure our ingestion pipeline, which is like been another big journey of like how we get the data from our Laravel application into ClickHouse. and so we're doing like dual entry and like that also affords us to not only audit, but, make sure our ingestion pipeline is like working and stable and that ingestion pipeline. What we settled on was from Laravel to Kafka.

And then we have a piece called vector that sits in the middle and that batches the events from Kafka into Clickhouse. So that's the ingestion pipeline. That's been like running super smooth. I just ran some audits. I started doing audits like last week and discovered there was like a bunch of data missing. Come to find out what was happening was.

We track our API requests, but we also have the ability for you to upload spreadsheets. And each line of the spreadsheet basically equates to an API request. So we need to track in this request log the spreadsheet uploads, because it counts against your credits and stuff we bill for. So it gets tracked in the request table. And those were making it to MariaDB, but not to ClickHouse.

Turns out the problem is that I had a non-nullable field for the IP address in ClickHouse. And ClickHouse has an IP address data type, but I was sending an empty string for spreadsheets because it's not a request. There's not an IP address associated with it. So vector was like,

failing to insert into click house, like, silently failing. So like we had an observability hole and this is why we wanted to do dual entry for a while so that we can like make sure we have things set up appropriately. like I've got to

Chris Gmyr (14:48)
Yeah,

this is a great use case for that, by the way. It's like, that's why you do this sort of thing.

TJ (14:51)
Yeah.

Yeah, so now we found an observability hole, I'm working on patching up. So now we have visibility into these failures. But I then just set a default value for the IP address. And now spreadsheet inserts are working. Then we come to find out we also insert the way we handle credits. It's not ideal, but we insert into the request table a negative integer.

is your credit so that when we like some like our calculation is like a some query to like get all of your usage it would like add negative amount of requests and so that's kind of how we handle like totaling credits inside of this like request query but I had the data type for like

the field as an unsigned integer. So again, Vector was failing to insert this negative amount because my data type was unsigned. So I make the query, fix the data type. Now we're getting all the data. Now we should be getting all the data inserted into ClickHouse. Because the events were happening in Kafka, but again, Vector was just silently failing to insert these records because of data type wrangling.

Chris Gmyr (16:07)
that it would fail silently like that because I would expect it to not and then get pushed back on Kafka and like the DLQ or just sit there waiting for you know the fix to be issued.

TJ (16:20)
Right. So

it's not that vector like silently fails. It's that we weren't listening for things in vector. So like we weren't observing the failures in vector, but like vector will throw like there were logs saying like, you know, once I enabled the right log levels for everything, there were logs saying that it was like failing to insert into click house, but like we weren't observing those logs appropriately. ⁓ so like

Chris Gmyr (16:47)
okay, gotcha.

TJ (16:48)
we needed to add, I don't think I had the right log levels enabled. And I just don't think I had things configured correctly to get the right logs. And then we also weren't observing the logs. So now I've got to go in and patch that up, get that into Grafana so we can view error rates and things like that. So we got to fix the observability there.

Chris Gmyr (17:00)
Mm-hmm.

TJ (17:10)
Now that I've got all the data in there, I did a few days worth of auditing of some of our like high traffic customers. And the cool thing was, out of, I dunno, six or eight customers that I checked, there were only a few that had differences and they were only off by like one to five requests, which is like totally fine. Yeah. Like that's a totally fine thing.

Chris Gmyr (17:32)
Yeah, that's not bad.

TJ (17:36)
And Matias went and did some additional auditing later, and found that there were actually more requests tracked in ClickHouse than there were in MariaDB. And they weren't like duplicates. So what it sounds like is happening is that we do have timeout failures on inserts occasionally into MariaDB. And so it sounds like.

our data pipeline for collecting requests for ClickHouse is just already more accurate and more performant. that was kind of a crazy discovery is that the new system is already proving better and more accurate because we're still getting these timeout inserts. So.

I think that kind of covers everything about the project up until this point. So I've been doing auditing. And the big push I'm on right now is to start writing, switching over the reads. So we've been doing this dual entry. So we've been writing the click house. But now we have to update all of the areas where we read. So this has been.

a slog this week doing it. The overall strategy, I think, is sound. What I've done is I've identified, we have basically this what we call the request repository, which is the central layer to access this request database. So because performance is a concern, there's areas where it's got raw SQL queries, because it's just more performant than using

eloquent and the query builder like.

So we have this request repository, but we've identified two pieces of functionality that access the request database directly. So now I've got to centralize everything to the request repository. Then I guess let me talk about the strategy, and then I'll talk about the work that I have to do. So the strategy is going to be to create a request repository interface.

the MariaDB.

like concrete to that interface in the service container and then create a click house request repository. Then I can like feature flag the reads to then like that feature flag. If enabled, it just swaps the two implementations in the IOC container. So that way I can like feature flag our internal users and like go into the production system and start auditing like the reads and everything and make sure all of that stuff works. And then

The cool thing is because all of the reads go through this request repository class, just swapping the IOC container, like the implementation of the IOC container, like everywhere that is going to have reads is going to start working that way, which is going to be really nice. then for testing, but then for deployment too,

that makes it really easy to roll out for deployment because we could just have like an end variable or something to say like what request repository to use. So if we need to like roll back, we can roll back. But also continue dual entry into the old system in case there are any issues. We do have a fallback still to like our previous system, which it works, but it's like very non-performing.

Chris Gmyr (20:44)
Yeah, I love that solution for the implementation. That's super smart, especially with the feature flag and conditionally changing those interfaces on runtime or based on client or customer account. I think that's a great solution.

TJ (20:44)
⁓

like the

IOC container is so powerful for this kind of stuff. And so the work that's been involved is I've found these rogue queries that aren't going through the request repository that I need to centralize to the request repository. And then I need to write the clickhouse implementation of a request repository, which isn't so bad.

what I found out were the things that were using the rogue queries, those don't have any tests around them. So I've got to go write tests to then make sure when I swap out for the request repositories that those things continue to work. And then everywhere in the code base was newing up the request repository. So I have to go around to all of those and resolve that out of the IOC container so that

those pieces of functionality continue to work. So it's actually gone real sideways on the implementation side. I've been really trying to leverage Cloud Code to do a lot of the heavy lifting, but it is just, I'm fighting it. It's dropping the ball. I'm like, hey, write these tests. And it's just writing not helpful tests around these things. It's like,

I don't know why it's struggling so hard with this. like, I think, I think I gave up yesterday and today I'm just gonna like.

write it myself. Like do go go the yeah, the old fashioned way and just write the code myself because I've spent like two days trying to get Claude to like do the heavy lifting and it's just

Chris Gmyr (22:18)
Do it the old fashioned way.

TJ (22:29)
I'm now spending time.

messing around with the AI instead of actually getting work done when I could just be actually moving things forward, writing code by hand.

Chris Gmyr (22:39)
sometimes you just gotta go that way

TJ (22:41)
Yeah, it's it's been a nightmare. But the cool thing is, is like, it's iterated enough that I

I know exactly like what I'm going in to do at this point.

I don't know. We'll see. I'd love to like take another shot at it and like really prompt out like a little bit further and see if I can get it to do it. But I don't want to like burn the time anymore. Like I've already burned to like I really should have given up yesterday and started like handwriting the code yesterday. But I don't know. It's going to be today. So hopefully we'll we'll make up some time.

Chris Gmyr (23:18)
Yeah, well, even if you can get a small handful of tests done, maybe it can take over from there, or maybe it will help with at least the implementation changes that you need. So maybe it just needs a little push in the right direction.

TJ (23:28)
Yeah.

Things where it's super helpful is like, yeah, go around and like find everywhere where we're newing up this class and resolve it out of the container. Then like from the interface. And so it did all of that. No problem. Like that, that's super useful stuff. but yeah, like it's done a horrible job writing the tests. I mean, these are actually, they're both artists and commands and those are, I feel like.

Artisan commands are really tricky to test. I think in general, so, I'm not surprised that it's struggling a bit, but it's been painful. but I'm, I'm excited to like push on the project. I'm hopefully I'd really like to get the reads wrapped up by the end of next week. which I think is, is very feasible, because I've got.

some very exciting other initiatives coming down the pipeline that I'll talk more about in the future.

so. I'd like to get this project wrapped up. But it's also...

Thinking about deploying it, while I've had like really positive feedback, thinking about deploying it is like a little scary for me because it is like, it is our billing. So like, it's not the mechanics of doing the billing, but like it's how we calculate our billing. And so there's like, I'm very paranoid about rolling it out because.

Yeah, it's how we calculate our billing and that's sensitive and important.

Chris Gmyr (24:52)
Yeah, and probably more of a bigger potential issue for larger clients. But have you seen like a big discrepancy for them between the two technologies? Because I think like as maybe like a bigger account, if I'm getting, I don't know, 100,000 requests for the month.

right on your older platform and now in the newer platform, even though it's calculating or logging those and calculating them right, if I'm going to get like an increase of another 50,000, so 100,000 to 150,000, are you guys going to put any sort of like buffer in for that or notify specific clients of like, hey, we actually like improved our

TJ (25:23)
Mm-hmm.

Chris Gmyr (25:47)
billing and tracking mechanism and this was like an issue before but now we're tracking them clearly but just letting you know that the newer higher number you know is more accurate. I'm not sure if you'd have any sort of like those situations but I can see there being a potential issue on the customer side if things are no longer consistent of what they've seen for the last like three, six, twelve months, you two years type of thing.

TJ (26:14)
you know, I in my audits

I was only seeing like differences of like one to five requests. I mean, and this was across like, this was also only across like a two day span. but I don't think the, numbers are going to be like that significantly different between the two. so I don't think anyone's going to get surprises of like, wow, are you, like where we've done nothing in our usage just jumped by like 50,000.

like what the heck. I don't think that's gonna be something that happens but...

Maybe something to think about.

Chris Gmyr (26:48)
Yeah. Yeah. And I think maybe even, uh, well, a potential option is to just do, uh, maybe a previous lookup as you're creating the new bill or invoice and be like, you know, how far away of a percentage is this from the last bill? Cause if you say like, okay, we're going to have, um, a little bit of a difference that we're okay with maybe like three to 5%. But if something goes over

5%, 10%, 20%. Let's raise that internally first before we actually send the invoice or bill or notification to the client. Maybe something like that to have some checks and balances before it goes out.

TJ (27:30)
Yeah, maybe.

Hmm. Yeah, something to think about for sure.

But one of the cool things that I discovered in the auditing was the calculation query to sum up their month to date was taking 60 seconds in a MariaDB database, anywhere between 60 and 70 seconds for a pretty high volume customer.

that causes issues, being that slow. But I take that exact same query and query click house and it's finishing in like milliseconds. Yeah, so it's like, and for the numbers to like match up was like so dope just to see like that huge of a difference in query time is like, that's exactly why we're making the switch. And like, it's nice to see the choice pan out, you know, like.

Chris Gmyr (28:04)
That's so awesome.

TJ (28:24)
That was the hopeful delivery is that we're going to be able to do these queries really efficiently now. And yeah, to see the queries go from 60, 70 seconds down to milliseconds was just super dope.

Chris Gmyr (28:36)
Yeah, yeah, that's awesome. And it opens up so much more possibility in the future. Like if you're only doing like daily calculations, like you could drop it down to hourly or even just live calculations. If it's that fast, like why not? You know?

TJ (28:50)
Yeah, there's definitely affordances we've made in our systems due to the query times being really long. We cache real heavily and like...

it'd be, there's things we do to account for the fact that those queries are slow, but now that those queries are gonna be fast, yeah, what other affordances does that give us to maybe make our system more accurate or more performant in other places too, or undo some complexities that we had because the system was slow? Do we have opportunities to just make things more simple in the code base by like,

maybe querying a little bit more often. It's like so performant.

But it also gives us some ideas around refactoring our developer center and exposing request debugging and stuff like that, too. That was just something that was almost unapproachable before because the query times would be so poor. But now that we have a performance system, maybe we can give you your query logs that you can just go through the history of all your queries, something like what Stripe does. That could be dope, too.

Chris Gmyr (29:53)
Yeah, that would be awesome. It's a huge upgrade just right there.

TJ (29:57)
Yeah.

Yeah. So yeah, return of the Clickhouse project. But again, like hoping to wrap it up by the end of next week. Hopefully it doesn't bleed like much further than that. ⁓ Yeah. Move on to some fun stuff.

Chris Gmyr (30:07)
Sweet. I love it.

Looking at the Grafana dashboards, once you click the button to take that live and everyone goes over to the reads for click house and seeing the response times drop. And that's going to be a good feeling, I'm sure.

TJ (30:23)
Yeah, I hope to see some really fun cliffs on things. But what we're going to do, yeah, I think I mentioned it to you, is we're going to continue to dual entry once we switch over reads and everything.

Chris Gmyr (30:29)
Mm-hmm.

TJ (30:36)
If we need to fall back, we can always kind fall back. And that's the nice thing about the IOC containers. If we need to fall back, like swap out an ENV variable or something, or push a quick code change, like a one line code change, and we're back on the old system.

Chris Gmyr (30:52)
Yep, 100%.

TJ (30:54)
Sick. ⁓ Moving on from Clickhouse, man, we can talk some Prism a little bit. Yeah. So I've been in the midst of this massive streaming output refactor, which is like the thing that's holding back 1.0 because I know this is going to introduce, like, basically it's a breaking change for all of streaming output.

Chris Gmyr (30:54)
Awesome.

Yeah. your cow.

TJ (31:17)
so.

We're introducing like three new streaming output methods. So like we have as stream, which gives you, it's a generator that gives you raw, like the raw system level events. Then we have as broadcast, which broadcasts Laravel events that you could like hook up web sockets to. Then we have as event stream response, which is like a standard server sent event response type using.

Prisms events, and then we have as data stream response. And the data stream response is Vercell's data protocol for building with their AISDK UI components, which is actually like super, it's super cool. Makes building chat UIs like so trivial. I was very impressed. So.

That's kind of where the direction of streaming is going, just making it easy for people to even just return from their controller or their route, like the prism invocation. And then we're going to add some hooks in there to be able to hook into on stream end. So if you want to persist messages or do stuff, you have the opportunity to do that too. But like I was saying, I've got to touch all these providers. And so.

Chris Gmyr (32:28)
Mm-hmm.

TJ (32:28)
Claude rather successfully migrated, Anthropic, OpenAI. Let me see what is done so far.

it's successfully done, open AI, Claude, Gemini, Olama, and Grok with a Q. but I started having it try to do like XAI and Maestro and, it's just like,

really breaking those. So I'm gonna like again, I've had like four different attempts with Maestro and it's made the same similar mistakes every time. So like I'm just gonna have to go in there and like hand convert it.

which I feel like is such a like first world complaint. Like, man, I've got to, I'm going to actually go write this code now. Jeez.

Chris Gmyr (33:11)
I got a right code at work. Got a right code on the side project. Like, what's going on?

TJ (33:14)
Yeah.

Yeah. I feel like I haven't like actually written code in months. Just I've been prompting my way through everything. so, what I'm finding too is that I'm finding a lot of holes in the provider test suites. like, it'll like for maestro, for example, like it did the conversion and all the tests were like, it went and updated the test with like

checking for the new events instead of the old chunks. And the test passed, but then I go and actually make an API request to the server send event endpoint. And it comes back with two events. Instead of like, it should be coming back with dozens of events. ⁓ But the test still passed. So I'm like, all right, there is definitely some like,

Chris Gmyr (33:51)
Mm-hmm.

TJ (33:59)
big holes in our testing for certain providers. So I think that's going to be something I have to go in and do too is like the tests. I went through the test file a little bit earlier because now I've been up so early with taking the kid to school. I've been coming home and working on Prism. The tests are there, but the assertions are not.

useful or they're not as useful. I've got to go in, I think, and just like augment the test with better assertions and

Hopefully, it'd be nice if the test failures were enough to get it to, like, actually write the provider correctly. But like I said, I'm on, like, four attempts of, like, very, like, making variations on the prompt or just, like, letting it have another attempt at it. Because, like, every time it does it, it, like, approaches it a little bit different.

And so I was just kind of like hoping that it would approach it well. and it's just not so. I like I've, down to like four providers left. It's not really doing a good job on any of those four providers, but like once those four providers are done, like I'm ready to merge the new streaming output PR, ⁓ which is great. but one of the things I'm definitely a little concerned about is I got.

Chris Gmyr (35:07)
Thanks.

TJ (35:14)
I'm a little burned out on this project. I've been working on it. I've been like thinking about it since before Larycon. I've been working on it.

Chris Gmyr (35:22)
Yeah, I know we've

talked about just the stringing feature for a long time.

TJ (35:27)
Yeah, like I've been thinking about it for a long time, but I've been thinking about it because I know it's holding up 1.0. I've actually been thinking about it pretty hard since like starting to prep for Lair Con because I was like hoping to like tag 1.0 at Lair Con or in Denmark, but.

I've been like, I've been really like working on this and thinking about it a lot for a while and like, I'm kind of burned out on it. And so like one of my concerns is that I.

I kind of leaned into the vibe code enos of having Claude like converts the other providers. And so I've done manual, like on every provider that it's done so far, I've also done like manual testing on, and I know the streams are being returned correctly, but I'm not sure I feel great about the code quality. because I did let it kind of vibe a bit on doing them.

I don't know what to do about that. Like, do I merge it because it works? I think, I think this is the plan. I think I'm going to not sweat that too much. I'm going to get the PR done with all the providers updated. I'm going to merge it, get it into people's hands as soon as possible. Because if I need to iterate on it more before I tag 1.0, I want to get that out of the way.

before I tag 1.0. So I kind of want to let this simmer.

I want to let this feature simmer for a little bit before I tag 1.0, just so that I don't have to tag 2.0 soon after. ⁓

Chris Gmyr (36:50)
Yeah, definitely. That's

kind of what I was thinking, too, is if it's working, you can always make it better in the future. Clean things up, make it look a little nicer. But if the functionality is there, yeah.

TJ (36:59)
And while I'm letting it simmer, yeah.

Yeah. And while I'm letting it simmer, can like go in and like refactor internally. I can always refactor internally. Like at the end of the day, the events are like, the events are the events and they need to be emitted like through the generator. So.

Yeah, think that would be the plan is like, just get it, like not worried too much about the code quality, get it merged, and then I can go back and start cleaning things up.

Chris Gmyr (37:25)
Yeah, I think that's a good call. Just get it out, get it in people's hands. And like you said, like you don't really use some of the providers all that often. So getting the people who are using them and building with them actively right now would also be a benefit and not just working off the separate branch. So yeah, a hundred percent. Get it out to people when ready, clean it up. You're going to have bug fixes and tweaks and stuff like that anyways for a little bit.

TJ (37:43)
Yeah.

Chris Gmyr (37:55)
And then once that settles down, then you're rolling up to 1.0.

TJ (37:59)
Yeah, there's, I feel like I've definitely been neglecting the repo a little bit. There's, there's stuff I want to get to. but I'm, I'm really trying to make this push for 1.0 and getting like the streaming stuff done. I'm, really hoping, I was hoping to get it done this week, but being that I've got to like go in and manually work a bunch of these, like a handful of these providers,

that I think it's gonna like, I think it's gonna take me through next week to finish it up. excited to get it out there. Like I think this is a huge update to streaming output.

So I'm excited to get it out. We'll get through it, think, hopefully next week. But there's a draft PR, so if you want to go check it out. then it's been an expensive token month, Testing, streaming, and all these providers. I am just owing everybody money from tokens.

Chris Gmyr (38:47)
No, I gotta get this thing done so you're just not bleeding money and blood tokens.

TJ (38:52)
Yeah, like tokens everywhere. So sponsor, this is where the sponsorships rig of Prism like really come into play. is supporting like that kind of like this kind of stuff where I do have to like make tons of requests to providers to like make the integration and test things out. because like once I get everything right, then I can like cash fixtures of the responses. But like I got to

Chris Gmyr (39:13)
Mm-hmm.

TJ (39:16)
test live requests until everything's like settled and correct and then I can capture it. So yeah, that's where the sponsorships like make a huge impact. tons of appreciation for everyone that's a sponsor.

Chris Gmyr (39:29)
Yeah, that's awesome. Love it. We're getting closer.

TJ (39:32)
They're getting closer, So on that, you want to wrap things up? Sick. So thank you everyone for listening to the Slightly Caffeinated podcast. Show notes, including all the links from things we've talked about, and social channels are down below and also available at slightlycaffeinated.fm. Thank you all for listening so much. We'll catch you next week.

Chris Gmyr (39:35)
Yeah, let's wrap up.