The Future of AI Accelerators

Disclaimer: The Futurum Tech Webcast is for information and entertainment purposes only. Over the course of this webcast, we may talk about companies that are publicly traded and we may even reference that fact and their equity share price, but please do not take anything that we say as a recommendation about what you should do with your investment dollars. We are not investment advisors and we do not ask that you treat us as such.

Transcript:

Daniel Newman: Hello, I’m Daniel Newman, principal analyst and founding partner at Future Home Research. Welcome to this webcast. Today I’m going to be joined by Groq CEO, Jonathan Ross, and we’re going to be talking about the future of AI accelerators and how this technology can help organizations reduce uncertainty. Jonathan, you just came back from Washington DC. It’s been a really interesting couple of years since the pandemic. Chips are back in vogue. Talk to me about what’s going on. What brought you to DC?

Jonathan Ross: Well, like you mentioned, chips are cool again, and I was in DC actually talking about how America could be competitive with chips. So I was invited by Eric Schmidt to his new SCSP.AI conference, and there were a lot of illustrious people in attendance.

Daniel Newman: Yeah, it’s been pretty interesting over the last few months when the CHIPS and Science Act passed to see all of those CEOs from leading technology companies all go to Washington. Of course, we know a handful that really championed a lot of companies are participating. Many are probably now thinking, what opportunities does this provide for us? Groq is a really exciting startup. As an investor advisor, I got behind the company immediately, I think after the first time I talked to you. I’m like, yes, I want to work with that company. They’re doing really interesting things. Talk a little bit about sort of the genesis, why you founded the company, the opportunity that you saw.

Jonathan Ross: So it started off with software. Every chip that that’s made today is actually designed by hardware engineers, and that would be like having auto mechanics, design cars. The comfort of the seat wouldn’t matter. Also, you probably wouldn’t have air conditioning because that just slows the engine down. Why would you want that? You probably want a driver to actually design a car and then have mechanics help build it. And in our case, I’m a software engineer. A lot of our team are software engineers, and so we approach building this chip that we built from the ground up, as a user of the chip as opposed to as someone who’s optimizing for the building of the chip. So it’s resulted in a very different architecture, but it’s very easy to program.

Daniel Newman: So can you share some examples of customers that are working with Groq today?

Jonathan Ross: Sure. So one of the first customers is Argon. So publicly they announced that we were able to get a 200X performance boost over a 100 GPUs from NVIDIA on drug discovery. They’ve actually found candidate drugs that have gone to testing in actual labs as part of this, and that’s a huge deal. Now, the next one was also same lab Argon. They’re actually able to get a nuclear fusion reactor control loop working on our chip. And this is something with a very tight deadline. You have to give an answer within a millisecond. And on that one, we were able to meet the deadline. GPUs were not, and we were 600 times faster even if you ignored the fact that the GPUs couldn’t do it.

And then lastly, entanglement, they’re doing cybersecurity anomaly detection and so on, and they just had this amazing report come out about them from the US Army. And so they are 700 to a thousand times faster with much higher accuracy and lower false positives than anything else. And they were able to do this on our hardware and they weren’t able to do this on other hardware. And that company’s fascinating because they actually designed their algorithms to run on quantum computers and they couldn’t find… They did some stuff on quantum computers, but they were unable to find a quantum computer that performed as fast on the quantum algorithms as our chips. And so they were actually using our chips instead of quantum computer.

Daniel Newman: Yeah, it sounds like you guys actually unearthed something about where quantum and application specific chips can actually do a better job, because I think the misnomer about that was often the quantum was going to do these things and that your accelerators couldn’t. And it turns out that with the right algorithm, the actual chips are better than the quantum machines. But it sounds like Groq is really getting a ton of momentum in the federal space, in the government space. And obviously that’s kind of interesting to me, Jonathan too, because we talked a little bit earlier about the disruption in an organization, the status quo. Oftentimes federal government is thought of as maybe slower to move, and it looks like they’re moving very quickly in this particular case in terms of adopting technologies that could help them solve these problems more quickly.

Jonathan Ross: Federal’s been moving very quickly. I mean, the CHIPS Act just got passed. That’s been a big deal. And there’s a lot of money not just for the manufacturing, but also for science and other applications where they’re going to be buying a lot more compute. But going back to those examples, just to highlight every single one of those was actually based on a report. So these are 200X, 600X, 700 to 1000X performance boosts, and that’s based on things that are actually certified and reported by agencies where the reputation matters. It’s not 200% 600… Those are 200X, 600X.

Daniel Newman: Yes. Exponential.

Jonathan Ross: Exactly… Well, in a couple of orders of magnitude, not just one order, not just two, but verging on three.

Daniel Newman: You mentioned a lot of these federal examples. I think they’re very interesting. Of course, it made a little tie into quantum, and so that gets me thinking all the different applications where a chip, like what you’re building could really make a difference. You got things like in financial services, you’ve got anti-money laundering, you’ve got high frequency trading, you’ve got drug discovery, which Argon sounds like is doing a little bit with. You’ve got cybersecurity for organizations and enterprises. You have, like you said, advertising and stuff.

Jonathan Ross: Autonomous vehicles, molecular drug discovery. These are just some of the things that are chips are being used for.

Daniel Newman: So yeah, talk about that. Are there some other applications and industries where Groq has some momentum and is gaining a lot of interest?

Jonathan Ross: So some of the most exciting… Like I said, molecular drug discovery, finances is particularly large. So what we’re discovering is that what matters the most is not necessarily just how fast the chip performs, although our chip performs really fast. What matters is how quickly a developer can take the code that they’ve written and get it into production. Because if it takes too long, then they’ve already missed the window and no one ever thinks, oh, my code isn’t running as fast as it could. I could run it twice as fast. No one cares what the care about is. Can I get the answer quickly enough? So because we started with the compiler, what happens in a lot of these cases is someone comes to us with some code, they’ve been trying to get it working on a GPU, an FPGA or a CPU, and they just can’t get the performance that they need, and then they come to us and we’re able to get it running quickly and performing.

Daniel Newman: Yeah, so that’s pretty interesting. So when you’ve seen success, some of these case studies have been written up, some of these selections by major US based laboratories. What is the timeline from time to value? Because I think that was something, at least in conversations that we’ve had in the past, that that’s really where you’ve seen a lot of is when a customer can quickly identify time to value, couple of months typically speaking?

Jonathan Ross: So if you’re working with say, one of the large GPU providers, it’s going to take a minimum of six months. Usually it gets closer to 12 months depending on how important they believe you are. And that’s a huge problem because your problem might be important to you, but it might not be as important to them. When you’re working with Groq, you can just use our compiler. It gets you most of the way there. We’ll answer questions. We’ve seen customers where they were highly valued by the GP maker, so they got their stuff done in six months by the GP vendor, but in our case, we were able to do it in 10 days and we were seven times faster, and then it continued to get faster after that. But being able to get a result that performant that quickly enabled them to move on, and they don’t have to depend on being considered important by their vendor in order to make progress.

Daniel Newman: So Jonathan, you’ve alluded to throughout our conversation a number of different tools, things that Groq has done to make it simpler for companies to work with you. Talk a little bit about what the developer kit and all the developer tools that Groq are building. What what’s out there?

Jonathan Ross: So there are really three tools. The first is GroqFlow. With GroqFlow, you literally just add one line of code to your TensorFlow or PYTorch model and it will compile for our chip. You don’t have to do anything. We convert the numerics. We do everything for you. Number two is actually a GroqView, which is our debugger. Now, if you’ve ever debugged on a CPU or good luck on a GPU, mostly people just give up and they won’t use something like GDB and they’ll just print FD bug because it’s easier than using the tools. In ours, we actually have a visualization that will show the data moving throughout the chip, and so you can see exactly what’s going on a cycle by cycle basis and step through. And then number three is for performance benchmarking, there is no tool.

When you compile for our chip, we just tell you what the performance is. You actually don’t have to run anything. You don’t have to do anything. Normally you have to run something a thousand times to take the average. We just tell you that this program’s going to run for this many nanoseconds and it’s plus or minus zero. There’s no variation. Now, in addition to that, we also are putting 60 models up on GitHub, which all compile on our chip. These are common models that are used throughout the industry, and we’re going to keep adding to that as great references that you can use to get things going. But they cover all sorts of different use cases from image classification to natural language processing, to you name it. They cover the whole gamut of what machine learning can do.

Daniel Newman: It sounds to me like comfort and complacency with technology has become a bit of an overstayed asset within organizations. They spend a lot of time, they overstay their welcome and then they don’t disrupt themselves. They don’t change, they don’t take that permission to do something that could create, if I’m not mistaken, orders of magnitude of improvement in removing the uncertainty of bringing down costs and improving the performance in their businesses. To me, you bring a really big point, but also to me, it feeds that whole thesis that you need license to innovate, you need license to change, and technologies that are capable of it.

Jonathan Ross: So one of the things that we did very early on at Groq was we decided our mission is to drive the cost of compute to zero.

Daniel Newman: And?

Jonathan Ross: Well, everyone hated it. They hated it. They’re like, we’re a business. We’re trying to make money. You can’t drive the cost of compute to zero, you’re going to make us bankrupt. But if you look at the history of compute, that’s what’s happened. And when we say drive the cost of compute to zero, what we mean is we sell our chip for the same amount as NVIDIA or AMD or Intel. We don’t charge less per chip, but in these examples where we’re giving 200X, 600X, 1000X the performance, we’re giving you 200, 600, 1000 times the performance per dollar. And so it’s approaching free.

Daniel Newman: So it could be a thousand times more or it could be a thousand times less?

Jonathan Ross: And the first thing that the group that does pricing for your company will do is they’ll say, you know what, this is insane. You need to charge a thousand times as much. But the realization is every time you make compute cheaper… And this has been true for 50 to 60 years, you make it a thousand times cheaper. People buy a hundred thousand times as much of it. So it’s a hundred times as much money to make. It’s called Jevons Paradox. It was discovered in the 1850s.

Daniel Newman: So from my experience, and it sounds like from your experience, change is one of the biggest challenges for organizations to make those big, important quantum type leaps forward. And as you think about that, Jonathan, you have these customers though that have taken the risk. They are partnering, they’re going big, they’re taking chances. I’m just interested a little bit, are there some common traits in those customers that you’ve found to be your best and most interesting early adopters?

Jonathan Ross: Well, yeah. They tend to be innovators. They tend to be early adopters. That tends to be the thing that ties them together. And the reason is often they feel that they can’t just take it easy. They’re in a competitive market, they’re a leader in their competitive market, and they’re not willing to just stick with status quo just because it’s easy. They actually want to do something that gives them an advantage.

Daniel Newman: All right, so here you are, some of the biggest laboratories in the world, some of the biggest financial institutions, some of the most important players in the healthcare space, relying on Groq technology. Where do you take the business? Where does it go from here?

Jonathan Ross: So the future of Groq is that someday we do want to provide all the compute for humanity. So when I see a TV show and it takes place 20, 30, 40 years in the future, I’m always thinking they’re actually using Groq chips, right? That’s what the future is. And what’s so different about our chips is the seventies you used to see these things with tape drives, right? And when we look at the way that chips work today and how unpredictable they are in their performance, it’s kind of like having a tape drive. It’s really old concept. In 20, 30, 40 years, no one’s going to put up with having a chip where you don’t know how long a program runs for. Especially when you’ve got multiple chips working together, that’s just going to seem weird and antiquated, like tape drives. So 20, 30, 40 years, I think the majority of compute will actually be running on Groq chips.