Name: From Idea to Production: How Tabby Ships AI in the Real World
Uploaded: 2026-05-06T13:46:09.908Z
Duration: 1 h 21 min 8 s
Description: From Idea to Production: How Tabby Ships AI in the Real World

Transcript for "From Idea to Production: How Tabby Ships AI in the Real World": Hi, everybody. So good to see everybody who has joined. I'm Nadia from JetBrains. I'm the head of business development for JetBrains in MENA. Thank you for joining us today at this exciting fireside chat with Tabby. For those who don't know, everybody here knows Tabby. But for those who don't know, JetBrains, we are the company behind such, leading products as IntelliJ, PyCharm, Rytr, and, we are incredibly delightful to hold this event today with our long term friend, with our client, Tabby, to share all things Adjantic AI in software development. Very quickly, JetBrains has been in MENA Region based in Dubai since last year, since 2025. We have a team on the ground with me, with Ibrahim, who is gonna be joining Tabby today in about two seconds, also with our amazing marketing manager, Reedy, who has organized an event. So thank you, Reedy, and also Nicholas and Andre from the sales side. With that said, I'm not gonna take any more of your time. We're gonna jump into the meat of this meeting, to the most exciting side. So please welcome Denise, the head of AI agent platform, the engineering manager at Tabby, and my colleague, Ibrahim, the solution engineer for JetBrains MENA based with us in Dubai. Hi, guys. So great to see you. Hello. Such a nice introduction. Hi, Nadia. Thank you for the introduction. Awesome. Okay. So, I'll take it from here. Thank you guys for, everyone that is joining. Big shout outs to the marketing team, Riddhi, and everyone at JetBrains that made this possible to happen. It's truly very exciting. I just wanna mention, I love how even though this is virtual, the the fact that I just pressed on stage, I I really do feel like I'm on stage in front of everyone. So, this is gonna be exciting. I'll just do a quick intro about myself before we jump into this. My name is Ibrahim Amran. I am heading all the solutions operations here in The Middle East for JetBrains. Most of the products I am handling right now are, a lot to do with IDE services, AI agents, AI assistants, and, some upcoming products that we have as well. I'm very excited about today's conversation that I'm gonna be having with, Dennis. And, before I jump into it, maybe, Dennis, if you'd like to introduce yourself. Yeah. Thank you. And I'm Denis Zaknoff. I'm I'm head of, AI platform in Taabe. So I, do agents. I implemented, in Taabe, and I participate in part of, agentic transformation for Tabby. So we are living in such, wonderful times where we can do a lot of things with AI, with AI agents. We can, have a lot of value implementing them in operations, in software development cycle, and, even doing new customer facing products based on AI. So we do a lot of all of this. And, actually, I'm very excited to share, opinion, and experience with you. Amazing. I'm actually very excited to hear all about it, Dennis. I know you and I have spoken, earlier briefly, but, you know, we didn't really get to dive deeper into, you know, this topic that I'm about to open up with you. So I'm really looking forward to it, especially as a Tabby user myself. You know, sometimes, I'm a too much of a user, let's say. And they get the best of me, but that just goes to say how awesome your platform is. And, without further ado, I think, it's time for us to jump straight into it. So maybe I'll go ahead and start with, like, the first topics that I wanna go ahead with and, ask you some questions. What do you think about that? Yes. But if you said that, I also mentioned, I'm using, JetBrains products from I I don't think, I think, from the beginning of my career and maybe even earlier. So I'm a fan of your team too. So Amazing. Alright. Well, I'm glad that we we brought each other together on this. Before I jump into it, just real quick, I wanna mention to everyone that's watching that there is a q and a section. Please feel free to ask questions that will be directed towards myself or towards Dennis, and we'll be we'll try our best to answer them, on the on the line over here with you guys. So without further ado, let's go ahead and get started with it. I wanna start by talking about how AI is helping companies today. Right? Everyone nowadays, it's the hot topic. Everyone is coming up with some use case for AI. Some are overly ambitious, and some are actually very obvious and not being utilized as much. So I wanna talk about, you know, where AI can help today, when it has to do with operations, effectiveness, and new customer AI customer facing agents. Can you tell me a little bit about that, you know, when it comes to operations? Yeah. So it can help it can help in a lot of ways, actually. I think the most, direct way is automating of your operations, because, if, there is a lot of, things, their operations, for for example, for fintech, operations are a big cost center. And AI can speed up, some repetitive work, for example. And, this work definitely takes a lot of efforts from and AI can be easily implemented, on operations and, to and reduce some cost of it. So I think, it's, like, the most obvious way to start, with operations. But, if you dig deeper, AI also can, transform all your products and all your software development life cycle. If you implement, AI not only obvious way in software development and coding, but also in product research, in so in design and so on, and, modify your company into one knowledge base. It can increase speed of your new products, implementation of, some of your ideas several times. So it also very, it it can bring a lot of value this way too. Yes. So the main things, is, you start with operations as a an oblivious way, and, then you can modify all your company, like, doing a lot of things faster, after that. And, also, you can use, new, you can use your AI, and, create, new products based only on AI. For example, I don't know, agents, helping you with your finances and so on. So there are a lot of ideas. So how can you implement AI centered products? So I think these three ways. Operations, AI, facing, prod AI customer facing products, and, software development life cycle. Nice. I like I like how you frame that. So it's I would say, this is more of, like, a bottom up approach, and just start with the simple things before you build your way up. You know? And, don't overcomplicate it when it comes to actually, you know, utilizing AI as a tool to actually optimize your process rather than replace it. Yeah. Like, And I other thing I'm sorry. Like, every other, yes. thing. in product development, you start with small and you scale it, scale it, make it more complicated, and, take more value with it. Awesome. Awesome. I like I like that you mentioned a few things, actually. So, like, you know, development effectiveness and, I think you said AI customer facing agents and also, like, you know, new AI customer facing agents and operations. I'm curious. In in your experience, which of these delivers, the fastest ROI, and why do companies sometimes start with the wrong ones? Yeah. I as I said, operations are the fastest because, if you have, if you have, a lot of operations, if you have if your company is a customer facing company, you have a lot of operations. For example, for fintech, it's anti fraud, KYC, a lot of customer support, a lot of things, going behind of this customer support, and a lot of repetitive work. And if you have a lot of repetitive work, it's a good candidate to, you know, automate it. And, AI agents are a good tool to up to automate and also to have a brain behind this automation because, LLM is like a brain for these, tools, for automations. And I think it can be more it can be not so, but easily implemented. It can be evaluated, so you can calculate, how effective does it work, if, it, does the work properly. And you can also measure the impact quite easily quite an easy way. So it's like the main candidate to to start this, and it will be the fastest way. But when you dig deeper, it's, of course, software development life cycle because it can, modify your, it can change your team, and you can deliver much, much faster things into production. Okay. So, essentially, what you're saying circles back to what you mentioned at the as the first answer, which is start, first of all, with the operations. But, eventually, your main value proposition will evolve into something that, goes towards the software development life cycle. So, it is something that you would say, for example, will contribute, inevitably to, you know, the live product or the finished product, usually. But at first, it is definitely if you want to get the fastest ROI and, you know, you wanna get the fastest results as an organization that is trying to adapt AI, the best approach would be to start from somewhere like operations. Yes. Yes. Check where you can increase speed of operations, help your customer support agents to, do more job to, create faster and, more complex, things and so on. Answer. questions complex. It's an interesting take that you're given giving, Dennis, because, you know, as a head of AI like yourself, I'm sure you're more, immersed into the engineering department. And so when you wanna make internal changes, like, for example, changing something that has to do with the customer so customer support and so on, then you'll definitely have to have some conversations with some business executives in the organization and have to talk them into that. And that does lead me to my next question. How do you help a skeptical executive distinguish between AI driven efficiency that truly changes unit economics and revenue, versus automation that just shifts work around. Yeah. You you told a very interesting thing. Yes. It's as, an engineer myself, I but implementing this agents, I need to, dig deeper into operations, into business stuff, and so on. So it's like emerging of different but not so different, of course, words, but it's it's an interesting way, in this aspect. But, actually, as you you you asked, about, skeptical, stakeholders. But in my, experience, stakeholders are not so skeptical. We're eager to use AI, in different ways and the problem, often is not to not to, you know, to tell him to use this AI, but, to tell him to tell them how to use it properly, how to bring value with it. But, because, the the more problem I think then stakeholder is it's is not so skeptical with AI, but, he's too eager to use it. And, the stakeholder wants to implement it right now because it worked on couple of examples. For example, he tried it. He tried, I don't know, Chargept. It, answers is very good quality to him and, with I don't know. He tried three or four examples. And after that, he, made makes a decision. Unless to implement it in production, I have, 1,000 requests per minute. Let's do it this way. You know, when you do this, you need to evaluate it very carefully. So, the main challenge is, to evaluate it and to to, bring, this knowledge how to evaluate these agents and to stakeholders too. Yes. Okay. Noted. Yeah. Actually, you know what? Now that I think about it, I I kind of agree with you. A lot of the times, I do find myself having to contain the ambitions of the executive rather than try to convince them to use AI. Because, like you mentioned, you know, when you're not on the engineering side and you're just you know, your objective is I need. to get to business, I need to get things to production. And Yeah. And you see a foreign concept like that, and it's like, oh, it's it's generating new results. Ship it to production right now. Let's, let's get it to, you know, tell tell the prompt to make me money right now. And, Yeah. Yeah. Yeah. I understand I understand how it can be a challenge to, you know, try to contain that excitement of, of executives. So. you would. say. the best next step with that would be to get them to evaluate further on how actually this would apply to their business case? Yes. Because, usually, this this approach that I saw a cool tool, I tried it couple of times. Let's strip it. It doesn't work. What works is, more systematic approach. Collect the right your use cases, prioritize your business value, how to what do you do, how do you do it, and, test it in real processes and implement it not on the whole, flow what you have, but, one by one. One use case by another. Measure the impact. every time. Measure it even, it's in production for, a decade because something can change and, the agent worked, yesterday can be broken because you introduced new fields in your data and so on. Because, it will then hallucinate, and you need a lot of things to make it right. No. Yeah. Actually, I couldn't agree more, and, that's why I'm thankful for folks like you in the world that can, you know, back these statements up when I actually try to do the same, with a lot of the people I work with. And they're just like, hey. Can I can I just, you know, do everything right now? But I'm curious. Where do you see the biggest gap today between where companies think where AI will help them and where it actually does? Yes. It's, it's, the same couple of questions couple of examples. You you see, it's a gap between the demo, which works, and a real production which, doesn't work at all because, it has a lot of corner cases, a lot of tools to implement in your agent. And, this step from demo to production is a really complex and maybe underappreciated thing. Okay. Yeah. No. That, I I couldn't agree more with that as well. So you have to basically, to sum up everything that you're saying, will evolve around, you know, just focusing on, you know, the small wins and to increase ROI, the fastest you would go for operations. And also when you're trying to take something to production, make sure you evaluate it as much as possible, before just fully relying on AI. While AI is very beautiful, at the end of the day, it is still, you know, not as, overly ambitious as people make it seem, and that's why we should actually treat it with such, delicacy, which I completely agree with. And I have, so many real life use cases and examples that I can, you know, come up with from the top of my head that align with exactly what you're saying. But you're mentioning, you know, basically taking something from test environment or, you know, while testing it out to production. So this brings me to my next question regarding, you know, like, AI agent life cycle. So, like, you know, the stages that you would go from an idea to production. Can you walk me through that life cycle, like, in-depth? Like, what does done look like at each stage? How do teams know when to move from prototyping to production? Yeah. I think it's, quite similar with the standard, software development production life cycle, actually. First of all, you need to, you need to check what what you have and, you need to think what you need to to do, what you need, to achieve with an agent in your system. You need to understand, what should agent do and with with what tools can, it do it. And this leads us to the first step to, prepare your internal infrastructure to the agent. Because, for example, if you have people and, have a lot of internal knowledge what is inside their heads and, not on paper, not, on, some documentation, they use some tools which can work, can, be, also broken. You need to systematize all of that to prepare your tools to be able to answer, to your agent, to have API, for it, you need to, systematize your knowledge base, and check is the it does agent even can agent even work in this, environment? Do you have enough tools and enough documentation to prepare your agent? After that, you need examples. It's enough. Several examples are enough on this stage, actually, just to create your first version. And second, and and and first step is to, create an agent using these examples, to build your agent for prompt it. But, actually, this stage looks like, the main stage of building React agent, but it's not. Because I think what comes next is the main stage. You need to prepare, your evaluation dataset. It should contain, all the examples your agent will face, in your product environment production. And after that, you need to run your agent on this dataset in your sandbox. You need to check the results of your agent, and you need to check errors. And your agent won't work for the from the first time. You need to fix it, add new tools, and run it again and again and again. And this evaluation is what what is it? It's a result of your agent. It's the sample result, which you have in production and some way to compare, does agent have a good answer or not. For example, we have anti fraud a anti fraud AI agent helping human agents to check is it fraud or not. It has a structured result. It can ask for, is it fraud, Who is responsible for fraud? Or is it not fraud? And who is not responsible? And, you can check, the results, compare it with the golden dataset, the right answers, and say, overall, statistically, are your agent working good or not? But also you can implement your unstructured agent answering agent answering with some text And how to check, did it right answer or did it wrong transfer? You need to also implement a way how to judge it. So you implement also second agent judging first agent, LLM judge. And also check the result of it. Also, there is a different topic, how to judge a judge. It's an interesting one. And you have this cycle. You have, ability to evaluate your agent and the results, and also you can return and fix it. You do it several times. And after some time, statistics shows you it's okay for production. You roll it out using a beta test, a b AB testing, of course, and, then check results production, roll out to prod, and checking checking the metrics every day because, everything can be broken after some release. So it's, almost as a product development cycle. Yeah. So, basically, just like any other, you know, feature or technology that you would like to implement into your solution or your platform, pretty much, you know, build it and then make sure you go through the unit testing. And, essentially, thank you for confirming to me, Dennis, that AI isn't just magic. It's also software. Yes. Yes. And then also it also needs to go through the essential the same process. One interesting point that you're mentioning to me right now that I've actually never heard about before is, actually having an AI agent that will judge the AI agent that you are testing out. That's that's an interesting one. Can I ask you about that for a second? If you could summarize it very quick. This is genuinely, Yes. like, Of course. interest. Yeah. Yeah. So yeah. You can have a some structured data, for example, as an anti fraud. And you, maybe you don't need any judge because it's, it's a deterministic way, was answer right or not. And you can can have statistics how precision and recall of this agent, for example, if it's classification. But if you have an agent doing something, or, answering, making some answer, some text, you don't know how to evaluate it because the text can be right. It's, it's a lot of ways how to write, the same thing differently, for example. And there are some metrics how to evaluate it. But, the most, the most right and, basically, near to production way is to test it with other agent comparing one result with the golden result and saying was the result, same or not. But it's simplicity because same or not is also not, the exact answer. It can it can use other tools and other artifacts, which an agent had in had, when it works. So, with LLMA with second LLM judge is just an agent. But its input is, like, golden dataset and answer of your agent. And the result is, was it right or not? Nice. Okay. Must be convenient to have, you know, an AI agent on your side that's not biased or attached to the feature that they built and will actually be honest with you. Because I'm sure that some software developers sometimes do have that attachment to the to the feature that they. built and to try to push it to irrespectively of, you know, how well. it works. But I'm curious it it come sorry. Go ahead. sorry. But in its order, who will judge, a judge, who will judge an LN's judge's answer? You need third agent to to judge yours. So your, LN judge, you know. It's. just. a joke. You don't need it. There there are ways how to judge, your LN judges also, but they're quite. But. I I bet I bet right now at this point, I mean, if we're there, we might as well. Right? There there's gonna there's coming a time where, you know, even, like, I was talking to someone the other day, and they were mentioning how their whole company is based on, using AI to clean datasets. And he's telling me that we did it so well, we're actually going out of business. Because we clay we got to the point where right now, we're getting the AI to tell the AI to clean the dataset. And so we put ourself out of business because, you know, an AI is telling an AI what to do right now. So, like, the other element is telling the other element how to clean a dataset. And the the whole point behind the the business defeated. its own purpose, but, good luck to them. But yeah. We're we're on this point. When agents, are using agents and, yes, multi agent systems are a thing right now. Yeah. Absolutely. And, I think we're making the best out of it, hopefully. But with that being said, you know, I'm sure that there comes times where you reach a point whether you want to decide whether, you know, the AI that you have, the AI agents that you are embedding in whichever part of your business is worth continuing to work on or not. And I'm curious. Like, at what point in the life cycle should a company decide to kill an agent project rather than to continue investing. Yeah. Yeah. I see. I think at every point, the company can decide this because it's all about value. If your potential agent brings a lot of value, you can invest a lot, even if something goes wrong. You can, you you can, do, you can fix your infrastructure to for your agent to be able to work inside of it. You can do a lot of iterations of evaluation if, you can't, manage your agent to, do appropriate answer. You can even, you you can even fine tune the LMs if you need, it for your agent. It depends on your budget and the value, which your agent will bring. If your agent will bring not so much value, you don't need to do it because only for the brand, already for doing a agent just just to be AI, it's not, it's not worth it. Okay. So, basically, there there isn't a right or wrong answer there, and it's just essentially it depends on your objectives and how much value you think it'll bring. It's it's always about money. You know? It's always about money. It's always about money. Yeah. I guess, like, you don't have to be an engineer in your business to figure out whether you want to implement AI or not, in in your part of the business. Right? Which actually, they'll take me to the next topic. You know, let's say that, I'm Ibrahim, and I'm a, how about we say, you know, sales specialist. And I'm working on, you know, like, implementing different technologies for my company when it comes to sales. And I identified an area in this part where that I can automate. You know? I I can actually oh, okay. Wait. What if I can get an AI agent to do this for me? In a situation like this, like, I I wanna talk about who should build agents. You know? Do I have to be a developer, or or is it okay, for example, as a nontechnical domain expert for me to, to to to set something up on my own? Like, I'm sure you've seen both models, developer led and domain expert led. What are the pros and cons, and who do you think should, you know, do this as as someone that can be a software developer and is a software developer? That's a really good question, actually, because, this paradigm is changing very fast. Not from long ago, I would say developers needs to be a part of it, at least, at least a part of it and very important parts. From now now, I'm not so sure. So developers are often, so engineering guys are often stronger in system thinking, how to build an architecture of an agent, make it reliable, and so on. And so for very complex agents, it's better. But, the must have, think is the domain knowledge. So the guys who knows what, what an agent should do, are very needed for this, implementation of an agent. Yeah. For to be to be able to implement sales agents, you need a salesperson, very good one, because, the salesperson need to teach teach your agent, basically. So you need both of things, but, now there are a lot of instruments to be able to build such agents without coding. And maybe even for if you are talking about simple agents, maybe without even some technical knowledge. Especially when you can, ask your coding agent, code code, or something like that to build, an agent and to test it even, if you have, enough, examples, of course, and if you have your evaluation dataset as answers. So I think domain knowledge is more important right now than, this, engineering knowledge for building an agent. What engineered, engineers, should do is build a good enough platform for non tech people to understand, an agent and to be able to, evaluate it, to check its results and so on. Maybe I'll I'll show you our platform, actually. Wait a minute. Where is it? So we have a lot of, can you see? I'll I'll do. this way. Yep. I can see. Yeah. We have a lot of aspects, one of our main agents. And, you see you you see a lot of iterations of these agents. I showed you a lot of iterations of an agent. I can show you an agent and interface how to build it. And we wouldn't be able to build to iterate with such speeds without, deep domain knowledge using all the engineers, for example. So it's, like, very important to be able for your node tech team to be able to participate in agent tool. Yes. And some interface from prompting and so on. Undeterministic agents agents deterministic tools. We are not so user friendly yet, but, where but non tech people are already able to use it. So yeah. Wow. This is this is such a cool deep dive. So long story short, what you're saying is that, you still do need to be a software developer to build these agents. But whether you have to be an expert in that domain or not, right now is not very, like, necessary. It is preferred to be an expert in that domain, but also as an engineer, I can still build an agent, within a domain that I'm not an expert in and be able to succeed on it. Am I getting this right? And then also, essentially, how we should the direction we should go towards is we should actually build a platform for nontechnical people and non that are only domain experts to be able to build these AI agents and test them the way they want to. I think we need to be able to build a platform for not only tech people to be able to build agents in their domains, Yeah. like transferring sales guy, domain knowledge into into an agent directly without that guy, lurking around, you know. Yeah. I, I agree. I mean, you know what? As a as a software developer myself, I also always would find that sometimes there's people in different domains that have nothing to do with tech. They have brilliant ideas. And, you know, they, they, they find they get frustrated trying to bring them to life because they don't have the technical background, but they have the industry background. They have the field background. And they're always saying, like, if I could just automate this workflow or if I could just bring this to life, I bet that I'll be able to help a very big chunk of people that struggle with this very specific use case like me every day. So, yeah, I think it's very interesting what you just, showed me, very eye opening. And, I I I hope we do take that that direction for sure so that everyone will be able to enjoy, the benefits of building an AI agent and not just us engineers. So, I'm curious about something right now, since we're talking about, you know, nontechnical people or, you know, low codes, no codes, five coders. They're maturing fast also. So even those people that we're talking about that are, you know, that have these amazing ideas, They do sometimes attempt to execute them. And while I see a lot of examples of a lot of them succeeding, a lot of them also do fail. You know, the nontechnical people, the the Vibe coders, let's say. You know, the no code and low code agent builders, platforms. So a lot of people are using those, the ones you're mentioning. Like, you know, at what point do they become sufficient for production level agents? Like, what is the ceiling? Yeah. I think, the main answer is evaluation again. I'm talking I I know I'm talking a lot of about evaluation, but, actually, the difference with the ceiling is to, be able to gather a representative dataset to be able to check the result agent gets on this dataset. And, if you have that, and if if you would check it in production time to time with, by Chrome or something. It's a good agent, I think. So the main thing is evaluation again. Okay. Fair. I mean, you know, let's say, for example, someone comes up to me, with an with a with an agent, and it looks impressive in a demo, but it actually can't be maintained by people that use them. How would I avoid, that trap? Is is requirements gathering a key over here? So you're talking about then, you built an agent and it works fine, but you can't maintain it, It looks impressive. in? a demo. Uh-huh. Yeah. Like, it got demoed to me, and it looks impressive. But, eventually, like, you know, it ends up not being useful for the people that actually need to use that agent. How do I avoid that? Is it also evaluation I just have to keep evaluating as well? Yeah. I think evaluation is also an answer, but, also, your agent need to be simple. You need to keep things simple. Same in in code. You need to keep things simple. If you built an agent, using, I don't know, 100 sub agents and so on, your agent doing something primitive, don't do it because it you wouldn't be able to maintain it, of course. So an agent needs to be, simple and transparent also. I think, it will help you to maintain it. So, you need to. to you need to be able to see what it's, do. It's it's, your agent needs to be, well structured and, yes. I think that's. an answer. Okay. So when it comes to simplicity and, you know, like, adopting, let's say, let's let's switch it up a bit. I wanna I wanna talk a little bit about, like, you know, as an organization, how we can adopt AI agents easier. What would you say is, like, the minimum, viable infrastructure a company needs before building their first production agent? What do, like, what did and and also what do teams, overbuild, too early? I think, you so you as a company, for example, if you are talking about operations, if you your company have some infrastructure already. Your company have, interfaces. Your human agents are looking for things up, looking for, I don't know, his history, transactions, looking for, tickets in Jira or something like that. And you need to, transfer these tools into tools for the agent. They need to be accessible by API, they need to be stable, and, they need to be without, garbage data. It's, they need to return only data your agent really needs because, otherwise, it will be, your agent will run out of context, or it will hallucinate and so on. So it's very important to keep your data, for your agents, quite, strict strict and only needed, data you need. Same if your agents does some actions. Also, you need to prepare your tools, to get result to get answer from your agents, to implement human, in the loop somewhere if you your human needs to evaluate answer of your agent in in production. And you need to prepare tools. The main thing is these tools. Second thing is, where your agent will be inserted, what it will be having as input and as output. And first thing is, again, a way to create evaluations. So these are, the basic thing you need, to be able to implement an agent. You said about overbuilding. I think, you can, I think the main thing you need to build your simple agent to insert it in production and see how it works? It's hard, not to do everything, from first time. Very complex agent doing everything and not to test any parts in production. Don't do that. Fair. So, essentially, we're going back to the to the word of the day, I want to say, which is evaluate. Evaluate and. evaluate. No. That's that's super fair. And, I absolutely agree. I feel like sometimes people do release some features into production way before they evaluate, and it ends up harming them so much more than, benefiting them. I'm just gonna take a quick pause over here and, just check if, you know, we have some, questions and, see, you know, maybe if, you and I can, try to answer them. We do have, some lovely spectators today, and they do have some interesting questions. I think you can see them as well, Dennis. Mhmm. So I'll just go ahead and see this one because it kind of does have to do with evaluation, and I actually do like it. I'm just gonna share over here, and it's from our friend Kirill Galkan. I hope I pronounced that correctly. So since AI evaluation is much more complex than traditional unit testing, how do you balance stakeholder pressure for speed with the need of for a robust evaluation framework? What is the lean way to do this for an MVP? What a great question, really. Yes. Absolutely. Excellent. Yes. I think, evaluation is, it's, yes. I think it's more complex than unit testing. It has a lot of com a lot, common in, machine learn with machine learning, actually. When you do when you do machine learning model, for example, if it's classification with classical machine learning, you also train your model, have a testing dataset, have have learned dataset, train dataset, testing dataset, and to test it, and, not to to be sure, not, what your model is capable of. And, a a agent evaluation is common with that. You also have a dataset, like, test dataset. You run it, you run your AI agent as a model on it and check the result, precision recall, and so on. You if, your stakeholder pressures too much on this, you can you can, split your things, split your agent into several parts. And to have the most primitive part bringing, at least, at least part of the value on the table and to do it first. With splitting of your agent, for for example, by topics or something like that, some fields, clusters, where your agent will be applicable. I think it's the only way to speed things up. But, if, even, this isn't enough, the only answer is, if this agent will be in production, it will ruin your some part of what you want to automate automatize. Okay. Awesome. I hope that answers the question for, Kirill. I I answered it for me. That's for sure. That was a very interesting and, complex one. I do have another interesting question over here. It's very simple, but I'm actually curious about it too, so I'm gonna share it. And it's by Hassan Al Zane. What AI agent frameworks do you use? Oh, it's a great one too. So, I showed you, part of our agent platform. In heart of it, is, Google edk. It's a framework from Google, for agents building. So we use it as a part of this interface and so so on with our with a lot of our differences, is it? But, it's, and, it's an agent of our platform. So if you heard of it, it's pretty great, actually. It's it, is becoming, even greater. And for for evaluation, actually, we use our own frameworks. Okay. So you guys are actually evaluating, you know, basically, when it comes to anything that you want to add. You're actually using your own framework to evaluate the AI agents? Yes. Yes. Exactly. Wow. Actually, when, for example, we are talking about LLM agents, we are using, the same platform to build LLM judges, for example, and to use them to evaluate. You know? Yeah. It's, if you have one platform, you can do all the agents here. Yeah. It's? like an inception of agents. Okay. Yes. I have another one that I'm finding interesting also. And, again, you guys are, flooded us with some beautiful questions. I'm just trying to pick out, which ones are just, you know, have more votes. So I'm just gonna go ahead with this one by m m Rudula Rahati. What degree of autonomy should AI agents have in client facing products? It really depends on your product, I think, because, in some ways, for example, we are when we are talking about, again again, about anti fraud or something, very, strict things, you always, consider your agent only as a tool for your human agent, for your, specialist facing the client. In in, in this cases, this is the only way because, you can't do wrong with it. But, they will increase your speed of answering clients a lot. In other cases, it's okay to answer on of on part of the questions from clients, directly. And, but in these cases, you need, a proper way to be able to escalate to humans and to have, a very capable human team, which will also, check your, you check your dialogues with, clients. I that was an awesome answer. I do have another question that came from the chat that I feel is very interesting. So I'm just gonna go ahead and read it out loud if that's okay. I'll actually try to display it over here for you to be able to, for me to be able to share it. So I'll just go ahead and do that. So given that MCP servers are becoming used more and more, security question is often being neglected. Does Tabby have MCP servers for internal tools, like for Grafana, Jira, etcetera? And if yes, how do you guys secure it? Or is it a matter of zTNA and managed based on the employees' device network settings? Or it's based on employees' device work settings, you said? Yeah. So it's not see it on the screen. question on the screen yet. Yeah. It's a really great one. Yes. We're not we are we don't have MCP servers for external tools. I mean, like, like Jira, for example. We are we have, MCP, and, we are letting, our agents to use only internal tools for this. If there's a question about this. And, about, if we are talking about, security settings, you have, you can have, a security no. You you can have, a client, like MCP client. You you can have, some service user for your agent, and it can be different for the different agents you have. And you have all the excesses or for this user, in your tools you are using. So system users is the answer how to use agents inside, some services, that need to be authenticated. Okay. Nice. So pretty much keeping it simple. Long story short. Simple. Simple. Simple. Yes. Yes. Keep it simple. Yes. But, also, about security is, great question also in different ways because, as a fintech, for example, we need to use our local LLMs. So we don't use, cloud. We don't use, anything. We have our servers our servers, and we are running, LLMs on these servers inside Tabby, for example. So it's also for the same purpose of security and effectiveness action. So you guys are training your own models? We are not, no. We are not training, our own models. We have open source models running on our, own servers. And open source models right now are, very good, actually. They're quite concurrent with the top tier models. Mhmm. Very nice. I'll I'll keep that in mind if I wanna start a, a pay a pay buy now, pay later solution. But thank you for these answers. We're gonna take a break from the q and a, guys, and I just wanna jump back into the topics that you and I were discussing, Dennis, if that's okay with you. We're just, you know, enjoying the show so much, and I feel like, you have more to share regarding Tabby. So I do have a few topics I would like to cover. I know we are going overtime, everyone that's watching, but, you know, I'm just enjoying this conversation a lot. So for whoever feels the same way, including yourself, this, then please feel free to stay and enjoy the show. And I won't bother you too much, Dennis. I just have a few more questions that I wanna get out of the way. You know, back to evaluation. You were mentioning a lot today evaluation evaluation evaluation. And, also, I can see a lot of people are asking me questions on the q and a about evaluation and testing. My topic is going to evolve around, what stops you from delegating everything, to AI right now when it comes to, you know, like, if we're talking about testing and evaluating agents, how come you're not delegating everything to agents right now? If you built, you you mean to delegate them evaluation or to delegate, like, everything without, no. Sorry. really? To delegate the evaluation, like, what do we need to properly evaluate? Mhmm. Yeah. Because, if you just, do the agent, for example, I don't know. It can do every operation work. Without evaluation, it will just it it will answer wrong, because, it doesn't because, when you build an agent, you don't know, from the beginning every aspect. And, there are a lot of common cases in every, in everything. And there when humans solving something, they're using, tools, they're thinking, and they can judge, actually, a lot better than Netherlands even even right now. And Netherlands often providing, right away solutions, which can be wrong because of these corner cases. So, you can't just implement something and to run it because, it will it will definitely, go wrong if you have a lot of requests and a lot of usages of this tool. You can do this in your in your private work, or example, if you're, doing something for to automate your your private life. But you can't do this in pro in, company production, for example, because when especially when fails, cost, you money or, client, or clients, or their trust. So, you need to evaluate this. And, also, to to be able to evaluate, you need, people to check, is your agent okay or not. You can't do it without people. Yep. Yeah. So you you do need that realistic and human element, before you can fully rely on AI agents even when it comes to evaluating. Fair. So then my next question would have to do with, like, you know, also being realistic when it comes to evaluation. What's a red flag in an evaluation setup that would tell you that the team is fooling themselves about their agents' real world performance? It's, when you try to evaluate but not on production dot data, but on or on something artificial data. Artificial inputs and outputs, maybe mocking some tools, which can be very it it can help you build an agent for for start, but you can't ship it without testing in prod. Got it. Okay. So, essentially, also, it just goes back to, being realistic, and using real world data instead of the data that you would just use to convince yourself that, okay. This AI agent is ready. Let's just listen to our eagerness and our ambitions and deploy, deploy, deploy, and, have to learn that lesson the hard way. Dennis, thank you so much for all the wonderful answers that you have been giving me. I'm just gonna have, a few minutes right now to, just one or two more q and a's, before we move forward. And, after that, we will be moving to the next stage of the webinar. So, I'm just gonna take a few seconds over here to look at some interesting questions that we are receiving right now. I see over here a question by Mohammed Al Amri. Any system architecture tips already used at Tabi? Microservices, modular, monolith. I'm assuming he means monolithic repositories. What would you have to respond quite that? used in Tabby, actually. Tabby is a very big company. Different teams can use different approaches, microservices. Yes. Of course, they are used in Tabby. Okay. Nice. That was a quick, one. even model monoliths are used. So in different teams, different approach. Ibrahim, can I also, I see, this question? It. was several times, I think. Which open source models, are being used? Yeah. I was just gonna share that as well, actually, because, I'm seeing. that people are asking. it. So, yep, please go ahead. Yeah. It's it's a it's a very interesting question too. The answer is, right now, for for for some of, cases, we use, Quen 3.5. And for some cases, we can use gamma. But it's like, it's like very interesting evolution of our approaches because we tested different models. And, the best model, we want to use, changes every maybe every month. Every couple of months at least. So we check it a lot. We test it a lot. It's all it's also a reason we need to evaluate a lot, because, the evolution of models of open source models, and not on the open source is amazing. We are becoming smarter. We are becoming way cheaper. It's very interesting. Nice. Okay. I mean, honestly, it sounds good that you guys are doing it every few months rather than every few weeks because I've had people, you know I think at the beginning, of the, AI but yeah. yeah. I was saying it's it's it's good that you don't have to do it every few weeks. I think I if I remember correctly, at the beginning, you know, when AI models were just coming out and a lot of updates were rolling. out, you would have to go through the change that frequently. But, yeah, I mean, every few months is, I think, acceptable considering the amount of work that it's helping you guys optimize and do. Awesome. I do have another interesting one here. And this also, I am interested in knowing the answer for myself, by Mbongini, Zwangobani. I'm sorry if I mispronounced that. What tools do you recommend, using for building in house agentic workflows with nontechnical teams? So maybe he has the ideas, and he can bring that platform to life. Yes. The first of all, you need to use some, some codec agent, of course. So I personally use, cold code, for example, for coding. Having this, you can implement any agent or agent, workflow builder or something like that. So, yeah, basically, you need to use some codec agent, of course. I use code code personally. Second, you need to use, something as core of your agentic system. We use Google ADK, and I'm, I I really happy how it evolved. It was, it was not very, very perfect from the beginning, but it's evolving very fast. I think, guys from Google are spending a lot of, efforts on it. So I recommend Google SDK also. It also have an interface already. You can check this. And, of course, you you can check, long chain and long graph, as part of your system as well. Nice. Okay. Well, and, Bongini, to the first to the gentleman, that answered this asked this question. Now you have the answer. I'm hoping that soon enough, nontechnical teams will be able to build in house agentic workflows, thanks to you. Yeah. Let's hope so. I mean, you know, I always see the the ambitious projects, and a lot of them end up coming to life. So I hope, I wish you all the best, with being able to build these awesome agents. Dennis, it's been an absolute pleasure. Thank you so much for sharing all these wonderful insights. I'm always so excited to see what Tabby has for the future. Once again, I'm a very good Tabby user. I actually even have the subscription Tabby plus where I get to pick the four categories and, you know, I'm using the Tabi card. So all your all your products are very cool. If you have anything to do with them, then I personally thank you for releasing such a great product out into the world. And I really appreciate you for, joining us in this conversation. Pleasure all was fine. Thank you very much, and thank you for good questions. Really, guys, you're the best. Thank you. Amazing. I look forward to hearing from you again, Dennis. And, you know, hopefully, we'll work together sometime again soon. Thank you very much. Awesome. Okay. So that was Dennis from Tabi, everyone. Again, thank you for everyone's attention until now. I really hope you guys are enjoying this webinar. Before I introduce the next speaker, and we do have another speaker, if you enjoyed this conversation and if you want to go even deeper into the tools behind teams like Tabby, we're hosting a virtual master class next week called IntelliJ Idea Unlocked, where a JetBrains developer advocate goes live to show you features that top engineering teams at leading organizations like Citibank, Google, Samsung, and x rely on every single day. And just to add on top of that, the first 50 attendees to join live will win a complimentary forty five day IntelliJ Ultimate license. The QR code is displayed on the screen right now, and the link is in the chat. Grab your spot, and let's get back into this session as we have one more conversation that you won't want to miss. I'll just give you guys a few seconds so you can scan that QR code. And, also, the link is shared if I'm not mistaken. It's going to be in the chat. Awesome. Okay. So coming up next, we have Cenk. Before we wrap things up, I want you folks to give it up for Cenk. He is an engineer also at Emirates MBD, a very awesome bank here in The UAE, and he has some awesome stuff to present to us today. Let's give it up for Cenk. Hi, Cenk. How are you? Hi, Ibrahim. Long time. I'm fine. How about you? I'm awesome. Always happy to see you, Zeng. I'm gonna go ahead and remove myself from the stage and give it to. you. It's all yours right now. Okay. Hi, everyone. Hi, Ibrahim Nadia again. Thanks with the the bringing us to to the stage. Today, let me share my screen first. Is it visible? Okay. So today, I have built one product for my, personal use, and I'm using the same tool at my office, because at Emirates Abide, we have, so many engineers. In my team, we have more than 100 engineers in multiple squads. So these two, actually, I am utilizing it, to track their GitHub contribution. And recently, all of you, we are using utilizing a lot of AI tools, and we would like to know which engineer is using which AI tool efficiently. So I created a simple presentation. Before that, let me introduce myself. I'm senior engineering engineer at Emirates MBD. This is my eighth year. Eight years, I'm working in this, Emirates MBDA. I'm mostly, supporting for the mobile applications. I'm sure most of you are already using the the MBDAX application, and we build also major applications. I'm also the founder of Mobile Guild Community. If you are interested in mobile engineering, almost every month we are organizing big event. You can, just send me a request and you can, invite you. So today, we will, shortly introduce what engineering plan is, is for and what how it is sent into the engineering fleets. So these two, I have designed for the engineering fleets who is, managing multiple engineers, and it's not specialized on any certain of, domain. It can be used for back end, front end, mobile, or any kind of engineering. So we have a challenges once, the teams are growing up. Once you have a lot of engineers, you need as a lead, you need to understand, how many engineers are working on, the product, how efficiently they are working, how efficiently they are utilizing their self and their tools. And Jira, GitHub, of course, they have an insight. We are not this tool is not reinventing the wheel. However, with given insights and with better interface and with also utilizing the AI, this tool is helping the engineering fleet how to utilize your Scott. So we have main features, AI utilization analytics, the contributor intelligence, PR bottleneck train writing, pin velocity, AI generated summaries, and delivery risk reduction. So before this two, personally, lots of, engineering did what we were doing. We were going to the terminals, GitHub contribution, insights. We were trying to fetch the statistics. We were trying to understand what's going on. But most of the job is manual and takes a lot of time. As engineering needs also, you are not sure what exactly you need to do. For example, you have a 100 plus engineers. One engineer is not doing to code review. What exactly you need to do? So it takes a lot of time for engineer lead to understand, to analyze, and to focus. So with with this tool, how we are utilizing, we are doing AI engineering score, usage efficiency, delivery impact, collaboration quality, code movement velocity, and PR quality. So before, introducing to the tool, how you can use the tool, either you can register your repo or you can register your engineers if you are, managing multiple scots. You can pick your engineers one by one because in the same scot, you might have multiple engineers who is working on, multiple repos in different technologies. This tool supports both, feature. For demo purpose, I choose the JetBrains Kotlin repo, which I can show you here. This is the most, popular repository from the JetBrains. As you can see, it has more than 50,000 stars, and so many contributors are working on this repo. So let me see if I'm audible. If you are ready, we can jump to the tool. Let me refresh my page. So the setup is very easy, actually. In in less than a few minutes, you can set up your tool. All you need to do is you need to authenticate, your GitHub user of or if your company allows you to use the admin, you can, authenticate the admin, and you can publicly deploy internally in your company. So these photos are the real contributors of the Kotlin repository. So we have a real time intelligence feature, AI detection engine, and the smart analytics. So it can detect the total stars, contributors, PR turnaround, and weekly commits of your repo. And each time, an admin or any engineer or any lead querying or using this tool, it has a MongoDB insight. So it even you don't have Internet connection or you need to check historical query, you can use your internal data. You don't need to each time go to the GitHub. So how I how I build this tool is, whatever I was doing before manually, I just train the AI. I put, this as the MD skills markdown file. So I kept it very minimum, very simple. So for total AI usage, I put some description. I the deduction rate for the commits, which engineering, how they are utilizing the AI, and I, created the confidence score. So this confidence score is calculated based on the keywords which is coming from the commits. And, I edit repository growth, commits by day. As you can see, Sunday is off. And, this is very, interesting area, especially for the leads, and it's public for any engineer in your team. So you can see most active contributor. You can see why there is a gap between the PRs. You can see who is doing the most code review. You can see the PR velocity, average PR 13, and you can see which, contributors are utilizing the AI less than the others. Because these are the most topics that I personally, follow-up in my company. So that is why I built this tool because I didn't want to go to the repository's tools and check what to do. I want tool to tell me what exactly I need to do. Now I'm also I'm adding two more features. I'm adding the Teams channel integration and Outlook integration. If I want to send appreciation appreciation to my engineers, I want this to to do it. Also, if I want to send warning, like, for the low activities low quality activities, I want to to send and, send email to the engineers and start the communication. So if you can use the team, Microsoft Teams or Slack MCP, you can easily achieve this. I have other products which I did. So last features, you can see your contributors and you can sort them so that you can see who is the, sending the most commits, least commits in a given time. There is a time range option also. You can see who is doing the most, line of code change, LLC, which engineer raised how many PR. So that is for, personally, our biggest problem. I really want to chase engineers who is not raising the PR or who is not doing the code review because code review is very important, thing for engineers to do. And as I said, you can connect multiple squads. You can connect multiple repos. Now I I will this is in my local. I haven't published yet. Next week, I'm planning to publish the tool, under our website, and then you can use it. It's under Apache license that it's free, and you can just easily integrate. So we can do create session if you want, guys. Awesome. Yes. It's the representation. like an amazing solution, Zank. And you know what? I think that this is just awesome to be followed up by my conversation with Dennis. It just goes to show, what he meant when he was saying that, you know what? When you want to take things into your own hands and implement some AI agents into your organization, start. with the operations, you know, and, build your way up from there. And, I really I really appreciate you sharing this, this tool with us, today, and I wish you the best of luck. I hope you guys are doing making the best out of it in your organization. And maybe one day we'll see it in ours and, in some different organizations as well. So, Yeah. thank. That's it. you for answering that. Yeah. In terms of q and a, we'll we will take just a few minutes to, to, to, you. know, see if anyone has any questions for you, and we'll see, if you would like to answer them. Actually, I don't see any questions at all. Okay. You know what? I have a question for you, Zeik. Just what led you to actually take matters into your own hands as an engineer at Emirates MBD? I'm sure that you have a very busy day to day, a busy nine to five. What made you just one day think to yourself, I'm gonna drop everything, and I'm gonna take this into my own hands, and I just want to implement this solution right now? Very good question. Actually, not only this tool, we are now building a lot of tools. And if you go to any enterprise, now a lot of engineers. I call it as a product engineering. And I think that is the new era. That is the new, future. So but the thing is now the building product, building dashboard is very evil. As long as you have the MCPs, you have the APIs, everybody can build, this kind of products very easily. But I want to be I want to differentiate this tool because as a lead, you are leading maybe 50 people, 100 people, dozens of people. Right? The problem is you come to your office. There is a lot of dashboards. What you need to do? So which dashboard you need to focus? Sometimes you are tired. Sometimes you forget to look at the dashboard. So how I want to utilize it? I want dashboard to tell the lead what to do exactly. I want dashboard to generate the tasks. For example, maybe you are leading your team differently. You can generate your own tasks. Ibrahim, what would Ibrahim do if he leads the 50 people in his team? For example, you would, send appreciation in a different way. You would send morning in a different way. You would, look at maybe different KPIs. So I think that could be the nice, future for the leads. Whatever you your daily tasks, you can utilize the AI tool to do on behalf of yourself. And then you can focus different classes. So that motivates you. That's that's awesome. And, honestly, Yeah. I think, it's it's it's very it's an honor to, you know, learn about a solution from someone such as yourself. I think we need more leaders like you in the world that that take matters into their own hands, and let's just make sure to make everyone's life easier, and more efficient. So, best of luck to you, and, thank you so much for sharing this tool. Everybody, give it up to Zhenk. You know, if you, enjoyed this conversation with him, feel free to connect with him on LinkedIn, and I'm sure he'll be more than happy to, you know, follow-up this conversation some more with you folks, you know, through some different, communication channels. So just one more thing before we wrap up. On my screen, as you can see, there's a short survey we'd like you to fill out, not just for our records, but to help us understand where you are in your journey and how we can actually support you better going forward. And because we genuinely want to make sure you walk away with something useful today, we're also giving away complimentary thirty day licenses for both JetBrains all product pack and JetBrains AI Ultimate to a number of today's attendees. Our team will be reaching out personally after the session, so keep an eye on your inbox. As you can see, the survey is shared right now on the screen. So while you guys scan that QR code and best of luck to everyone that could get some AI Ultimate and some old products box, I just wanna give a closing note right now, and I wanna say once again, thank you everyone for joining this webinar. You know, JetBrains has been around for twenty five years, and, you know, what we care about mostly and the way we like to position ourselves is developer experience organization. Everything we do in our tools is to make sure that the developer, the person behind the awesome code that is reaching many different applications in this world today, is having a good experience while building it. So I hope you guys had a good experience today learning about us more as an organization and learning that we have a presence right now in The Middle East, and also learning more about the wonderful tools that Tabby is bringing into this world and the wonderful things that Tabby is doing. I look forward to seeing a lot of you in next week's webinar. And until then, stay tuned on all of our channels, which is our website, our LinkedIn, and, you know, so many more channels that you could find online. And, actually, one last thing. Very big thank you to really, very big thank you to the marketing team here at JetBrains for this initiative, for bringing us out here. Huge round of applause on my end for a lovely event. And once again, my name is Ebrahim Imran. I'm a solutions engineer here at JetBrains. I'm the head of solutions right now heading all the operations. Please feel free to reach out to me on LinkedIn. And thank you once again, everyone, for joining today's webinar. Have a great day.