Name: Snowflake&Komodo Webinar
Uploaded: 2026-05-19T18:25:26.323Z
Duration: 1 h 7 min 17 s
Description: Snowflake&Komodo Webinar

Transcript for "Snowflake&Komodo Webinar": Great. It looks like we are live. Welcome, everybody, and thank you for attending. We have a a pretty solid audience today. I believe we're a little over a thousand registrants. So, super excited to to meet in the conversation. Quick introduction. I am Chris Reimer. I have a solution engineer with the stuff like team. I've been with stuff like for about three years now. Been almost twenty years now in the enterprise data management space, with a number of different organizations, worked up and down the analytics scale. So, familiar with a lot of the the growth and the evolution in the space as well as the challenges and the lessons we've learned over the last several years collectively. Super happy to introduce, Dinesh, who will be here participating and presenting majority of the content. So, Dinesh, I'll let him do a quick few words of intro, and then we'll, outline the agenda for this session today. Hey. Thanks, Chris, for setting up the stage. Hi, everyone. I'm Dineshwar Nuswami. I am a staff data architect in Komodo Health, where I've been working for the past, four point five years now. So in my role, I'm focusing mainly on building the scalable, secure, and intelligent data platforms that power product and partnerships, from foundation infrastructure all the way to, real world insights for delivery. So my work spans on sending data model, optimizing pipeline, and then structured and unstructured sources, how we can, efficiently handle the data from different providers. That is my core, expert area. I've been with DataWell for the past twenty four years. Yeah. And that's it all to you, Chris. Awesome. Thanks, Suresh. So quick outline of the agenda, the intention for the session today. And for our audience that's attending, you'll notice there's a q and a section, that is open for for posting your questions at any point during the conversation. We'll do our best to tackle those in some combination combination of of real time as well as a dedicated session, or respond towards the end of the session where we'll, we'll try and address those. Obviously, we may not have time to address everyone's individual question. You're welcome to reach out to Dinesh and myself, on LinkedIn. It's probably the easiest pathway. But Dinesh and I will also be presenting, a very similar presentation at, at the Snowflake Summit happening in June of this year. And I believe there is a link to register if you've already done so, in the resources section of the webinar. But I think that covers the majority of logistics. So let's jump in and look at the agenda. So, I realized we have different folks from different functional areas, different industries attending today. First thing we're gonna do is give you an overview of Komodo Health. For those not in the health care and life sciences industry, Komodo may be an entirely new organization. So we're gonna start off with the grounding on what Komodo does, what their value prop is, and then transition into what the underlying infrastructure that they build on Snowflake, and other tools is to support that, that go to market initiative. We're gonna focus on, how Komodo was progressing prior to the introduction and leveraging, of Facebook functionality within the Snowflake, environment. We're then gonna highlight some of the challenges that we attempted to address, and how icebergs help those. We'll dive into a little bit of detail specifically on individual functional areas where we're leveraging icebergs to implement some of those solutions. You can see that's gonna focus on the following two items. Number one is around managing the archival data, where we have large volumes of data that might easily be online, but not necessarily in in fully hot state. We're also then gonna talk about how Iceberg has helped implement, data sharing, across a number of different architectures between Komodo and Komodo's customers. We'll come back and start to review the overall impact that this has had, and then as I alluded to, the q and a session at the end with whatever sound we have reading. So with that, Dinesh, I will hand it over to you and tell us a little bit about, about Komodo. Yep. Thanks, Chris. So let ex let me tell you what we are. Right? So Komodo is a leading health care insight and technology company. We're really seeing the evident standard to the most accurate, complete, and patient centric intelligence. We bridge patient journey with health outcomes by combining deeply curated data with AI powered purpose built software solution. This integrated approach empowers our customer to accelerate their operations enabling faster data driven decisions, streamline workflows, and target actions to close the gaps in healthcare And this, overall, helps to reduce the overall burden of disease. So main Comoros motto is, like, to reduce the overall burden of disease in the world. That's what we are trying to achieve. Yeah. Thanks, Jason. Go next slide, please. Yeah. So what we are doing in Comoros Healthcare, how we are trying to reduce the overall burden of disease to to achieve that. So Comoros creating an created an health care map. That health care map is continuously being being updated by getting the data from different providers. And this is being a single source of truth for our end of product. Right? So what what it does, every decision in health care today involves millions and sometimes hundreds of millions of individual data points. Right? But all data points is not needed for the, the health care industry to take a decision point. That that is becoming a more challenges. So what we try to do, like, we try to unlock that. Comodo tried to unlock that. We help our customers unlock patient insights faster by doing three things exceptionally well. First, we separate valuable data from the noise, helping team team focus on what drives real outcomes. Second, we connect the dots across fragmented person fashion journeys, giving a holistic view of the health care experience. And third, we support decisions that meet a higher evidentiary threshold, ensuring insights are accurate, trusted, and patient centered. In today's healthcare environment, data quality is not just a priority, it's a foundation for everything for clinical research to operational strategy. That's why we built the Comora Healthcare Map, the industry most complete and continuously updated view of the identified longitudinal patient journeys. It protects patient privacy. It delivers insights at scale, and it reflects the real world health care dynamics, how people interact with the systems, what treatment they receive, and what outcomes follow. When you start with clean, connected, and context rich data, your insights are value truly valuable, and you by using that, you are trying to reduce the gaps and it'll enable you to make the decision decision making faster. And, ultimately, the final goal is, like, to improve the patient experience. Can you move on to next slide? Yeah. Yeah. So that we have a product. Now we are looking into the different products. Map enhance is one of our product. So what we are trying to bring to the market using our map enhancement project is like, it's a new depth to the, healthcare map. When working with traditional claims data, we are often missing key pieces of the puzzle, things like lab results, clinical variables, and biomarker data. So what we are trying to do, like, we are more importantly, claims do not give us the full picture of the patient journey, including that what happened before diagnosis or what care was provided after. So for, manufacturers, especially those working in rare disease area, this blind spot is costly. With such a limited patient population, the margin of error risk and not having the right data can directly impact trial design, launch success, and ultimately, patient outcomes may go wrong. That's why in this context, more really is more by combining claims their claims data with other forms of real world data, we were we are delivering richer insights, enabling stronger commercial strategy, more precise clinical trial design, and, deeper outcomes research. These are the outcomes the customer get benefited by using our map enhanced. With map enhanced, we are adding a new layer of depth to Comoros Healthcare Map, enriching it with multidimensional insight that provide greater context across the full patient journey. We are doing this through the powerful streams of data enhancement. Comoros built enhancement like we are using the EHR, lab data and patient level clinical characteristics. Comodo partner data proprietary and specialized datasets from a curated network of industry leading data providers. Together, they give our customers a more complete, more connected, and more actionable view of the populations they are trying to reach. Yeah. Next slide, Chris. So these are the softwares product by, Comora Health to create the analytics on top of our data. Right? So here we have, like name it like Aperture, Iris, Pulse, and Sentinel. Sentinel is our custom, analytical sandbox. And then we have Maplap Enterprise. Right? What is Maplap Enterprise? Maplap Enterprise is a is a product. It's a kind of a high code environment where it will help data scientists and the programmers to write the query and create the insight. They can do the data driven, insight and take the decision on top of it. Right? So that's why, like, the Maplab enterprise, help us to, help us the customers to to who are highly, technical. Right? They can use the product. So let's deep dive into a little bit of information I just want to give what this Maplab enterprise is capable of. As I explained that it is built for data scientists and researchers and enterprise users who want to direct access on our Comoros Healthcare map data, but with the ability to shape and explore on their own data business business, business logic. Right? So what is going to give to the customer? It's going to provide a customer sandbox sandbox like environment where they can perform custom progress build, run rapid, epidemiological queries, test and iterate on real, world evidence hypothesis, and collaborate securely across teams and stakeholders. Since you are come you are entering in the sandbox, it has well secured environment. Data is not leaked at all. It is like full full governance in place. And just a quick quick clarification on that, Dinesh. So not only has Komodo built to the underlying foundational area in the health care map, which consolidates all of those various sources, that you're that you're requiring from third parties, You have a suite of analytic and visualization tools cater into a range of different personas depending on whether, you know, think of a a traditional data scientist or machine learning engineer is going to want to have a very different type of interaction with that underlying data than someone who's struggling to come to terms with the basics of of writing a SQL query, for example. Can you can tell us a bit more about you know, for those that don't have a background in health care and life sciences space, give us a bit more detail on sort of the scale of the magnitude of the the health care map, the challenge that you guys are managing from a data volume perspective, data complexity. You don't have to go into into any explicit detail that that's proprietary, but just giving a sense for those other data practitioners in the in the audience of of what scale we're working with here. So you you you want to know about, to give information about the data, how we are trying to, update your health care map? Yeah. If we think about the volumes of data under management, the variety, the different types of sources you're working with, you know, that could be in the context of, you know, terabytes terabytes of data or or even simply in the context of of coverage. You know, how many how many patient charges are we talking about monitoring and and reporting on here? Yeah. So we have, like, three hundred three hundred three hundred plus millions of patient data available in the health care map. Right? And, the data sources we are getting from different providers, like, in the market, we are trying to get the data from them. And then we are trying to curate that as per our business needs and how it's sensible for our product to run. It has been curated based upon our business values, and then it is stored as a table in in Snowflake for us to power up our products. Right? And as you rightly said, like, there are different sort of products available. You beat you beat me on that, Chris. Yes. I'm going to speak about, like, how we are using artificial intelligence, on top of health care data, using our health care map. And our entire product suite is built on top of Snowflake. Right? How Snowflake is powering us to enable our product, working very efficiently. We can discuss in detail when we go to next further slides. Great. Yeah. Thank you, Chris. So this is the map plug is the entire product suite where it consists of, like, map plug interface as we discussed earlier and MAPView and then MAPAI. So MAPView is meant for the no code environment where the commercial team or the business person, if they want to get an insight about their, clinical products or about the drugs. Right? Or they want to know about the rare disease insights. So they they don't want to write a code. Just they can advocate they can go and create the COVID, and they hit the run button. It will provide the, analytical insights for them so that they can take the decision faster. So map view, it's a no code, and low code, so high code environment is for is a Maplock enterprise. Right? Yeah. Move to next slide, Trish. I want to explain about map AI. So recently, we announced this, the public preview of this product available in the market, and we did our, one day summit as well last week, in Boston on this. So map AI is is nothing, but we are empowering the each and everyone in the market. Like, not only, like, okay, data scientists can get the insights. Even the commercial team that they can ask the natural language questions, and we're trying to generate the insights based upon the question asked. So what we are trying to do, like, we are enabling our the health care, or the life science industry where they can try to take a right decision in a faster way. And instead of writing a code, they can ask the questions in an actual language process where we will take care of the, creating the, the respective queries and and view the actual insights. Right? So this, is going to open up a huge, insights to the end customers who are going to use it, and this is going to be a game changer in the healthcare industry or in the life changing. Yeah. Really, really interested to see how you're leveraging AI, within the Snowflake stack and and how that's standing on your product offerings on the go to market side. That this is fundamentally also powered by the Alcatel map. Right? Everything sort of relies on that foundational data there. And and what we've seen as we've worked together over the last two years or so, are are a couple key drivers in terms of everything that Komodo builds from a customer facing application suite really does rely on that health care map on that foundational layer. So any efficiencies, any optimizations that that can be realized at that layer are multiplying orders of magnitude in terms of their payback because each of those downstream applications or products or pipelines that you've built out of the health care map are benefiting from that. So we got a couple high level topics here that will come up in more detail as we get into the architectures we've implemented moving forward. And and these are relatively straightforward. I mean, these are kind of table stakes at a surface level when we just described on things like eliminating data redundancies. Anyone who spent time as a practitioner in the field, knows that, you know, duplicating data is is a challenge, is expensive, is cumbersome, is error prone, introduces all kinds of additional challenges moving forward. So I I don't think anyone would push back on that, but we'll talk in a little bit more detail on how we've actually been able to realize that in the real world. That obviously directly enables simplification of governance. Right? So the platform has to provide for that, but the way that you're deploying to figure that platform, needs to needs to keep that in mind. So particularly in industries where we're working with sensitive highly sensitive information, you know, health care perhaps more than more than any other industry, as in as a huge proportion of highly sensitive information, knowing that you have confidence that that data is only to expose the appropriate parties, the appropriate applications, the appropriate users, absolutely critical. The cost of of inadvertent breach or exposure of data there, is is something that we hope to never discover or understand. Seamless integration and this applies, I think, both on the on the upstream side on provisioning new data sources, figuring out how to integrate those into the health care map, as well as how to integrate those downstream products, the applications we think you have built and and continue to evolve on, how those can integrate with that health care map. Obviously, when we think about things like Map dot ai, with that type of an interface, it's gonna have very different needs and very different demands of the health care map than an application that simply exposes a simple SQL API was. But we don't have the option or the luxury of partitioning or fragmenting that health care map to support different individual use cases. We really need a solution there that's going to support all of them. And as you alluded to when when I asked the question earlier around scale, north of 300,000,000 patient lives within the platform, they think about the population of The US. That's bordering on a 100% coverage, which is which is pretty incredible to just wrap our heads around that there is a single dataset that allows us to gain that kind of valuable insights over, that looks every every patient or every potential patient, but certainly an extremely high percentage of that. So maintaining all of these objectives as we scale the platform, as the data volume continues to grow, the number of patient lives in the platform continues to grow, efficiency is is obviously absolutely critical. So with that said, let's, let's take a look at, a fairly abstracted view of Komodo's current architecture as it relates to leveraging Iceberg. And it's important to note that this is this is not an architecture that we designed on day one. Snowflake has continued to expand its its support, participation in the Iceberg ecosystem, as that open table format has has grown in popularity and obtained more traction within the data engineering, the data analytics space. Snowflake has continued to to aggressively pursue supporting that. There's a number of different steps we've taken over the last couple years. So I'll highlight a few and then, you know, welcome you to to add additional color if I if I skip over any, Dinesh. But if we think back to, I believe it was 2023 when Snowflake first announced support for for reading iceberg tables. In the sense of if you had an existing iceberg ecosystem and you wanted to make that data available through a Snowflake interface, you did have the ability to create an external table, read that data, and expose it through Snowflake. So there was some amount of functionality there, but relatively limited in terms of the amount of control that you could exert over that data through Snowflake. The range of functionality that a lot of Snowflake users have come to expect in terms of how they interact with data objects, was not necessarily there at that point in time. There was a real fundamental turning point in early twenty twenty four, so the start of last year, which I think is really where we we started our journey together on exploring how Komodo could leverage Snowflake. That was the first iteration of Snowflake support for building and publishing its own Iceberg catalog. So you now have the ability within Snowflake to actually create and manage Iceberg tables with, I won't say, a 100% feature parity, but pretty significant feature parity at the time, with the things that you could do with those iceberg tables within Snowflake. A lot of that functionality in terms of things like time travel, cloning those tables, creating shares on top of those tables, that was pretty strong right out of the gate, but has continued to expand to the point where there's almost perfect feature parity, between iceberg tables created and managed in stuff like, versus legacy, FPN or or stuff like proprietary tables. So we actually did start down that path where everything that Komodo was pursuing was stuff like managed and stuff like curated iceberg tables. That's evolved a fair bit with the introduction of Polaris catalog, which is an open source Apache format iceberg catalog. So although this was initially built and released by stuff like it is in fact open source. And what this does is it really opens up your flexibility to be able to interact with essentially any other compute engine. Any other database engine that's compliant with that iceberg standard is going to have the ability to read and write, from any of the tables that are exposed through that Polaris catalog. So this is this is sort of where we are today in terms of of Komodo's default, approach, in terms of our implementing all of the new workloads, all of any of your datasets or your data objects that are going to be residing in expert format. And keep me honest here if I'm if I'm overstating or misstating, but sort of the default mentality I think to match is is that Komodo is is looking towards Polaris as your default catalog for for all things iceberg. Yeah. That is our, future strategy. Thanks, Chris, for, setting up this. Yes. As Chris explained, like, yeah, the main thing what we are seeing, like, we want to avoid the data silos. Right? So by getting into iceberg, it really help us to do that. We always maintain a gold image copy for our entire health care map which is try to, provide the insights in a unique way for all of our products, which help us since we are maintaining this, single golden copy of data, you are there is in the complexity of the data quality. Right? When we were in the FDM tables, we are trying to, create, like, multiple copies of data based upon the product or based upon the needs, because we won't allow the other the the engines to read the the data part. So now with the iceberg, I don't want to create my data quality check-in each and every data silos. Now I'm I need to make the data quality check on my main main goal copy. And security and governance is, is being very useful. If you have multiple tables, you have to look through it here. Okay. Who access who is actually table? Because we have a different format as in place for each and every table side. So we need to keep monitoring that. But now being an iceberg, it's a one single copy. You can reduce your security and governance. You can focus on only one, only one single copy of data based upon your product or based upon your health care map, whatever you need. For Comoro, it's going to be health care map. Right? And this easily enabled us, the Snowflake, power of Snowflake replication of Iceberg as well. Right? So when we started the journey, the replication is in works with Snowflake, but myself and Chris were trying to design the solution for this for the, for sharing the data with the customer through the iceberg. Then, like, I need to tell that Chris helped us a lot to pull the strings and make that happen. So now we are able to replicate the iceberg tables as well from our account to to customer account through private testing if they are in a different region, or you can share the data within the same region using direct share. And one more update with yeah. Sorry. Go ahead, Chris. Oh, sorry. I'll I'll let you finish that last point, then I had a a couple of clarification questions I was gonna ask you. So Yeah. And then, like, we were we did our performance, testing. The the moment when we thought about, like, okay, can we start leveraging the iceberg table, do we need to really use Snowflake to manage the catalog, or do we need to use the excellent catalog? So we didn't directly get into the conclusion of, okay, we need to use store Snowflake managed catalog. We did our testing on these Snowflake, other third party catalogs like Blue and other other catalogs, and we try to run our pipeline jobs on top of it or try to run our product queries on top of it. We are not we are not able to get the performance or the response time same as, like, the FDM. And then we try to convert those tables into not, like, manage the catalog tables, and then we try to run our queries. We were on par same as, like, FDM. Like, there are minimum of two to three minutes delay, not same like, one hour delay or three three hours delay like the external catalog. So we were able to achieve what we can do within FDM using the iceberg table as well, the top on the performance front. What if, Chris, you're asking something? Right. Right. Yeah. So I was just gonna ask for a bit more clarification. So when we talk about reduction in storage cost, I I think safe safe to say that that's primarily due to a reduction in the the number of different copies of data, that are being maintained inside Komodo. Is that is that a fair statement? Sorry. Can I use that one? So the reduction in storage cost or the cost of the the data volume under management is is is more due to the reduction in the number of copies of data. Right? Yeah. Since data silos are avoided, it's eventually going to help you to reduce the storage cost as well. Okay. And and even in a relatively modern architecture that you had prior to adopting iceberg, as a key component of that, Can you give us a sense of some of the some of the more challenging or problematic pipelines? Like, how many copies of data were we potentially talking to? Are we talking about two or three copies of the data that you needed to maintain governance and security and synchronization on? Or are we talking about 10 or 20? Like, what what kind of scale are we looking at optimizing there? So product wise, the data is going to come for our health care map. Right? So that is the base. But for the pipeline to curate and, get the data as per the product requirement, we have to maintain, like, different copies. Right? So each and every downstream teams as well, like, they will try to get one copies of the data, and then they'll try to pair on top of it. Suppose if I take an example of data from Steam. Right? So if they want to do something, they want to take a copy of data and then try to do some insights on top of it. Right? So that's why the efficiency data I cannot say exactly how much. Right? So that we avoided it because Iceberg played a key role for us. Okay. So one source, you can you can do whatever you want with it. So no data silos anymore. That is the main, aspect or or the main goal, main main one of the goal is using Iceberg too. Yeah. Perfect. Okay. And then the the second question I was gonna ask, so you mentioned a pretty good parity in terms of query execution, which obviously is is fundamental. Right? Every interaction ultimately relies on on a query against that underlying dataset. So, you know, if there were a major hit in terms of query ex execution performance, that would sorta undermine the viability of of Iceberg as a as a as a potential protocol. I'm curious if you can say a little bit more about parity in in other ways other than query execution. So I I've mentioned a few points earlier in terms of, you know, your ability to interact with an iceberg table and stuff like the same way you would with any other, legacy format or FBN, format table. Any any any issues you run into? I know there have been a few recent evolutions in terms of, operations that are supported with those iceberg tables, but I think we're pretty close to to future parity. Is that Yes. So other than performance, right, so like, FDM tables, so you can do, like, partition tuning, variable in FDM tables. Can we do the same partition tuning inside the iceberg? Yes. We can do the same. So whenever you are creating an iceberg table, you can do, like, partition byte to us to it so that your partition pruning will happen. And even the same cluster data also is is is applicable on the iceberg table. It's it's it's the same feature parity like FTM. And the other thing which I want to tell, like, I already attached that point is, like, replication part. So it's it's needed for the to build the DIA strategy as well as well as to transfer the data between our account to another account maintained by by end user customer account or within our within within Snowflake or within that account. So we want to share the data. So that is possible because that is easily available in FDM, but the same feature parity is working in, Iceberg as well. And the other thing which I want to highlight, like, cloning. Right? So when you share the data from one account to other account in in native table, and then you can clone that shared table within the target account, on top of that, you can operate. Right? So that, feature parity also implemented by Snowflake recently, and it has been released as part of the 09/2010 release version. And that is working effectively as well. Because in in in in our use cases, we will share the data, the whole copy of data to different accounts. And from there, we want to clone it and try to process some data on top of it. Right? So it's a zero copy clone. It's not like a actual data copy. It's a zero copy clone in Snowflake always. The same exact thing is working in Snowflake as well. The data sits in your catalog your your S3 layer, and everything works in us like your table. You can play around with, like, okay, granting access and blocking access like normal table. So we are going to discuss about this use case, after this. There, I will try to explain, how we are using IZBERG to unlock the potentials, what we used to face, in before IZBERG. Right. So and just before we skip off to that, the use case, it's worth mentioning because I'm I'm sure some of the audience may not be familiar with the the concept of of cloning tables in Snowflake. Even though we spent a few minutes talking about the challenges and the overhead associated with creating copies of data or silos of data, which are generally something that are great to avoid as much as we can, A clone really gives us the ability to interact with data as if we had created a copy without having to worry about the associated challenges in the overhead in terms of things like maintaining synchronization between those two copies. Right? Is the data or a given record identical between those two or has it evolved over time or is there lag or latency between those two copies? With this concept of the clone that goes away, because you're not actually physically copying that data. You can almost think of it as two independent views of that data that can be maintained independently. But we don't have to worry about synchronizing the the shared components of those, those two different paths or those two different branches. So super super interesting to see that you're also seeing, you know, feature parity there or pretty close to it, across the board for for really important powerful features like that where you do want to leverage what feels like a copy of that data with an incurring corresponding cost. So, David, let's dive in. I know we talked about we're gonna look at two specific use cases, within that architecture. So first thing here is around storage archival or data life cycle policies. I'll I'll let you do a brief intro around this. Thank you, Chris. So before Iceberg, we want to archive our data from Snowflake. Right? Archival means, like, we want to, move the data into s three bucket if the tables are not being used, like, more than six months or more than three months. Right? So we have our own method of identifying, like, based upon the products or based upon the teams, we will decide what is the threshold to hold the data within Snowflake environment. So in order to do that, since how you can get the, data, you need to scan the query history to identify when it was lost access. Right? So to scan the query history, you know that scanning query history, if you have millions of, queries run per day, then consider that how much bigger your query history is going to be. Scanning that is going to be a huge compute spend task for us. Right? And to get the output is going to take time. It's it's a it's a costless operation for us. But we cannot get rid of that because I need to query the table, only query the place for me to find out to the yearly access. And then based on that, I will decide what is the date it has been accessed and to try to map it my mapping table to see whether it's meeting meeting the archit, threshold has been defined by me or by by our or my or by my organization. If it matches, then what it does, like, we need to unload the data. So as you already know that, in Snowflake, if it's unload, you need to copy into s three. Right? So you're unloading data from Snowflake into s three and then dropping the table inside Snowflake. What if, like, okay, we are we are tired of it, but we have seen a scenario like, okay, after six months, one of our team come in and say, hey. I want this data to be restored again because we need this data immediately for one of our project. In order to restore, then it's a tedious it's a complex process where you have to, load the data back from yesterday into Snowflake. But what we I we did like when the data moved to yesterday bucket, we enabled the, life cycle policy to push the data to Deep Glacier as well. So think that we need to get the data from Deep Glacier into standard bucket. And from standard bucket, you need to load the data into Snowflake, and then you need to grant the record permissions as we have here. So consider that this is a very complex process on the Octave. How efficiently we really use the complexity once we think about iceberg in place there. Right? So these are the, comp complex problems we were, handling in in snow, in the snowflake negative tables. I can just ask yeah. Sorry. Go ahead. Sorry. I was just gonna ask for some clarification there. So with this process of of managing, you know, data life cycle, was it something that you're running daily or hourly, or what what was the scale of of this this process? Good catch. I missed that. So we were trying to run the job, weekly to identify the list of, tables to implement the life cycle policy so that we'll be, moving the unwanted, data out of Snowflake in the in the s three. Right? Right. And then it will help us to manage our storage as well within Snowflake. Yeah. Okay. And so the real challenge here, I think, as you as you alluded to is that, you know, there's a certain subset of data that based on recency of access or, you know, the time stamp associated with that data was curated or generated. The the frequency of usage of value was delivering didn't necessarily warrant keeping it in hot storage in Snowflake. So the goal was, hey. Let's put it into, you know, whether that's Deep Glacier or some other long term archival format at lower price point. But then the challenge is you never know when an end user is going to say, hey. Oh, I need that. I'm I'm doing, you know, an audit. I need to track back and see what the state of a given record or a given data point was at some point in the past where that then needed to be brought back into Snowflake. So Yeah. Great. Okay. And then so moving forward, so just leveraging Iceberg to try and solve that. Yeah. So, you know, after, thinking about Iceberg in place, how we change our architecture is like, okay. Let's convert all the tables into Iceberg table. Right? So now we know that this particular, based upon my threshold, I can find out, okay, whether this table is not used for more than six months, then what I can do, like, simply I will try to, remove the access from the cable from the road. Right? And then what we will what I will what we will do, like, we will try to enable the intelligent clearing on the SD bucket where the iceberg table has been placed. Right? So here, like, we did our testing as well. We try to put the data into DeepGlacier, but Snowflake NASH catalogs supports still instant glacier retrieval. You cannot read directly from Iceberg table from the DeepGlacier. So the limitation is there. Like, you can read from, Glacier Instant Retrieval. Why is it so? Because Snowflake wants to get the response time within a within a fair amount of time. Right? But when you are in deep glacier, the restoration from deep glacier into the standard bucket itself, like, if period depends upon your data, the restoration time varies. If it isn't, like, find find a terabyte, then it's the minimum time what quoted by databases, like, twenty four hours to forty eight hours it would take to restore. So that's the limitation where Snowflake is putting that, okay, we should not end up giving the error to the customers. Let's try to stop it under, Glacier instant October. So you it's a Instant Retriever. Right? So this enable us to save the cost as well in s three package as well as we can directly allow them to read with their data directly using iceberg. So in the end of restoration, what will happen? So I can just grant the access on the iceberg table. I only do the entire restore process again. Right? And one more thing you can observe here. I I'm not unloading data from Snowflake into yesterday. Because already while creating it, I created this as iceberg table. So I'm I'm sharing my compute on that, and there is the complexity of that. So this one enabled us and it reduced the, life cycle of the job as well. Because really, it does, like, four or five step process. So now it's, like, one or two step process to automate. Right. You're welcome. You may add anything on this? Yeah. So it it makes perfect sense. And I see there's there's several questions in the q and a that are that are relating specifically to this scenario. So I'll I'll try and summarize or or touch on a few of the questions in in summarizing the statements you've already made. So so in taking advantage of this in this case, I believe, when we initially set this up, this was a Snowflake managed iceberg table. So the catalog was within Snowflake. The data storage was landing in you guys being primarily an AWS shop was landing in s three. Obviously, each customer has the flexibility to choose their their cloud provider of choice. But in this case, we'll we'll use, you know, AWS and s three vocabulary. So in this case, by leveraging that Snowflake managed iceberg table and specifying that that would get stored in s three buckets, You were then able to set within AWS a life cycle policy on those buckets that gave you more granular flexibility to decide at one point in time data was stale and should be migrated either to a slightly lower cost, lower performance tier like Glacier Instant Retrieval, where it's still accessible, can still be read in real time, albeit at a slightly somewhat lower performance level. Or if you decided that at a certain point in time, if the data reaches, I'll invent a number and say, you know, it's thirty days old since it was last accessed or thirty days old since it was created. We'd like to move that into Deep Glacier. Still online, it just can't be queried directly through that iceberg table because the infrastructure in leveraging in s three, that Deep Glacier has, as you mentioned, an hours long, process for recovering and making that data readable. So depending on where in that life cycle policy, I won't necessarily call it deprecating the data, but migrating it from true hot storage down to long term, almost offline archive. I'll think of that as cold storage. You have the ability to define that policy directly within AWS, and the only impact the only limitation would be any data that gets moved out into, Deep Glacier would need to be revised and brought back into either Glacier Instant Retrieval or into a fully online, you know, s three bucket, before it could be consumed and queried through that iceberg too. Is there a way? Yeah. Yeah. So to add little bit, context to it, like, we enable incident clearing, incident clearing will decide that based upon the access of the, particular file. Right? It really works based upon the file. So if the file was not being accessed, then incident clearing will decide, okay. It has to go to the instance retrieval, and there are multiple, layers we have to try to push it. Right? Yeah. But it's not like capital upgrading it, but your response time is going a little bit delayed when it's not in standard bucket. Right. Yeah. Okay. Great. So unless you find the balance between performance and responsiveness for that data versus the cost associated with persisting it long term. Yes. So this particular thing, like, it will it will it will save us a huge MPN life cycle time for the job. Right? Because it has multiple steps. There's a scanning is not we are not going to, we can't avoid here. Scanning used to happen, but, yes, it it reduced, the cost very much. Right. Yeah. That's and that's that's important to call it as well. It's not only is the cost of storage, you know, optimized for, you know, whatever your business SLAs are, but it it removes that entire process that you guys were maintaining of continuous scanning to identify what subsets of data, had had exceeded those thresholds and should be offloaded. So, again, reducing both the process and and the overall cost of storage. Thanks for your patience. Yes. Great. Okay. So let's, let's dive into the other one, and this one's this one's really interesting because it's more directly customer facing. I think drives drives more obvious value to the customer than sort of the operational efficiencies we're gaining with life cycle management there. So, walk us through how how we're leveraging those for your for data sharing. Yeah. Thanks, Chris. So what we what we're here in FeetN, we are trying to leverage, the data sharing capability. Like, we allow customers to run the, use our products and create the insights, and the insight will get stored inside the, Snowflake environment. Right? How they can get the data into their environment? If they are in the same region, as I explained earlier, like, you can do the direct sharing. And or else if they are in a a different region, you are going to use the private listing under the code that it's going to do the replication. Right? And even with this, like, as the other one, however, the private listing allows you to do the sub DB application as it's no need to transfer the entire, database. You can select the schema, and then you can select the needed tables as well. Right? And then you can share the data. So by by sharing that information, what we are trying to do, like, we are trying to reduce the data availability into the customer environment, and they can, do the insights faster. When Iceberg came into play, we want this particular functionality to work because this is our this we need. We we we promise our customer that, yes, whatever the insights you are generating, you can you can access it immediately because that is your data. You can access it whenever you want to. And if time to market is is very, very important for them. Right? So how we can reduce the time to market by leveraging the iceberg table? Two ways we can we can try to do this. If it is iceberg, we can allow them to directly use any DB query engine. It's supported as API. They can directly call, connect, and then query the iceberg table. Or else, if there will be a complaint okay. Some organization may have HIPAA compliant. We won't allow you to read anything from our data. Okay. How we can enable it? It's my my data. Right? Okay. I can transfer I can replicate the iceberg table to the private listing into your account in your account in the customer account, and then there is available in their account. Now when they can create the curated, inside data, on top of them, they can generate the reports, and it can be presented to their marketing the leadership team immediately to take the decision faster. Right? So if and what if the the multi the, replication or the delta channel is not available, the the we need to go the traditional way. What's the traditional way? Like, okay. I'm loading data into s three. From yesterday, scan it because each and every organization has their own regulations. Scan. If this can success, okay, then transfer it to the customer bucket using the bucket replication. During the bucket replication, there might be a chance that the packet loss can happen. So there might be chance of data loss as well. So by leveraging this, replication as well as the delta sharing on iceberg format, it allows us to avoid this data loss, risk the limit, the risk of data loss. Right. Right? So And then So maybe sort of summarize that at a at a little bit of a higher level. So in this case, we're talking about stuff like managed catalog or stuff like managed iceberg tables. So when we replicate let's let's take the more interesting scenario and you have a customer who's in a different region or potentially even an entirely different cloud, and you'd like to make that data available to them in iceberg format, you can leverage the same data sharing pipelines and the same mechanisms that you did previously for, Snowflake at the end tables or any other Snowflake objects or applications that you wanted to share. So, again, back to this this notion of future parity, you can you can use those iceberg tables in very much the same way you did with the with the legacy snowflake tables. Can you say a little bit about, complicates the issue a little bit because we don't have a corresponding visual here. But as we've explored leveraging Polaris together, that also changes the the scope a little bit because right now, this requires you to have a Snowflake account running in both the source and the target region. And when you replicate, you get the benefit of Snowflake taking care of all the replication of the data, the replication of the catalog, and all the contents of that catalog. Then on the client side in that secondary or in that target region, those applications are still accessing an iceberg table, but they're as accessing it through a Snowflake managed catalog. As we talked about leveraging Polaris, that gives you a little more flexibility still. Just and I'm throwing this out here off the cuff. I know we didn't prepare for this specifically, but I think it'd be an interesting evolution to talk about. Just Yeah. Let me, let me give you some information about that one as well. Right? So as Snowflake always, tells to the customer that, okay, that is a vendor lock in, going forward because Polaris catalog come into play. Right? So you can manage as I explained earlier, in Snowflake managed catalog, table iceberg tables, your performance is on par. Right? But when you go for the third party, your performance is is little bit you cannot achieve what you are getting it inside these Snowflake APN tables. So what we are, myself and Chris, were went on a detailed, discussion what good what good kind of session, like, okay. We will try to register our Snowflake managed catalog Iceberg table into the Polaris catalog as a cataloging database over there, and this will enable us, the customer, to read the data from the Polaris catalog because Polaris catalog is a open format catalog, environment. It's a open format. Any DB engine which is capable of querying the iceberg table using the rest API, they can read the data directly. And then it will help us to resolve the Doctor capability issues as well. So you can try to build the the app. Right? So how we can how we can build the the app? So in US West, your database, you you are creating a Snowflake managed catalogized work table and the metadata maintained by Snowflake. And it has referenced everything to that particular account and that particular region. The moment I move that catalog, ratio that Snowflake magic catalog in the Polaris catalog, then it's a open source format. In the event of failure in US West, how I can bring up US East? I don't want to create my, entire Snowflake iceberg managed catalog again. I can directly create the external table responding to the Polaris catalog because it is an open format. I can create that as external table, and I can make my customer read the data for the particular outage period. We won't allow a write because as you know, the external table is read only. It's not a write. So we will make the products, everything is available, and they can read the data for the time period. And then once the issue is resolved, you can fall back to to US West. So that's So if we think about the I'm sorry. I was just gonna say if we think about the trade offs there and I I see a bunch of questions in the q and a around, sort of iceberg fundamentals. And and the goal of the the session today, I should have highlighted initially, was was not sort of one zero one or 100 level enablement. It's it's sort of targeted at the intermediate audience. But to try and give some context around the trade offs of leveraging Polaris versus leveraging and stuff like managed expert catalog, for example, Really, what we're getting here is the data storage. And there were several questions here that I'm trying to answer in the context of the points you were making. So, the data storage layer, we get to decide where we want that to be persistent. So in your case, the Nash and Ford Komodo, that's residing in s three buckets on AWS. So that is where the data is physically written on disk. The second layer is the catalog. It tells any client applications that wanna query and then drag with that data where it's physically located along with all of the other structural and relevant metadata that's required to to build a query plan and execute that query. Now we've spent a lot of time talking about different catalog options, Snowflake, Iceberg catalog being one, which is Snowflake's proprietary implementation of a catalog to an open source table format. So we talked about iceberg tables. That's the format that the data is written in that cloud storage, in this case, s three. Catalog in in the Snowflake iceberg catalog is how clients that can talk to Snowflake can now directly and seamlessly interact with that data sitting in s three. The other option we talked about here with Polaris is an open source, so vendor agnostic, vendor independent implementation of a catalog. So now what that means is you don't need to talk to Snowflake to be able to interact with that data sitting in iceberg table format. You can connect to that catalog with any compliant compute engine or database engine. And there's a ton of them out there, Spark, Trino, lot of different players in the ecosystem that have various levels of of iceberg compliance. But that Polaris option, although you do lose a little bit of the stuff like proprietary functionality around data shares and sharing cross region and replication, what that gives you is is true flexibility, total vendor, hypnosticity. So if at some point in the future, all of your data is in iceberg sample format and you're using Polaris as your catalog, if a compute engine comes along in a year or a couple years that outperforms Snowflake, you can very easily and very seamlessly say, hey. We're gonna start leveraging that for the workloads where it makes sense. We're not tied into only client applications or only catalog implementations that are supported by Snowflake. So not only is it maintaining the performance you need, but it's giving you that feature flexibility to pick the best of breed point solutions and then run through that analytics stack. Rightly said, Douglas. Great. Okay. So I know we're getting a little bit short on time. I think we got about five or so minutes left. Really quickly, just wanted to touch on the time line, and I think we've made a couple different references to this as we've touched on few of the prior slides. Well, this is, this is a pretty hot topic, you know, evidenced not only by the the number of people registered for the session. We're interested in hearing more. But just in terms of the time line for, for when we see an iceberg really gain a lot of traction here, from a snowflake perspective, this is something that's that's really gotten a lot of additional attention, a lot of effort in terms of teacher development and support within the last, I'll say, sixteen to eighteen months or so. One of the great things about, you know, getting really an active participation from Komodo, Dinesh has been selfishly, from Snowflake's perspective, our ability to talk to a customer who's using this in the real world, solicit your feedback and your input on what features are important as we pursue that feature parity with, you know, legacy Snowflake tables, which ones are top of mind, which ones are most important for Komodo. So, I I I think it's safe to say that that's been a a great exercise to engage in where you have the ability to influence the Yeah. The direction of that product evolution. It's been it's been a great symbiotic relationship. But I highlight this that this is still really dynamic space. So there's there's a lot of change still happening. There's a lot of participation from different players in the space. And, where we are today is is super exciting, but, you know, there's there's certainly more to come. And, you know, even in a couple months when we have this this presentation session at Summit, we may actually have some, some significant new content to add to this one as well. So k. Great. Last thing I'll touch on, and I think some of these will be will be fairly evident from some of the problem statements that we've highlighted so far. I won't read through through Lerat's quote, who's your your CTO Yeah. At Komodo there. Apart from the fact that I laid, and he's kinda reinforcing a few of the points that I just made of saying, look, first and foremost, is is performance. Right? The the benchmark, the SLAs that we've established with our customers based on all of the pipelines, all of the tables, all of the data products you've built on legacy Snowflake is not something where there's flexibility, to sacrifice performance. So that is the first criteria. If Iceberg did not beat that performance SLA, that would have been the end of the conversation. And, obviously, given where we are today, I think it's safe to say that not only does it not that performance level, but so Laurent's point towards the end of the quote here is giving you guys a significant degree of additional flexibility, not only in terms of where you take the Komodo platform in the future, but your ability to interact with a diverse range of customer applications and and meeting different customer technical requirements. Yep. So here are the, impact. I think we already discussed about what are the impacts, in detail. But I think quickly we we can go to the next slide to to to speak about what is the impact to our end customers. These are the benefits. Right? So input query performance, and it's going to lower the complete spend. One one thing, like, as I explained in the archival, we are reducing the complete spend as well because we are cutting short the number of steps. And faster the job cycles, there is no multiple pipelines, like, silos of data is going to go, we are going to run here after. And the reliable data sharing, we discussed more about this. Right? So you can rely on the data which has been shared with your end customers. And then accelerated insights by sharing the data immediately to your customer and customer able to generate the insights immediately, not to wait for, like, okay, six hours to to get the data or or a day to get the data. So now the insight is being created instantly for them. What core benefits, our custom value to our customers, as part of this, iceberg initiative. Right? So it's a deeper, more, considering insights, we obtain information immediately. Right? And value on day one, that is a very main thing. Each and every customer want to get the value immediately so that we are able to achieve. And then we are trying to innovate with other partners like, like, and these and other data scientists. Like, we are allowing them to directly query the data in, like, the format because they are not tied to okay. I I don't have snowflake. How I can bring the data from the enrollment? Okay. Use any data pertaining. If you have Snowflake enrollment and you need a performance we have a good performance, okay, then we will replicate the data to you. So that's the strategy we'll try to play around. And, that these are the key value we are getting to our customers immediately by leveraging the iceberg. Yes. Great. That's fantastic. And great to see that this is actually having direct benefit not only to Komodo in terms of operational efficiency and the and the cost of managing the, you know, the health care map platform and all of your other operational pipelines, but, actually, in terms of delivering value to your customer. Right? It's one thing to say, we're maintaining the value we delivered, but doing it in a more efficient manner. And and I think it's safe to say that, you know, on a pretty broad range of scenarios here, not only are you doing it in a more efficient more efficient way, but you're actually delivering even better value whether that's through lower cost of operation, lower latency in the data, more comprehensive insights. Like, this is this is a best of both scenarios. So I would highlight that to to other attendees on the call here who are thinking about, hey. Where does iceberg fit within our ecosystem? What's the real value here? Is this an internal operational improvement? Is this something that's gonna deliver value to to our downstream consumers? And and your customers or your consumers could be internal teams. Or if you're a an organization who's building data products or applications on stuff like those could be external entities. But there's real opportunity there to leverage iceberg, not only on the operational efficiency side, but in in improving your end customer experience. So I know we're right at the end of our window here, and, we're hoping to leave, you know, five or ten minutes for the q and a. Maybe what I'll do if, if our admin team will give me the thumbs up, I'll run just a couple minutes late, and I'll try and touch on a few of the questions that have popped up. I know there have been several. I've tried to address some of those at least, at least briefly in, in passing. So I will just do a quick scan here and see if there are any that, that pop out. So couple of questions around catalogs in general. Hopefully, some of the content we've touched on has has clarified a little bit of that. But we think about a price per there's there's two fundamental components. There's the persistence and the storage there, which could reside on on just about any technology. I think the predominant leaders in that space right now are gonna be object storage on one of the cloud hyperscalers. So, you know, Dinesh, in your context, that's s three. On top of that, we need a catalog. And there's I would say there's two classes of catalogs. So there's, vendor specific catalogs, and and stuff like has one of those. And the reason why it's vendor specific is that we wanna be able to support a whole range of proprietary stuff like features and functions, like table cloning, like data sharing through the stuff like marketplace, all of the stuff like governance features, that exist. Some of those are only possible when we extend the iceberg standard. So that's where the vendor proprietary, the vendor specific catalog implementation set. So you do have storage agnosticity where that data sitting in s three could be read by other iceberg catalogs as well. So at some point in the future, if you decided, hey. I'd like to leverage, you know, Trino or I'd like to, you know, set some lines up in Dremio that leverage those iceberg tables. That is an open standard and open format. Any compliant engine or catalog will be able to read, but your choice of catalog will determine, what operations you can support in terms of read and write as well as advanced features, a few of the likes we've just that we've just spoken about there. So when I said these two classes of catalogs, the vendor specific catalog generally is going to give you enhanced functionality, but is vendor specific, which would require, a change or an evolution if you wanted to move away from a given vendor. The other option is the true open source catalogs, and that's where Polaris sits. So it is defined by it it's true open source in the very sense of the the term. It is open source and restricted specifically to the iceberg standard for catalog. So, the good news is that it's standard. It's compatible. It's read and write from any compliant compute engines or database engines that would be the client application interacting with that catalog. The trade off is that you can't extend that standard until there's agreement within the working group that, the new features will be added. So I knew that ring Oh, sure. Go ahead, Dipesh. So when we speak about the catalog, if you did remember yesterday, we were discussing, like, external and internal. That plays a major role. If it's a external, it's just like a third party. You can do read and write operations. If it's you are creating an internal, which is really, like, internal to Polaris catalog and your catalog is not managed by Snowflake, then it is only read only. You can do read only from Snowflake. Right? So if it's actually the read write, and if it's internal, it's only read only. So that you don't need to be aware that when you are following your journey towards iceberg, you should be aware of this. Right. Okay. Couple other questions around, storage and things like egress charge. I think this will ultimately come down to whether we're taking data that already exists in Snowflake and moving that out into an external iceberg table. In that case, there would be a one time egress, but you also have the option of when I create a new table. So I might have, you know, Snowflake adjusting data, from a Kafka topic, for example, and I can actually have that data land directly into an iceberg table that I have defined to live in s three and whatever, you know, zone or region I I've chosen. So in that case, there is no egress. The one and the initial break of that data is directly into an s three database per table. So it ultimately comes down to whether we're migrating existing infrastructure or whether we're creating that new. Quickly going through. I'll maybe try and tackle one more here. A few other questions around data formats, which is a little bit in the weeds, but I do see these questions from from a few different folks on the audience. So I'll I'll touch on that briefly, and and, Dinesh, if you wanna add additional context as well. So when we talk about the data format, how is that data actually being read? Important to note that iceberg is an OpenTable standard, and it supports a couple different file formats. And it it starts to become a pretty nuanced conversation when we talk about catalogs because we have things like business catalogs and semantic catalogs versus table catalogs. A level below that, we have table formats that dictate, okay. Great. What is the structure of my metadata and my manifest files versus the actual raw data and how that gets written? Level below that are file formats that are supported by that table standard. So I believe, Iceberg currently supports Parquet, ORC, and Avro as I know. Yes. File formats. Right. So any of those file formats, if your data is already existing in that format, it's simply a matter of saying, great. Can I wrap an appropriate, table structure around that, add the additional metadata required to make that a viable iceberg table? So not necessarily a major lift for those that are asking of the transformation or manipulation of that data. If it's in one of those formats, it's pretty close to being iceberg. It's not necessarily compatible right now, but but iceberg adjacent, I think I can describe it as. So it's, it's it's not likely to be a major lens to be able to to migrate that or port that into an iceberg compatible workload. Mhmm. So great. With that, I know we're a couple minutes past here. I don't wanna go too much longer. We will try and follow-up with the questions that didn't get answered in the q and a here. As I mentioned, feel free to reach out to Dinesh or I. LinkedIn is probably the easiest option. We'd be, be happy to to carry on the conversation there. For any of those that are attending summit, we will be presenting, an updated version of this, at the, at at a summit dedicated talk track. So it'd be great to see anyone attending at that session. Happy to chat in person. And I believe there's some questions around logistics of the recording. The recording will be made available, as a follow-up to everyone who is registered, so you will be able to follow-up on this and and review this for for additional detail if you missed anything at the time. And with that, appreciate everybody for joining, and especially thank you to you, Dinesh, for, not only being a a great active partner in in helping stuff like, you know, evolve our offering, but certainly in in volunteering your time to come and and have the conversation today. It's been been great, and I'm sure super valuable for, for the audience who's attended here. Yeah. I hope so, Fraser. And thank you, Fraser, as well for your partnership, because, without, you and Snowflake, we were not able to achieve this journey. And thank you very much for that. Awesome. Thanks.