LLM State of Play

Important

If you skip this section, the only thing I would urge you to take away is that LLMs are getting more capable by the day and the cost of using them has plummeted. There’s also a mix of as-good-as-best-in-class opensource models now available, with variants that are viable to self-host, bringing with it more choice for users that may not wish to utilise API endpoints of the big foundation-model vendors. This, together with some other use-cases I hope to demonstrate in here, should convince those on the fence to embrace this innovation for their business.

Given the pace of innovation in LLMs, it’s a somewhat moveable topic, so it’s perhaps useful to take a snapshot of where we currently stand in Q1 2025. In this field, I defer to better experts than myself and Simon Willison’s uncanny ability to offer precient observations¹ makes him a go-to source. In a recent review², he made some of following observations:

• while 2023 was a ‘slow’ year for LLMs, with OpenAI maintaining dominance for most of that year with GPT-4, 2024 saw the pace of innovation spike, with as many as 18 labs³ producing GPT-4 equivalent models.
• 2024 was also the year of multi-modal LLMs, taking LLMs beyond text and gaining capabilities in processing images, audio and video.
• pricing for usage of the most capable models has continued to decline precipitously (Figure 1) making the economics of using this technology all the while more compelling. Willison describes being able to label 68k photos with descriptions for a sum total of $1.68, leveraging Gemini 1.5 Flash 8B.
• he also points to the step-change in the sophistication of models that is now possible to run on local hardware, as even more capability is gettting packed into ever-smaller variants. He actively records these milestones as he experiments, offering a valuable time-line of progress, such as these, from November through January.
  • Qwen 2.5 Coder
  • Llama 3.3 70B
  • Mistral Small 3

• per model ranking in Figure 2, the presence of top-ranked opensource models is predominantly down to the efforts of labs from China, including DeepSeek and Alibaba (Qwen).
• January 2025 marked the arrival of DeepSeek V3 and R1 models, which you may recall rattled people’s assumptions around the possibility of being able to train these cutting-edge models for much less in compute cost than was previously assumed. It turns out that the low training cost posted at the time was taken a little out of context.
• Most compellingly, DeepSeek have been the first to properly opensource their methods (not just weights), which is a step-change.
• A note from Anthropic’s CEO⁶, in which he argues for even stronger efforts to limit advanced chips for China, he posits that the interest in their reasoning R1 model was elevated, since they were the first to expose the Chain-of-Thought reasoning to users. You’ll recall, OpenAI’s o1 only gives you the final answer.
• In that same note, Amodei argues “DeepSeek-V3 is not a unique breakthrough or something that fundamentally changes the economics of LLM’s; it’s an expected point on an ongoing cost reduction curve. What’s different this time is that the company that was first to demonstrate the expected cost reductions was Chinese.”. Take from that what you will.
• The top-ranked models (Figure 2) are predominantly reasoning (inference-time compute) ones — OpenAI o1 and o3, DeepSeek R1, Qwen QwQ, Claude 3.7 Thinking and Gemini 2.0 Thinking are all examples of this pattern in action — or so-called MoE (mixture of experts)⁷, with Deepseek-V3 using as many as 256 different experts, per their paper⁸. Technical readers may find this discussion⁹.
• Willison also observes that these models are getting much better at handling PDFs and he offers well-documented approaches to leveraging this¹⁰. For instance, Gemini and Claude can both accept PDFs directly. For other models (including OpenAI) you need to break them up into images first—a PNG per page works well. These capabilities and reduced cost will only accelerate the sophistication of RAG (retrieval-augmented-generation) systems, a foundational capability for any business looking to use these technologies, as we’ll elaborate on lower down.

LLMs as a Developer Tool

The ability to leverage LLMs, as a developer tool, has dramatically changed the time-frame (and economics) of developing a model that can be iterated upon to solve a business problem. Convinced of the value they can bring, the challenge becomes how can we securely and reliably get LLMs to perform complex tasks with real-world data.

For most LLM applications out there, your development steps will involve the selection of a foundation model, which you further have to optimize by using prompt engineering, fine-tuning, or RAG¹¹. The operational aspect of these three steps is the most critical to understand, something we’ll touch on in the LLMOps section below. For those keen to move beyond the abstract, I can whole-heartedly recommend the LLM Engineer’s Handbook¹², which leads the reader through an end-to-end project implementation, which I personally think is the best way of gaining a clear understanding of how these technologies work and, more importantly, offers a spring-board for implementing a solution particular to your own business.

Understand Different AI User-types

How you leverage LLMs as a business and the requisite sophistication of your LLMOps capability depends on the type of user you identify as. A helpful blog from AWS (linked below the diagram) depicts three main user-types along a continuum, with most users, as of today, probably identifying as ‘consumers’ — tapping into API end-points to feed an LLM some task. The ‘providers’ are the ones building models from scratch, where obviously much fewer organisations have such capabilities. In the middle are the ‘fine-tuners’, those that recognise the power of LLMs and that are prepared to build a level of expertise to be able to refine models for needs specific to their business use-cases.

• Providers - train models that are general-purpose. Each trained model needs to be benchmarked against many tasks not only to assess its performances but also to compare it with other existing models, to identify areas that needs improvement. Providers might become fine-tuners to support use cases based on a specific vertical (such as the financial sector).

• Fine-tuners - want to solve specific tasks (e.g. sentiment classification, summarization, question answering) as well as pre-trained models for adopting domain specific tasks. They need evaluation metrics generated by model providers to select the right pre-trained model as a starting point. they must curate and create their private datasets since publicly available datasets, even those designed for a specific task, may not adequately capture the nuances required for their particular use case.

• Consumers - consume general purpose or fine-tuned models in production, aiming to enhance their applications or services through the adoption of LLMs. robust monitoring and evaluation framework, model consumers can proactively identify and address regression in LLMs. Consumers might become fine-tuners to achieve more accurate results.

As a proving ground for developing AI use-cases, the AWS ecosystem is no bad starting point, since they do the heavy-lifting around security on your behalf, allowing you to focus on evolving a solution that can add value. Their Bedrock platform offers a single API to various different LLM foundation-model providers and is best suited to the ‘consumers’ identified above. It offers an out-of-the-box solution that allows you to quickly deploy an API endpoint powered by one of the available foundation models.

Alternatively, their SageMaker is a multi-functional platform enabling you to customize your ML logic fully and is aimed at the ‘fine-tuners’, if not the ‘providers’. This incidentally is the system leveraged by the end-to-end project contained in the LLM Engineer’s Handbook, linked to earlier, for its greater versatility.

Importance of the RAG Pattern

RAG is an efficient way to provide an FM (foundation model) with additional knowledge by using external data sources and is depicted in the diagram on the left:

• Retrieval: Based on a user’s question (1), relevant information is retrieved from a knowledge base (2).
• Augmentation: The retrieved information is added to the FM prompt (3.a) to augment its knowledge, along with the user query (3.b).
• Generation: The FM generates an answer (4) by using the information provided in the prompt.

From left to right (on left) are the retrieval, the augmentation, and the generation steps. In practice, the knowledge base is often a vector store, allowing for hybrid of key-word and semantic search.

Some Common RAG Use-cases:

• Employee training and resources – In this use case, chatbots can use employee training manuals, HR resources, and IT service documents to help employees onboard faster or find the information they need to troubleshoot internal issues.
• Industrial maintenance – Maintenance manuals for complex machines can have several hundred pages. Building a RAG solution around these manuals helps maintenance technicians find relevant information faster. Note that maintenance manuals often have images and schemas, which could put them in a multimodal bucket.
• Product information search – Field specialists need to identify relevant products for a given use case, or conversely find the right technical information about a given product.
• Retrieving and summarizing financial news – Analysts need the most up-to-date information on markets and the economy and rely on large databases of news or commentary articles. A RAG solution is a way to efficiently retrieve and summarize the relevant information on a given topic.

With experience, an organisation’s sophistication in using these technologies will grow. As articulated in this blog¹⁵:

A more scalable option is to have a centralized team build standard generative AI solutions codified into blueprints or constructs and allow teams to deploy and use them. This team can provide a platform that abstracts away these constructs with a user-friendly and integrated API and provide additional services such as LLMOps, data management, FinOps, and more. Establishing blueprints and constructs for generative AI runtimes, APIs, prompts, and orchestration such as LangChain, LiteLLM, and so on will simplify adoption of generative AI and increase overall safe usage. Offering standard APIs with access controls, consistent AI, and data and cost management makes usage straightforward, cost-efficient, and secure.

Outerbounds (Metaflow) Use-case — New Web Technologies leveraging AI

To illustrate the changing sophistication of websites, a furniture retailer site is used. For the past 30 years, the same basic website design entails giving users an ability to filter available stock by price, type etc. with a query to the backend and the front-end refreshing to show the available products. The only real change in this model up until recently has been slicker photos of the products.

However, with the arrival of LLMs, it’s now possible to allow users to place products into a picture of their own room and to interactively swap out products using natural language. Certain objects added within an image can be queried based on where you click within that image. LLMs can be used to improve product description summaries and a whole host of other things (blue squares on right). Various other modelling opportunities with collected data present themselves (yellow squares on right), taking the degree of sophistication around customer interaction to new heights.

While all the focus these days seems to be on LLMs, a much broader set of models will become important in the coming years. This includes:

• Multi-modal - where text, audio, images and potentially video are handled by a single model. Use-cases, based on our simple furniture retailer, might include an image-generator (show me a leather sofa) and object-recognizer (what is this lamp that has been placed in the image).
• Hybrid Models - this is where pre-existing machine-learning models can be further enhanced with richer data-sets aided by LLMs that can help surface that data from unstructured sources (see Figure 6).

• Advanced Data & ML
  • causal marketing model - a sub-domain of models dedicated to better understanding the cause and effect of interactions with end-consumers intended to prompt some action. Should we offer them a discount? Are they more likely to desire a leather sofa?
  • LTV estimate - life-time value calculation is the predicted revenue generated by a customer over the entire relationship with a company. This will be an important factor in deciding the level of customer acquisiton cost you might be able to afford. LTV = ARPU x Customer Lifetime or LTV = ARPU/User Churn
  • Logistics optimzer - which products are most profitable to sell to consumers and easily available/deliverable.
  • Inventory forecasting - how much inventory of each product should we ideally be holding?

LLMOps is more than an Extension of MLOps

A single prompt fed into an LLM might produce varying responses over time. This can lead to inconsistencies in applications powered by LLMs. LLMOps implements strategies to manage output consistency. Unlike traditional ML models, which operate on structured data, LLMs handle the vast and often messy world of text and code. This introduces a new layer of complexity, which demands special techniques for data ingestion, pre-processing, and training.

While MLOps addresses the principles and practices of managing various ML models, LLMOps focuses on the distinct aspects of LLMs, including their large size, highly complex training requirements, prompt management, and non-deterministic nature of generating answers.²⁰

I’m not about to go down a rabbit-hole and teach you about LLMOps, when others have done that more eloquently (follow the linked footnote above). What I will do, however, is point you towards a group of practitioners in this field (Applied LLMs) that have assembled an invaluable resource mid 2024 that runs through their experiences and ‘lessons-learned’ from a year shipping LLM applications²¹, with a video version here²². They split their learnings under Tactical, Operational and Strategy headings. We’ve tried to summarise the tactical part in the grey box within the embedded PDF below. I would encourage you to read the full note, linked above.

Takeaways from the piece I found most compelling include:

• The model isn’t the product, the system around it is - For teams that aren’t building models, the rapid pace of innovation is a boon as they migrate from one SOTA (state-of-the-art) model to the next, chasing gains in context size, reasoning capability, and price-to-value to build better and better products.
• calibrate risk tolerance based on the use-case - for less critical applications, such as a recommender system, or internal-facing applications like content classification or summarization, excessively strict requirements only slow progress without adding much value.
• Focus on model evaluation - LLMOps is about production monitoring and continual improvement, linked by evaluation. The better your evals, the faster you can iterate on experiments, and thus the faster you can converge on the best version of your system.
• Empower everyone to use new AI technology - while it may seem expensive to have a team spend a few days hacking on speculative projects, the outcomes may surprise you.

LLM Evals and Monitoring

Among the practitioners alluded to earlier, Hamel Husain blogs²⁴ extensively on LLMs, often with very pertinent content based on his direct consulting experience in implementing AI systems with clients. His big insight (per Figure 7) is that improvement requires process — you need to be evaluating your LLMs constantly in order to iterate for their improvement. Evaluation of a customized LLM against the base LLM (or other models) is necessary to make sure the customization process has improved the model’s performance on your specific task or dataset.

He’s not a big fan of off-the-shelf model evaluation tools. Rather, there can be lots of domain-specific data that needs capturing, so it can often make sense to customise a data-reviewing tool using a simple front-end like Gradio, Streamlit or Shiny. Such an app can also encompass labelling, allowing for capturing human review. A central message is to remove any friction in the process of being able to look²⁵ at your data produced by your LLM app.

LLM-as-a-judge can be helpful as a bolt-on capability to a functioning eval system, but it is really important that it is aligned to a human. Use experts to label LLM-as-a-judge output until this judge-human alignment is sufficiently good. The important thing is not to revert to using this approach too early before exhausting the prompt-engineering and fine-tuning steps. He’s also adamant that you need to design metrics that are specific to your business, along with tests to evaluate your AI’s performance. The data you get from these tests should also be reviewed regularly to make sure you’re on track. There’s a presentation²⁶ on this approach worth watching.

Mental model for improving AI systems

The key insight is that you need both quantitative and qualitative feedback loops that are FAST. You start with LLM invocations (both synthetic and human-generated), then simultaneously:

1. Run unit tests to catch regressions and verify expected behaviors. People tend to skip this stuff, however it is the foundation for good evaluation systems. You can potentially use LLMs to synthetically bootstrap test-cases.
2. Collect detailed logging traces to understand model behavior. Log your traces to a database of some sort and spend time examining them.

These feed into evaluation and curation (which needs to be increasingly automated over time). The eval process combines:

1. Human review
2. Model-based evaluation
3. A/B testing

The results then inform two parallel streams:

1. Prompt engineering improvements - with a minimal evaluation system, you want to begin to iterate on your AI with prompt-engineering. Determine if your test-coverage is good. Are you logging your traces correctly?
2. Fine-tuning with carefully curated data - by having a system to capture human evaluation (labelling good and bad cases), this curated data has the potential to be leveraged for fine-tuning a model, driving further improvement.

A useful case-study from the wild on LLM usage²⁸ at Clearwater Analytics and their rigorous LLM evaluation process is available from this blog²⁹. This is evidence of the lengths to which companies, utilising LLMs, will go to ensure that their solutions are particularly robust. We’ve captured the essence of the Clearwater evaluation process in the box below, but we would point you back to the linked blog for more detail. We also have an embedded one-pager lower down, summarising a series of really thoughtful pieces put out by their engineering team. For those looking for a larger-scale evaluation framework that already fully-formed, we would urge your to study this Sagemaker-based³⁰ approach.

Better Flow Controls for LLM Apps

We’ve covered alot of ground here, so we’ll close-out with a final nod towards some novel opensource tooling designed to aid the development process of LLM Apps, especially ones that expect to leverage calling out to agentic systems. Those with greatest experience in developing LLMs articulate that common friction points with GenAI applications can include logically modeling application flow, debugging and recreating error cases, and curating data for testing/evaluation. The developers of Burr say these problems all got easier to reason about when they modeled applications as state machines composed of actions designed for introspection. An application will hold state and make decisions off of that state. You can therefore capture your application as a state-machine, modifying the state as you go. The nice thing about this tooling is that it’s light-weight, it doesn’t seek to obfuscate function calling (as some other agentic frameworks do) and it has nice byproduct properties allowing for data-gathering for evaluation. We’ve tried to capture the essence of the solution in the one-pager embedded below.

That concludes our intial AI Strategy series. We’ve probably just scratched the surface on various topics, but hopefully we’ve sketched for you an outline for an approach to get started with leveraging AI for your business. We hope to follow-up with another series that gets more technical in nature and that builds upon this foundational perspective we’ve shared.

Recap So Far

By way of recap of this AI strategy blog-series, so far we have covered off skaffolding a strategy, planning through documentation, the importance of finding a relevant business-application for experimentation and ultimate productionisation, as well as the necessity of evolving a solid platform on which to be able to iterate on AI solutions, scale them, and weave them into the fabric of your business. We will continue that journey with two more blog-posts in this series, focusing initially on all things DataOps-related, before diving into the world of LLM-based application development and its promise, pointing to important lessons-learned so far from respected practitioners in this field.

Composable Data Systems

Wes McKinney, the original author of the pandas library and a long-time advocate for better opensource data analysis tooling, penned an important blog⁴ at the end of 2023 that took stock of the culmination of many years of effort to arrive at a better, more composable tool-set.

While compute and networking speeds have soared over the past decade, data-analytics has lagged badly - the bottleneck often being CPU-bound, given a typical, up to now, need to serialize and de-serialize data moving between different ecosystem layers. Without a standardized solution for data interchange and in-memory computation, systems pay a steep penalty both in computational cost and development time to interoperate with each other. While libraries like pandas had been transformational for the data-analytics profession, it was generally accepted that it also had performance, scale, and memory-use problems. In McKinney’s words, pandas had to do everything for itself, and this was an enormous burden for a fully volunteer-based open source project.

Hence his involvement in various parallel projects to solve these problems, culminating in The Composable Data Management System Manifesto⁵ and a more detailed version articulated by Voltron Data (the driving force behind the open-source Ibis project) who have assembled a well-articulated technical series called The Composable Codex which brings a useful historical context to what has needed to happen to unleash better productivity around the data ecosystem, all enabled by open standards for exchanging and operating on data, allowing for more composable data systems being built that are modular, interoperable, customizable, and extensible (MICE). While we summarise some key concepts from this manifesto below, I would urge readers to read the Codex in full for a greater understanding of the importance of this seismic shift in thinking that will underpin how companies perform DataOps into the future.

Even before the rise of LLMs, there was a real need for faster, more scalable, more cost-effective machine learning, as volumes of data grew exponentially. A before and after illustration of this shift to a better data ecosystem is probably well illustrated by NVIDIA’s tongue-and-cheek portrayal of a day in the life of a data-scientist, prior to GPUs. The data ecosystem changes are obviously more than just GPUs, but are probably best summarised as Accelerated Workflows, in general, enabled by tooling that operates on layers and as standards between those layers envisaged by the composable data system.

• Standards + Layers
• Ibis as Composable UI
• Substrait as IR Standard
• Evolving Data Connectivity

The Composable Codex has a concept of Layers (1, 2 and 3) and Standards (A, B and C).

Layers:

• User Interface [1] - Users interact with this UI in order to initiate operations on data. This is typically exposed as a language frontend or API.
• Execution Engine [2] - this engine performs operations on the data, as specified by users.
• Data Storage [3] - the layer that stores data that is available to users.

Standards:

• Substrait [A] - is a format standard for describing compute operations on structured data.
• ADBC (Arrow Database Connectivity) [B] - applications code to this API standard (like JDBC or ODBC), but fetch result sets in Arrow format.
• Arrow [C] - is focused on a standardized memory representation of columnar data.

Cycling through the series of tabs on the left, The Composable Codex envisages a minimal viable data system as broken down into three main layers: UI, a query execution engine and a data-storage layer. Between those layers are standards, intended to ease friction between those layers and enhance interoperablitity for data (common data-structures), queries (common formats) and systems (serialization/data-interchange). On the second tab, we see projects like Ibis which works by decoupling the dataframe API (python) from the backend execution. Most backends support a SQL dialect, which Ibis compiles its expressions into, enabling queries (driving complex ETL) to be written once and work against multiple backends. Ibis uses Apache Arrow to provide a common data format for data interchange between Ibis and backends that is fast. Ibis is also able to compile queries to Substrait, an evolving cross-language serialization protocol. On the right-most tab, we see an evolution of status-quo systems (squareDB) towards using this fully-integrated modern data-tooling (triangleDB) resulting in vastly improved data-stack performance.

Embedded Databases as viable Serverless Solutions

The client/server architecture for databases has been around for a very long time, and has been proven to be a successful commercial model, which is why they are the norm in large-scale production use cases. However, they aren’t well adapted to serverless use-cases, since they require to be on all the time, incurring server costs, even if they aren’t being actively used. It turns out that new embedded database model solves this problem, so we should sit up and take note. There are a number of very informative blog-posts ¹⁴ on embedded-databases that are recommended reading. We also include a one-pager (embedded below) that discusses some of the tools out there that deliver compelling value across relational, graph and vector data-model paradigms.

An embedded database is an in-process database management system that’s tightly integrated with the application layer. The term “in-process” is important because the database compute runs within the same underlying process as the application. A key characteristic of embedded databases is how close the storage layer is to the application layer. Additionally, data that’s larger than memory can be stored and queried on-disk, allowing them to scale to pretty huge amounts of data (TB) with relatively low query latencies and response times.

The evolution of modular data-tooling has been an important driver of the new database models, with Arrow as a de-facto standard for efficiency gains with in-memory and on-disk capabilities that are core features of this new class of embedded database. The embedded architecture is still relatively new, at least for OLAP databases, nevertheless vendors are delivering rich feature-sets as open-source solutions, even as they figure out their monetization strategies.

Being able to dispense with traditional databases in favour of cheaper object-stores, such as S3 is a game-changer. It means you can work with ever-scaling amounts of data, that working with AI-based solutions requires, benefit from the low-cost object-store model and get to keep the query-processing power that embedded-databases bring. You also don’t have to compromise on query latency, with the advent of more performant variants such as S3 Express.

The Ubiquity of DuckDB

Wes McKinney also posited here¹⁵ that the ability to add cutting-edge analytic SQL processing to almost any application is a disruptive and transformative change for our industry — in effect, that every DBMS will have the same vectorized execution capabilities that were unique to Snowflake ten years ago.

From Figure 5, sitting alongside the likes of Snowflake on the execution-engine layer of our composable data system is DuckDB, which has become a ubiquitous component of most data-stacks for its versatility. First introduced in 2019, akin to SQLite, it allows you to store your data in a single .duckdb database file, making it easily portable within other projects. It dispenses with the client-server convention — all you need to do is point to where the single DuckDB database file is stored (that could be remotely), or start an in-memory session without even requiring to use a physical database. In fact, here is a great illustration¹⁶ of what’s possible with DuckDB.

Alluding to our earlier discussions around the importance of standards, DuckDB has an ability to run SQL queries directly on Python dataframes, enabling it to query Pandas, Polars and Apache Arrow dataframe objects as though they were SQL tables. In the case of the latter, it leverages a zero-copy-mode that dispenses with the data-serialisation overhead of old, often a bottleneck for data-tooling. With DuckDB, you can directly query DBMS systems like MySQL and Postgres, open data files like JSON, CSV, and Parquet files stored in cloud storage systems like Amazon S3, and modern open table formats like Apache Iceberg and Delta Lake. For illustration, below is a basic query running on DuckDB that encapsulates a data pipeline to push parquet files from a public to private bucket and then write an aggregation query to update that private data-source.

Below are a selection of one-pagers relevant to our discussions derived from a library I maintain around these technologies for my own and my clients’ reference. These are intended as summaries and don’t look to cover a topic thoroughly, but often link off to more detailed resources.

• Embedded Databases for Serverless
• Modular Data Tooling
• New Generation of Pipeline Tools

Download PDF File

Download PDF File

Download PDF File

New Generation of Pipeline and Orchestration Tools

The Composable Codex envisaged three layers of a minimum viable data-system — data, execution and UI (expression). The DAGWork’s variant is presented below, where they include an asset layer that is responsible for structuring code into assets that are meaningful to the business and, for producing them, the orchestration layer that triggers their computation.

• Data: the physical representation of data, both inputs and outputs
• Execution: perform data transformations
• Expression: the language to write data transformations
• Asset: an object in persistent storage that captures some understanding of the world (e.g., a dataset, a database table, an ML model, dashboards)
• Orchestration: operational system for the creation of assets

Pipeline solutions like Airflow, Dagster, Hamilton and Kedro, all use a directed-acyclic-graph (DAG) to represent a pipeline, specifying how ‘assets’ (a node within the DAG) relate and depend upon eachother. Imperative systems (like Airflow) require developers to expressly tell the orchestrator ‘what to do’. Declarative ones (like Dagster and Hamilton), declare what can be computed and each function infers its dependencies.

Common to all tools we profile here is the ability to run and test locally, but also then the possibility of running them in-production in the cloud, usually without any need to adjust the code much, if at all, making them efficient to work with. Refer to the ‘New Generation of Pipeline Tools’ tab above, for an embedded one-pager that discusses these tools that are integral to the composable data system in more detail.

Envisioning a Tech Stack

Up to this point, we haven’t talked much about technology, for good reason — there are important structural and cultural elements that needed to be in place in an organisation so that the technology lands on fertile ground. Working with technology in pursuit of, let’s say, a machine-learning driven business, is complex and involves knitting together many systems and tools and then potentially running them in clusters in the cloud, which have a host of complexities themselves. But as we have laid-out in the first blog of this series, it’s possible to do things in stages and, in fact, you can get a long way with some basic components in place.

Platform-engineering is about simplifying that journey for users in general — not just developers or data-scientist, per se. For machine learning to pervade the culture of an organisation, everyone in an organisation should be able to discover and experiment with these solutions. Having a common platform to do this can be transformative, since it reduces the barriers to entry for those less tech-savvy, but nonetheless motivated to learn more technically in the field of machine-learning and AI more generally.

In my own learning journey, I’ve experimented with lots of technologies and built-out my own blueprint platform (Figure 1), including leveraging Backstage¹, in order to reap the benefits that I’m articulating here. More on that later.

For organisations with geographically spread businesses, or indeed private-equity firms with disparate portfolio companies ripe for collaboration in machine-learning, then establishing one of these platforms centrally and hosting it in the cloud for each business to access their own tech artifacts securely, could be a good solution to seeding the organisation with a culture towards machine-learning more broadly.

Let’s not get ahead of ourselves though. We need to think about what other primitives need to be in place before a solution like Backstage can thrive. Let’s address some of the components that sit under Backstage in our simplified system in Figure 1.

Platforms for Provisioning MLOps

In pursuing an MLOps (machine learning operations) solution for usage in an investment research context, I’ve experimented with many different tools in what has become a very crowded space. Many of these tools are the same technologies, repurposed for a new set of tasks - think vector databases. Nevertheless, combining a mix of these tools in a modular way, while limiting the cognitive overload for developers that now have to contend with new complexities that micro-services and cloud-native technologies have unleashed, is challenging.

I’ve tended to follow the well-beaten track of open-source solutions and many of these pervade in the most innovative parts of the ecosystem, but hosting them in conjunction with eachother generally requires you to immerse yourself in the world of kubernetes, which brings with it a whole other set of technical challenges.

The alternative is to go all-in with cloud-solutions, where the likes of AWS offer a suite of pre-built machine-learning tools, but these, at scale, can become very expensive.

The stage of MLOps maturity, something I introduced in the first blog of this series, has an important bearing on the sophistication of the solution too. In the Initial phase, disparate tooling used for different parts of the value-chain doesn’t all have to fit seamlessly together. There are more manual interventions in the chain to get things working. However, by the Scalable phase, everything needs to be fully integrated and automated.

There’s a thorough review of the MLOps tooling landscape from Neptune.ai³ where they review 90 tools across 17 categories. An important take-away from the article is the number of open-source solutions that continue to thrive in this space. The modus operandi seems to be one to lure developer interest via open-source and then upsell users to selected enterprise features and offer support.

An organisation’s size could also be a determinant of the complexity of a solution and whether it is Saas or self-hosted. At one end of the range are companies experimenting with ML models or just starting to scale their first few deployments. At the other end are businesses that need powerful capabilities to develop and operate sophisticated ML models in production that drive mission-critical business applications deriving value from massive amounts of harvested data.

A common thread underpinning most of these solutions is that they all mostly leverage containerising workloads, a pivotal technology introduced by Docker way back in 2013. The ubiquity of containers then drove the industry towards solutions around orchestrating these containerised components, where it seems the consensus has largely coalesced around Kubernetes⁵ as the preferred technology upon which to build platforms, even ones capable of running across different cloud providers.

Internal Developer Platforms

While cloud native drove huge improvements in areas like scalability, availability and operability, it also meant setups (applications or running workloads) have became a lot more complex. Expecting developers to have deep knowledge around trouble-shooting code run on cloud-based clusters makes them less productive. Using the lingo, it’s not realistic that every developer (or data-scientist, in our context) can be full-stack. Enabling a team of developers (or data-scientists) to better operate in the cloud but retaining high-enough levels of controls around meeting a given security posture has driven a shift towards what is commonly referred to as ‘platform engineering’.

The open-source IDP (internal developer platform) I have most experience with is Backstage, as per Figure 1, where it’s used as a front-end to control running existing workflows and creating new ones (via argo-workflows). I’ve included a one-pager on key features of that solution below, so I’m not going to elaborate too much here except to say that it is a versatile solution that offers a single pane-of-glass that sits within an organisation allowing users to document and control any resources running either internally or across various cloud-providers. As you can see, data-science oriented businesses like Zalando⁶ have used this platform as the basis for building-out their whole MLOps stack.

One of the easier ways I have found to get up-and-running with Backstage, in conjunction with other core operating components (ArgoCD, Gitea etc..) is to leverage the idpbuilder tool. We have a separate one-pager on that below (see other tab), but in short it enables you to develop IDPs locally, running them initially on a kind cluster, allowing for debugging combining various components together ahead of a production-grade deployment on a larger cloud-based cluster. That local-to-cloud handover isn’t quite seamless yet, but the tool is very promising. Various stacks (combinations of tools) are contained at github.com/cnoe-io/stacks, allowing new users to get a sense of the capabilities.

Onyxia Datalab is another interesting open-source platform for data-science developed by INSEE, the French National Institute of Statistics and Economic Studies. It resembles Backstage in part, especially Service Catalog, which is essentially a libary of tooling as helm-charts, that get launched into a kubernetes environment on your users’ behalf. They have control allocating RAM, CPU and GPU to specific containers. The system is already tightly-integrated with S3 for data and Vault for secret management. For teams that have restrictions on exposing their data to cloud-based Saas systems, this might be an ideal proving-ground system for intial proof-of-concept work.

Importance of a fast Feedback Loop

In addition to the MLOps platform solutions, as we have learned, there are many other Saas⁷ offerings that, akin to idpbuilder, offer this ‘develop locally, deploy anywhere’ hybrid capability. However, while idpbuilder is concerned with running-up technology stacks, these new-generation MLOps Saas solutions are all about creating encapsulated workflow jobs or pipelines, often defined in Python, utilising some @decorators to bring special functionality, like specifying cloud-based assets such as container runtimes with GPU access or defining a serverless cloud-based function (such as AWS Step Functions) to perform some batch job. I’m most familiar with Metaflow, but for a deeper dive into comparable solutions, have a look here⁸.

What each of these solutions promise is an ability to build and debug these pipelines locally and then deploy them to run on more robust, more available or faster remote compute. Particularly in this era of LLM application development, none of us has much access to local GPU, so its imperative we have a way of running pipelines efficiently, often in a distributed way. Figure 3 depicts a typical 7-step MLOps process that often forms a working ML system. Generally speaking, the idea is to be able to get each of the steps working efficiently, with as short-as-possible feedback loops, so that developers can fix or refine those parts accordingly.

On certain cloud-based infrastructure set-ups, let’s say we have Kubeflow running on an EKS cluster, it can sometimes take 30 minutes to have a stable running cluster and then troubleshooting jobs, in particular those looking to distribute workloads across multiple workers in parallel, can be a time-consuming and frustrating process.

I have been recently prototyping with one of the new-breed of serverless GPU platforms called Modal. It’s well worth your time spending 30 minutes watching this MLOps on Modal⁹ run-through, rather than me trying to explain things here. The bottom-line is that their infrastructure is laser-focused on this fast feedback loop concept, in terms of being able to run workloads fast in a distributed way, but only having to pay for the seconds of actual GPU-usage, as opposed to you having to secure, then manage a fleet of VMs with GPUs. You can have deployments on web endpoints that will only start on receipt of an inference request, ensuring compute spend is kept to a minimum. There’s another good run-through here¹⁰, especially pertaining to running multiple workloads in parallel for faster execution.

MLOps Iterative Process

These are the actual processes involved in performing machine-learning operations, as opposed to the components used for MLOps. This workflow is modelled on the Practioners Guide to MLOps¹¹ from Google. The processes can consist of the following:

1. ML development concerns experimenting and developing a robust and reproducible model training procedure (training pipeline code), which consists of multiple tasks from data preparation and transformation to model training and evaluation.
2. Training operationalization concerns automating the process of packaging, testing, and deploying repeatable and reliable training pipelines.
3. Continuous training concerns repeatedly executing the training pipeline in response to new data or to code changes, or on a schedule, potentially with new training settings.
4. Model deployment concerns packaging, testing, and deploying a model to a serving environment for online experimentation and production serving.
5. Prediction serving is about serving the model that is deployed in production for inference.
6. Continuous monitoring is about monitoring the effectiveness and efficiency of a deployed model.
7. Data and model management is a central, cross-cutting function for governing ML artifacts to support auditability, traceability, and compliance. Data and model management can also promote shareability, reusability, and discoverability of ML assets.

The Importance of Envisaging Personas

An integral part of the problem-to-solve discovery (Step 1) in the domain reimagination process above is the ‘user unmet need’. That user could be internal (an employee) or external (a customer). Georgian, a Canadian VC specialsing in B2B software investments, operate a ‘growth network’ of potential portfolio investments and they produced a useful analysis of internally-facing personas and their intended GenAI use-cases. I’ve linked to the original blog-post in Figure 5.

Based on my own engagement with clients in the finance and VC domains, I’m including a few one-pagers that I assemble as part of a larger deck, aggregating various considerations that a persona in those fields might consider. This could be an exercise prior to delineating-out user needs and associated problems-to-solve, per the McKinsey framework described above. The Finance Executive persona (below) draws from a useful report from Celent¹⁰ which echoes what we saw in Figure 2, where the tendency among corporates (and banks, in this case) is to develop GenAI applications for an internal audience, only planning a roll-out for external users much later, when presumably rough-edges and hallucination risks can be mitigated.

Change Management Required to Embed AI Usage

In the first installment of this blog-series, we posited that the professionals involved in the day-to-day business of running a function, product or solution are the ones that will have the ideas for how things can be done differently, aided by AI, so it makes sense that they are also the ones that have a sense for what is viable, technologically-speaking, too — and develop the requisite skills to be able to build prototypes more easily. Enabling that in reality shouldn’t be underestimated and relies on the less glamorous and more arduous task of change management.

The framework presented below borrows from Five Frames of Performance and Health, Beyond Performance 2.0, a seminal publication from partners and practitioners at McKinsey. Why the delineation between Performance and Health, you might ask? I suppose the analogy in their book is useful: for someone looking to run a marathon for the first time, for each performance step, there’s an equivalent health step that determines the actions to be taken for that objective to become a reality.

This all might seem a little overkill, but it acknowledges the fact that as much as AI might be truly tranformational for businesses reliant on knowledge-workers, expecting them to understand it and fully embrace it without requisite support, isn’t going to be realistic. Digital transformation will probably only succeed with culture transformation within an organisation, requiring some form of change-management.

A central idea behind this Five Frames approach in delivering successful change at scale is that leaders should put equal emphasis on performance- and health-related efforts — where Performance is ‘What an enterprise does to deliver to its stakeholders in financial and operational terms’, while Health is ‘How effectively an organization works together in pursuit of a common goal’.

The five performance frameworks are:

1. Aspire: Strategic objectives. Create a compelling long-term change vision, roll back the future to a mid-term aspiration, and guard against biases in the process.
2. Assess: Skillset requirements. Forecast skill “demand” and understand skill “supply” dynamics, and then identify how any gaps will be closed.
3. Architect: Bankable plan. Define the portfolio of initiatives to deliver on your strategic objectives and fulfill your skill requirements, then programmatically sequence actions and reallocate resources to deliver.
4. Act: Ownership model. Establish strong governance; decide how to scale up your portfolio of initiatives; monitor progress and dynamically adjust as plans are implemented.
5. Advance: Learning infrastructure. Institutionalize processes and expertise to enable knowledge sharing, continuous improvement, and continuous learning to characterize the day-to-day workings of the organization going forward.

The five health frameworks are:

1. Aspire: Health goals. Objectively check your health, choose where to be exceptional, and target any ailing areas in need of immediate improvement.
2. Assess: Mindset shifts. Pinpoint the helping and hindering behaviors related to health priority areas, explore the underlying mindset drivers of these behaviors, and name and reframe the critical few “root-cause” mindsets.
3. Architect: Influence levers. Reshape the work environment to influence needed shifts in mindsets and behaviors by hardwiring health interventions into performance initiatives and interactively cascading the change story.
4. Act: Energy generation. Mobilize influence leaders, make the change personal for a critical mass of leaders, and maintain high impact two-way communications.
5. Advance: Leadership placement. Prioritize roles by value-creation potential based on the go-forward strategy, match the best talent to priority roles, and operationalize the talent match process to ensure it’s regularly revisited.

Setting up Machine Learning projects for success

Leading on from the points above, part of the health of an organisation is having effective pillars in place that can underpin efforts to make machine-learning adoption across a business more effective and more likely to succeed. There are degrees to which this can be implemented. At a basic level, following industry best-practices is a good start and this eminently sensible guide ¹² offers a good starting point. When it comes to educating the workforce on best-practices, there’s little point in reinventing the wheel. There are some excellent open-source resources out there from which to learn, among them those from the Turing Institute¹³. I’ve taken the liberty of summarising some of the points around ML project structure from both in the one-pager below. I would urge you to delve into the source too. By all means these can be honed for a given industry or organisation, but as a foundation, they’re solid.

In the next blog-post in this series, I’ll be talking about platforms, which is where an organisation will begin to formalise their own best-practices around working with AI.

Forming a Strategy around Machine Learning / AI

If you were to believe the hype, every business executive these days is ‘doing’ AI. It has become a byword for sophistication in every product or service we might want. AI has become a veneer for everything - irrespective of whether a process involving the usage of artificial-intelligence was actually invoked in the production of a given product or the provision of a given service.

The MAD (ML, AI & Data) Landscape¹ produced by venture capital firm Firstmark, is considered a key annual survey of the AI ecosystem. Solutions tackling every niche are battling out for what Gartner forecast² to be a $300bn market for AI developer tools within three years. As a business setting out on an AI journey, where are you supposed to start?

While there has been a degree of hype around Large Language Models (LLMs), unleashed by OpenAI’s release of ChatGPT onto the market in late 2022, the effect has been to shine light on the value that can be derived from a more rigorous approach to machine learning and AI, especially one that is knitted into the fabric and culture of a business.

Stage your adoption

While it was offered in the context of working with Large Language Models (LLMs), Georgian’s ‘Crawl-Walk-Run’ analogy (below left) applies more generally to an approach to AI adoption that can be calibrated to an organisation’s overall level of sophistication with these technologies. It’s acknowledged that there should be a staged approach towards adopting AI and adapting a business to then leverage AI. As a slightly more formal categorisation, the MLOps Maturity Model (below right) from AWS is appealing for its articulation around a phased approach to working with Machine Learning / AI:

• Initial phase — a secure experimentation environment, data-enabled, for internal users to build proof-of-concepts around solving a specific business problem with ML. AWS present their Sagemaker solution, but this could just as easily be a self-hosted JupyterHub, with connections into S3-hosted data assets, allowing users to collaborate on Jupyter notebooks.
• Repeatable phase — this typically involves productionising what has proved effective as a PoC in the phase above. Code in notebooks get formalised as more robust pipelines, split-out typically as feature, training and inference. In conjunction, code repository structures are formalised as are data repository (S3 bucket) structures, for containing models and model-artifacts. This approach supports complete auditability of every experimentation. An AWS sample environment with these structures, deployable via CDK is available at aws-samples/aws-enterprise-mlops-framework. I will discuss other lighter-touch MLOps solutions like Modal in part three of this blog-series.
• Reliable phase — even though the models have been generated via the ML pipelines, they need to be tested before they get promoted to production. An automatic testing methodology is introduced, for both the model and triggering infrastructure, in an isolated staging (pre-production) environment that simulates production.
• Scalable phase — after the productionisation of the first ML solution, scaling of the MLOps foundation to support multiple data science teams to collaborate and productionize tens or hundreds of ML use cases is necessary.

Documentation as a Framing Mechanism

The landscape around AI/ML is evolving rapidly. There are new terminologies, new technology categories - a seemingly unending stream of solutions to problems we didn’t even know we had. As with most things, it can be helpful to break them down into smaller addressable parts and to acknowledge who your end audience is. That’s what I try to do with the documentation hierarchy, presented below, which I find helpful when engaging with clients. Levels 1 through 5 differ in terms of technical detail, tackling high-level concepts for business at the top and delving into technical implementation detail for technologists towards the bottom. Spanning horizontally are the delineated components of a MLOps value-chain. Think of the elements in Level 2 as the assets one draws on when evolving a machine-learning capability, while Level 3 are the processes undertaken (think in terms of a continuous life-cycle or fly-wheel) that sweat those assets for the delivery of business value-add.

1. High-Level Conceptual Docs - answering questions like, what problems are we trying to solve with ML/AI?, do these technologies open new markets for our business? if we don’t respond, can competitors leap-frog our position technologically?

2. ML/AI Value-chain - this is our approach to breaking-down the various components that come together to form a system for implementing ML/AI solutions.

3. MLOps Processes Lifecycle - there are many varied MLOps-process lifecycle frameworks in the wild. We have found this particular model from Google’s Practitioners Guide to MLOps⁵ to be more instructive than others.

4. Proof of Concepts - these are proof-of-concepts involving combining different technology stacks and solutions either within a delineated part of the value-chain or across multiple parts.

5. Projects / Implementations - these are short project descriptions that depict approaches to encapsulating business processes as self-contained pipelines or where technologies are bundled as app-sets to solve particular business requirements.

This documentation hierarchy could also form as a framework for planning an AI strategy within an organisation. Level 1 is where high-level corporate aspirations can be set for what transformations are envisaged with these new technologies. Planning around building machine-learning operational capabilities and processes can be formed in Levels 2 and 3, respectively. Then proof-of-concepts (Level 4) involving different coupling of technologies can be explored followed by actual project MVPs (minimum-viable-products) (Level 5), which have been prioritised for implementation at the higher levels. Based on feedback and learnings from these early implementations, project-planning and prioritisations at Level 1 will adjust accordingly, perhaps elevating certain successful MVPs to more widely-scoped production-ready implementations.