Embedded Analytics: Integrating Analysis With the Business Workflow
Author: Donald Farmer
Date: 2023-04-30T05:09:14.677062+00:00
Description: Over the past 10 years, data analytics and data visualization have become essential components of an enterprise information strategy. And yet, the adoption of data analytics has remained remarkably static, reaching no more than 30% of potential users. This book explores the most important techniques for taking that adoption further: embedding analytics into the workflow of our everyday operations. Author Donald Farmer, principal of TreeHive Strategy, shows business users how to improve decision-making without becoming analytic specialists. You'll explore different techniques for exchanging dat
Publisher: O'Reilly Media

Embedded Analytics

Integrating Analysis with the Business Workflow

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

Donald Farmer and Jim Horbury

Embedded Analytics
by Donald Farmer and Jim Horbury

Copyright © 2023 TreeHive Strategy. All rights reserved.

Printed in the United States of America.

Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.

- Acquisitions Editor: Michelle Smith
- Development Editor: Jeff Bleiel
- Production Editor: Kate Galloway
- Interior Designer: David Futato
- Cover Designer: Karen Montgomery
- Illustrator: Kate Dullea

- June 2023: First Edition

Revision History for the Early Release

- 2022-09-01: First Release

- 2022-10-25: First Release

See http://oreilly.com/catalog/errata.csp?isbn=9781098120931 for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Embedded Analytics, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the authors and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-098-12087-0

Chapter 1. Introduction to Embedded Analytics

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 1st chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the authors at feedback.embeddedanalytics@gmail.com.

Most of us are familiar today with business intelligence (BI). At one time, it was a new and exciting capability, but now, thanks to self-service technologies, the cloud and the power of in-memory processing, richly featured analytic applications, data visualisations, reports and dashboards are available to almost any business user who wants wants them.

However, each of these capabilities typically depend on separate applications. To perform business analysis, you need to open your BI suite. If you want to create a special charting, you may a look to a data visualisation application.

Embedded analytics takes a somewhat different approach. The aim of embedding is to integrate visualisations, dashboards, reports and even predictive analytics or artificial intelligence capabilities inside your everyday business applications. So if you are managing a production line, preparing a budget or reviewing HR issues you can have analytic insights ready to hand to guide you.

For a business user, this means that the analytic process takes place in the context of a business process. It also means that the data and the analytic experience becomes part of your everyday work. This makes analysis more accessible to a business user.

In this chapter, we will review some of the benefits of analytics for enterprises and we’ll also consider what makes for a successful implementation. With that knowledge, we will also look at how we could map out a strategy to take us from our current situation to being a more fully-informed, analytics-driven enterprise.

Analytics for business users and consumers

There are several advantages to embedded analytics which business users and IT teams appreciate.

From the point of view of an IT team, who are very concerned with issues such as security and data governance, embedding your analytics in another application which is already secured has a simple but significant benefit. There is only environment and one login to be managed; the analytic application does not add an additional layer of complexity, We will look at this topic in more detail in Chapter 8.

Business users will be most familiar with their existing operational applications. Embedding analytics in this well-understood environment affords a non-specialist user insights and information in a way that works with their existing skills and practices.

In addition to being familiar, it is also more efficient to embed analytics in a commonly used application. If we need to switch from our working application to an analytic application, we interrupt our workflow. In fact, we lose focus a little while we’re doing it, in addition to the simple hassle of opening and navigating to another application.

How does this work in practice?

In a call centre, an operator taking customer calls for support can see on one screen not only the customer record, but (thanks to the embedded analytics) but also trends and patterns related to this and similar cases. These patterns - perhaps the customer is calling increasingly often over time - may prompt the operator to route the call to a specialist who can better diagnose and resolve the underlying issues at hand. The result should be a better resolution for the customer and more efficient handling by the call center.

A salesperson on the road with an application that helps them select the next customer to visit and optimise the offers they make could see, embedded in that application, even on a mobile device, a recommendation driven by machine learning.

Other applications may sound more advanced, but in principle are very similar. A doctor may use an embedded algorithm to assist with diagnosis. A production-line manager in a busy factory needs may need to compare the performance of different machines in order to see where improvements can be made. An HR person, interviewing an employee with performance issues, may need to compare their work and achievements with peers within and across organizational boundaries in order to suggest a best course of action.

Embedded analytics for consumers

We can see some of these principles not only in our work but in our personal use of technology as consumers. Although we grapple with a different set of challenges, as consumers we still make decisions and take actions. As our consumption increasingly moves on line, so do our decisions and actions. In e-commerce, you expect to be given recommendations of products that may interest you, the famous example being “customers who bought this also bought that.” These recommendations are in fact analytics, embedded inside the web experience of e-commerce site. We don’t think, when browsing for books or clothes online, that it’s an analytic experience. Yet analytics are driving our interactions in a way that is so seamlessly embedded, we don’t even notice it.

The same is true if we’re watching movies online. Our watch-list likely includes recommendations made by the artificial intelligence engines of the streaming company. Again, the analytics are there, but seamlessly embedded.

In business we often want to be making decisions that are consciously data-driven. Some users are very interested in their data and like to explore it and make new finding. However, we can learn from a consumer experience when we’re dealing with users who must make operational decisions very quickly with the least possible distraction. In that case, an alert or a recommendation can be served up without drawing unnecessary attention to the analytics behind it.

So, one thing we need to think about with embedded analytics is what we’re trying to achieve. Are we looking for a simple business outcome? Or are we informing a user with an analytic experience that enriches their use of data, their analytic thinking and their understanding of the business?

What success looks like

From this, we can say that what makes for a good embedded analytic experience is somewhat variable. In many cases, the best analytic experience will be one that is more or less invisible to the user, like the recommendation engine.

In other cases, we wish to prompt the user into a mode of analytic thinking. Having the analytics or data visualisations right there where they are currently working, makes that process more seamless, and we might hope, will also make our process more effective.

So in each case, there’s a slightly different measure of success. For some scenarios we have specific business goals to achieve, but often we just want people to be better informed in their work.

Measurable business outcomes

Ultimately, in the business use case of analytics, the business outcome should be straightforward in principle: we make better decisions. Therefore, the most important outcome for embedded business analytics should be smarter, more effective, more impactful business decisions.

Perhaps not surprisingly, measuring a “decision” can be tricky, but in general we should be focusing on a business outcome. For example, we may be looking for increased sales numbers, lower costs or improved customer satisfaction. Those are the ultimate measures of this success of our process, although it may be difficult to draw a clear line from one decision to the overall outcome.

There are also measures of the process itself. For example, how quickly a decision can be made, how quickly we can process an order, how many customers or clients our call centre can deal with in a day. If we have a recommendation engine, we can measure how often the recommendations are actually accepted, and does that lead to an increase in sales?

Always, it’s important to measure an outcome in terms that makes sense to the business. Simply measuring the process itself doesn’t tell us much about its ultimate effectiveness.

Engagement and adoption.

Having said that, there are still important ways in which the process of analysing can have business advantages in itself, however tricky to measure. Among these merits are the engagement of the user with data and the adoption of analytic software.

It has often been said by industry analysts that business intelligence software still only reaches between 25% and 30% of business users. BI software vendors have struggled for over 20 years to increase user adoption. However, the vendors themselves are perhaps part of the problem! It is now time to recognise that one of the barriers to adoption is the nature of their specialised tools, requiring a business user to drop out of their business experience and drop into an analytic experience. This breaks their workflow and disrupts rather than deepens their thinking.

Vendors are beginning to realize that one way to increase the adoption and the engagement with their software is to embed it in the experience that people already know and use. The advantage of more engagement and adoption should be that your business decision makers are better informed. We might also hope that business users take an interest in the data, sharing tips and tricks and insghts with other users.

Even though engagement and interest may not factor directly into decisions they make, having greater situational awareness of the business environment, and of the current situation and trends within their department within their company, should help. In fact, sometimes we should be wary of putting precise metrics on these outcomes because there’s a common habit of managing to the metric where we focus too much on narrow measures of business value and miss the bigger picture.

Engagement and adoption may also value in increasing employee satisfaction with their work by giving a broader understand and more business context to what may be a narrowly focussed task.

Spreadsheets and analytics

I have mentioned before that it’s important for embedded analytics that the new features live within already familiar software. For many of the people with an interest analytics - such as financial analysts, marketers and sales operations - their daily work involves a lot of spreadsheets. In fact, for many business people, the spreadsheet is their default analytic tool. Even when they have reports and dashboards, they will often reach for the Export button so they can load the data into a spreadsheet and do new work.

I don’t want to break that paradigm. I think it’s reasonable to recognise that spreadsheets are extremely powerful, flexible, and simple to use. Most people learn to use spreadsheets, including quite advanced features, without any training; they just pick up those capabilities in the course of their daily work. So there’s definitely a case for embedding analytics into spreadsheets, or extending spreadsheets with analytics.

One way of doing this is simply to provide better visualization features than the spreadsheet can offer natively. Or we extend the spreadsheets functions with new capabilities, often integrating with external – and more powerful – analytic engines, especially to handle more data than spreadsheets are designed for.

Another approach is to accept the capabilities, simplicity and limitations of spreadsheets, but then embed your into another application. For example, I may be able to build a useful mortgage and loan calculator in a spreadsheet – a very common scenario for a financial advisor – but with the right tools, I could embed that calculator in my web page, or another application, so that my existing or potential clients can use it easily when they browse my site.

It’s a little secret of business intelligence vendors, especially of self-service BI apps, that their number one data source is actually the spreadsheet, not the enterprise data warehouse or cloud applications; just the traditional spreadsheet, where the data already exists, and where they can do a lot of the shaping and review of the data that’s necessary before they build an analysis tool.

Later, I’ll talk in some detail about several of these scenarios in more detail.

A game plan for embedded analytics

It may be tempting just to dive into embedded analytics and start inserting charts and visualisations and recommendations all over your operational systems, hoping to inform your users better. Let’s not rush into it! We need a strategy.

I think of this as being more like a gameplan than a traditional roadmap. By a gameplan, I mean the sort of strategy that a football or basketball team learn before going into the game. They know the strengths and weaknesses of their opponents and their own team. Therefore they can work out a rough plan of how they’re going to approach the problem and how they might respond to developments during the course of the game. But of course, they never know exactly what’s going to happen. They certainly can’t predict if the other team are going to have a good or bad day. Maybe this is the day your opponents are at their best: or the day at which they stumble and fall at every opportunity. So any plan must be very flexible.

My goal with a gameplan is to give businesses a strategic tool with which they can set a course for a specific technology – such as embedded analytics - while being very flexible in how they get there. But first, you need to know where you are starting from. There are three steps to the gameplan.

Understanding where you are.

The first step of a game plan is to orient yourself, taking stock of your current capabilities, data assets, current skill sets and demands. I think it’s fair to say that when we look at more traditional maturity surveys of an entire business, they try to give a single answer that would apply across departments, but that is often too broad a measure to be useful. Different divisions, departments and all have very varying strengths and weaknesses, skills and practices and indeed, in many enterprises today, tools. They may even have different platforms, especially in businesses which have grown by merger and acquisition.

So the first step in designing a game plan is to understand where we are, remembering that there’s not one answer but rather multiple definitions of where we are whatever organisational structure makes sense for our business.

Setting a goal.

The same principle, that there is no one-size-fits-all strategy even within a company, is true for goal setting. We know where we are once we have oriented ourselves in the gameplan but what we want to achieve with analytics will also be different for different departments.

The manufacturing team or the production team may well have a very different level of analytics from the sales and marketing team. So we need to understand not only where ever team starts from, but where they need to end up.

Now you can see why we need that strategic flexibility. The route from our origin to our destination is not only going to be different between teams, but as we implement tactical solutions for each there are likely to be synergies and intersections (and conflicts!) along the way.

So we have unique starting points, varying ending points and different journeys.

Mapping out the journey to success.

The journey that we take from our present situation, to our goal can be quite long. It’s important therefore that we break that journey up into achievable steps.

My strong suggestion is to identify a series of projects, each one taking not more than one business quarter. Defining each of these steps in such a way that even if you only complete the first one you will have achieved something. Every step should deliver business value and be in many ways complete in itself. This achieves two very important goals.

The first is that you’re constantly adding business value. If for some reason the project has to pause, or even if the project is cancelled, you have still achieved concrete goals.

The other reason is that this continuous process of adding value encourages engagement and adoption with business users because they see the value being added. This approach also increases the commitment and sponsorship of business executives who see in progress and a return on investment.

It is much more satisfying to see new features and new capabilities becoming available every quarter than it is to have to wait for one year. In the world of embedded analytics, I think it is straightforward to identify more concrete, less abstract steps, that make up this incremental gameplan.

However, before we start discussing the architecture and development of embedded solutions we do need to understand a little more the decision making process that we are trying to improve.

Chapter 2. Analytics and decision-making

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the authors at feedback.embeddedanalytics@gmail.com.

As we consider the role of embedded analytics in business, it’s useful to loop back to the question we asked in the introduction. What is analytics for?

One answer is that analytics helps us to keep in touch with our business, providing a sort of broad situational awareness for managers and planners. I think that’s one sensible goal, but there is a more practical and direct reason why we need analytic insights. We need to make better decisions.

To make better business decisions, we need better business information. This has been understood systematically since the first methodical research at the Carnegie Institute of Technology in the 1950s and the 1960s. Even before that, when people first started to create basic financial reports or management reports, the intention to provide better information was implicit. In those days, this process wasn’t called analytics, it was called decision support. In truth, I think that’s still a good name for the practice, because our purpose is indeed to support better decisions.

What is a decision?

As so often, a word we likely use every day can be a little tricky to define. In a business context, I rather like the definition given by James Taylor and Neil Raden in their 2007 book Smart Enough Systems.

A decision ... is the act of determining a course of action.

I like this because it so clearly links the act of deciding to the act of doing. Of course, sometimes we may decide not to act, but in business even inaction is a choice with consequences.

Having said that one goal is to support decisions, we do need to understand a little more about the decision-making process. There’s a wealth of academic research in the field of decision science, often related to probability and game theory, but it’s rather outside our scope, being somewhat abstract. For our purposes, we don’t need a scientific model, but a useful account of an important pattern you can see in most modern enterprises.

Figure 2-1. Focus on Decisions

In Figure 2-1, I’ve laid out two dimensions of decision-making: the number of decisions that we make on the vertical axis and the business value of an individual decision along the horizontal axis. We can see that decisions range from a very high number of individually low-value decisions to a small number of individually high-value decisions: that is a range from operational to strategic decisions.

These decisions vary not only in their value and frequency but also in the data used to support them and the people who make them. Understanding this variance is important for us when we come to design embedded analytic experiences.

In this chapter, we will consider the different kinds of decisions which are made in modern businesses, especially where they can be assisted by embedded analytics. We will also discuss a design pattern for analytics which will be particularly useful when embedding components which need to be impactful but not distracting. We also need to discuss the role of ambiguity in analytics, because it is rare to find our decisions supported by perfect data.

Let’s look first at strategic decisions.

Executive and strategic decisions

In any business, there are some high-value decisions to be made. What new product will we build? Should we acquire another company? What new markets will we enter? These decisions each have a significant business impact, but they are not made on a daily basis. These high-value strategic decisions are typically made by executive teams and the matter may be resolved at an annual or quarterly board meeting, or during a monthly management review. Strategy in a well-governed business is not something which changes day-to-day.

The data which supports such decisions is highly aggregated. The quarterly board meeting does not get hung up on the details of every sales transaction, but rather they consider global or departmental sales over the preceding period, summed up, averaged and trended.

Not only is data for strategic decisions aggregated in this way, it is also augmented with a lot of external information. Executive decisions are rarely made by looking at only one data source or one narrow domain of business. In fact, strategy is informed by all sorts of data which may not even be captured in a structured format. For example, a strategic decision may be influenced by competitive intelligence about rival businesses, the state of the economy, questions of political stability and the hopes of the business leaders.

Strategic decisions are made with data which is highly aggregated and richly augmented. Executives rarely make these decisions using just one source of information and very rarely will there be an app involved. Nevertheless, we will look at executive decisions in more detail in Chapter 7 when we consider how spreadsheets can be embedded. Spreadsheets remain an executive tool of choice.

Operational decisions

If we look now at the other end of the spectrum, at operational decisions, we can see that there are very many of them. Indeed, there will be thousands of operational decisions made every day in a large business.

An operational decision is a choice made in the course of business which is very limited in scope and individually has a low business value compared to the entire enterprise. Which customer do I visit next? What products do I present to them? What discount may I offer? What action do I take when this alarm goes off? Do I approve a loan for this customer?

Of course, in an important sense, operational decisions are your business in action. This is how your business operates.

Operational decisions are typically made with a relatively limited amount of data from well-defined data sources. There may be no external information used at all. In all these ways, they are almost exactly the opposite of a strategic decision. The decision-maker, rather than an executive, may be a salesperson, machine operator, call-centre representative or clerk.

Operational decisions are a prime use case for embedded analytics. If I need to respond to an alarm, or to make an offer to a customer on the phone right now, it’s extremely useful for me to have some context or recommendation which helps me to make a better decision. Otherwise I will likely just respond instinctively to a relatively low information event.

For these reasons, throughout this book, we will often focus on operational decisions and how we can improve and enhance them with embedded analytics.

Tactical management decisions

As the chart shows, in the range between operational and strategic work, we have tactical decisions. These decisions are typically made by those we sometimes call (not always as a compliment) middle-managers. Despite their reputation as bureaucrats, the middle-manager role is extremely important in business decision-making. They work within a scope which aggregates operational decisions to a point where patterns, trends and exceptions can be found. From there, the tactical decision-maker sets the direction of the business over a workable period of time which may be days, weeks or months, rather than the years-long outlook of strategic planning.

For example, a sales manager - unlike the VP of Sales, will indeed need to access to all individual sales transactions perhaps digging in to detail if even one item looks notably wrong or exceptional. In this way, tactical decisions are made with aggregated operational data and perhaps with some augmentation from sources such as HR data. Nevertheless, the scope of tactical thinking is much tighter than strategic decision-making.

Middle-management decisions are the key focus of traditional business intelligence. It is for this kind of work that most reporting and dashboard tools have optimised their user experience over the last 20 years. In fact, this middle-management focus is also the reason why business intelligence has been limited in its adoption, because there are a limited number of tactical decision-makers.

The limits of business intelligence

In 2017, the Gartner analysts Cindi Howson and Rita Sallam said, “Pervasive business intelligence remains elusive, with BI and analytics adoption at about 30% of all employees. Data and analytics leaders wanting to extend the reach and impact of BI and analytics in their organizations should deploy modern BI platforms, leveraging mobile and embedding capabilities.”

I think there is a reasonable case to be made for extending the reach of BI, but I’m convinced that we’ll do that by focussing on improving decision-making, not just by increasing adoption in itself.

In contrast, there are a very large number of operational decision-makers and those are the people we are trying to reach with embedded analytics. That’s not to say there no role for embedded analytics in the world of the tactical manager or strategic executive: we will look at some of those scenarios when we consider reporting and dashboard solutions.

For now, how we can start to support decision-makers with data and insight? There are some patterns we can share which will help ...

A design pattern for the analytics experience

In 2007, Marcia Bates, a leading researcher at the Department of Information Studies, UCLA, analysed the components of browsing behaviour, specifically in a library. Her paper, What is browsing - really? is fascinating. Bates explores the research in some detail but I, and design teams I have worked with, have found a simplified model to be useful in our own work.

Please note that as I describe this model here, all the good points really come from Bates’s work - it’s a paper worth reading. Any inadequacies are the result of my own over-simplification.

Nevertheless, whether we are considering analytics just for information or, as in our case, as an input to decision-making, the following pattern, shown in Figure 2-2, works well.

Figure 2-2. A simplified model of the analytic experience

Orientation

The first step in browsing - and in analysis - is to know where you are: to orient yourself in the information environment. In a library that would involve knowing which subject section you are in and how to find your way to other sections.

When working with analytics objects embedded in an application there are similar questions. What am I looking at? Let’s say it is an analysis of product sales. What products are included? What geographies? What years are shown? We should not overwhelm the user with helpful indicators because they are more likely to be lost in the detail than helped. (Think of how difficult it can be to make sense of a road junction with too many lights, signs and markings.) But 2 or 3 key pieces of information should help a user determine what they are looking at.

Orientation should also afford a user some sense of how they can move through the information space. For example, if I’m looking at today’s call volume, how could I find out the numbers for yesterday or the week before? Is that information available?

It is beyond our scope here to describe all the ways in which user experience can provide these capabilities. In fact, we don’t need to, because when we embed analytics in a host environment or application it is most useful if the orientation and navigation features of the analytic components more or less follow those of the host application. In this way, the experience is more familiar and simple to learn while reducing the cognitive burden for the user of switching between environments.

Glimpsing

The original example of a library (or a bookstore, if you prefer) serves us well here. If you are looking for a book on how to run good meetings, you can surely orientate yourself in the business books section. But what about that meetings book? There will likely be hundreds of volumes on all sorts of topics, perhaps organised by author.

If we have time, patience and some knowledge of the system, we may look up a catalog. However, we’re much more likely to browse - thus the title of the paper - not looking at every book carefully in turn, but rather letting our vision drift over the shelves until something catches our eye. We glimpse, perhaps the word meeting in the title. And, just as we glimpse the title we will also take in the author’s name, design features of the cover, and so on.

In analytics, glimpsing is an essential step in making sense of data, especially when we are looking for insight or new information. In the scatter plot shown in Figure 2-3, it is easy to see that one or two values - outliers - are standing apart from the majority of data points. We can glimpse what may be interesting or important.

Figure 2-3. A scatter plot where outliers can be easily glimpsed

Similarly, in this time series visualization we can glimpse several useful characteristics. The overall trend shows sales increasing over time. There is one short period where sales show a significant drop before recovering; why might that be? And we can glimpse a recent spike in values: perhaps that’s a useful data point to explore.

Figure 2-4. A time series chart enabling spikes and dips to be clearly seen

As you can see, certain visualizations are particularly useful for enabling these quick insights. In fact, choosing the right visualization for your data should involve a consideration of how the information will be glimpsed and then, having seen something potentially interesting, how it will be examined. This is the next step in the model.

Examining

Back in our library, having glimpsed a book about meeting culture, it is time to see if it will be a useful choice for us. So we take the book from shelf and examine it. Note that we don’t read it, there in the library. We may look up the table of contents, flick through the chapters, read the short marketing blurb and enthusiastic quotes on the back page. Has it got diagrams? Exercises? An index? Examining helps us to understand if this is indeed as useful as we might hope.

In our visualisations, we can do something conceptually similar. Having glimpsed a data point or a trend we think is important, it is time to examine it.

Some tools make this easier than others. In the example below, by simply hovering my mouse over the data point in question, I can bring up a small panel with more information. That’s certainly useful and better than overloading the visualisation with all that extra detail presented up-front.

Figure 2-5. Examining a data value

Here’s another example. Rather than just hovering over the visualization for more information, I can click on the data point in question and now the whole view changes. I have gone from looking at sales by region to examining sales by one particular sales person in that region.

Figure 2-6. Drill-through in action

We call this kind of interaction, drill-through. It’s a critical capability of any business intelligence tool.

The navigational scope of embedded analytics

In these examples, when we examine some feature of the data that we glimpsed, we are given more information about it. Some business intelligence tools enable you to do much more. For example, having examined sales at the level of the region, you might expand your view, look at the sales for the entire enterprise and then trace a new path into another, different region. Strategic and tactical decision makers do this frequently, because they are examining data at a higher level and need to make comparisons across a much wider span of the business.

However, if I am a salesperson, trying to select which customer I visit next and I’m using an embedded visualization to help with that decision, I am unlikely to gain much insight by comparing my own work to overall sales at a global level, or in another region. I will do best to focus on scenarios which remain admittedly small-scale but relevant to my own decisions.

When embedding analytics, we must bear in mind, that we’re not embedding a complete business intelligence environment which can address any issue in the enterprise. We are supporting a specific class of user as they make specific decisions.

To enable this tighter focus - and to prevent distracting users - embedded analytics are typically much more restricted in scope than traditional, general purpose, business intelligence tools.

Deciding

When Marcia Bates investigated browsing in the library, the last step was called Acquiring. Taking out a library book to read it. The person browsing literally acquired an object to use.

In our case, we are not going to take our data to the front desk to be stamped and later returned. But, having glimpsed something interesting and having examined the data, we now have an insight and it is time to so something with it. It’s time to take an action - to make a decision.

As I emphasised that we are supporting decisions in our work, you can imagine this step is absolutely critical to the success of embedded analytics. In fact, this subject is so important that we a large part of Chapter 3 will discuss how we model the components of a decision and then enable the decision to be made with analytic insight.

For now, I think it is fair to point out that this final step has not been handled well in analytic tools. In fact, as someone who has designed several leading analytic applications in my career, I have to admit my own failings here. For a long time our focus was on the challenges of our day - getting good quart data into the hands of the business user and enabling them to explore it and visualise it. I think we did a good enough job, but the technology and user needs have moved on. Today, we need to integrate this decision-making step more directly into the user’s workflow.

There are four ways in which users can move from Examining to Deciding.

- Users can simply make the decision themselves - in their heads, if you like - and then go to another tool or another operational environment to take the action. This is how users made decisions with traditional business intelligence tools.

- It is common for dashboards and some reports to enable users to share a visualisation in email, or through a chat application, or even clipping or sharing the analytic object into a presentation slide. Sometimes this capability is tightly integrated, but often it is only a loose capability. It still requires the user to make an observation in the analytic tool and take action elsewhere.

- Some business intelligence tools enable extensions to the analytic environment which can integrate links to an operational application. This is often through a process called write-back. We will discuss write-back in more detail in Chapter 4. In a sense, this approach is the very opposite of embedded analytics: it embeds operational capabilities in the analytic toolset rather than analytic capabilities into the operational environment. There are good scenarios for this which we will discuss later.

- Finally - and our main focus here - if the analytic object, such as a visualization, is embedded in an operational application, the user can take the action right there on the same screen, without switching environments or losing focus. As a salesperson, I can see some outliers in my customer analytics, perhaps clients I have not called in a long time. Right there, in my customer management application I can decide to initiate a phone call, make notes and deal with those customers very directly.

This model of browsing is powerful and practical, in part because the experience is very natural for us. We use it in a library. In fact, with a little thought you will recognise the pattern in many of our activities such as choosing a movie to stream or grocery shopping. However, I do want to emphasise that the pattern is simplified from real-world experience, in order to make it a little easier to apply in our design work.

One important aspect of data and decision-making that this browsing pattern glosses over, is the common problem that our data does not always lead clearly or cleanly to a single, simple decision. Data, in the real world, is ambiguous.

Ambiguity and analytics

Human beings are not comfortable with ambiguity. Research into our neurological reactions to ambiguity show our amygdala - which controls our flight or fight instinct - lighting up as we become emotionally and physically uncomfortable with uncertainty.

In addition top of this discomfort, ambiguity is simply not persuasive. As J.M. Coetzee has put it, If you do not already think in probabilistic terms, predictions emerging out of the probabilistic world seem vacuous. General McArthur did not leave Manila saying, I will probably return.

Business Intelligence has mostly sought to remove ambiguity and uncertainty by favouring clarity over complexity. The best known example is the claim of data warehouse architects to deliver a single version of the truth. This implies, in its very phrasing, the existence of alternatives to be suppressed. Certainty is rewarded.

Even where uncertainty is unavoidable, we often prefer the rather artificial clarity of a fixed value. It’s common to see dashboards which predict budgets and expenses down to the very cent.

Figure 2-7. A dashboard of unrealistic precision

Of course, we don’t believe that we will ever behold a Gross Margin in January of exactly $1,501,360.56 but we lack the tools and, for many people without a background in probability, we lack the vocabulary to bring more realistic, more ambiguous, outcomes into focus.

At best, everyone involved understands this as a game of self-deception. Nevertheless, behind the comfortable illusion lurks the danger of the fallacy of false precision. We may place undue trust in these forecasts because they look accurate, forgetting our own part in the trick.

Today, we are only fooling ourselves. But as machine learning and artificial intelligence are innately probabilistic, our businesses are rapidly reaching the point where such illiteracy will severely limit our work and success. We don’t all need to become data scientists, but we do need to get better at communicating uncertainty in our work.

Here’s an example of a visualization which shows ambiguous data in a helpful, easily understood way.

Figure 2-8. A simple, but compelling, visualization of uncertain data

As you can imagine, we cannot precisely know what the world population will be in the future. But we can make some reasonable predictions and this visualisation shows that, while also showing in a simple way, the range of potential variance around that prediction - the ambiguity. Yet this is not a technical chart, with complex formulas and detailed visualizations of error. Almost any reasonably data-literate user can understand this.

Is this better than the previous, more detailed, unambiguous prediction? I think it is. But more importantly, I think it can lead to better decisions. For one thing, we can see and interpret very easily the potential range of sales of different products. We can do so quickly without a demanding cognitive burden - which means we can interpret it and get on with our job, while still being well informed.

- We can orient ourselves easily: seeing past sales and future sales side-by side.

- We can glimpse numerous scenarios and see which are most promising, but also which are least certain.

- We can examine a product line to see in more detail what lies behind its numbers.

- Finally, if properly integrated with our operational application, we can make some smart decision.

Ambiguity is a complex issue. A full treatment of the implications for business decisions is somewhat out of scope for us. However, understanding that the problem exists and that there are ways to address it will be very useful when we come to design embedded analytic experiences.

Summary

In this chapter we have reviewed the different kinds of decisions made in the modern enterprise. We have seen that strategic decisions are made with a wide variety of inputs and so rarely benefit from embedded applications – but look out for spreadsheets, which are still an important tool for executives.

Tactical and operational decisions ( the decisions of line managers and their staff) are those which benefit most from embedded analytics. The simple pattern described – Orientiation, Glimpsing, Examining and Deciding – can greatly help in making analytics accessible and effective.

Finally, we also considered how we can deal with ambiguous and uncertain data, which is increasingly important as we make more use of predictions and other machine learning features in our work.

Now it is time to get started, planning our embedded analytics …

Chapter 3. Data for embedded analytics

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 4th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the authors at feedback.embeddedanalytics@gmail.com.

One of the first questions we have to ask when building an analytic solution which will be embedded inside an operational application is, “Where will our data come from?” And the answer to that is not always simple. Your data will not necessarily come from the operational application itself. In fact, very often we want to complement or augment the application with insights from other systems.

As we do so, we have to think of three different scenarios for each data source that we wish to use.

- The first is simply where does data come from? Which system has the best data to support our decisions or to support the application that we are enhancing?

- The second question to ask is how will we connect to that data? There are different ways of connecting (even to the same data source), which may be more or less appropriate for the scenario in hand.

- And the third question to ask is how will we manage and govern the connectivity, the security, and the performance of the connection to that data source?

In this chapter, we will look at several different data sources that can be useful for embedded analytics and we will consider those questions What is the nature of the data source and why would we use it? How will we connect to it?

CSV and other text files

There are few data sources as simple – in theory – as a text file of data. They are found in almost every business in some scenario because they have some great advantages. For one thing, they are widely supported by data analysis tools ranging from the simplest spreadsheet to the most advanced data science workbench. CSV files are equally well supported as an export format. Connecting to a CSV file typically only involves having read permissions to access the folder in which it stored. So for many situations, they are a convenient solution.

This convenience comes at a cost, however.

Security

The only security mechanism for CSV files is to protect access to the folder in which they are stored. There is no support for row or column protection. It is important therefore that if you are using CSV files as a source they must be exported with the strictest security and privacy controls in place. Do not save sensitive data to files.

Performance

Most data movement tools, such as ETL applications for data warehouses, are optimized in some way for text files, especially for reading or writing data in bulk. These optimizations often take the form of read and write buffers which load data into memory for parsing. Business intelligence tools rarely have these optimizations in place. They will read text files easily, but performance is likely to be poor compared to a native application driver.

Parsing

Not all text files are CSV files – comma separated values. Some are delimited by tab characters, some by pipe or vertical bar characters. Highly customized delimiters are rare. And, ironically, ASCII control codes, intended to be a standard, are rarer still. Numerous issues can arise, such as the use of decimal separators (which in some countries are commas) or less common thousands separators, such as the Indian Lakh (1,00,000.) If your analytics environment does not support these characteristics, you will need to ensure that your data export process renders the text files in an easily readable format.

Operational data sources

An operational data source is simply the data which is stored by an operational application. All modern applications to some extent, create, store and manage data. For example, an application for processing orders generates the data for each order, and perhaps links data for each customer. That will need to be persisted somewhere. Operational data sources focus on integrating that data in such a way that the performance of the application is maximized and the integrity of the application is ensured. In many cases, the ease of use – especially the responsiveness of the application - is enhanced by having a robust data store which can persist large volumes of data for record keeping with good performance.

Many applications use very simple structures for operational data sources. For example, many use an underlying operational database. Developers are familiar with the connectivity to that database and they’re quite familiar with the table structure, because typically they’re working with the details of the operational application daily. And because you’re connecting to your operational data source, consistency is pretty much guaranteed.

Sometimes, you may use a copy of the operational source. For example, you may take advantage of a technology such as replication. The advantage of that for a developer is that the work they do with analytics doesn’t affect the operational application itself. So, for example, you could extract all the records for an individual customer, aggregate them do some analysis on them and present that to the operator. You don’t have to look up all the details of the original database because you’re using a replicated copy which improves performance and preserves the consistency of the original data source.

As we said, operational data structures are optimized for efficiency and for integrity. But the implication of this is that they are not optimized for analytics. In practice, the joins between tables in a highly normalized database can be very complex, tricky to navigate and often don’t provide the best performance. In such cases trying to execute analytics over these tables becomes an extra load on the database. Replication can help with that, but you may find it better to use another system such as the data warehouse, which we will discuss later.

However, when it comes to the connectivity to an operational system, this is typically very simple because your operational application is already connected to it! When we connect to a replicated version of the operational data source connectivity will also typically be simple because the replicated version is just another copy of the original system.

The situation does get a little more difficult if we are trying to combine two operational systems: one in which we are performing operations, and one from which we are sourcing data to enhance the analytics.

You may have a purchase management system which handles orders and invoices and various aspects of the order-to-delivery process. The customer data may be in quite a different application: it may be in a CRM system, or a financial system. Bringing those two together involves embedding one within the other, but they’re both operational systems and therefore not particularly optimised for analytics. Our connectivity to the second operational system may not be suitable.

In fact, for any operational system, there are typically two ways to connect: through a bulk connector or through an API. A bulk connector is optimised for extracting a large amount of data. An example of this could be connecting with OLEDB or JDBC, to the operational database underlying the application. With such a connector, you can ask for a full table of information, or for many tables at a time, and you can probably issue a query against it and return a result set which may be small or actually very large. That kind of connectivity can be very useful for analytics because typically, we’re looking at larger volumes of data to analyze.

Nevertheless, we also need to consider the other way to connect to operational systems: through API’s if they are supported. Web developers will be very familiar with this, because most of the connectivity between web applications involves API calls. However, API’s are typically not designed for returning large volumes of data: often only single records or the answer to a simple query and the result set may be returned as an XML document rather than as a flat table.

As a result, we have to be careful in choosing the kind of connectivity that we want to use for analytics. It will not always be the case that an OLEDB connector is the best one. Many of them are very simple and performance may not be good. Equally, it will not always be the case that the API is inappropriate - some API’s do have calls which enable bulk data extraction. In other words, we have to know our system pretty well, to understand the volume of data we are trying to extract, the characteristics of the source system and the workload we ask it to deliver for us.

Analytic data sources

There are three types of data sources we might consider as analytic.

Data Warehouses

The first and most common example would be the data warehouse - an architecture specifically intended to enable data analysis. Its data is denormalized to make it simple to query and to improve the performance of large analytic queries. The architecture is designed to integrate data from multiple data sources so that work does not need to be repeated. Most importantly, I think the data warehouse represents not just a data structure, but a business model. That is to say, a logical representation of how a business operates.

A data warehouse will have dimensions which represent how the business thinks about its customers. It will include a Geography dimension, which represents how the business thinks about its divisions and its offices and it stores our factories. A data warehouse nearly always includes a Time dimension which represents the calendars, working days, holidays and seasonal divisions relevant to the business. It may include a Personnel dimension, which reflects the structure of the company’s hierarchy. All of these pre-designed and pre-populated dimensions, make analytic queries very simple for the developer. Even better, the data warehouse should be optimised for analytic performance.

In fact, a data warehouse or it’s simpler, more focused cousin, the data mart, is an excellent source of data for embedded analytics. But not every enterprise has a data warehouse and certainly not every small and medium business. Even those enterprises which do have a data warehouse may not want to make it available for operational applications, because in some cases the warehouse is so focused on management, financial or tax reporting: mission critical business analysis, which may be sensitive or governed in such a way that the business does not wish to make the data directly available to other systems.

Sometimes, we can work around that by creating downstream replications of specific subject areas and even the answers to specific queries which we regularly used.

In memory engines

The data warehouse technology with which we are familiar has been with us since the 1990s.

Around 2005, another technology started to appear, which had many of the advantage of a data warehouse, but offered greater performance, greater flexibility and had advantages for both large and small businesses. This was the development of in-memory engines, which took advantage of the very high performance of memory as compared to disk for performing complex queries. With 64-bit systems (and cheap memory chips) emerging in the mid 2000s, this enabled companies to perform analysis on the desktop, which previously had only been possible on the largest corporate systems.

Building on these desktop capabilities, in-memory technologies are very popular in self-service business intelligence, and in fact, are the key to performance and flexibility for many of the most popular self service applications, such as Microsoft Power BI, Salesforce Tableau, and Qlik. However, self-service applications were in the past very self-contained and focused on their own analytic capabilities. Their connectivity to other systems was entirely focused on extracting data into their own memory space. Developers didn’t think very much about how they would share that data with other systems. Having said that, many of the in-memory BI applications today now have embeddable capabilities. So, it is more often than not, possible to take a chart from a BI tool and embed it inside an operational application. There will be benefits for developers in terms of familiarity and performance and simplicity. But there are some restrictions because the systems are often not designed from the start to be embedded. Originally, the embedding ability (and often the API’s) are afterthoughts, which have been added later. Nevertheless, for developers familiar with these systems, and where the systems are already deployed in the business or in the enterprise, it’s a very attractive proposition to reuse this functionality and embed it inside operational applications. This scenario is often well supported by the vendors and their communities. And it’s an option that is well worth considering.

Data lakes

The data lake architecture, and perhaps in its more modern incarnation, the data lake house is, like the data warehouse, focused on the storage of large volumes of data, which we make available for purposes such as data science, and to some extent analytics.

Unlike the data warehouse, the data lake does not have a built-in business model, a logical semantic architecture, which represents how the business thinks about itself - with rules, well-defined processes for governance or easily-navigated data structures. In fact, the data lake is intended to store data in its most raw format without any processing. This is a great advantage for data scientists, who often want to see data in its native state, because they can use that raw data to do more detailed and more complex analyses that they could if the data had already been cleaned up and built into a data model for reporting.

In the data lake, the data scientist can see all the raw detail - with all its errors and complexities.

Data Integration pipelines

Another source of data for an embedded system may be the output of a data integration pipeline. That is to say, rather than reading data from a table in an in-memory system or a data warehouse or an operational database, that data will be read from a pipeline running in a data science environment. The pipeline may perform numerous operations of integration, cleansing and preparation on the data from whatever source it comes. This is a very popular scenario for data science. But it is limited in use for embedded analytics, because the process is a little more fragile and a little more difficult to govern than a data warehouse or an in-memory system. This is because the pipeline is more dynamic and more volatile, only delivering data while it is running.

A scenario where this is popular and useful is when we are presenting data which is continuously changing - streaming data. This could include data from machines, such as automated lathes, in manufacturing, or an e-commerce ordering system where hundreds of orders may be placed every minute, or from environmental sensors.

In such cases, the analytic scenario is somewhat different. An application offering streaming analytics must include methodologies for handling interruptions to the stream. Similarly it must be able to display changing values without users having to manual refresh the display. Finally, methodologies for governance and security may be different from the analytic governance that we’re used to.

Writing back to sources

A final point has to be made about the handling of data in embedded analytics. We have been talking so far about data extracted from source systems and presented in the embedded analytic environment purely for the purposes of information and augmentation. This is a read only scenario. There are times when we may want to make changes to the analytic system from within the embedded application - what we call writeback.

Writeback is often used for planning and simulation applications, where we want to insert numbers that represent future estimates of sales or staffing or growth, or whatever the parameters are that we are planning around. It is relatively unusual for systems designed for analysis – such as a data warehouse to enable direct writeback. Such systems are too concerned with data integrity, data history and compliance to allow ad-hoc changes. But often they will allow writeback to special tables, which can be set up as parameter tables specifically for the purpose .

Summary

We have seen that data can be sourced for embedded analytics from many different kinds of systems: operational analytic and streaming. There is no one best solution for a typical embedded scenario, but it is fair to say that most developers of embedded applications will start with a simple data warehouse or data mart scenario if that already exists within your organization.

The following table summarizes some of the benefits and weaknesses of the approaches we discussed.

Table 3-1. Data for embedded analytics

Data Source | Benefits | Weaknesses

CSV Files

Widely supported, often human readable.

No security mechanism.
May be difficult to parse.
Performance may be poor.

Operational database

Developers are familiar with the connectivity and table structure.
Because you are connecting to operational data, consistency is guaranteed.
If replication is available, you may use a copy of the operational system for analytics.

Operational table structures are optimized for efficient transactions, not for analytics. Joins can be complex.
Analytics is an extra load on the database and can affect performance, although replication can help.

Views and aggregate tables

Analytic queries are simplified because the joins were developed in advance.
Filtering is simple.
We can apply role-based access rules simply or build them into the view. (Like the Secured Views of some cloud data warehouses.)
Aggregating data in advance improves performance.

Response time may still be slow if the underlying joins are complex.
Views and aggregate tables must be maintained separately from the operational tables: another burden for administrators.
Aggregate tables must be updated, often using triggers. These updates have to be carefully scheduled or they will affect the performance of the operational system.

Data mart or Data warehouse

High performance as the architecture and schema is optimized for analytic queries.
Simple to integrate with BI and analytics tools as the schema is designed for analysis.
Load on the transactional source system is reduced.

Loading the data warehouse using ETL is a specialized practice.
The load and refresh cycle of the warehouse (often scheduled to run overnight) may be too slow for analytics embedded in an operational application.
When schema changes are needed, this can be a complex process requiring a lengthy process of specificying, implementing and testing.

In-memory engine

Very high performance.
Should be closely integrated with the embedded analytics environment.

Support for writing back to the database is often limited.
Data updates may be occasional and often require full, rather than incremental, updates.

Data lake

If using a data “lakehouse” architecture, you will find many of the benefits of a data warehouse.
Data is generally stored in its native format which can be good for predictive analytics.
Data may be continuously updated.

If you are not using a lakehouse model, you may find performance for business intelligence poor because so much transformation is required to conform and shape data.
Data lakes are generally shared with data scientists, data engineers and analysts. Security can be a concern.

In general, I recommend starting with a data mart or data warehouse connection if this already exists in your organization’s architecture. You’ll find it gives the best balance of accessibility, performance and governance for your needs. A data lake or lakehouse, if well managed, offers some similar advantages. However, if you don’t already have a formal data warehouse or lakehouse in place, it is a significant effort to build one and likely beyond the scope of your embedded analytics project on its own.

In this case, you may find it practical to start with views and aggregate tables, defined for your purposes and hosted within the operational database. The administrator can provision access and ensure security and good governance, performance should be acceptable and data access should be quite possible with a suitable driver.

Having reviewed the data available, in the next chapter, we’ll consider the different types of embeddable objects and how we can integrate them visually with applications for the most effective support.

About the Authors

Donald Farmer is the Principal of TreeHive Strategy, a strategic advisory practice specializing in data, analytics and innovation strategy. His background is very diverse, with over 30 years of deeply practical experience in industries ranging from fish-farming to archaeology to advanced manufacturing. He has worked in award-winning startups in the UK and Iceland, and spent 15 years at Microsoft and at Qlik leading teams designing and developing new enterprise capabilities in data integration, data mining, self-service analytics and visualization.

Donald is an advisor to investors, venture funds, incubators and startups around the world. He mentors executives and speaks globally at events and private workshops. He lives with his wife, Alison, an artist, in a recycled-polystyrene house near Seattle with one of the world’s most beautiful treehouses—the tree hive—in their wild garden.

Jim Horbury is the Solutions Practice Director at InterWorks, currently living and working in Hampshire, UK. Having spent almost two decades working in technical and creative direction roles in advertising, Jim was an early proponent of data-driven response measurement and drove the development of ad-tech reporting solutions for some of the UK’s largest travel and tourism organizations.

After a move into telecoms, big data and machine learning, it was a natural step to put his accumulated knowledge into action as a consultant. Jim now leads a global team of solution architects working with some of the world’s best-known brands, designing and leading the development of innovative and scalable analytics solutions.

Outside of work, Jim tries to recapture his youth with mountain-biking, running, and climbing.

Table of Contents

- 1. Introduction to Embedded Analytics

- Analytics for business users and consumers

- What success looks like

- Measurable business outcomes
- Engagement and adoption.
- Spreadsheets and analytics

- A game plan for embedded analytics

- Understanding where you are.

- Setting a goal.

- Mapping out the journey to success.

- 2. Analytics and decision-making

- Executive and strategic decisions

- Operational decisions

- Tactical management decisions

- A design pattern for the analytics experience

- Orientation
- Glimpsing
- Examining
- Deciding

- Ambiguity and analytics

- Summary

- 3. Data for embedded analytics

- CSV and other text files

- Operational data sources

- Analytic data sources

- Data Warehouses
- In memory engines
- Data lakes

- Data Integration pipelines
- Writing back to sources
- Summary

- About the Authors