Data Visualization on the Web with Python and JavaScript

Scrape, Clean & Transform Your Data

SECOND EDITION

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

Kyran Dale

Data Visualization on the Web with Python and JavaScript

by Kyran Dale

Copyright © 2023 Kyran Dale Limited. All rights reserved.

Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.

- Acquisitions Editor: Michelle Smith

- Development Editor: Shira Evans

- Production Editor: Greg Hyman

- Interior Designer: David Futato

- Cover Designer: Karen Montgomery

- Illustrator: Kate Dullea

- July 2016: First Edition

- December 2022: Second Edition

Revision History for the Early Release

- 2021-09-29: First Release

See http://oreilly.com/catalog/errata.csp?isbn=9781098111878 for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Data Visualization on the Web with Python and JavaScript, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-098-11180-9

Chapter 1. Visualizing Data with Matplotlib

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 10th chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at sevans@oreilly.com.

As a data visualizer, one of the best ways to come to grips with your data is to visualize it interactively, using the full range of charts and plots that have evolved to summarize and refine datasets. Conventionally, the fruits of this exploratory phase are then presented as static figures, but increasingly they are used to construct more engaging interactive web-based charts, such as the cool D3 visualizations you have probably seen (one of which we’ll be building in [Link to Come]).

Python’s Matplotlib and its family of extensions (such as the statistically focused Seaborn) form a mature and very customizable plotting ecosystem. Matplotlib plots can be used interactively by IPython (the Qt and Notebook versions), providing a very powerful and intuitive way of finding interesting nuggets in your data. In this chapter we’ll introduce Matplotlib and one of its great extensions, Seaborn.

Pyplot and Object-Oriented Matplotlib

Matplotlib can be more than a little confusing, especially if you start randomly sampling examples online. The main complicating factor is that there are two main ways to create plots, which are similar enough to be confused but different enough to lead to a lot of frustrating errors. The first way uses a global state machine to interact directly with Matplotlib’s pyplot module. The second, object-oriented approach uses the more familiar notion of figure and axes classes to provide a programmatic alternative. I’ll clarify their differences in the sections ahead, but as a rough rule of thumb, if you’re working interactively with single plots, pyplot’s global state is a convenient shortcut. For all other occasions, it makes sense to explicitly declare your figures and axes using the object-oriented approach.

Starting an Interactive Session

We will be using a Jupyter notebook for our interactive visualization. Use the following command to start a session:

$ jupyter notebook

You can then use one of the Matplotlib magic commands within the IPython session to enable interactive Matplotlib. On its own, %matplotlib will use the default GUI backend to create a plotting window, but you can specify the backend directly. The following should work on standard and Qt console IPython:1

%matplotlib [qt | osx | wx ...]

To get inline graphics in the Notebook or Qt console, you can use the inline directive. Note that with inline plots, you can’t amend them after creation, unlike the standalone Matplotlib window:

%matplotlib inline

Whether you are using Matplotlib interactively or in Python programs, you’ll use similar imports:

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

NOTE

You will find many examples of Matplotlib using pylab. Pylab is a convenience module that bulk-imports matplotlib.pyplot (for plotting) and NumPy in a single namespace. Pylab is pretty much deprecated now, but even were it not, I’d still recommend avoiding this namespace and merging and importing pyplot and numpy explicitly.

While NumPy and Pandas are not mandatory, Matplotlib is designed to play well with them, handling NumPy arrays and, by association, Pandas Series.

The ability to create inline plots is key to enjoyable interaction with Matplotlib, and we achieve this in IPython with the following “magic”2 injunction:

In [0]: %matplotlib inline

Your Matplotlib plots will now be inserted into your IPython workflow. This works with Qt and Notebook versions. In the Notebooks, the plots are incorporated into the active cell.

AMENDING PLOTS

In inline mode, after an IPython cell or (multi-line) input has been run, the drawing context is flushed. This means you cannot change the plot from a previous cell or input using the gcf (get current figure) method but have to repeat all the plot commands with any additions or amendments in a new input/cell.

Interactive Plotting with Pyplot’s Global State

The pyplot module provides a global state that you can manipulate interactively.3 This is intended for use in interactive data exploration and is best when you are creating simple plots, usually containing single figures. pyplot is convenient and many of the examples you’ll see use it, but for more complex plotting Matplotlib’s object-oriented API (which we’ll see shortly) comes into its own. Before demoing use of the global plot, let’s create some random data to display, courtesy of Panda’s useful period_range method:

from datetime import datetime

x = pd.period_range(datetime.now(), periods=200, freq='d')

x = x.to_timestamp().to_pydatetime()

y = np.random.randn(200, 3).cumsum(0)

Creates a Pandas datetime index with 200 day (d) elements, starting from the current time (datetime.now()).

Converts datetime index to Python datetimes.

Creates three 200-element random arrays summed along the 0 axis.

We now have a y-axis with 200 time slots and three random arrays for the complementary x values. These are provided as separate arguments to the (line)plot method:

plt.plot(x, y)

This gives us the not particularly inspiring chart shown in Figure 1-1. Note how Matplotlib deals naturally with a multidimensional NumPy line array.

Figure 1-1. Default line plot

Although Matplotlib’s defaults are, by general consensus, less than ideal, one of its strengths is the sheer amount of customization you can perform. This is why there is a rich ecosystem of chart libraries that wrap Matplotlib with better defaults, more attractive color schemes, and more. Let’s see some of this customization in action by using vanilla Matplotlib to tailor our default plot.

Configuring Matplotlib

Matplotlib provides a wide range of configurations, which can be specified in a matplotlibrc file or dynamically, through the dictionary-like rcParams variable. Here we change the width and default color of our plot lines:

import matplotlib as mpl

mpl.rcParams['lines.linewidth'] = 2

mpl.rcParams['lines.color'] = 'r' # red

You can find a sample matplotlibrc file at the main site.

As well as using the rcParams variable, you can use the gcf (get current figure) method to grab the currently active figure and manipulate it directly.

Let’s see a little example of configuration, setting the current figure’s size.

Setting the Figure’s Size

If your plot’s default readability is poor or the width-to-height ratio suboptimal, you will want to change its size. By default, Matplotlib uses inches for its plotting size. This makes sense when you consider the many backends (often vector-graphic-based) that Matplotlib can save to. Here we use pyplot to set the figure size to eight by four inches, using rcParams and gcf:

# set figure size to 8 by 4 inches

plt.rcParams['figure.figsize'] = (8,4)

plt.gcf().set_size_inches(8, 4)

Points, Not Pixels

Matplotlib uses points, not pixels, to measure the size of its figures. This is the accepted measure for print-quality publications, and Matplotlib is used to deliver publication-quality images.

By default a point is approximately 1/72 of an inch wide, but Matplotlib allows you to adjust this by changing the dots-per-inch (dpi) for any figures generated. The higher this number, the better the quality of the image. For the purpose of the inline figures shown interactively during IPython sessions, the resolution is usually a product of the backend engine being used to generate the plots (e.g., Qt, WXAgg, tkinter). See here for an explanation of backends.

Labels and Legends

Figure 1-1 needs, among other things, to tell us what the lines mean. Matplotlib has a handy legend box for line labeling, which, like most things Matplotlib, is heavily configurable. Labeling our three lines involves a little indirection as the plot method only takes one label, which it applies to all lines generated. Usefully, the plot command returns all Line2D objects created. These can be used by the legend method to set individual labels.

plots = plt.plot(x,y)

plots

Out:

[<matplotlib.lines.Line2D at 0x9b31a90>,

<matplotlib.lines.Line2D at 0x9b4da90>,

<matplotlib.lines.Line2D at 0x9b4dcd0>]

The legend method can set labels, suggest a location for the legend box, and configure a number of other things:

plt.legend(plots, ('foo', 'bar', 'baz'),

loc='best',

framealpha=0.5,

prop={'size':'small', 'family':'monospace'})

Sets the labels for our three plots.

Using the best location should avoid obscuring lines.

Sets the legend’s transparency.

Here we adjust the font properties of the legend.4

Titles and Axes Labels

Adding a title and label for your axes is as easy as can be:

plt.title('Random trends')

plt.xlabel('Date')

plt.ylabel('Cum. sum')

You can add some text with the figtext method:5

plt.figtext(0.995, 0.01,

u'© Acme designs 2021',

ha='right', va='bottom')

The location of the text proportionate to figure size.

Horizontal (ha) and vertical (va) alignment.

The complete code is shown in Example 1-1 and the resulting chart in Figure 1-2.

Example 1-1. Customized line chart

plots = plt.plot(x, y)

plt.legend(plots, ('foo', 'bar', 'baz'), loc='best,

framealpha=0.25,

prop={'size':'small', 'family':'monospace'})

plt.gcf().set_size_inches(8, 4)

plt.title('Random trends')

plt.xlabel('Date')

plt.ylabel('Cum. sum')

plt.grid(True)

plt.figtext(0.995, 0.01, u'\u00a9 Acme Designs 2021',

ha='right', va='bottom')

plt.tight_layout()

This will add a dotted grid to the figure, marking the axis ticks.

The tight_layout method should guarantee that all your plot elements are within the figure box. Otherwise, you might find tick-labels or legends truncated.

Figure 1-2. Customized line chart

We used the tight_layout method in Example 1-1 to prevent plot elements from being obscured or truncated. tight_layout has been known to cause problems with some systems, particularly OS X. If you have any problems, this issue thread may help. As of now, the best advice is to use the set_tight_layout method on the current figure:

plt.gcf().set_tight_layout(True)

Saving Your Charts

One area where Matplotlib shines is in saving your plots, providing many output formats.6 The available formats depend on the backends available, but generally PNG, PDF, PS, EPS, and SVG are supported.

Saving is as simple as this:

plt.tight_layout() # force plot into figure dimensions

plt.savefig('mpl_3lines_custom.svg')

You can set the format explicitly using format="svg", but Matplotlib understands the .svg suffix. To avoid truncated labels, use the tight_layout method.7

Figures and Object-Oriented Matplotlib

As just shown, interactively manipulating Pyplot’s global state works fine for quick data sketching and single-plot work. However, if you want to have more control over your charts, Matplotlib’s figure and axes OOP approach is the way to go. Most of the more advanced plotting demos you see will be done this way.

In essence, with OOP Matplotlib we are dealing with a figure, which you can think of as a drawing area with one or more axes (or plots) embedded in it. Both figures and axes have properties that can be independently specified. In this sense, the interactive pyplot route discussed earlier was plotting to a single axis of a global figure.

We can create a figure by using Pyplot’s figure method:

fig = plt.figure(

figsize=(8, 4), # figure size in inches

dpi=200, # dots per inch

tight_layout=True, # fit axes, labels, etc. to canvas

linewidth=1, edgecolor='r' # 1 pixel wide, red border

)

As you can see, figures share a subset of properties with the global pyplot module. These can be set on creation of the figure or through similar methods (i.e., fig.text() as opposed to plt.fig_text()). Each figure can have multiple axes, each of which is analogous to the single, global plot state but with the considerable advantage that multiple axes can exist on one figure, each with independent properties.

Axes and Subplots

The figure.add_axes method allows precise control over the position of axes within a figure (e.g., enabling you to embed a smaller plot within the main). Positioning of plot elements uses a 0 → 1 coordinate system, where 1 is the width or height of the figure. You can specify the position using a four-element list or tuple to set bottom-left and top-right bounds [bottom(h*0.2), left(w*0.2), top(h*0.8), right(w*0.8)]:

fig.add_axes([0.2, 0.2, 0.8, 0.8])

Example 1-2 shows the code needed to insert smaller axes into larger ones, using our random test data. The result is shown in Figure 1-3.

Example 1-2. A plot insert with figure.axes

fig = plt.figure(figsize=(8,4))

# --- Main Axes

ax = fig.add_axes([0.1, 0.1, 0.8, 0.8])

ax.set_title('Main Axes with Insert Child Axes')

ax.plot(x, y[:,0])

ax.set_xlabel('Date')

ax.set_ylabel('Cum. sum')

# --- Inserted Axes

ax = fig.add_axes([0.15, 0.15, 0.3, 0.3])

ax.plot(x, y[:,1], color='g') # 'g' for green

ax.set_xticks([]);

This selects the first column of our random NumPy y-data.

Removes the x ticks and labels from our embedded plot.

Figure 1-3. Inserted plot with figure.add_axes

Although add_axes gives us a lot of scope for fine-tuning the appearance of our charts, most of the time Matplotlib’s built-in grid-layout system makes life much easier.8 The simplest option is to use figure.subplots, which allows you to specify row-column layouts of equal-sized plots. If you want a grid with different-sized plots, the gridspec module is your go-to.

Calling subplots without arguments returns a figure with single axes. This is closest in use to using the Pyplot state machine. Example 1-3 shows the figure and axes equivalent to the pyplot demo in Example 1-1, producing the chart in Figure 1-2. Note the use of “setter” methods for figure and axes.

Example 1-3. Plotting with single figure and axes

figure, ax = plt.subplots()

plots = ax.plot(x, y, label='')

figure.set_size_inches(8, 4)

ax.legend(plots, ('foo', 'bar', 'baz'), loc='best', framealpha=0.25,

prop={'size':'small', 'family':'monospace'})

ax.set_title('Random trends')

ax.set_xlabel('Date')

ax.set_ylabel('Cum. sum')

ax.grid(True)

figure.text(0.995, 0.01, u'\u00a9 Acme Designs 2021',

ha='right', va='bottom')

figure.tight_layout()

Calling subplots with arguments for number of rows (nrows) and columns (ncols) (as shown in Example 1-4) allows multiple plots to be placed on a grid layout (see the results in Figure 1-4). The call to subplots returns the figure and an array of axes, in row-column order. In the example, we specify one column so axes is a single array of three stacked axes. We make use of Python’s handy zip method to produce three dictionaries with line data. zip takes lists or tuples of length n and returns n lists, formed by matching the elements by order:

letters = ['a', 'b']

numbers = [1, 2]

zip(letters, numbers)

Out:

[('a', 1), ('b', 2)]

In the for loop, we use enumerate to supply an index i, which we use to select an axis by row, using our zipped labelled_data to provide the plot properties.

Note the shared x- and y-axes specified in the subplots call. This allows easy comparison of the three charts, particularly on the now normalized y-axis. To avoid redundant x labels, we only call set_xlabel on the last row, using Python’s handy negative indexing.

Example 1-4. Using subplots

fig, axes = plt.subplots(

nrows=3, ncols=1,

sharex=True, sharey=True,

figsize=(8, 8))

labelled_data = zip(y.transpose(),

('foo', 'bar', 'baz'), ('b', 'g', 'r'))

fig.suptitle('Three Random Trends', fontsize=16)

for i, ld in enumerate(labelled_data):

ax = axes[i]

ax.plot(x, ld[0], label=ld[1], color=ld[2])

ax.set_ylabel('Cum. sum')

ax.legend(loc='upper left', framealpha=0.5,

prop={'size':'small'})

axes[-1].set_xlabel('Date')

Specifies a subplot grid of three rows by one column.

We want to share x- and y-axes, automatically adjusting limits for easy comparison.

Switch y to row-column and zip the line data, labels, and line colors together.

Labels the last of the shared x-axes.

Now that we’ve covered the two ways in which IPython and Matplotlib engage interactively, using the global state (accessed through plt) and the object-oriented API, let’s look at a few of the common plot types you’ll use to explore your datasets.

Figure 1-4. Three subplots

Plot Types

As well as the line plot just demonstrated, Matplotlib has a number of plot types available. I’ll now demonstrate a few of the ones commonly used in exploratory data visualization.

Bar Charts

The humble bar chart is a staple for a lot of visual data exploration. As with most of Matplotlib charts, there’s a good deal of customization possible. We’ll now run through a few variants to give you the gist.

The code in Example 1-5 produces the bar chart in Figure 1-5. Note that you have to specify your own bar and label locations. This kind of flexibility is beloved by hardcore Matplotlibbers and is pretty easy to get the hang of. Nevertheless, it’s the sort of thing that can get tedious. It’s trivial to write some helper methods here, and there are many libraries that wrap Matplotlib and make things a little more user-friendly. As we’ll see in Chapter 2, Panda’s built-in Matplotlib-based plots are quite a bit simpler to use.

Example 1-5. A simple bar chart

labels = ["Physics", "Chemistry", "Literature", "Peace"]

foo_data =   [3, 6, 10, 4]

bar_width = 0.5

xlocations = np.array(range(len(foo_data))) + bar_width

plt.bar(xlocations, foo_data, width=bar_width)

plt.yticks(range(0, 12))

plt.xticks(xlocations, labels)

plt.title("Prizes won by Fooland")

plt.gca().get_xaxis().tick_bottom()

plt.gca().get_yaxis().tick_left()

plt.gcf().set_size_inches((8, 4))

Here we create the middle bar locations, two bar_width’s apart.

This places tick labels at the middle of the bars.

Figure 1-5. A simple bar chart

Bar charts with multiple groups are particularly useful. In Example 1-6, we add some more country data (for a mythical Barland) and use the subplots method to produce grouped bar charts (see Figure 1-6). Once again we specify the bar locations manually, adding two bar groups—this time with ax.bar. Note that our axes’ x-limits are automatically rescaled in a sensible fashion, at increments of 0.5:

ax.get_xlim()

# Out: (-0.5, 3.5)

Use the respective setter methods (set_xlim, in this case) if autoscaling doesn’t achieve the desired look.

Example 1-6. Creating a grouped bar chart

labels = ["Physics", "Chemistry", "Literature", "Peace"]

foo_data = [3, 6, 10, 4]

bar_data = [8, 3, 6, 1]

fig, ax = plt.subplots(figsize=(8, 4))

bar_width = 0.4

xlocs = np.arange(len(foo_data))

ax.bar(xlocs-bar_width, foo_data, bar_width,

color='#fde0bc', label='Fooland')

ax.bar(xlocs, bar_data, bar_width, color='peru', label='Barland')

#--- ticks, labels, grids, and title

ax.set_yticks(range(12))

ax.set_xticks(ticks=range(len(foo_data)))

ax.set_xticklabels(labels)

ax.yaxis.grid(True)

ax.legend(loc='best')

ax.set_ylabel('Number of prizes')

fig.suptitle('Prizes by country')

fig.tight_layout(pad=2)

fig.savefig('mpl_barchart_multi.png', dpi=200)

With a width of 1 for our two-bar groups, this bar width gives 0.1 bar padding.

Matplotlib supports standard HTML colors, taking hex values or a name.

We use the pad argument to specify padding around the figure as a fraction of the font size.

This saves the figure at the high resolution of 200 dots per inch.

Figure 1-6. Grouped bar charts

It’s often useful to use horizontal bars, particularly if there are a lot of them and/or you are using tick labels, which are likely to run into one another if placed on the same line. Turning Figure 1-6 on its side is easy enough, requiring only that we replace the bar method with its horizontal counterpart barh and switch the axis labels and limits (see Figure 1-7).

Example 1-7. Converting Example 1-6 to horizontal bars

# ...

ylocs = np.arange(len(foo_data))

ax.barh(ylocs-bar_width, foo_data, bar_width, color='#fde0bc',

label='Fooland')

ax.barh(ylocs, bar_data, bar_width, color='peru', label='Barland')

# --- labels, grids and title, then save

ax.set_xticks(range(12))

ax.set_yticks(ticks=ylocs-bar_width/2)

ax.set_yticklabels(labels)

ax.xaxis.grid(True)

ax.legend(loc='best')

ax.set_xlabel('Number of prizes')

# ...

To create a horizontal bar chart, we use barh in place of bar.

A horizontal chart necessitates swapping the horizontal and vertical axes.

Figure 1-7. Turning the bars on their side

Stacked bars are easy to achieve in Matplotlib.9 Example 1-8 converts Figure 1-6 to a stacked form; Figure 1-8 shows the result. The trick is to use the bottom argument to bar to set the bottom of the raised bars as the top of the previous group.

Example 1-8. Converting Example 1-6 to stacked bars

# ...

bar_width = 0.8

xlocs = np.arange(len(foo_data))

ax.bar(xlocs, foo_data, bar_width, color='#fde0bc',

label='Fooland')

ax.bar(xlocs, bar_data, bar_width, color='peru',

label='Barland', bottom=foo_data)

# --- labels, grids and title, then save

ax.set_yticks(range(18))

ax.set_xticks(ticks=xlocs)

ax.set_xticklabels(labels)

# ...

The foo_data and bar_data bar groups share the same x-locations.

The bottom of the bar_data group is the top of the foo_data, providing stacked bars.

Figure 1-8. Stacked bar chart

Scatter Plots

Another useful chart is the scatter plot, which takes 2D arrays of points with options for point size, color, and more.

Example 1-9 shows the code for a quick scatter plot, using Matplotlib autoscaling for x and y limits. We create a noisy line by adding normally distributed random numbers (sigma of 10). Figure 1-9 shows the resulting chart.

Example 1-9. A simple scatter plot

num_points = 100

gradient = 0.5

x = np.array(range(num_points))

y = np.random.randn(num_points) * 10 + x*gradient

fig, ax = plt.subplots(figsize=(8, 4))

ax.scatter(x, y)

fig.suptitle('A Simple Scatterplot')

randn gives normally distributed random numbers, which we scale to be within 0 and 10 and to which we then add an x-dependent value.

The equally sized x and y arrays provide the point coordinates.

Figure 1-9. A simple scatter plot

We can adjust the size and color of individual points by passing an array of marker sizes and color indices to the current default colormap. One thing to note, which can be confusing, is that we are specifying the area of the markers’ bounding boxes, not the circles’ diameters. This means if we want points to double the diameter of the circles, we must increase the size by a factor of four.10 In Example 1-10, we add size and color information to our simple scatter plot, producing Figure 1-10.

Example 1-10. Adjusting point size and color

num_points = 100

gradient = 0.5

x = np.array(range(num_points))

y = np.random.randn(num_points) * 10 + x*gradient

fig, ax = plt.subplots(figsize=(8, 4))

colors = np.random.rand(num_points)

size = np.pi * (2 + np.random.rand(num_points) * 8) ** 2

ax.scatter(x, y, s=size, c=colors, alpha=0.5)

fig.suptitle('Scatterplot with Color and Size Specified')

This produces 100 random color values between 0 and 1 for the default colormap.

We use the power notation ** to square values between 2 and 10, the width range for our markers.

We use the alpha argument to make our markers half-transparent.

Figure 1-10. Adjusting point size and color

MATPLOTLIB COLORMAPS

Matplotlib has a huge variety of colormaps available, the choice of which can significantly improve the quality of your visualization. See the colormap docs for details.

Adding a regression line

A regression line is a simple predictive model of the correlation between two variables, in this case the x and y coordinates of our scatter plot. The line is essentially a best fit through the points of the plot, and adding one to a scatter plot is a useful dataviz technique and a good way to demo Matplotlib, NumPy interaction.

In Example 1-11 NumPy’s very useful polyfit function is used to generate the gradient and constant of a best-fit line for the points defined by the x and y arrays. We then plot this line on the same axes as the scatter plot (see Figure 1-11).

Example 1-11. Scatter plot with regression line

num_points = 100

gradient = 0.5

x = np.array(range(num_points))

y = np.random.randn(num_points) * 10 + x*gradient

fig, ax = plt.subplots(figsize=(8, 4))

ax.scatter(x, y)

m, c = np.polyfit(x, y ,1)

ax.plot(x, m*x + c)

fig.suptitle('Scatterplot With Regression-line')

We use NumPy’s polyfit in 1D to get a line gradient (m) and constant (c) for a best-fit line through our random points.

Use the gradient and constant to plot a line on the scatter plot’s axes (y = mx + c).

Figure 1-11. Scatter plot with regression line

It’s generally a good idea to plot confidence intervals when doing line regression. This gives an idea of how reliable the line fit is, based on the number and distribution of the points. Confidence intervals can be achieved with Matplotlib and NumPy, but it is a little awkward. Luckily, there is a library built on Matplotlib that has extra, specialized functions for statistical analysis and data visualization and, in the opinion of many, looks a lot better than Matplotlib’s defaults. That library is Seaborn, which we are going to take a quick look at now.

Seaborn

There are a number of libraries that wrap the powerful plotting abilities of Matplotlib in a more user-friendly guise11 and, as important for us data visualizers, play nicely with Pandas:

- Bokeh is an interactive visualization library with the Web in mind, producing browser-rendered output and therefore playing very nicely with IPython Notebook. It’s a great achievement, with a design philosophy similar to D3’s.12

But for the kind of interactive, exploratory dataviz necessary to get a feel for your data and suggest visualizations, I recommend Seaborn. Seaborn extends Matplotlib with some powerful statistical plots and is well integrated with the PyData stack, playing nicely with NumPy, Pandas, and the statistical routines found in Scipy and Statsmodels.

One of the nice things about Seaborn is that it doesn’t hide the Matplotlib API, allowing you to tweak your charts with Matplotlib’s extensive tools. In this sense, it’s not a replacement for Matplotlib and the relevant skills, but a very impressive extension.

To work with Seaborn, simply extend your standard Matplotlib imports:

import numpy as np

import pandas as pd

import seaborn as sns # relies on matplotlib

import matplotlib as mpl

import matplotlib.pyplot as plt

Matplotlib provides a number of plotting styles which can be invoked by calling a use method with a style key. Let’s set the current style to Seaborn’s default, which will provide a subtle gray grid to the charts:

matplotlib.style.use('seaborn')

You can checkout all available styles and their visual effects here.

Many of Seaborn’s functions are designed to accept a Pandas DataFrame, allowing you to specify, for example, the column values describing 2D scattered points. Let’s take our existing x and y arrays from Example 1-9 and use them to make some dummy data.

data = pd.DataFrame({'dummy x':x, 'dummy y':y})

We now have some data with columns of x ('dummy_x') and y ('dummy_y') values. Example 1-12 demonstrates the use of Seaborn’s dedicated linear regression plot lmplot, which produces the chart in Figure 1-12. Note that for some Seaborn plots, to adjust figure size we pass a size (height) in inches and an aspect ratio (width/height). Note also that Seaborn shares pyplot’s global context.

Example 1-12. Linear regression plot with Seaborn

data = pd.DataFrame({'dummy x':x, 'dummy y':y})

sns.lmplot(data=data, x='dummy x', y='dummy y',

height=4, aspect=2)

plt.tight_layout()

plt.savefig('mpl_scatter_seaborn.png')

The x and y arguments specify the column names of the DataFrame data which define the coordinates of the plot points.

To set figure size, we provide the height in inches and an aspect ratio of width/height.

Seaborn shares the pyplot global context, allowing you to save its plots as you would Matplotlib’s.

Figure 1-12. Linear regression plot with Seaborn

As you would expect from a library that places an emphasis on attractive-looking plots, Seaborn allows a lot of visual customization. Let’s make a few changes to the look of Figure 1-12 and adjust the confidence interval to the standard error estimate of 68% (see Figure 1-13 for the result):

sns.lmplot(data=data, x='dummy x', y='dummy y', height=4, aspect=2,

scatter_kws={"color": "slategray"},

line_kws={"linewidth": 2, "linestyle":'--',

"color": "seagreen"},

markers='D',

ci=68)

Provide the scatter plot component’s keyword arguments, setting our points’ color to slate gray.

Provide the line plot component’s keyword arguments, setting line width and style.

Sets the plot markers to diamonds using Matplotlib marker code D.

We set a confidence interval of 68%, the standard error estimate.

Figure 1-13. Customizing the Seaborn scatter plot

Seaborn offers a number of useful plots beyond Matplotlib’s basic set. Let’s take a look at one of the most interesting, using Seaborn’s FacetGrid to plot reflections of multidimensional data.

FacetGrids

Often referred to as “lattice” or “trellis” plotting, the ability to draw multiple instances of the same plot on different subsets of your dataset is a good way to get a bird’s-eye view of your data. Large amounts of information can be presented in one plot, and relationships between the different dimensions can be quickly apprehended. This technique is related to the small multiples popularized by Edward Tufte.

FacetGrids require the data to be in the form of a Pandas DataFrame (see [Link to Come]) and in a form referred to by Hadley Whickam, creator of ggplot, as “tidy,” meaning each column in the DataFrame should be a variable and each row an observation.

Let’s use Tips, one of Seaborn’s test datasets,13 to show a FacetGrid in action. Tips is a small set of data showing the distribution of tips by various dimensions, such as day of the week or whether the customer was a smoker. First let’s load our Tips dataset into a Pandas DataFrame using the load_dataset method:

In [0]: tips = sns.load_dataset('tips')

Out[0]:

total_bill   tip     sex smoker   day    time  size

0         16.99  1.01  Female     No   Sun  Dinner     2

1         10.34  1.66    Male     No   Sun  Dinner     3

2         21.01  3.50    Male     No   Sun  Dinner     3

3         23.68  3.31    Male     No   Sun  Dinner     2

...

To create a FacetGrid, we specify the tips DataFrame and a column of interest, such as the smoking status of the customer. This column will be used to create our plot groups; in this case, 'smoker=Yes' and 'smoker=No'. We then use the grid’s map method to create multiple scatter plots of tip size against total bill.

g = sns.FacetGrid(tips, col="smoker", height=4, aspect=1)

g.map(plt.scatter, "total_bill", "tip")

map takes a plot class, in this case scatter, and two (tips) dimensions required for this scatter plot.

This produces the two scatter plots shown in Figure 1-14, one for each smoker status, with tips and total bills correlated.

Figure 1-14. A Seaborn FacetGrid using scatter plots

We can include another dimension of the tips data by specifying the marker to be used in our scatter plots. Let’s make it a red diamond for females and a blue square for males:

pal = dict(Female='red', Male='blue')

g = sns.FacetGrid(tips, col="smoker",

hue="sex", hue_kws={"marker": ["D", "s"]},

palette=pal, height=4, aspect=1, )

g.map(plt.scatter, "total_bill", "tip", alpha=.4)

g.add_legend();

Adds a marker color (hue) for the sex dimension with diamond (D) and square (s) shapes, and uses our color palette (pal) to make them red and blue.

Figure 1-15. Scatter plot with diamond and square markers for sex

We can use rows as well as columns to create subsets of the data by dimension. Combining the two allows, with the help of a regplot,14 five dimensions to be explored:

pal = dict(Female='red', Male='blue')

g = sns.FacetGrid(tips, col="smoker", row="time",

hue="sex", hue_kws={"marker": ["D", "s"]},

palette=pal, height=4, aspect=1, )

g.map(sns.regplot, "total_bill", "tip", alpha=.4)

g.add_legend();

Adds a time row to separate tips by lunch and dinner.

Figure 1-16 shows four regplots producing a linear-regression model fit with confidence intervals for Female and Male hue-groups. The plot titles show the data subset being used, each row having the same time and smoker status.

Figure 1-16. Visualizing five dimensions

We can achieve the same effect using the lmplot we saw in Example 1-12, which wraps the functionality of FacetGrid and regplot for convenience. The following code produces Figure 1-16.

pal = dict(Female='red', Male='blue')

sns.lmplot(x="total_bill", y="tip", hue="sex",\

markers=["D", "s"],

col="smoker", row="time", data=tips, palette=pal,

height=4, aspect=1

);

Note the use of a markers keyword as opposed to the kws_hue dictionary we used with the FacetGrid plot.

lmplot offers a nice shortcut to producing FacetGrid regplots, but FacetGrid’s map allows you to use the full panoply of Seaborn and Matplotlib charts to create plots on dimensional subsets. It’s a very powerful technique and a great way to drill down into your data.

Pairgrids

Pairgrids are another rather cool Seaborn plot type that provide a way to quickly assess multidimensional data. Unlike with FacetGrids, you don’t divide the dataset into subsets that are then compared by designated dimensions. With Pairgrids, the dataset’s dimensions are all compared pair-wise in a square grid. By default all dimensions are compared, but you can specify which ones get plotted by providing a list to the vars parameter when declaring the Pairgrid.15

Let’s demonstrate the utility of this pair-wise comparison by using the classic Iris dataset, showing some vital statistics for a set containing members of three Iris species. First we’ll load the example dataset:

In [0]: iris = sns.load_dataset('iris')

In [1]: iris.head()

Out[1]:

sepal_length sepal_width  petal_length  petal_width  species

0           5.1         3.5           1.4          0.2   setosa

1           4.9         3.0           1.4          0.2   setosa

2           4.7         3.2           1.3          0.2   setosa

...

To capture the relationship between petal and sepal dimensions by species, we first create a PairGrid object, set its hue to species, and then use its mapping methods to create plots on and off the diagonal of the pair-wise grid, producing the charts in Figure 1-17.

sns.set_theme(font_scale=1.5)

g = sns.PairGrid(iris, hue="species")

g.map_diag(plt.hist)

g.map_offdiag(plt.scatter)

g.add_legend();

Tweaks the font size using Seaborn’s set_theme method (see here to see the full list of available tweaks).

Sets the markers and subbars to be colored by species.

Places histograms of the species’ dimensions on the grid’s diagonal.

Uses standard scatter plots to compare the dimensions of the diagonal.

As you can see in Figure 1-17, a few lines of Seaborn goes a long way in creating a richly informative set of plots correlating the different Iris metrics. This plot is known as a scatter-plot matrix and is a great way of finding linear correlations between pairs of variables in a multivariate set. As it stands, there is redundancy in the grid: for example, plots for sepal_width-petal_length and petal_length-septal_width. PairGrid gives you the opportunity to use the redundant plots above or below the main diagonal to provide a different reflection of the data. Check out some of the examples at the Seaborn docs for more info.16

I’ve covered a few of the Seaborn plots in this section, and you’ll be seeing a few more when we explore our Nobel Prize dataset in the next chapter. But Seaborn has a lot of other very handy and very powerful plotting tools, mainly of a statistical nature. For further investigation, I’d recommend starting with the main Seaborn documentation. There are some nice examples, a well-documented API, and some good tutorials that should complement what you’ve learned in this chapter.

Figure 1-17. Pairgrid summation of Iris measures

Summary

This chapter introduced Matplotlib, Python’s plotting powerhouse. It’s a big, mature library with lots of documentation and an active community. If you have a particular customization in mind, chances are there’s an example out there somewhere. I’d recommend firing up a Jupyter Notebook and playing around with a dataset.

We saw how Seaborn extends Matplotlib with some useful statistical methods and that it has what many consider to be superior aesthetics. It also allows access to the Matplotlib figure and axes internals, allowing full customization if required.

In the next chapter we’ll use Matplotlib along with Pandas to explore our freshly scraped and cleaned Nobel dataset. We’ll use some of the plot types demonstrated in this chapter and see a few useful new ones.

1   If you have errors trying to start a GUI session, try changing the backend setting (e.g., if using OS X and %matplotlib qt doesn’t work, try %matplotlib osx).

2   IPython has a large number of such functions to enable a whole slew of useful extras to the vanilla Python interpreter. Check them out on the IPython website.

3   This was inspired by Matlab.

4   See the docs for more details.

5   See the Matplotlib website for details.

6   As well as providing many formats, it also understands LaTex math mode, which means you can use mathematical symbols in the titles, legends, and the like. This is one of the reasons Matplotlib is much beloved by academics, as it is quite capable of journal-quality images.

7   More details are available on the Matplotlib website.

8   The handy tight_layout option assumes grid-layout subplots.

9   It’s questionable whether stacked bar charts are a particularly good way of appreciating groups of data. See Solomon Messing’s blog for a nice discussion and one example of “good” use.

10   Setting marker size, rather than width or radius, is actually a good default, making it proportional to whatever value we are trying to reflect.

11   It’s generally agreed that Matplotlib’s defaults aren’t that great and making them better is an easy win for any wrapper.

12   Both D3 and Bokeh tip their hats to the classic visualization text, Leland Wilkinson’s The Grammar of Graphics (Springer).

13   Seaborn has a number of handy datasets, which you can find on GitHub.

14   regplot is equivalent to lmplot, used in Example 1-12. The latter combines regplot and FacetGrid for convenience.

15   There are also x_vars and y_vars parameters enabling you to specify nonsquare grids.

16   For the curious, there’s a D3 example which builds a scatter-plot matrix at the bl.ocks.org site.

Chapter 2. Exploring Data with Pandas

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 11th chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at sevans@oreilly.com.

In the previous chapter, we cleaned the Nobel Prize dataset that we scraped from Wikipedia in [Link to Come]. Now it’s time to start exploring our shiny new dataset, looking for interesting patterns, stories to tell, and anything else that could form the basis for an interesting visualization.

First off, let’s try to clear our minds and take a long, hard look at the data to hand to get a broad idea of the visualizations suggested. Example 2-1 shows the form of the Nobel dataset, with categorical, temporal, and geographical data.

Example 2-1. Our cleaned Nobel Prize dataset

[{

'category': u'Physiology or Medicine',

'date_of_birth': u'8 October 1927',

'date_of_death': u'24 March 2002',

'gender': 'male',

'link': u'http://en.wikipedia.org/wiki/C%C3%A9sar_Milstein',

'name': u'C\xe9sar Milstein',

'country': u'Argentina',

'place_of_birth': u'Bah\xeda Blanca ,  Argentina',

'place_of_death': u'Cambridge , England',

'year': 1984,

'born_in': NaN

},

...

}]

The data in Example 2-1 suggests a number of stories we might want to investigate, among them:

- Gender disparities among the prize winners

- National trends (e.g., which country has most prizes in Economics)

- Details about individual winners, such as their average age on receiving the prize or life expectancy

- Geographical journey from place of birth to adopted country using the born_in and country fields

These investigative lines form the basis for the coming sections, which will probe the dataset by asking questions of it, such as “How many women other than Marie Curie have won the Nobel Prize for Physics?,” “Which countries have the most prizes per capita rather than absolute?,” and “Is there a historical trend to prizes by nation, a changing of the guard from old (science) world (big European nations) to new (US and upcoming Asians)?” Before beginning our explorations, let’s ready our tools and load our Nobel Prize dataset.

Starting to Explore

To start our exploration, let’s fire up a Jupyter notebook from the command line:

$ jupyter notebook

We’ll use the magic matplotlib command to enable inline plotting:

%matplotlib inline

Then import the standard set of data exploration modules:

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

import json

import matplotlib

import seaborn as sns

Now we’ll make a few adjustments to the plotting parameters and the general look and feel of the charts. Make sure to change the style before adjusting figure sizes, fonts and the rest:

matplotlib.style.use('seaborn')

plt.rcParams['figure.figsize'] = (8, 4)

plt.rcParams['font.size'] = '12'

We’ll use the seaborn theming for our charts, arguably more attractive than Matplotlib’s default.

Sets the default plotting size to eight inches by four.

At the end of [Link to Come], we saved our clean dataset as a json file. Let’s load the clean data into a Pandas DataFrame, ready to begin exploring.

df = pd.read_json(open('data/nobel_winners_cleaned.json'))

Let’s get some basic information about our dataset’s structure:

df.info()

&lt;class 'pandas.core.frame.DataFrame'&gt;

RangeIndex: 858 entries, 0 to 857

Data columns (total 13 columns):

#   Column          Non-Null Count  Dtype

---  ------          --------------  -----

0   category        858 non-null    object

1   country         858 non-null    object

2   date_of_birth   858 non-null    datetime64[ns, UTC]

3   date_of_death   559 non-null    datetime64[ns, UTC]

4   gender          858 non-null    object

5   link            858 non-null    object

6   name            858 non-null    object

7   place_of_birth  831 non-null    object

8   place_of_death  524 non-null    object

9   text            858 non-null    object

10  year            858 non-null    int64

11  award_age       858 non-null    int64

12  born_in         102 non-null    object

dtypes: datetime64[ns, UTC](2), int64(2), object(9)

memory usage: 87.3+ KB

Note that our dates of birth and death columns have the standard Pandas datatype of object. In order to make date comparisons, we’ll need to convert those to the datetime type, datetime64. We can use Pandas’ to_datetime method to achieve this conversion:

df.date_of_birth = pd.to_datetime(df.date_of_birth)

df.date_of_death = pd.to_datetime(df.date_of_death)

Running df.info() should now show two datetime columns:

df.info()

...

date_of_birth     858 non-null datetime64[ns, UTC]

date_of_death     559 non-null datetime64[ns, UTC]

...

UTC is the primary time standard by which the world regulates clocks and time. It’s almost always desirable to work to this standard.

to_datetime usually works without needing extra arguments, but it’s worth checking the converted columns to make sure. In the case of our Nobel Prize dataset, everything checks out.

Plotting with Pandas

Both Pandas Series and DataFrames have integrated plotting, which wraps the most common Matplotlib charts, a few of which we explored in the last chapter. This makes it easy to get quick visual feedback as you interact with your DataFrame. And if you want to visualize something a little more complicated, the Pandas containers will play nicely with vanilla Matplotlib. You can also adapt the plots produced by Pandas using standard Matplotlib customizations.

Let’s look at an example of Pandas’ integrated plotting, starting with a basic plot of gender disparity in Nobel Prize wins. Notoriously, the Nobel Prize has been distributed unequally among the sexes. Let’s get a quick feel for that disparity by using a bar plot on the gender category. Example 2-2 produces Figure 2-1, showing the huge difference, with men receiving 811 of the 858 prizes in our dataset.

Example 2-2. Using Pandas’ integrated plotting to see gender disparities

by_gender = df.groupby('gender')

by_gender.size().plot(kind='bar')

Figure 2-1. Prize counts by gender

In Example 2-2, the Series produced by the gender group’s size method has its own integrated plot method, which turns the raw numbers into a chart:

by_gender.size()

Out:

gender

female     47

male      811

dtype: int64

In addition to the default line plot, the Pandas plot method takes a kind argument to select among other possible plots. Among the more commonly used are:

- bar or barh (h for horizontal) for bar plots

- hist for a histogram

- box for a box plot

- scatter for scatter plots

You can find a full list of Pandas’ integrated plots in the docs as well as some Pandas plotting functions that take DataFrames and Series as arguments.

Let’s extend our investigation into gender disparities and start extending our plotting know-how.

Gender Disparities

Let’s break down the gender numbers shown in Figure 2-1 by category of prize. Pandas’ groupby method can take a list of columns to group by, with each group being accessed by multiple keys.

by_cat_gen = df.groupby(['category','gender'])

by_cat_gen.get_group(('Physics', 'female'))[['name', 'year']]

Out:

name  year

269    Maria Goeppert-Mayer  1963

612  Marie Skłodowska-Curie  1903

Gets a group using a category and gender key.

Using the size method to get the size of these groups returns a Series with a MultiIndex that labels the values by both category and gender:

by_cat_gen.size()

Out:

category                gender

Chemistry               female      4

male      167

Economics               female      1

male       74

...

Physiology or Medicine  female     11

male      191

dtype: int64

We can plot this multi-indexed Series directly, using hbar as the kind argument to produce a horizontal bar chart. This code produces Figure 2-2:

by_cat_gen.size().plot(kind='barh')

Figure 2-2. Plotting multikey groups

Figure 2-2 is a little crude and makes comparing gender disparities harder than it should be. Let’s go about refining our charts to make those disparities clearer.

Unstacking Groups

Figure 2-2 isn’t the easiest chart to read, even were we to improve the sorting of the bars. Handily, Pandas Series have a cool unstack method that takes the multiple indices—in this case, gender and category—and uses them as columns and indices, respectively, to create a new DataFrame. Plotting this DataFrame gives a much more usable plot, as it compares prize wins by gender. The following code produces Figure 2-3:

by_cat_gen.size().unstack().plot(kind='barh')

Figure 2-3. Unstacked Series of group sizes

Let’s improve Figure 2-3 by ordering the bar groups by number of female winners (low to high) and adding a total winners bar group for comparison. Example 2-3 produces the chart in Figure 2-4.

Example 2-3. Sorting and summing our gender groups

cat_gen_sz = by_cat_gen.size().unstack()

cat_gen_sz['total'] = cat_gen_sz.sum(axis=1)

cat_gen_sz = cat_gen_sz.sort_values(by='female', ascending=True)

cat_gen_sz[['female', 'total', 'male']].plot(kind='barh')

Sums the male and female totals. The axis argument is 0 for index sum, 1 for columns.

Sorts the rows using the female field, from low to high.

Figure 2-4. Bars ordered by number of female winners

Ignoring Economics, a recent and contentious addition to the Nobel Prize categories, Figure 2-4 shows that the largest discrepancy in the number of male and female prize winners is in Physics, with only two female winners. Let’s remind ourselves who they are:

df[(df.category == 'Physics') & (df.gender == 'female')]\

[['name', 'country','year']]

Out:

name    country  year

269    Maria Goeppert-Mayer  United States  1963

612  Marie Skłodowska-Curie         Poland  1903

While most people will have heard of Marie Curie, who is actually one of the four illustrious winners of two Nobel Prizes, few have heard of Maria Goeppert-Mayer.1 This ignorance is surprising, given the drive to encourage women into science. I would want my visualization to enable people to discover and learn a little about Maria Goeppert-Mayer.

Historical Trends

It would be interesting to see if there has been any increase in female prize allocation in recent years. One way to visualize this would be as grouped bars over time. Let’s run up a quick plot, using unstack as in Figure 2-3 but using the year and gender columns.

by_year_gender = df.groupby(['year','gender'])

year_gen_sz = by_year_gender.size().unstack()

year_gen_sz.plot(kind='bar', figsize=(16,4))

Figure 2-5, the hard-to-read plot produced, is only functional. The trend of female prize distributions can be observed, but the plot has many problems. Let’s use Matplotlib’s and Pandas’ eminent flexibility to fix them.

Figure 2-5. Prizes by year and gender

The first thing we need to do is reduce the number of x-axis labels. By default, Matplotlib will label each bar or bar group of a bar plot, which in the case of our hundred years of prizes creates a mess of labels. What we need is the ability to thin out the number of axis labels as desired. There are various ways to do this in Matplotlib; I’ll demonstrate the one I’ve found to be most reliable. It’s the sort of thing you’re going to want to reuse, so it makes sense to stick it in a dedicated function. Example 2-4 shows a function to reduce the number of ticks on our x-axis.

Example 2-4. Reducing the number of x-axis labels

def thin_xticks(ax, tick_gap=10, rotation=45):

""" Thin x-ticks and adjust rotation """

ticks = ax.xaxis.get_ticklocs()

ticklabels = [l.get_text()\

for l in ax.xaxis.get_ticklabels()]

ax.xaxis.set_ticks(ticks[::tick_gap])

ax.xaxis.set_ticklabels(ticklabels[::tick_gap],\

rotation=rotation)

ax.figure.show()

Gets the existing locations and labels of the x-ticks, currently one per bar.

Sets the new tick locations and labels at an interval of tick_gap (default 10).

Rotates the labels for readability, by default on an upward diagonal.

As well as needing to reduce the number of ticks, the x-axis in Figure 2-5 has a discontinuous range, missing the years 1939–1945 of WWII, during which no Nobel Prizes were presented. We want to see such gaps, so we need to set the x-axis range manually to include all years from the start of the Nobel Prize to the current day.

The current unstacked group sizes use an automatic year index:

by_year_gender = df.groupby(['year', 'gender'])

by_year_gender.size().unstack()

Out:

gender  female  male

year

1901       NaN     6.0

1902       NaN     7.0

...

2014         2.0    11.0

[111 rows x 2 columns]

In order to see any gaps in the prize distribution, all we have to do is reindex this Series with one containing the full range of years:

new_index = pd.Index(np.arange(1901, 2015), name='year')

by_year_gender = df.groupby(['year','gender'])

year_gen_sz = by_year_gender.size().unstack()

.reindex(new_index)

Here we create a full-range index named year, covering all the Nobel Prize years.

We replace our discontinuous index with the new continuous one.

Another problem with Figure 2-5 is the excessive number of bars. Although we do get male and female bars side by side, it looks messy and has aliasing artifacts too. It’s better to have dedicated male and female plots but stacked so as to allow easy comparison. We can achieve this using the subplotting method we saw in “Axes and Subplots”, using the Pandas data but customizing the plot using our Matplotlib know-how. Example 2-5 shows how to do this, producing the plot in Figure 2-6.

Example 2-5. Stacked gender prizes by year

new_index = pd.Index(np.arange(1901, 2015), name='year')

by_year_gender = df.groupby(['year','gender'])

year_gen_sz = by_year_gender.size().unstack().reindex(new_index)

fig, axes = plt.subplots(nrows=2, ncols=1,

sharex=True, sharey=True, figsize=(16, 8))

ax_f = axes[0]

ax_m = axes[1]

fig.suptitle('Nobel Prize-winners by gender', fontsize=16)

ax_f.bar(year_gen_sz.index, year_gen_sz.female)

ax_f.set_ylabel('Female winners')

ax_m.bar(year_gen_sz.index, year_gen_sz.male)

ax_m.set_ylabel('Male winners')

ax_m.set_xlabel('Year')

Creates two axes, on a two (row) by one (column) grid.

We’ll share the x- and y-axes, which will make comparisons between the two plots sensible.

We provide the axis’s bar chart (bar) method with the continuous year index and the unstacked gender columns.

Figure 2-6. Prizes by year and gender, on two stacked axes

So the take-home from our investigation into gender distributions is that there is a huge discrepancy but, as shown by Figure 2-6, a slight improvement in recent years. Moreover, with Economics being an outlier, the difference is greatest in the sciences. Given the fairly small number of female prize winners, there’s not a lot more to be seen here.

Let’s now take a look at national trends in prize wins and see if there are any interesting nuggets for visualization.

National Trends

The obvious starting point in looking at national trends is to plot the absolute number of prize winners. This is easily done in one line of Pandas, broken up here for ease of reading:

df.groupby('country').size().order(ascending=False)\

.plot(kind='bar', figsize=(12,4))

This produces Figure 2-7, showing the United States with the lion’s share of prizes.

The absolute number of prizes will be bound to favor countries with large populations. Let’s look at a fairer comparison, visualizing prizes per capita.

Figure 2-7. Absolute prize wins by country

Prize Winners per Capita

The absolute number of prize winners is bound to favor larger countries, which raises the question, how do the numbers stack up if we account for population sizes? In order to test prize haul per capita, we need to divide the absolute prize numbers by population size. In [Link to Come], we downloaded some country data from the Web and stored it to MongoDB. Let’s retrieve it now and use it to produce a plot of prizes relative to population size.

First let’s get the national group sizes, with country names as index labels:

nat_group = df.groupby('country')

ngsz = nat_group.size()

ngsz.index

Out:

Index([u'Argentina', u'Australia', u'Austria', u'Azerbaijan',...

Now let’s load our country data into a DataFrame and remind ourselves of the data it contains:

df_countries = pd.read_json('data/winning_country_data.json',\

orient='index')

df_countries.loc['Japan'] # coutries indexed by name

Out:

gini                   38.1

name                  Japan

alpha3Code              JPN

area               377930.0

latlng        [36.0, 138.0]

capital               Tokyo

population        127080000

Name: Japan, dtype: object

Our country dataset is already indexed to its name column. If we add to it the ngsz national group-size Series, which also has a country name index, the two will combine on the shared indices, giving our country data a new nobel_wins column. We can then use this new column to create a nobel_wins_per_capita by dividing it by population size:

df_countries = df_countries.set_index('name')

df_countries['nobel_wins'] = ngsz

df_countries['nobel_wins_per_capita'] =\

df_countries.nobel_wins / df_countries.population

We now need only sort the df_countries DataFrame by its new nobel_wins_per_cap column and plot the Nobel Prize wins per capita, producing Figure 2-8.

df.countries.sort_values(by='nobel_wins_per_capita',\

ascending=False).nobel_per_capita.plot(kind='bar',\

figsize=(12, 4))

Figure 2-8. National prize numbers per capita

This shows the Caribbean Island of Saint Lucia taking top place. Home to the Nobel Prize–winning poet Derek Walcott, its small population of 175,000 gives it a high Nobel Prizes per capita.

Let’s see how things stack up with the larger countries by filtering the results for countries that have won more than two Nobel Prizes:

df_countries[df_countries.nobel_wins > 2]\

.sort_values(by='nobel_wins_per_capita', ascending=False)\

.nobel_wins_per_capita.plot(kind='bar')

The results in Figure 2-9 show the Scandinavian countries and Switzerland punching above their weight.

Figure 2-9. National prize numbers per capita, filtered for three or more wins

Changing the metric for national prize counts from absolute to per capita makes a big difference. Let’s now refine our search a little and focus on the prize categories, looking for interesting nuggets there.

Prizes by Category

Let’s drill down a bit into the absolute prize data and look at wins by category. This will require grouping by country and category columns, getting the size of those groups, unstacking the resulting Series and then plotting the columns of the resulting DataFrame. First we get our categories with country group sizes:

nat_cat_sz = df.groupby(['country', 'category']).size()

.unstack()

nat_cat_sz

Out:

category     Chemistry  Economics  Literature  Peace  \...

country

Argentina            1        NaN         NaN      2

Australia          NaN          1           1    NaN

Austria              3          1           1      2

Azerbaijan         NaN        NaN         NaN    NaN

Bangladesh         NaN        NaN         NaN      1

We then use the nat_cat_sz DataFrame to produce subplots for the six Nobel Prize categories:

COL_NUM = 2

ROW_NUM = 3

fig, axes = plt.subplots(ROW_NUM, COL_NUM, figsize=(12,12))

for i, (label, col) in enumerate(nat_cat_sz.iteritems()):

ax = axes[int(i/COL_NUM), i%COL_NUM]

col = col.order(ascending=False)[:10]

col.plot(kind='barh', ax=ax)

ax.set_title(label)

plt.tight_layout()

iteritems returns an iterator for the DataFrames columns in form of (column_label, column) tuples.

order orders the column’s Series by first making a copy. It is the equivalent of sort(inplace=False).

tight_layout should prevent label overlaps among the subplots. If you have any problems with tight_layout, see the end of “Titles and Axes Labels”.

This produces the plots in Figure 2-10.

A couple of interesting nuggets from Figure 2-10 are the United States’ overwhelming dominance of the Economics prize, reflecting a post-WWII economic consensus, and France’s leadership of the Literature prize.

Figure 2-10. Prizes by country and category

Historical Trends in Prize Distribution

Now that we know the aggregate prize stats by country, are there any interesting historical trends to the prize distribution? Let’s explore this with some line plots.

First, let’s increase the default font size to 20 points to make the plot labels more legible:

plt.rcParams['font.size'] = 20

We’re going to be looking at prize distribution by year and country, so we’ll need a new unstacked DataFrame based on these two columns. As previously, we add a new_index to give continuous years:

new_index = pd.Index(np.arange(1901, 2015), name='year')

by_year_nat_sz = df.groupby(['year', 'country'])\

.size().unstack().reindex(new_index)

The trend we’re interested in is the cumulative sum of Nobel Prizes by country over its history. We can further explore trends in individual categories, but for now we’ll look at the total for all. Pandas has a handy cumsum method for just this. Let’s take the United States column and plot it:

by_year_nat_sz['United States'].cumsum().plot()

This produces the chart in Figure 2-11.

Figure 2-11. Cumulative sum of US prize winners over time

The gaps in the line plot are where the fields are NaN, years when the US won no prizes. The cumsum algorithm returns NaN here. Let’s fill those in with a zero to remove the gaps:

by_year_nat_sz['United States'].fillna(0)

.cumsum().plot()

This produces the cleaner chart shown in Figure 2-12.

Figure 2-12. Cumulative sum of US prize winners over time

Let’s compare the US prize rate with that of the rest of the world:

by_year_nat_sz = df.groupby(['year', 'country'])

.size().unstack().fillna(0)

not_US = by_year_nat_sz.columns.tolist()

not_US.remove('United States')

by_year_nat_sz['Not US'] = by_year_nat_sz[not_US].sum(axis=1)

ax = by_year_nat_sz[['United States', 'Not US']]\

.cumsum().plot()

Gets the list of country column names and removes United States.

Uses our list of non-US country names to create a 'Not_US' column, the sum of all the prizes for countries in the not_US list.

This code produces the chart shown in Figure 2-13.

Figure 2-13. United States versus rest of world prize hauls

Where the 'Not_US' haul shows a steady increase over the years of the prize, the US shows a rapid increase around the end of World War II. Let’s investigate that further, looking at regional differences. We’ll focus on the two or three largest winners for North America, Europe, and Asia:

by_year_nat_sz = df.groupby(['year', 'country'])\

.size().unstack().reindex(new_index).fillna(0)

regions = [

{'label':'N. America',

'countries':['United States', 'Canada']},

{'label':'Europe',

'countries':['United Kingdom', 'Germany', 'France']},

{'label':'Asia',

'countries':['Japan', 'Russia', 'India']}

]

for region in regions:

by_year_nat_sz[region['label']] =\

by_year_nat_sz[region['countries']].sum(axis=1)

by_year_nat_sz[[r['label'] for r in regions]].cumsum()\

.plot()

Our continental country list created by selecting the biggest two or three winners in the three continents compared.

Creates a new column with a region label for each dict in the regions list, summing its countries members.

Plots the cumulative sum of all the new region columns.

This gives us the plot in Figure 2-14. The rate of Asia’s prize haul has increased slightly over the years, but the main point of note is North America’s huge increase in prizes around the mid-1940s, overtaking a declining Europe in total prizes around the mid-1980s.

Figure 2-14. Historical prize trends by region

Let’s improve the resolution of the previous national plots by summarizing the prize rates for the 16 biggest winners, excluding the outlying United States:

COL_NUM = 4

ROW_NUM = 4

by_nat_sz = df.groupby('country').size()

by_nat_sz.sort_values(ascending=False,\

inplace=True)

fig, axes = plt.subplots(COL_NUM, ROW_NUM,\

sharex=True, sharey=True,

figsize=(12,12))

for i, nat in enumerate(by_nat.index[1:17]):

ax = axes[i/COL_NUM, i%ROW_NUM]

by_year_nat_sz[nat].cumsum().plot(ax=ax)

ax.set_title(nat)

Sorts our country groups from highest to lowest win hauls.

Gets a 4×4 grid of axes with shared x- and y-axes for normalized comparison.

Enumerates over the sorted index from second row (1), excluding the US (0).

Selects the nat country name column and plots its cumulative sum of prizes on the grid axis ax.

This produces Figure 2-15, which shows some nations like Japan, Australia, and Israel on the rise historically, while others flatten off.

Figure 2-15. Prize rates for the 16 largest national winners after the US

Another good way to summarize national prize rates over time is by using a heatmap and dividing the totals by decade. The Seaborn library provides a good heatmap. Let’s import it and use its set method to increase the font size of its labels by scaling them:

import seaborn as sns

sns.set(font_scale = 1.3)

The division of data into chunks is also known as binning, as it creates bins of data. Pandas has a handy cut method for just this job, taking a column of continuous values—in our case, Nobel Prize years—and returning ranges of a specified size. You can supply the DataFrame’s groupby method with the result of cut and it will group by the range of indexed values. The following code produces Figure 2-16.

bins = np.arange(df.year.min(), df.year.max(), 10)

by_year_nat_binned = df.groupby('country',\

[pd.cut(df.year, bins, precision=0)])\

.size().unstack().fillna(0)

plt.figure(figsize=(8, 8))

sns.heatmap(\

by_year_nat_binned[by_year_nat_binned.sum(axis=1) > 2],\

cmap='rocket_r')

Gets our bin ranges for the decades from 1901 (1901, 1911, 1921…).

Cuts our Nobel Prize years into decades using the bins ranges with precision set to 0, to give integer years.

Before heatmapping, we filter for those countries with over two Nobel Prizes.

We use the continuous rocket_r heatmap to highlight the differences. Check out all the Pandas color palettes here.

Figure 2-16 captures some interesting trends, such as Russia’s brief flourishing in the 1950s, which petered out around the 1980s.

Now that we’ve investigated the Nobel Prize nations, let’s turn our attention to the individual winners. Are there any interesting things we can discover about them using the data at hand?

Figure 2-16. Nations’ Nobel Prize hauls by decade

Age and Life Expectancy of Winners

We have the date of birth for all our winners and the date of death for 559 of them. Combined with the year in which they won their prizes, we have a fair amount of individual data to mine. Let’s investigate the age distribution of winners and try to glean some idea of the winners’ longevity.

Age at Time of Award

In [Link to Come] we added an 'award_age' column to our Nobel Prize dataset by subtracting the winners’ ages from their prize years. A quick and easy win is to use Pandas’ histogram plot to assess this distribution:

df['award_age'].hist(bins=20)

Here we require that the age data be divided into 20 bins. This produces Figure 2-17, showing that the early 60s is a sweet spot for the prize and if you haven’t achieved it by 100, it probably isn’t going to happen. Note the outlier around 20, which is the recently awarded 17-year-old recipient of the Peace Prize, Malala Yousafzai.

Figure 2-17. Distribution of ages at time of award

We can use Seaborn’s displot to get a better feel for the distribution, adding a kernel density estimate (KDE)2 to the histogram. The following one-liner produces Figure 2-18, showing that our sweet spot is around 60 years of age:

sns.displot(df['award_age'], kde=True, height=4, aspect=2)

Figure 2-18. Distribution of ages at time of award with KDE superimposed

A box plot is a good way of visualizing continuous data, showing the quartiles, the first and third marking the edges of the box and the second quartile (or median average) marking the line in the box. Generally, as in Figure 2-19, the horizontal end lines (known as the whisker ends) indicate the max and min of the data. Let’s use a Seaborn box plot and divide the prizes by gender:

sns.boxplot(df, x='gender', y='award_age')

This produces Figure 2-19, which shows that the distributions by gender are similar, with women having a slightly lower average age. Note that with far fewer female prize winners, their statistics are subject to a good deal more uncertainty.

Figure 2-19. Ages of prize winners by gender

Seaborn’s rather nice violinplot combines the conventional box plot with a kernel density estimation to give a more refined view of the breakdown by age and gender. The following code produces Figure 2-20.

sns.violinplot(data=df, x='gender', y='award_age')

Figure 2-20. Violinplots of prize-age distribution by gender

Life Expectancy of Winners

Now let’s look at the longevity of Nobel Prize winners, by subtracting the available dates of death from their respective dates of birth. We’ll store this data in a new 'age_at_death' column:

df['age_at_death'] = (df.date_of_death - df.date_of_birth)\

.dt.days/365

datetime64 data can be added and subtracted in a sensible fashion, producing a Pandas timedelta column. We can use its dt method to get the interval in days, dividing this by 365 to get the age at death as a float.

We make a copy of the 'age_at_death' column,3 removing all empty NaN rows. This can then be used to make the histogram and KDE shown in Figure 2-21.

age_at_death = df[df.age_at_death.notnull()].age_at_death

sns.displot(age_at_death, bins=40, kde=True, aspect=2, height=4)

Removes all NaNs to clean the data and reduce plotting errors (e.g., distplot fails with NaNs).

Figure 2-21. Life expectancy of the Nobel Prize winners

Figure 2-21 shows the Nobel Prize winners to be a remarkably long-lived bunch, with an average age in the early 80s. This is all the more impressive given that the large majority of winners are men, who have considerably lower average life expectancies in the general population than women. One contributary factor to this longevity is the selection bias we saw earlier. Nobel Prize winners aren’t generally honored until they’re in their late 50s and 60s, which removes the subpopulation who died before having the chance to be acknowledged, pushing up the longevity figures.

Figure 2-21 shows some centenarians among the prize winners. Let’s find them:

df[df.age_at_death > 100][['name', 'category', 'year']]

Out:

name                category  year

101          Ronald Coase               Economics  1991

328   Rita Levi-Montalcini  Physiology or Medicine 1986

Now let’s superimpose a couple of KDEs to show differences in mortality for male and female recipients:

df_temp = df_temp[df.age_at_death.notnull()]

sns.kdeplot(df_temp[df_temp.gender == 'male']

.age_at_death, shade=True, label='male')

sns.kdeplot(df_temp[df_temp.gender == 'female']

.age_at_death, shade=True, label='female')

plt.legend()

Creates a DataFrame with only valid 'age_at_death' fields.

This produces Figure 2-22, which, allowing for the small number of female winners and flatter distribution, shows the male and female averages to be close. Female Nobel Prize winners seem to live relatively shorter lives than their counterparts in the general population.

Figure 2-22. Nobel Prize winner life expectancies by gender

A violinplot provides another perspective, shown in Figure 2-23.

sns.violinplot(data=df, x='gender', y='age_at_death',\

aspect=2, height=4)

Figure 2-23. Winner life expectancies by gender

Increasing Life Expectancies over Time

Let’s do a little historical demographic analysis by seeing if there’s a correlation between the date of birth of our Nobel Prize winners and their life expectancy. We’ll use one of Seaborn’s lmplots to provide a scatter plot and line-fitting with confidence intervals (see “Seaborn”).

df_temp = df[df.age_at_death.notnull()]

data = pd.DataFrame(

{'age at death':df_temp.age_at_death,

'date of birth':df_temp.date_of_birth.dt.year})

sns.lmplot(data=data, x='date of birth', y='age at death',

height=6, aspect=1.5)

Creates a temporary DataFrame, removing all the rows with no 'age_at_death' field.

Creates a new DataFrame with only the two columns of interest from the refined df_temp. We grab only the year from the date_of_birth, using its dt accessor.

This produces Figure 2-24, showing an increase in life expectancy of a decade or so over the prize’s duration.

Figure 2-24. Correlating date of birth with age at death

The Nobel Diaspora

While cleaning our Nobel Prize dataset in [Link to Come], we found duplicate entries recording the winner’s place of birth and country at time of winning. We preserved these, giving us 104 winners whose country at time of winning was different from their country of birth. Is there a story to tell here?

A good way to visualize the movement patterns from the winners’ country of birth to their adopted country is by using a heatmap to show all born_in/country pairs. The following code produces the heatmap in Figure 2-25:

by_bornin_nat = df[df.born_in.notnull()].groupby(\

['born_in', 'country']).size().unstack()

by_bornin_nat.index.name = 'Born in'

by_bornin_nat.columns.name = 'Moved to'

plt.figure(figsize=(12, 12))

ax = sns.heatmap(by_bornin_nat, vmin=0, vmax=8, cmap="crest",\

linewidth=0.5)

ax.set_title('The Nobel Diaspora')

Selects all rows with a 'born_in' field, and forms groups on this and the country column.

We rename the row index and column names to make them more descriptive.

Seaborn’s heatmap attempts to set the correct bounds for the data, but in this case, we must manually adjust the limits (vmin and vmax) to see all the cells.

Figure 2-25 shows some interesting patterns, which tell a tale of persecution and sanctuary. First, the United States is the overwhelming recipient of relocated Nobel winners, followed by the United Kingdom. Note that the biggest contingents for both (except cross-border traffic from Canada) are from Germany. Italy, Hungary, and Austria are the next largest groups. Examining the individuals in these groups shows that the majority were displaced as a result of the rise of antisemitic fascist regimes in the run-up to World War II and the increasing persecution of Jewish minorities.

Figure 2-25. The Nobel Prize diaspora

To take an example, all four of the Nobel winners who moved from Germany to the United Kingdom were German research scientists with Jewish ancestry who moved in response to the Nazis’ rise to power:

df[(df.born_in == 'Germany') & (df.country == 'United Kingdom')]

[['name', 'date_of_birth', 'category']]

Out:

name date_of_birth                category

119  Ernst Boris Chain    1906-06-19  Physiology or Medicine

484   Hans Adolf Krebs    1900-08-25  Physiology or Medicine

486           Max Born    1882-12-11                 Physics

503       Bernard Katz    1911-03-26  Physiology or Medicine

Ernst Chain pioneered the industrial production of penicillin. Hans Krebs discovered the Krebs cycle, one of the most important discoveries in biochemistry, which regulates the energy production of cells. Max Born was one of the pioneers of quantum mechanics, and Bernard Katz uncovered the fundamental properties of synaptic junctions in neurons.

There are many such illustrious names among the winning emigrants. One interesting discovery is the number of prize winners who were part of the famous Kindertransport, an organized rescue effort that took place nine months before the outbreak of WWII and saw 10,000 Jewish children from Germany, Austria, Czechoslovakia, and Poland transported to the United Kingdom. Of these children, four went on to win a Nobel Prize.

Summary

In this chapter, we explored our Nobel Prize dataset, probing the key fields of gender, category, country, and year (of prize) looking for interesting trends and stories we can tell or enable visually. We used a fair number of Matplotlib (by way of Pandas) and Seaborn’s plots, from basic bar charts to more complicated statistical summaries like violinplots and heatmaps. Mastery of these tools and the others in the Python chart armory will allow you to quickly get the feel of your datasets, which is a prerequisite to building a visualization around them. We found more than enough stories in the data to suggest a web visualization. In the next chapter we will imagine and design just such a Nobel Prize winner visualization, cherry-picking the nuggets gained in this chapter.

1   Anecdotally, no one I have asked in person or in talk audiences has known the name of the other female Nobel Prize winner for Physics.

2   See Wikipedia for details. Essentially the data is smoothed and a probability density function derived.

3   We are ignoring leap years and other subtle, complicating factors in deriving years from days.

About the Author

Kyran Dale is a jobbing programmer, ex-research scientist, recreational hacker, independent researcher, occasional entrepreneur, cross-country runner and improving jazz pianist. During 15 odd years as a research scientist he hacked a lot of code, learned a lot of libraries and settled on some favorite tools. These days he finds Python, JavaScript, and a little C++ goes a long way to solving most problems out there. He specializes in fast-prototyping and feasibility studies, with an algorithmic bent but is happy to just build cool things.

Table of Contents

1. Visualizing Data with Matplotlib

Pyplot and Object-Oriented Matplotlib

Starting an Interactive Session

Interactive Plotting with Pyplot’s Global State

Configuring Matplotlib

Setting the Figure’s Size

Points, Not Pixels

Labels and Legends

Titles and Axes Labels

Saving Your Charts

Figures and Object-Oriented Matplotlib

Axes and Subplots

Plot Types

Bar Charts

Scatter Plots

Seaborn

FacetGrids

Pairgrids

Summary

2. Exploring Data with Pandas

Starting to Explore

Plotting with Pandas

Gender Disparities

Unstacking Groups

Historical Trends

National Trends

Prize Winners per Capita

Prizes by Category

Historical Trends in Prize Distribution

Age and Life Expectancy of Winners

Age at Time of Award

Life Expectancy of Winners

Increasing Life Expectancies over Time

The Nobel Diaspora

Summary