Learning MySQL

SECOND EDITION

Get a Handle on Your Data

Vinicius M. Grippa and Sergey Kuzmichev

Learning MySQL

by Vinicius M. Grippa and Sergey Kuzmichev

Copyright © 2021 Vinicius M. Grippa and Sergey Kuzmichev. All rights reserved.

Printed in the United States of America.

Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.

- Acquisitions Editor: Andy Kwan

- Development Editor: Corbin Collins

- Production Editor: Beth Kelly

- Copyeditor: Rachel Head

- Proofreader: FILL IN PROOFREADER

- Indexer: FILL IN INDEXER

- Interior Designer: David Futato

- Cover Designer: Karen Montgomery

- Illustrator: Kate Dullea

- November 2006: First Edition

- September 2021: Second Edition

Revision History for the Second Edition

- 2021-08-25: First Release

See http://oreilly.com/catalog/errata.csp?isbn=9781492085928 for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Learning MySQL, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-492-08592-8

[LSI]

Preface

Database management systems are part of the core of many companies nowadays. Even if a business is not technology-focused, it needs to store, access, and manipulate data in a fast, secure, and reliable way. With the Covid-19 pandemic, areas that traditionally resisted digital transformation, like the judiciary systems in many countries are now being integrated through technology due to travel and meeting restrictions. And online shopping and working from home are more popular than ever before.

But it’s not just disasters that have propelled such far-reaching changes. With the advent of 5G, we will soon have many more machines connected to the internet than humans. Vast amounts of data are already being harvested, stored, and used to train machine learning models, artificial intelligence, and much more. We are living at the beginning of the next revolution.

And with the need to store more data, several database types have emerged to help with this mission, especially from the unstructured data world, including NOSQL databases like MongoDB, Cassandra, and Redis. However, we can see that traditional SQL databases remain strong. There is no sign that they will vanish in the near future. And in the SQL world, the one that is undoubtedly the most popular open source solution is MySQL.

Both authors of this book worked with many customers from all parts of the world. Along the way, we have learned lots of lessons and experienced a vast number of cases going from mission-critical monolith applications to simpler microservices applications. This book if full of the tips and advice we think most readers will find helpful for their daily activities.

Who This Book Is for

This book is primarily for people using MySQL for the first time or learning it as a second database. If you are entering the database area for the first time, the first chapters will introduce you to the database design, concepts, and how to deploy MySQL into different operating systems and in the cloud.

For those coming from another ecosystem like Postgres, Oracle, or SQL Server, the book covers backup, high-availability, and disaster and recovery strategies.

We hope all readers will also find this book to be a good companion for learning or reviewing fundamentals, from the architecture to “production environment” bits of advice.

How This Book Is Organized

We introduce many topics, from the basic installation process, database design, backups, and recovery to CPU performance analysis and bug investigation. We divide the book into four main parts:

1. Starting with MySQL

2. Using MySQL

3. MySQL in Production

4. Miscellaneous Topics

Let’s look at how we’ve organized the chapters.

Starting with MySQL

Chapter 1 explains how to install and configure the MySQL software on different operating systems. This chapter provides far more detail than most books do. We know that those initiating their career with MySQL are often unfamiliar with various Linux distributions and installation options. Running the “MySQL hello world” requires far more steps than compiling a hello world in any programming language does. You will see how to deploy MySQL in Linux, Windows, macOS, and Docker, and how to deploy instances quickly for testing.

Using MySQL

Before we dive into creating and using databases, we look at proper database design in Chapter 2. You will learn how to access your database’s features and see how the information items in your database relate to each other. You will see that lousy database designs are challenging to change and can lead to performance problems. We will introduce the concept of strong and weak entities and their relationships (foreign keys). This chapter also shows how to download and configure database examples such as Sakila, World, and Employee.

In Chapter 3, we explore the famous SQL commands that are part of the CRUD (create, read, update, and delete). We will see how to read data from an existing MySQL database, store data in it, and manipulate existing data.

In Chapter 4, we explain how to create a new MySQL database and create and modify tables, indexes, and other database structures.

Chapter 5 covers more advanced operations such as using nested queries and using different MySQL database engines. This chapter will give you the capability of performing more complex queries.

MySQL in production

With the concepts for installing and manipulating data in hand, the next step is to understand how MySQL handles simultaneous access to the same data. The ideas of isolation, transaction, and deadlocks are explored in the Chapter 6.

In Chapter 7, you will see more complex queries that you can perform in MySQL. And you will see how to observe the query plan to check whether the query is efficient or not. Finally, we explain the different engines available in MySQL (InnoDB and MyISAM are the most famous ones).

With Chapter 8 you will see how to create and delete users in the database. This step is one of the most important in terms of security since users with more privileges than needed can cause considerable damage to the database and the company’s reputation. You will see how to establish security policies, give and remove privileges, and restrict access to specific network IPs.

The Chapter 9 covers the MySQL configuration file and its options. This file contains the MySQL configuration like its buffer pool, the size of the redo log files, and all necessary settings to customize MySQL. Those familiar with MySQL will recognize the /etc/my.cnf configuration file. You will also see that it is possible to configure user access using special option files.

Databases without backup policies are headed for disaster sooner or later. In Chapter 10 we discuss the different types of backups (logical vs. physical), the options available to execute this task, and the ones that are more appropriate for large production databases.

At the end of the production part, Chapter 11 discusses the essential parameters you need to pay attention to when setting a new server. We provide formulas for that and identify whether the parameter value is the correct one for the database workload.

Miscellaneous topics

With the essential established, it is time to go beyond. Chapter 12 teaches how to monitor your database and collect data from it. Since database workload behavior can change according to the number of users, transactions, and data manipulated, identifying which resource is saturated and what is causing the problem is crucial.

Chapter 13 explains how to create a replication between two servers to provide high availability. We also introduce the cluster concept, highlighting two: InnoDB Cluster and Galera/PXC cluster.

Chapter 14 expands the MySQL universe to the cloud. You will see how to deploy DBaaS (also known as managed database service) in the most prominent three cloud providers: AWS, Azure, and Google Cloud.

In Chapter 15, we show the most commonly used tools to distribute the queries among different MySQL servers to extract even more performance from MySQL.

The last chapter, Chapter 16, introduces more advanced analysis methods, tools, and a bit of programming. In this chapter, we talk about MySQL shell, flame graphs, and how to analyze bugs.

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

TIP

This element signifies a tip or suggestion.

NOTE

This element signifies a general note.

WARNING

This element indicates a warning or caution.

Using Code Examples

Supplemental material (code examples, exercises, and more) is available for download at the Learning MySQL 2nd Edition GitHub repository.

If you have a technical question or a problem using the code examples, please send email to bookquestions@oreilly.com.

This book is here to help you get your job done. In general, if an example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but generally do not require attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: "Learning MySQL by Vincinius M. Grippa and Sergey Kuzmichev (O’Reilly). Copyright 2021 Vincinius M. Grippa and Sergey Kuzmichev, 978-1-492-08592-8.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com.

O’Reilly Online Learning

NOTE

For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed.

Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

- O’Reilly Media, Inc.

- 1005 Gravenstein Highway North

- Sebastopol, CA 95472

- 800-998-9938 (in the United States or Canada)

- 707-829-0515 (international or local)

- 707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/learn-mysql-2e.

Email bookquestions@oreilly.com to comment on or ask technical questions about this book.

For news and information about our books and courses, visit http://oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://youtube.com/oreillymedia

Acknowledgments

From Vinicius Grippa

Thanks to the following people who helped improve this book: Corbin Collins, Charly Batista, Sami Ahlroos, and Brett Holleman. Without them, this book would not achieve the excellence we strived for.

Thanks to the MySQL community members Shlomi Noach, Giuseppe Maxia, Jeremy Cole, Brendan Gregg, and its various bloggers (Planet MySQL, Several Nines, Percona Blog, MySQL Entomologist) for contributing with vast material and great tools.

Thanks to everyone at Percona who provided the means to write this book, notably Bennie Grant, Carina Punzo, and Marcelo Altmann, which helped me grow as a professional and human being.

Thanks to the O’Reilly staff who do such a classy job of publishing books and running conferences.

I want to thank my parents Divaldo and Regina, my sister Juliana, and my girlfriend Karin for having patience and supporting this project in many ways. A special thanks to Paulo Piffer, who gave me my first opportunity to work with what I love.

And last, to Sergey Kuzmichev, the co-writer of this book. Without his expertise, dedication, and hard work, this book would not be possible. I am grateful for having him as a colleague and the honor of working with him on this project.

From Sergey Kuzmichev

Chapter 1. Installing MySQL

Let’s begin our learning path by installing MySQL and accessing it for the first time.

Note that we do not rely on a single version of MySQL for this book. Instead, we drew on our collective knowledge of MySQL in the real world. The book’s core is focused on MySQL 5.7, MySQL 8.0, and Linux operating systems (mostly Ubuntu/Debian and CentOS/RHEL or its derivations) because those are what we consider the “current” versions capable of production workloads. The MySQL 5.7 and 8.0 series are still under development, which means that they will be getting newer versions with bug fixes and new features.

With MySQL becoming the most popular open source database (Oracle, which ranks first, is not open source), the demand for having a fast installation process has increased. Even with the source code available, the recipe for building software, installation is not easy to perform. It takes time to compile, and most of the time, it is necessary to install additional development libraries that expose a risk to production environments. Think of the source code as a chocolate cake recipe: you have the instructions on how to bake it, but sometimes you don’t to mess up your kitchen, or you don’t have the time to bake it, so you go to a bakery and buy a chocolate cake. For MySQL, when you want it ready to use without the effort involved in compiling it, you can use the distribution packages.

Distribution packages for MySQL are available for diverse platforms. They include Linux distributions, Windows, and macOS. These packages provide a flexible and fast way to start using MySQL. Returning to the chocolate cake example, suppose you want to change something. Maybe you want a white chocolate cake. For MySQL, we have what are called MySQL Forks, which include some different options available to choose from. The most notable forks are Oracle MySQL, Percona Server, and MariaDB. This chapter discusses these in detail.

MySQL Forks

In software engineering, a fork occurs when the source code is copied and starts its path of independent development and support from its source. The fork can follow the track close to its original version, as the Percona distribution does, or drift away like MariaDB. Because the source code is open and free, new projects can fork the code without permission from its original creator.

MySQL community

The MySQL community version, also known as the upstream or vanilla version, is the open source version distributed by Oracle. The community version drives the development of the InnoDB engine and new features, and it is the first one to receive updates, new features, and bug fixes.

Percona Server for MySQL

The Percona distribution for MySQL is a free, open source drop-in replacement for the MySQL community version, founded by Peter Zaitsev. The software includes additional customized features for security, high availability, and backup, generally available only for MySQL enterprise versions. The development closely follows the MySQL community version, focusing on improving performance and the overall MySQL ecosystem.

MariaDB Server

Created by Michael Widenius (often called Monty), the MariaDB server is by far the one that most drifted the farthest away from the other MySQL forks. In recent years, it has been developing its features and engines such as MariaDB ColumStore, and it was the first database to integrate Galera 4 clustering functionality. The MariaDB Foundation distributes it.

MySQL Enterprise

The MySQL Enterprise version is the only version so far with a commercial license (which means you need to pay to use it, like a Windows license). The MySQL Enterprise version contains all the community version functionalities plus exclusive features for security, backup, and high availability. Oracle also distributes it.

Installation Choices and Platforms

First, you must choose the MySQL version compatible with your operating system. You can verify whether the operating system is compatible with the MySQL version that you chose. The same support policies are available for Percona Server and MariaDB.

We often hear the question: is it possible to install an operating system that is not supported?

Most of the time, the answer is yes. It is possible to install MySQL on Windows 7, for example, but the risks of hitting a bug or facing an unpredictable behavior (like memory leaking or underperformance) are high. Because of these risks, we do not recommend doing this for production environments.

The next step is to decide whether to install a Development or a General Availability (GA) release. Development releases have the newest features, but we do not recommend them for production because they are not stable. GA releases, also called production or stable releases, are meant for production use.

TIP

We highly recommend using the most recent GA release because they have the latest stable bug fixes and performance improvements.

The last thing to decide is which distribution format to install for the operating system. For most use cases, a binary distribution fits most. Binary distributions are available in native format for many platforms, such as rpm packages for Linux or dmg packages for macOS. The distributions are also available in generic formats such as zip archives or compressed tar files (tarballs). On Windows, you can use the MySQL Installer to install a binary distribution.

NOTE

Watch out for whether the version is 32-bit or 64-bit. The rule of thumb is to pick the 64-bit version. Unless you are working with an ancient OS, you should not select the 32-bit version. 32-bit processors can handle a limited amount of RAM (4GB or less), whereas 64-bit processors are capable of addressing much more memory.

The installation process consists of four major steps. They are essential to install and set the minimum security requirements for the MySQL database.

1. Download the distribution that you want to install

Each distribution has its owner and, by consequence, its source. Some Linux distributions provide default packages in their repository. For example, on CentOS 8, the MySQL vanilla distribution is available from the default repositories. With the operating system having default packages, it is unnecessary to download MySQL from the website or configure a repository, and because of this, it facilitates the installation process. We will demonstrate how to install the repositories and download the files without the need to go to the website and download it during the installation process.

However, if you want to download MySQL from the website, the following links indicate where you can download them:

- MySQL

- Percona

- MariaDB

2. Install the distribution

Installing consists of the elementary steps to make MySQL functional and online, but not securing MySQL. For example, at this point, MySQL root user can connect without a password, which is something quite hazardous since the root user has privileges to do everything in the database like, for example, dropping a database.

3. Perform any necessary post-installation setup

This section is about making sure the MySQL Server is working correctly. It is essential to make sure that our server is secure, and the first step for this is executing the mysql_secure_installation process. This step will change the password for the root user, disable the access with the root user from a remote server and remove the test database.

4. Run Benchmarks

Some DBAs run benchmarks for each deployment to measure whether the performance is suitable for the project they are using. The most used is the sysbench tool. It is essential to highlight here that sysbench performs a synthetic workload, whereas when the application is running, we call it real workload. Synthetic workloads usually provide reports about the maximum server performance. Still, they cannot reproduce the real-world workload (with its inherent locks, different query execution times, stored procedures, triggers, and so on).

In the next section we’ll check the details of a few installation processes for the most used platforms.

Installing MySQL on Linux

The Linux ecosystem is diverse and has many options, like Red Hat(RHEL), Centos, Ubuntu, Debian, and others. This section focuses on only the most popular ones — otherwise, this book would be entirely about the installation process.

Installing MySQL on Centos 7

CentOS, the abbreviation for Community Enterprise Linux Operating System, was founded in 2004, and Red Hat acquired it in 2014. CentOS is the community version of Red Hat (their upstream source). So it’s pretty much identical, but it is free, and support comes from the community instead of Red Hat itself. Centos 7 was released in 2014, and its end-of-life is in 2024.

Installing MySQL 8

To install MySQL 8 on Centos 7 using yum, complete the following steps:

- Logging into Linux

Usually, for security reasons, users log into Linux servers as non-privileged users. Here is an example of a user logging into the Linux from a macOS terminal using a private key:

$ ssh -i key.pem centos@3.227.11.227

And this is the terminal after successfully connected:

[centos@ip-172-30-150-91 ~]$

- Becoming root in Linux

Now that we are connected to the server, we need to become root:

$ sudo su - root

And we will have the following in our terminal:

[root@ip-172-30-150-91 ~]#

Becoming root is important because to install MySQL it is necessary to perform tasks such as creating the MySQL user in Linux, configuring directories, and filing permissions. It is also possible to use the sudo command for all examples we will show. Still, there is a risk of forgetting to add sudo previously to the command and the installation process to be incomplete.

NOTE

This chapter will use the Linux root user in the majority of the examples (represented by # in the code lines). Another advantage of the # representation is that the # also means the comment character in Linux. If we blindly copy/paste data, it won’t run any real commands in the shell.

- Configuring the yum repository

We execute the following command to configure MySQL yum repository:

# rpm -Uvh https://repo.mysql.com/mysql80-community-release-el7.rpm

- Install MySQL 8 Community Server

Because the MySQL yum repository has multiple repositories configuration for multiple MySQL versions (5.7 and 8.0 major versions), first we have to disable all repositories:

# sed -i s/enabled=1/enabled=0/ /etc/yum.repos.d/mysql-community.repo

Next, we need to enable MySQL 8 repository and execute the following command to install MySQL 8:

# yum --enablerepo=mysql80-community install mysql-community-server

- Starting MySQL Service

Next, we start MySQL service with the systemctl command:

# systemctl start mysqld

It is also possible to start the MySQL process manually, which can be useful to troubleshoot initialization problems when MySQL is refusing to start. To start manually, indicate the location of the my.cnf file and which user can manipulate the database files and the process:

# mysqld --defauls-file=/etc/my.cnf --user=mysql

- Discovering the default password for the root user

When you install MySQL 8.0, MySQL creates a temporary password for the root user account. To identify the password of the root user account, execute the following command:

# grep "A temporary password" /var/log/mysqld.log

The command provides the output:

2020-05-31T15:04:12.256877Z 6 [Note] [MY-010454] [Server] A temporary

password is generated for root@localhost: #z?hhCCyj2aj

- MySQL Secure Installation

MySQL provides a shell script called mysql_secure_installation. This is a shell script available on Unix systems that enables you to improve the security of your server installation in the following ways:

- You can set a password for root accounts.

- You can remove root accounts that are accessible from outside the localhost.

- You can remove anonymous-user accounts.

- You can remove the test database, which by default can be accessed by anonymous users.

Execute the command mysql_secure_installation to secure MySQL Server:

# mysql_secure_installation

It will prompt you for the current password of the root account:

Enter the password for user root:

Enter the temporary password provided before in ??? and press Enter. The following message will show:

The existing password for the user account root has expired. Please

set a new password.

New password:

Re-enter new password:

NOTE

This section will cover only the basics of changing the root password to grant our access to the MySQL Server. We will show more details about creating, grant privileges, and password policy in Chapter 8.

You will need to enter the new password for the root account twice. More recent MySQL versions come with a validation policy, which means that the new password needs to respect minimal requirements to be accepted. The default requirements are:

- Passwords must be at least eight characters long

- and have at least one numeric character

- and at least one lowercase character

- and at least one uppercase character

- and at least one special (nonalphanumeric) character).

Next, it will prompt some questions about whether you want to do some initial setup changes. The questions are of the type yes(y) or no(n). To ensure maximum protection, we recommend removing anonymous users, disabling root login remotely, and removing the test database (basically yes for all options):

Remove anonymous users? (Press y|Y for Yes, any other key for No) : y

Disallow root login remotely? (Press y|Y for Yes, any other key for No)

: y

Remove test database and access to it? (Press y|Y for Yes, any other

key for No) : y

Reload privilege tables now? (Press y|Y for Yes, any other key for No)

: y

- Connecting to MySQL

This step is optional, but we use it to verify that we executed all the steps correctly. Use this command to connect to the MySQL Server:

# mysql -u root -p

It will prompt for the password of the root user. Type the password and press Enter:

Enter password:

If successful, it will show the MySQL command line:

mysql>

- Starting MySQL 8 upon server start (optional)

To set MySQL to start whenever the server boots up, use the following command:

# systemctl enable mysqld

Installing MariaDB 10.5

To install MariaDB 10.5 on Centos 7, we need to execute similar steps as the vanilla MySQL distribution ones.

- Becoming root in Linux

First, we need to become root. Refer to the section Becoming Root.

- Installing the MariaDB repository

The following set of commands will download the MariaDB repo and configure them for the next step. Note that in the yum commands, we are using the -y option. This option tells Linux to assume the answer yes for all subsequent questions:

# yum install wget -y

# wget https://downloads.mariadb.com/MariaDB/mariadb_repo_setup

# chmod +x mariadb_repo_setup

# ./mariadb_repo_setup

- Installing MariaDB server

With our repository configured, the next command will install the latest stable version of MariaDB and its dependencies:

# yum install MariaDB-server -y

The end of the output will be similar to this:

Installed:

MariaDB-compat.x86_64 0:10.5.8-1.el7.centos                                                                                         MariaDB-server.x86_64 0:10.5.8-1.el7.centos

Dependency Installed:

MariaDB-client.x86_64 0:10.5.8-1.el7.centos MariaDB-common.x86_64

0:10.5.8-1.el7.centos boost-program-options.x86_64 0:1.53.0-28.el7

galera-4.x86_64 0:26.4.6-1.el7.centos        libaio.x86_64

0:0.3.109-13.el7               lsof.x86_64 0:4.87-6.el7

pcre2.x86_64 0:10.23-2.el7                  perl.x86_64

4:5.16.3-299.el7_9              perl-Carp.noarch 0:1.26-244.el7

perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.el7

perl-Compress-Raw-Zlib.x86_64 1:2.061-4.el7  perl-DBI.x86_64

0:1.627-4.el7

perl-Data-Dumper.x86_64 0:2.145-3.el7       perl-Encode.x86_64

0:2.51-7.el7             perl-Exporter.noarch 0:5.68-3.el7

perl-File-Path.noarch 0:2.09-2.el7           perl-File-Temp.noarch

0:0.23.01-3.el7        perl-Filter.x86_64 0:1.49-3.el7

perl-Getopt-Long.noarch 0:2.40-3.el7        perl-HTTP-Tiny.noarch

0:0.033-3.el7         perl-IO-Compress.noarch 0:2.061-2.el7

perl-Net-Daemon.noarch 0:0.48-5.el7          perl-PathTools.x86_64

0:3.40-5.el7           perl-PlRPC.noarch 0:0.2020-14.el7

perl-Pod-Escapes.noarch 1:1.04-299.el7_9    perl-Pod-Perldoc.noarch

0:3.20-4.el7        perl-Pod-Simple.noarch 1:3.28-4.el7

perl-Pod-Usage.noarch 0:1.63-3.el7

perl-Scalar-List-Utils.x86_64 0:1.27-248.el7 perl-Socket.x86_64

0:2.010-5.el7

perl-Storable.x86_64 0:2.45-3.el7

perl-Text-ParseWords.noarch 0:3.29-4.el7    perl-Time-HiRes.x86_64

4:1.9725-3.el7        perl-Time-Local.noarch 0:1.2300-2.el7

perl-constant.noarch 0:1.27-2.el7            perl-libs.x86_64

4:5.16.3-299.el7_9

perl-macros.x86_64 4:5.16.3-299.el7_9       perl-parent.noarch

1:0.225-244.el7          perl-podlators.noarch 0:2.5.1-3.el7

perl-threads.x86_64 0:1.87-4.el7

perl-threads-shared.x86_64 0:1.43-6.el7      socat.x86_64

0:1.7.3.2-2.el7

Replaced:

mariadb-libs.x86_64 1:5.5.64-1.el7

Complete!

Most important is to observe the Complete! at the end of the log, indicating a successful installation.

- Starting MariaDB server

With MariaDB installed, let’s initialize with the systemctl command:

# systemctl start mariadb.service

And to verify the status of the MariaDB serice:

# systemctl status mariadb

mariadb.service - MariaDB 10.5.8 database server

Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled;

vendor preset: disabled)

...

Feb 07 12:55:04 ip-172-30-150-91.ec2.internal systemd[1]: Started

MariaDB 10.5.8 database server.

- Securing MariaDB 10.5

At this point, MariaDB will be running in insecure mode. In contrast to MySQL 8, MariaDB will have an empty root password so that we can access it instantly:

# mysql

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 44

Server version: 10.5.8-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input

statement.

MariaDB [(none)]>

We can execute the mysql_secure_installation for MariaDB like we do for MySQL 8. There is a slight variation in the questions, with one extra question:

Switch to unix_socket authentication [Y/n] y

Enabled successfully!

Reloading privilege tables..

... Success!

If we accept the change with a yes answer, the connection will change from TCP/IP to Unix socket mode. We will get into the differences of each type of connection later in the Unix socket file.

Installing Percona Server 8

Install Percona Server 8 on Centos 7 using the following steps:

- Becoming root in Linux

First, we need to become root. Refer to Becoming Root section.

- Install the Percona repository

We can install Percona yum repository by running the following command as a root user or with sudo:

# yum install https://repo.percona.com/yum/percona-release-latest.noarch.rpm

The installation creates a new repository file /etc/yum.repos.d/percona-original-release.repo. Now, we need to enable Percona Server 8.0 repository using the next command:

# percona-release setup ps80

- Install Percona Server 8

To install the server, we need to execute the command:

# yum install percona-server-server

- Initializing Percona Server 8 with systemctl

Once we’ve installed the Percona Server 8 binaries, we will start the service:

# systemctl start mysql

And to validate the status of the service:

# systemctl status mysql

mysqld.service - MySQL Server

Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled;

vendor preset: disabled)

Active: active (running) since Sun 2021-02-07 13:22:15 UTC; 6s ago

Docs: man:mysqld(8)

http://dev.mysql.com/doc/refman/en/using-systemd.html

Process: 14472 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited,

status=0/SUCCESS)

Main PID: 14501 (mysqld)

Status: "Server is operational"

Tasks: 39 (limit: 5789)

Memory: 345.2M

CGroup: /system.slice/mysqld.service

└─14501 /usr/sbin/mysqld

Feb 07 13:22:14 ip-172-30-92-109.ec2.internal systemd[1]: Starting

MySQL Server...

Feb 07 13:22:15 ip-172-30-92-109.ec2.internal systemd[1]: Started MySQL

Server.

At this point, the steps are similar to the vanilla installation where you need to get the temporary password and execute the mysql_secure_installation.

Installing MySQL 5.7

Install MySQL 5.7 on Centos 7 using the following steps.

- Becoming root in Linux

First, we need to become root. Refer to Becoming Root section.

- Installing the MySQL 5.7 repository

You can install Percona’s yum repository by running the following command as a root user or with sudo:

# yum localinstall https://dev.mysql.com/get/mysql57-community-release-el7-9.noarch.rpm -y

The installation creates a new repository file in /etc/yum.repos.d/mysql-community.repo.

- Installing MySQL 5.7 binaries

To install the server, execute this command:

# yum install mysql-community-server -y

- Initializing MySQL 5.7 with systemctl

Once we installed the MySQL 5.7 binaries, we will start the service:

# systemctl start mysqld

And to validate the status of the service:

# systemctl status mysqld

At this point, the steps are similar to the MySQL 8.0 vanilla installation where we need to get the temporary password and execute the mysql_secure_installation.

Installing Percona Server 5.7

Install Percona Server 5.7 on Centos 7 using the following steps.

- Becoming root in Linux

First, we need to become root. Refer to Becoming Root section.

- Install the Percona repository

We can install Percona yum repository by running the following command as a root user or with sudo:

# yum install https://repo.percona.com/yum/percona-release-latest.noarc

h.rpm

The installation creates a new repository file /etc/yum.repos.d/percona-original-release.repo. Now we need to enable Percona Server 5.7 repository using the next command:

# percona-release setup ps57

- Installing Percona Server 5.7 binaries

To install the server, we need to execute this command:

# yum install Percona-Server-server-57 -y

- Initializing Percona Server 5.7 with systemctl

Once we’ve installed the Percona Server 5.7 binaries, we will start the service:

# systemctl start mysql

And to validate the status of the service:

# systemctl status mysql

At this point, the steps are similar to the vanilla installation where you need to get the temporary password and execute the mysql_secure_installation.

Installing MySQL on Centos 8

The current version of CentOS is CentOS 8, and it is built atop RHEL 8. Typically, CentOS enjoys the same ten-year support lifecycle as RHEL itself. This traditional support lifecycle would give CentOS 8 an end-of-life date in 2029. However, in December of 2020, a Red Hat announcement planned to put a headstone on CentOS 8’s grave much sooner — in 2021. (Red Hat will support CentOS 7 alongside RHEL 7 through 2024). Current CentOS users will need to migrate either to RHEL itself or to the newer CentOS Stream project. Some community projects are arising, but at this point, the future of CentOS is uncertain.

However, we will share the installation steps here since many users are using RHEL 8 and Oracle Linux 8 in the industry.

Installing MySQL 8

The latest MySQL 8.0 version is available to install from the default AppStream repository using the MySQL module that the CentOS 8 and RHEL 8 systems enable by default. So there is a variation from the traditional yum method. Let’s see the details.

- Becoming root in Linux

First, we need to become root. Refer to Becoming Root section.

- Installing MySQL 8 binaries

Run the following command to install the mysql-server package and a number of its dependencies:

# dnf install mysql-server

When prompted, press y and then Enter to confirm that you want to proceed:

Output

...

Transaction Summary

=======================================================================

Install  50 Packages

Upgrade   8 Packages

Total download size: 50 M

Is this ok [y/N]: y

- Starting MySQL

At this point, we’ve installed MySQL on our server, but it isn’t yet operational. The package we installed configures MySQL to run as a systemd service named mysqld.service. To start MySQL, we need to use the systemctl command:

# systemctl start mysqld.service

- Checking if the service is running

To check if the service is running correctly, run the following command:

# systemctl status mysqld

If we successfully started MySQL, the output will show that the MySQL service is active:

# systemctl status mysqld

mysqld.service - MySQL 8.0 database server

Loaded: loaded (/usr/lib/systemd/system/mysqld.service; disabled;

vendor preset: disabled)

Active: active (running) since Sun 2020-06-21 22:57:57 UTC; 6s ago

Process: 15966 ExecStartPost=/usr/libexec/mysql-check-upgrade

(code=exited, status=0/SUCCESS)

Process: 15887 ExecStartPre=/usr/libexec/mysql-prepare-db-dir

mysqld.service (code=exited, status=0/SUCCESS)

Process: 15862 ExecStartPre=/usr/libexec/mysql-check-socket

(code=exited, status=0/SUCCESS)

Main PID: 15924 (mysqld)

Status: "Server is operational"

Tasks: 39 (limit: 23864)

Memory: 373.7M

CGroup: /system.slice/mysqld.service

└─15924 /usr/libexec/mysqld --basedir=/usr

Jun 21 22:57:57 ip-172-30-222-117.ec2.internal systemd[1]: Starting

MySQL 8.0 database server...

Jun 21 22:57:57 ip-172-30-222-117.ec2.internal systemd[1]: Started

MySQL 8.0 database server.

- Securing MySQL 8

As with installing MySQL 8 on Centos 7, you need to execute the mysql_secure_installation. The main difference is that there is not a temporary password for Centos 8, so when the script requests the root password, leave it blank and press enter.

- Start MySQL 8 upon server start (optional)

To set MySQL to start whenever the server boots up, use the following command:

# systemctl enable mysqld

Installing Percona Server 8

To install Percona Server 8 in Centos 8, we need to install the repository first. Let’s see the detailed steps.

- Becoming root in Linux

First, we need to become root. Refer to Becoming Root section.

- Installing MySQL

Run the following command to install the Percona repository:

# yum install https://repo.percona.com/yum/percona-release-latest.noarh.rpm

When prompted, press y and then Enter to confirm that you want to proceed:

Last metadata expiration check: 0:03:49 ago on Sun 07 Feb 2021 01:16:41 AM UTC.

percona-release-latest.noarch.rpm                                                                                                                                                                                                        109 kB/s |  19 kB     00:00

Dependencies resolved.

=======================================================================

=======================================================================

=======================================================================

====================================================

Package

Architecture                                                Version

Repository

Size

=======================================================================

=======================================================================

=======================================================================

====================================================

Installing:

percona-release

noarch                                                      1.0-25

@commandline                                                       19

k

Transaction Summary

=======================================================================

=======================================================================

=======================================================================

====================================================

Install  1 Package

Total size: 19 k

Installed size: 31 k

Is this ok [y/N]: y

Downloading Packages:

Running transaction check

Transaction check succeeded.

Running transaction test

Transaction test succeeded.

Running transaction

Preparing        :

1/1

Installing       : percona-release-1.0-25.noarch

1/1

Running scriptlet: percona-release-1.0-25.noarch

1/1

* Enabling the Percona Original repository

<*> All done!

* Enabling the Percona Release repository

<*> All done!

The percona-release package now contains a percona-release script that

can enable additional repositories for our newer products.

For example, to enable the Percona Server 8.0 repository use:

percona-release setup ps80

Note: To avoid conflicts with older product versions, the

percona-release setup command may disable our original repository for

some products.

For more information, please visit:

https://www.percona.com/doc/percona-repo-config/percona-release.html

Verifying        : percona-release-1.0-25.noarch

1/1

Installed:

percona-release-1.0-25.noarch

- Enable the repository for Percona 8.0

We can see that our installation created a new repository file in /etc/yum.repos.d/percona-original-release.repo. Now we need to enable Percona Server 8.0 repository using the next command:

# percona-release setup ps80

The command prompts us to disable the RHEL 8 module for MySQL; we can do this now by pressing y:

* Disabling all Percona Repositories

On RedHat 8 systems it is needed to disable dnf mysql module to install

Percona-Server

Do you want to disable it? [y/N] y

Disabling dnf module...

Percona Release release/noarch YUM repository

6.4 kB/s | 1.4 kB     00:00

Dependencies resolved.

=======================================================================

=======================================================================

=======================================================================

====================================================

Package

Architecture                                                   Version

Repository                                                       Size

=======================================================================

=======================================================================

=======================================================================

====================================================

Disabling modules:

mysql

Transaction Summary

=======================================================================

=======================================================================

=======================================================================

====================================================

Complete!

dnf mysql module was disabled

* Enabling the Percona Server 8.0 repository

* Enabling the Percona Tools repository

<*> All done!

Or manually with the following command:

# dnf module disable mysql

- Installing Percona Server 8.0 binaries

We are ready to install Percona Server 8.0 on our CentOS 8/RHEL 8 server. To avoid being prompted again about whether we want to proceed or not, we’ll add the -y to the command line:

# yum install percona-server-server -y

- Starting and Securing Percona MySQL Server

Now that we installed Percona Server 8.0 binaries, we will start and set the mysqld service to start at system boot.

# systemctl enable --now mysqld

# systemctl start mysqld

- Checking service status

It is important to validate if we completed all steps successfully. Let’s check the status of the service:

# systemctl status mysqld

● mysqld.service - MySQL Server

Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled;

vendor preset: disabled)

Active: active (running) since Sun 2021-02-07 01:30:50 UTC; 28s ago

Docs: man:mysqld(8)

http://dev.mysql.com/doc/refman/en/using-systemd.html

Process: 12864 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)

Main PID: 12942 (mysqld)

Status: "Server is operational"

Tasks: 39 (limit: 5789)

Memory: 442.6M

CGroup: /system.slice/mysqld.service

└─12942 /usr/sbin/mysqld

Feb 07 01:30:40 ip-172-30-92-109.ec2.internal systemd[1]: Starting MySQL Server...

Feb 07 01:30:50 ip-172-30-92-109.ec2.internal systemd[1]: Started MySQL Server.

NOTE

If you ever want to disable the option from MySQL starting up at boot, you can do so by running the following:

# systemctl disable mysqld

Installing MySQL 5.7

Install MySQL 5.7 on Centos 8 using the following steps:

- Becoming root in Linux

First we need to become root. Refer to the section Becoming Root.

- Disabling the MySQL default module

Systems such as RHEL8, Oracle Linux 8, and CentOS 8 enable the MySQL module by default. Unless this module is disabled, it masks packages provided by MySQL repositories, not allowing us to install a version different than MySQL 8. So, we are going to remove this default module:

# dnf remove @mysql

# dnf module reset mysql && dnf module disable mysql

- Configuring MySQL 5.7 repository

There is no MySQL repository for CentOS 8, so we’ll use CentOS 7 repository instead as a reference. Let’s create a new repository file:

# vi /etc/yum.repos.d/mysql-community.repo

And paste the following data into the file.

[mysql57-community]

name=MySQL 5.7 Community Server

baseurl=http://repo.mysql.com/yum/mysql-5.7-community/el/7/$basearch/

enabled=1

gpgcheck=0

[mysql-connectors-community]

name=MySQL Connectors Community

baseurl=http://repo.mysql.com/yum/mysql-connectors-community/el/7/$basearch/

enabled=1

gpgcheck=0

[mysql-tools-community]

name=MySQL Tools Community

baseurl=http://repo.mysql.com/yum/mysql-tools-community/el/7/$basearch/

enabled=1

gpgcheck=0

- Installing MySQL 5.7 binaries

With the module disabled and the repository configured, we need to run the following command to install the mysql-server package and its dependencies:

# dnf install mysql-community-server

When prompted, press y and then Enter to confirm that you want to proceed:

Output

...

Install  5 Packages

Total download size: 202 M

Installed size: 877 M

Is this ok [y/N]: y

- Starting MySQL

We’ve installed the MySQL binaries on our server at this point, but it isn’t yet operational. The package we installed configures MySQL to run as a systemd service named mysqld.service. To start MySQL, we need to use the systemctl command:

# systemctl start mysqld.service

- Check if the service is running

To check that the service is running correctly, run the following command:

# systemctl status mysqld

If we successfully started MySQL, the output will show that the MySQL service is active:

# systemctl status mysqld

● mysqld.service - MySQL Server

Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)

Active: active (running) since Sun 2021-02-07 18:22:12 UTC; 9s ago

Docs: man:mysqld(8)

http://dev.mysql.com/doc/refman/en/using-systemd.html

Process: 14396 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)

Process: 8137 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)

Main PID: 14399 (mysqld)

Tasks: 27 (limit: 5789)

Memory: 327.2M

CGroup: /system.slice/mysqld.service

└─14399 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid

Feb 07 18:22:02 ip-172-30-36-53.ec2.internal systemd[1]: Starting MySQL Server...

Feb 07 18:22:12 ip-172-30-36-53.ec2.internal systemd[1]: Started MySQL Server.

- Securing MySQL 5.7

At this point, the steps are similar to the vanilla installation where we need to get the temporary password and execute the mysql_secure_installation.

- Start MySQL 5.7 upon server start (optional)

To set MySQL to start whenever the server boots up, use the following command:

# systemctl enable mysqld

Installing MySQL on Ubuntu 20.04 LTS (Focal Fossa)

[source,subs=+quotes]://ubuntu.com/[Ubuntu] is a Linux distribution based on Debian and is composed mostly of free and open source software. Officially, there are three Ubuntu editions: Desktop, Server, and Core for IoT devices and robots. The version we will work on in this book is the Server version.

Installing MySQL 8

For Ubuntu, the process is slightly different since Ubuntu uses the apt repository. Let’s check.

- Becoming root in Linux

First, we need to become root. Refer to Becoming Root section.

- Configure the apt repository

On Ubuntu 20.04 (Focal Fossa), we can install MySQL using the apt package repository. First, let’s ensure that our system is up-to-date:

# apt update

Install MySQL 8

Then install the mysql-server package:

# apt install mysql-server -y

The apt install command will install MySQL but won’t prompt to set a password or make any other configuration changes. Unlike the CentOS installation, Ubuntu initializes MySQL in insecure mode.

For fresh installations of MySQL, you’ll want to run the DBMS’s included security script. This script changes some of the less secure default options for remote root logins and the test database. We will address this problem in the securing step after we initialize MySQL.

- Starting MySQL

At this point, we’ve installed MySQL on our server, but it isn’t yet operational. To start MySQL, we need to use the systemctl command:

# systemctl start mysql

- Checking if the service is running

To check that the service is running correctly, run the following command:

# systemctl status mysql

If we successfully started MySQL, the output will show that the MySQL service is active:

● mysql.service - MySQL Community Server

Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)

Active: active (running) since Sun 2021-02-07 20:19:51 UTC; 22s ago

Process: 3514 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)

Main PID: 3522 (mysqld)

Status: "Server is operational"

Tasks: 38 (limit: 1164)

Memory: 332.7M

CGroup: /system.slice/mysql.service

└─3522 /usr/sbin/mysqld

Feb 07 20:19:50 ip-172-30-202-86 systemd[1]: Starting MySQL Community Server...

Feb 07 20:19:51 ip-172-30-202-86 systemd[1]: Started MySQL Community Server.

- Securing MySQL 8.0

At this point, the steps are similar to the vanilla installation. However, for MySQL 8 on Ubuntu, MySQL is initialized unsecured, which means the root password is empty. We can execute the mysql_secure_installation:

# mysql_secure_installation

The mysql_secure_installation will take you through a series of prompts to make some changes to the MySQL installation’s security options, which are similar to those of the CentOS version and described in mysql_secure_installation.

There is a small variance here because in Ubuntu it is possible to change the validation policy, which manages password strength. In this example, we are setting the validation policy to MEDIUM(1).

Securing the MySQL server deployment.

Connecting to MySQL using a blank password.

VALIDATE PASSWORD COMPONENT can be used to test passwords

and improve security. It checks the strength of password

and allows the users to set only those passwords which are

secure enough. Would you like to setup VALIDATE PASSWORD component?

Press y|Y for Yes, any other key for No: y

There are three levels of password validation policy:

LOW    Length >= 8

MEDIUM Length >= 8, numeric, mixed case, and special characters

STRONG Length >= 8, numeric, mixed case, special characters and dictionary                  file

Please enter 0 = LOW, 1 = MEDIUM and 2 = STRONG: 1

Please set the password for root here.

New password:

Re-enter new password:

Estimated strength of the password: 50

Do you wish to continue with the password provided?(Press y|Y for Yes, any other key for No) : y

By default, a MySQL installation has an anonymous user,

allowing anyone to log into MySQL without having to have

a user account created for them. This is intended only for

testing, and to make the installation go a bit smoother.

You should remove them before moving into a production

environment.

Installing Percona Server 8

Install Percona Server 8 on Ubuntu 20.04 LTS using the following steps.

- Becoming root in Linux

First, we need to become root. Refer to Becoming Root section.

- Installing GnuPG, the GNU Privacy Guard

Oracle signs MySQL downloadable packages with GnuPG (GNU Privacy Guard). GnuPG is an Open Source alternative to the well known Pretty Good Privacy (PGP) by Phil Zimmermann. Most Linux distributions ship with GnuPG installed by default, but in this case we need to install it:

# apt-get install gnupg2 -y

- Fetch the repository packages from the Percona web page

Next, we are going to fetch the repository packages with the wget command from the Percona repository:

# wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb

- Install the downloaded package with dpkg

Once download, we will install with the following command:

# dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb

Once we install the package, we can check the repository configured in the /etc/apt/sources.list.d/percona-original-release.list file.

- Enable the repository

The next step is enabling Percona Server 8 in the repository and refreshing it:

# percona-release setup ps80

# apt update

- Installing MySQL 8 binaries

Then we install the mysql-server package with the apt-get install command:

# apt-get install percona-server-server -y

- Starting MySQL

At this point, we’ve installed MySQL on our server, but it isn’t yet operational. To start MySQL, we need to use the systemctl command:

# systemctl start mysql

- Check if the service is running

To check that the service is running correctly, run the following command:

# systemctl status mysql

At this point, Percona Server will be running in insecure mode. The mysql_secure_installation will take us through a series of prompts to make some changes to our MySQL installation’s security options, which are identical to the installation of MySQL 8 in Ubuntu described in mysql_secure_installation.

Installing MariaDB 10.5

Install MariaDB 10.5 on Ubuntu 20.04 LTS using the following steps.

- Becoming root in Linux

First, we need to become root. Refer to the section Becoming Root.

- Update the system with apt package manager

Let’s ensure our system is updated and install the software-properties-common package with the following command:

# apt update && sudo apt upgrade

# apt -y install software-properties-common

The software-properties-common package contains the common files for software properties like the D-Bus backend and an abstraction of the used apt repositories.

- Importing MariaDB GPG key

Run the following command to add the repository key to the system:

# apt-key adv --fetch-keys https://mariadb.org/mariadb_release_signing_key.asc

- Adding MariaDB repository

After importing the repository GPG key, we need to add the apt repository by running the following command:

# add-apt-repository deb [arch=amd64] http://mariadb.mirror.globo.tech/repo/10.5/ubuntu focal main

NOTE

There are different mirrors to download the MariaDB repository. In this example, we use the http://mariadb.mirror.globo.tech.

- Installing MariaDB 10.5 binaries

The next step is the installation of the MariaDB Server:

# apt install mariadb-server mariadb-client

- Checking if the service is running

To check if the MariaDB service is running correctly, run the following command:

# systemctl status mysql

At this point, MariaDB 10.5 will be running in insecure mode. The mysql_secure_installation will take us through a series of prompts to make some changes to our MySQL installation’s security options, which are identical to the installation of MySQL 8 in Ubuntu described in mysql_secure_installation.

Installing MySQL 5.7

We will install MySQL 5.7 on Ubuntu 20.04 LTS using the following steps.

- Becoming root in Linux

First, we need to become root. Refer to the section Becoming Root.

- Update the system with apt package manager

Let’s ensure our system is updated and install the software-properties-common package with the following command:

# apt update -y && sudo apt upgrade -y

- Add and configure MySQL 5.7 repository

Add the MySQL repository by running the following command:

# wget https://dev.mysql.com/get/mysql-apt-config_0.8.12-1_all.deb

# dpkg -i mysql-apt-config_0.8.12-1_all.deb

In the prompt, we choose “ubuntu bionic” as shown in Figure 1-1 and click OK.

Figure 1-1. Choose “ubuntu bionic.”

The next prompt shows MySQL 8.0 chosen by default. Press ENTER and in the first option as shown in Figure 1-2.

Figure 1-2. Choose the MySQL & Cluster option

In the next option, as in Figure 1-3, choose MySQL 5.7 and press Ok.

Figure 1-3. Choose the MySQL 5.7 option

After returning to the main screen, we finish pressing Ok to exit, as shown in Figure 1-4.

Figure 1-4. Exit the main screen pressing the Ok option

Next, we need to update the MySQL packages:

# apt-get update -y

And validate Ubuntu policy to install MySQL 5.7:

# apt-cache policy mysql-server

Let’s check the output to observe which MySQL 5.7 version is available:

# apt-cache policy mysql-server

mysql-server:

Installed: (none)

Candidate: 8.0.23-0ubuntu0.20.04.1

Version table:

8.0.23-0ubuntu0.20.04.1 500

500 http://br.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages

500 http://br.archive.ubuntu.com/ubuntu focal-security/main amd64 Packages

8.0.19-0ubuntu5 500

500 http://br.archive.ubuntu.com/ubuntu focal/main amd64 Packages

5.7.33-1ubuntu18.04 500

500 http://repo.mysql.com/apt/ubuntu bionic/mysql-5.7 amd64 Packages

- Installing MySQL 5.7 binaries

Now, that we saw that the MySQL 5.7 version is available (*5.7.33-1ubuntu18.04*), let’s install it:

# apt-get install mysql-client=5.7.33-1ubuntu18.04 -y

# apt-get install mysql-community-server=5.7.33-1ubuntu18.04 -y

# apt-get install mysql-server=5.7.33-1ubuntu18.04 -y

The installation process will prompt us to choose the root password, as shown in Figure 1-5.

Figure 1-5. Unlike other installations processes, we define the root password now

- Checking if the service is running

To check if the MySQL 5.7 service is running correctly, run the following command:

# systemctl status mysql

At this point, MySQL 5.7 will have the root user with a password set. However, it is still valid to run mysql_secure_installation since we need to set the password policy, remove remote root login and anonymous users, and remove the test database. The details are in mysql_secure_installation.

Installing MySQL on macOS Big Sur

MySQL for macOS is available in some different forms. Since most of the time MySQL is installed on macOS for development purposes, we will demonstrate only how to install using the native macOS installer (DMG). Be aware that it is also possible to use the tarball to install on macOS.

Installing MySQL 8

First, download the MySQL dmg file from the website.

NOTE

According to Oracle the macOS Catalina packages work for Big Sur.

Once downloaded, we will execute the package to start the install procedure, as shown in Figure 1-6.

Figure 1-6. MySQL 8.0.23 dmg package

Next, we need to authorize MySQL to run, as seen in Figure 1-7

Figure 1-7. MySQL 8.0.23 authorization request

Figure 1-8 shows the installer’s welcome screen.

Figure 1-8. MySQL 8.0.23 initial screen

Figure 1-9 shows License agreement.

Figure 1-9. MySQL 8.0.23 license agreement

Even with open source software, it is necessary to agree with the license terms; otherwise, we can’t proceed.

And now, we can define the location and customize the installation, as shown in Figure 1-10.

Figure 1-10. MySQL 8.0.23 installation customization

We are going to proceed with the standard installation. After clicking Install, we might get prompted to put the macOS user password to run the installation with higher privileges, as Figure 1-11 shows.

Figure 1-11. macOS authorization request

Now that we’ve installed MySQL, the installation process will prompt us to choose the password encryption. We are going to use the newer authentication method(default option), as shown in Figure 1-12, which is safer.

Figure 1-12. MySQL 8.0.23 password encryption

The last step consists of creating the root password and initializing MySQL, as seen in Figure 1-13.

Figure 1-13. MySQL 8.0.23 root password

We’ve installed the MySQL Server, but it is not loaded (or started) by default. To start, open System Preferences and search for the MySQL icon, as shown in Figure 1-14.

Figure 1-14. MySQL in the System Preferences

Click the icon to open the MySQL panel, and you should see something like in Figure 1-15.

Figure 1-15. MySQL start options

Besides the obvious, which is to start the MySQL process, there is the configuration panel (with the MySQL files location) and an option to re-initialize the database (we already initialized it in the installation). Now it is necessary only to start the MySQL process. You might be asked for the administrator password again.

With MySQL running, it is possible to validate the connection and whether the MySQL Server is running correctly. We can use the MySQL Workbench to test or install the MySQL client using brew:

$ brew install mysql-client

Once we’ve installed the MySQL client, we can connect with the password we defined in the section Figure 1-11. In the terminal, run the following:

$ mysql -uroot -p

Enter password:

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 8

Server version: 8.0.23 MySQL Community Server - GPL

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type help; or \h for help. Type \c to clear the current input statement.

mysql> SELECT @@version;

+-----------+

| @@version |

+-----------+

| 8.0.23    |

+-----------+

1 row in set (0.00 sec)

Installing MySQL on Windows 10

Oracle provides a MySQL installer to facilitate the installation. Note that MySQL Installer is a 32-bit application, but it can install MySQL in 32-bit and 64-bit binaries. To initiate the installation process, we need to execute the installer file and choose the type of installation we are going to make, as shown in Figure 1-16.

Figure 1-16. MySQL 8.0.23 Windows installation customization

We will proceed with the Developers profile and install everything. The other options offer alternatives to the installation process. We are not going into those details because we don’t recommend using MySQL for production systems, mainly because the MySQL ecosystem is developed for Linux.

Next, the installer will check if all requirements are satisfied, as shown in Figure 1-17.

Figure 1-17. Installation requirements

After that, we click Execute. It might be necessary to install Microsoft Visual C++, as Figure 1-18 indicates.

Figure 1-18. Installation requirements

With Microsoft Visual C++ in-place, click Next, and the installer will show the products that are ready to install, as shown in Figure 1-19.

Figure 1-19. The installer is ready to install the MySQL software

After clicking Next, we arrive at the section where we will configure MySQL properties. We will the default settings for TCP/IP, ports, and X protocol port, as shown in Figure 1-20.

Figure 1-20. We can customize the MySQL port and other settings in this section

In the next phase, we will choose the authentication method. We will use the newer version that is more secure, as seen in Figure 1-21.

Figure 1-21. We will proceed with SHA-256 based passwords

Next, we define the root user password and whether we want to add additional users to the MySQL database, as shown in Figure 1-22.

Figure 1-22. Configuring users

With the users configured, we define the service name and user that will run the service, as seen in Figure 1-23.

Figure 1-23. Configuring the service name

When we click Next, the installer begins configuring MySQL. Once the MySQL installer finishes its execution, you should see something like Figure 1-24.

Figure 1-24. If the installation went fine, there are no errors

After this, our database server is operational. Since we chose the Developer profile, the installer will go through the MySQL router installation. The MySQL Router is not essential for this setup, and since we don’t recommend Windows for production, we will skip this part. We will dive into the details of the MySQL router in “MySQL Router”.

Now, we can validate our server using Workbench, as shown in Figure 1-25.

Figure 1-25. We can see a MySQL connection option

Double-click the connection and Workbench will prompt us to input the password, as shown in Figure 1-26.

Figure 1-26. We need to fill the password to connect

And, finally, we can start using MySQL in our Windows platform, as seen in Figure 1-27.

Figure 1-27. We can begin testing our environment

The Contents of the MySQL Directory

During the installation process, MySQL creates all necessary files to start the server. MySQL stores its files under a directory called data directory. It is also widespread that DBAs refers to this directory as datadir, which is the MySQL parameter name that indicates the directory. You will see that MySQL parameters can change much of the content stored in this directory to another location. The reasons for it are various, and the two most common ones are performance and disk usage. The default location for Linux distributions is the /var/lib/mysql directory. We can also discover its location by running the following command in the MySQL instance:

mysql> SELECT @@datadir;

+-----------------+

| @@datadir       |

+-----------------+

| /var/lib/mysql/ |

+-----------------+

1 row in set (0.00 sec)

MySQL 5.7 default files

The following list briefly describes the files typically found in the data directory.

The REDO log files

MySQL creates the redo log files as ib_logfile0 and ib_logfile1 in the data directory. MySQL writes to the redo log files in a circular fashion. Because of the circular fashion-writing, the files do not grow beyond their configuration size (configured by innodb_log_file_size). As any other RDBMS database that is ACID-compliant, the redo files are fundamental to provide data durability and the ability to recover from a crash scenario.

The auto.cnf file

MySQL 5.6 introduced the auto.cnf file. It has only a single [auto] section containing a single server_uuid setting and value. The server_uuid creates a unique signature for the server, and the replication layer uses it to communicate with different servers to replicate data.

WARNING

MySQL automatically creates the auto.cnf file in the data directory when initialized and should not be changed.

The *.pem files

In short, these provide a way of using encrypted connections for the communication between a client and the MySQL Server. Encrypted connections are a fundamental part of the network security layer to avoid access by an element that is not authorized to read it. MySQL 5.7 enables SSL by default and creates the certificates as well. However, it is possible to use certificates provided by the Certificates Authorities (CA) available in the market.

The performance_schema directory

The MySQL Performance Schema is a feature for monitoring MySQL Server execution at a low level during runtime. When we can use Performance Schema to monitor a particular metric, we say that MySQL has instrumentation. For example, performance Schema instruments can provide the number of users connected:

mysql> SELECT * FROM performance_schema.users;

+-----------------+---------------------+-------------------+

| USER            | CURRENT_CONNECTIONS | TOTAL_CONNECTIONS |

+-----------------+---------------------+-------------------+

| NULL            |                  40 |                46 |

| event_scheduler |                   1 |                 1 |

| root            |                   0 |                 1 |

| rsandbox        |                   2 |                 3 |

| msandbox        |                   1 |                 2 |

+-----------------+---------------------+-------------------+

5 rows in set (0.03 sec)

NOTE

Many people are surprised to see NULL in the user column. The NULL value is used for internal threads or for a user session that failed to authenticate. The same applies to the host column in the performance_schema.accounts table:

mysql> SELECT user, host, total_connections AS cxns

-> FROM performance_schema.accounts ORDER BY cxns DESC;

+-----------------+-----------+------+

| user            | host      | cxns |

+-----------------+-----------+------+

| NULL            | NULL      |   46 |

| rsandbox        | localhost |    3 |

| msandbox        | localhost |    2 |

| event_scheduler | localhost |    1 |

| root            | localhost |    1 |

+-----------------+-----------+------+

5 rows in set (0.00 sec)

Although instrumentation has existed since MySQL 5.6, it was MySQL 5.7 that gained many improvements and became a fundamental part of the DBA tools to investigate and troubleshoot issues at the MySQL level.

The ibtmp1 file

When the application needs to create temporary tables or MySQL needs to use an on-disk internal temporary table, MySQL creates them in a shared temporary tablespace. The default behavior is to create an auto-extending data file named ibtmp1 that is slightly larger than 12 MB (controlled by the innodb_temp_data_file_path parameter).

The ibdata1 file

The ibdata1 is probably the most famous file in the MySQL ecosystem. For MySQL 5.7 and older, the ibdata1 file holds data for the InnoDB data dictionary, the doublewrite buffer, the change buffer, and undo logs. It may also contain table and index data if tables if we disable the innodb_file_per_table option. When the innodb_file_per_table is enabled, each user table has a tablespace and a dedicated file. Note that it is possible to have multiple ibdata files in the MySQL data directory.

For MySQL 8.0 series, some of these components were removed from ibdata1 and allocated into separated files. The remaining components are the change buffer table and index data if tables are created in the system tablespace (disabling the innodb_file_per_table).

The mysql.sock file

This is a Unix socket file that the server uses for communication with local clients. This file only exists when MySQL is running, and removing it or creating the file manually may lead to problems.

NOTE

A UNIX socket is an inter-process communication mechanism that allows bidirectional data exchange between processes running on the same machine. IP sockets (mainly TCP/IP sockets) are a mechanism allowing communication between processes over the network.

You can connect to MySQL Servers on Linux using two methods: TCP protocol or a socket. For security purposes, if the application and MySQL are on the same server, you can disable remote TCP connections. There are two ways to do this in MySQL Server: set the bind-address to 127.0.0.1 instead of the default * value (accepts TCP/IP connections from everyone) or modify the skip-networking parameter, which disables network connections to MySQL.

The mysql directory

The mysql directory corresponds to the MySQL system schema, which contains the MySQL Server’s information as it runs. For example, it includes information on users and their privileges, time zone tables, and replication. We will use these tables along the way in this book. You can see the files named according to their respective table names with the ls command:

# cd /var/lib/mysql

# ls -l mysql/

-rw-r-----. 1 vinicius.grippa percona    8820 Feb 20 15:51

columns_priv.frm

-rw-r-----. 1 vinicius.grippa percona       0 Feb 20 15:51

columns_priv.MYD

-rw-r-----. 1 vinicius.grippa percona    4096 Feb 20 15:51

columns_priv.MYI

-rw-r-----. 1 vinicius.grippa percona    9582 Feb 20 15:51 db.frm

-rw-r-----. 1 vinicius.grippa percona     976 Feb 20 15:51 db.MYD

-rw-r-----. 1 vinicius.grippa percona    5120 Feb 20 15:51 db.MYI

-rw-r-----. 1 vinicius.grippa percona      65 Feb 20 15:51 db.opt

-rw-r-----. 1 vinicius.grippa percona    8780 Feb 20 15:51 engine_cost.frm

-rw-r-----. 1 vinicius.grippa percona   98304 Feb 20 15:51 engine_cost.ibd

...

-rw-r-----. 1 vinicius.grippa percona   10816 Feb 20 15:51 user.frm

-rw-r-----. 1 vinicius.grippa percona    1292 Feb 20 15:51 user.MYD

-rw-r-----. 1 vinicius.grippa percona    4096 Feb 20 15:51 user.MYI

MySQL 8.0 default files

MySQL 8.0 releases brought a few changes in the core of the data directory structure. Part of these changes are related to implementing the new data dictionary; others intend to bring better database management. The following list describes the new files and changes:

The undo tablespace files

MySQL(InnoDB) uses undo files to undo the transactions that need to be rolled back and ensure isolated transactions whenever it needs to perform a consistent read.

From MySQL 8.0, the undo log files were separated from the system tablespace (ibdata1) and placed on the data directory. It is also possible to set another location by changing the innodb_undo_directory parameter.

The dblwr files (introduced in version 8.0.20)

The doublewrite buffer is responsible for writing pages flushed from the buffer pool to the disk before MySQL writes the pages to the datafiles. The doublewrite file names have the following format: #ib_page_size_file_number.dblwr. It is possible to change the doublewrite files’ location by modifying the innodb_doublewrite_dir parameter.

Example:

#ib_16384_0.dblwr

#ib_16384_1.dblwr

The mysql.ibd file (introduced in version 8.0)

Dictionary tables and system tables store data and metadata which MySQL 5.7 held previously in the mysql directory inside the datadir. Now it is protected by the InnoDB mechanisms to ensure consistency.

Using the Command-Line Interface

The mysql binary is a simple SQL shell with input line-editing capabilities. Its use should be straightforward (we already used a few times during the installation process). To invoke, run the following:

# mysql

We can extend its functionality by executing queries in it:

# mysql -uroot -pseKret -e "SHOW ENGINE INNODB STATUS\G"

Or we can execute more advanced commands piping them with other commands to perform more complex tasks. For example, we can extract a dump from one database, send it across the network, and restore it into another MySQL Server in the same command line:

# mysql -e "SHOW MASTER STATUS\G" && nice -5 mysqldump

--all-databases --single-transaction -R --master-data=2 --flush-logs

--log-error=/tmp/donor.log --verbose=TRUE | ssh mysql@192.168.0.1 mysql

1> /tmp/receiver.log 2>&1

MySQL 8.0 introduced the MySQL shell, which is way more powerful than its predecessor. MySQL Shell supports Javascript, Python, or SQL languages, providing development and administration for the MySQL Server. We go into more details about MySQL shell in “MySQL Shell”.

Using Docker

With the advent of virtualization and its popularization with cloud services, many platforms have emerged, including Docker. Born in 2013, Docker is a solution that provides a portable and flexible way to deploy software. It provides resource isolation through the use of Linux features like cgroups and Kernel namespaces.

Docker is useful for DBAs who often need to install a specific version of MySQL, MariaDB, or Percona Server for MySQL to run some experiments. With Docker, it is possible to deploy a MySQL instance in seconds to perform some tests. Once you finish the tests, you can destroy the instance and release the operating system’s resources to other tasks. All the processes of deploying a virtual machine, installing packages, and configuring it are not necessary when using Docker.

Installing Docker

One advantage of using Docker is that all the commands are the same in all operating systems after the Docker service is running. The commands being the same means that the learning curve for using Docker is faster compared to learning different Linux versions such as Centos and Ubuntu, for example (installing a package is different — for instance, yum vs. apt-get).

The process for installing Docker service is, in some way, similar to installing MySQL. For Windows and macOS operating systems, it is a binary installation, and after that the service is up and running. For Linux server-based operating systems without a graphic interface, the process requires configuring the repository.

Installing Docker on CentOS 7

The CentOS packages for Docker are, in general, older than the ones available to RHEL and the upstream. At the time of writing, the Docker version provided by regular CentOS repositories is 1.13.1, whereas the upstream stable version is 20.10.3. There is no difference for the purposes of this book, but we always recommend the latest version for production environments.

Execute the following command line to install the Docker package from the default CentOS repository:

# yum install docker -y

If we want to install Docker from the upstream repository to ensure that we are using the latest release, we need to configure it first. Here are the steps:

1. Install yum-utils to enable the yum-config-manager:

# yum install yum-utils -y

2. Use the yum-config-manager to add the docker-ce repository:

# yum-config-manager \

--add-repo \

https://download.docker.com/linux/centos/docker-ce.repo

3. Install the necessary packages:

# yum install docker-ce docker-ce-cli containerd.io -y

4. Start the docker service:

# systemctl start docker

5. Enable the docker service to auto-start after the system reboot:

# systemctl enable --now docker

6. To validate whether the Docker service is running, execute the systemctl status command:

# systemctl status docker

7. To verify that Docker Engine is installed correctly, we can run the the hello-world container:

# docker run hello-world

Installing Docker on Ubuntu 20.04 (Focal Fossa)

To install the latest Docker release from the upstream, we will first remove older versions of Docker (called docker, docker.io, or docker-engine). Uninstall them with this command:

# apt-get remove -y docker docker-engine docker.io containerd runc

With the default repository removed, we can initiate the installation process.

1. Make sure that Ubuntu is up-to-date with the command:

# apt-get update -y

2. Install packages to allow apt to use a repository over HTTPS:

# apt-get install -y \

apt-transport-https \

ca-certificates \

curl \

gnupg-agent \

software-properties-common

3. Next, add Docker’s official GPG key:

# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo

apt-key add -

4. With the key in place, let’s add the Docker stable repository:

# add-apt-repository \

"deb [arch=amd64] https://download.docker.com/linux/ubuntu \

$(lsb_release -cs) \

stable"

5. Now, use the apt command to install the Docker packages:

# apt-get install -y docker-ce docker-ce-cli containerd.io

6. Ubuntu will start the service for us, but we can check by running this:

# systemctl status docker

7. To make the Docker service auto-start when the OS reboots:

# systemctl enable --now docker

8. Check the Docker version we installed:

# docker --version

If we want to verify that Docker Engine is installed correctly, we can run the the hello-world container:

# docker run hello-world

Deploying MySQL container

Once we have the Docker engine installed and running, the next step is deploying the MySQL Docker container.

WARNING

We designed the following instructions to get a test instance running quickly and easily; you do not want to use these for production deployments.

To deploy the latest MySQL version with Docker:

# docker run --name mysql-latest  \

-p 3306:3306 -p 33060:33060  \

-e MYSQL_ROOT_HOST=% -e MYSQL_ROOT_PASSWORD=learning_mysql \

-d mysql/mysql-server:latest

The Docker engine will launch the latest version of the MySQL instance and be remotely accessible from anywhere with the specified root password. Installing MySQL with Docker means that you do not get any tools, utilities, or standard libraries available in a traditional host (bare metal or virtual machines). We need to either deploy these tools separately or use command lines shipped with the Docker image if we need them.

Next, connect to the MySQL container using the MySQL client:

# docker exec -it mysql-latest mysql -uroot -plearning_mysql

Since we mapped the TCP port 3306 in the container to port 3306 on the Docker host with the parameter -p 3306:3306, we can connect to the MySQL database from any MySQL client (Workbench, MySQL shell) available that can reach the host (hostname or IP) and the port 3306.

Let’s look at a few commands to manage the container.

To stop the MySQL Server Docker Container, run:

# docker stop mysql-latest

To start the stopped MySQL Docker container, don’t try to use docker run to start it again. Instead, use this:

# docker start mysql-latest

To investigate an issue — for example, if the container isn’t starting — access its logs using this command:

# docker logs mysql-latest

To remove the Docker container that we created:

# docker stop mysql-latest

# docker rm mysql-latest

To check which and how many Docker containers are running in the host:

# docker ps

It is possible to customize MySQL parametrization using command-line options to the Docker engine. To configure the InnoDB buffer pool size and the flush method, run the following:

# docker run --name mysql-latest  \

-p 3306:3306 -p 33060:33060  \

-e MYSQL_ROOT_HOST=% -e MYSQL_ROOT_PASSWORD=strongpassword   \

-d mysql/mysql-server:latest \

--innodb_buffer_pool_size=256M \

--innodb_flush_method=O_DIRECT

To run another MySQL version different from the latest MySQL version, first check that it is available in the official Docker Hub. Let’s say we want to run the MySQL 5.7.31 version; the first step is checking the official MySQL Docker Images in Docker Hub to see if it exists.

Once its existence is confirmed, run it with the following command:

# docker run --name mysql-5.7.31  \

-p 3307:3306 -p 33061:33060  \

-e MYSQL_ROOT_HOST=% -e MYSQL_ROOT_PASSWORD=learning_mysql   \

-d mysql/mysql-server:5.7.31

It is possible to run multiple MySQL docker instances, though one potential problem of running multiple MySQL versions in Docker at the same time is TCP port conflicts. In the previous example, we mapped different host ports for the mysql-5.7.31 container (3307 and 33061). Also, the name of the container needs to be unique.

Deploying MariaDB and Percona Server Containers

As demonstrated for deploying MySQL container in the previous section, MariaDB and Percona Server follow the same rules. The main difference is that they use different Docker images and have their own official repositories.

To deploy a MariaDB container:

# docker run --name maria-latest  \

-p 3308:3306 \

-e MYSQL_ROOT_HOST=% -e MYSQL_ROOT_PASSWORD=learning_mysql   \

-d mariadb:latest

And for Percona Server:

# docker run --name ps-latest \

-p 3309:3306 -p 33063:33060 \

-e MYSQL_ROOT_HOST=% -e MYSQL_ROOT_PASSWORD=learning_mysql \

-d percona/percona-server:latest \

--innodb_buffer_pool_size=256M \

--innodb_flush_method=O_DIRECT

NOTE

We are mapping different ports for MariaDB (-p 3308:3306) and Percona (-p 3309:3306) because we are deploying all the containers in the same host:

# docker ps

CONTAINER ID   IMAGE                                      COMMAND

CREATED              STATUS                 PORTS

NAMES

5e487dd41c3e   percona/percona-server:latest

"/docker-entrypoint.…"   About a minute ago   Up 51 seconds

0.0.0.0:3309->3306/tcp, 0.0.0.0:33063->33060/tcp

ps-latest

f5a217f1537b   mariadb:latest

"docker-entrypoint.s…"   2 minutes ago        Up 2 minutes

0.0.0.0:3308->3306/tcp

maria-latest

If you are deploying a single container, you can use the port 3306 or any custom port you might want to use.

Using Sandboxes

In software development, a sandbox is a testing environment that isolates code changes and allows experimentation and testing before deploying to production. DBAs primarily use sandboxes for testing new software versions, performance tests, and bug analysis, and the data present in MySQL is disposable.

In 2018, Giuseppe Maxia introduced DBdeployer, a tool that provides an easy and fast way to deploy MySQL and its forks. It supports diverse topologies such as master x slave (source x replica), master x master (source x source), Galera cluster, and Group Replication.

NOTE

We discuss the concept of high availability in Chapter 13, but we want to clarify some aspects of the terminology used in this book. It is common in the MySQL area to hear the terms master and slave. The origins of these words are clearly negative. Oracle, Percona, and MariaDB therefore decided to change this terminology to source and replica. In this book, we will use both sets of terms because you will encounter both them, but be aware that these companies will implement the following terminology for the upcoming releases:

Old

New

master

source

slave

replica

blacklist

blocklist

whitelist

allowlist

Installing DBdeployer

The tool is developed in the Go language, works with macOS and Linux (Ubuntu and CentOS), and standalone executables are provided. Get the latest version here:

# wget https://github.com/datacharmer/dbdeployer/releases/download/v1.5

8.2/dbdeployer-1.58.2.linux.tar.gz

# tar -xvf dbdeployer-1.58.2.linux.tar.gz

# mv dbdeployer-1.58.2.linux /usr/local/bin/dbdeployer

If you have your /usr/local/bin/ directory in the $PATH variable, you should now be able to run the dbdeployer commands:

# dbdeployer --version

dbdeployer version 1.58.2

Using DBdeployer

The first step to use dbdeployer is downloading the MySQL binary you want to run and then unpacking it into the binaries directory. We will use Linux - Generic tarballs since they are compatible with most Linux distributions, and we will store our binaries in the /opt/mysql directory:

# wget https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.11-linux

-glibc2.12-x86_64.tar.gz

# mkdir /opt/mysql

# dbdeployer --sandbox-binary=/opt/mysql/ unpack mysql-8.0.11-linux-glibc2.12-x86_64.tar.gz

The unpack command will extract and move the files to the appropriate directory. With the --sandbox-binary argument, we can define where we want to place the downloaded binaries in the operating system. In this case, we determined the /opt/mysql/ directory. The output below shows the expected output of this operation:

# * dbdeployer --sandbox-binary=/opt/mysql/ unpack*

mysql-8.0.11-linux-glibc2.12-x86_64.tar.gz

Unpacking tarball mysql-8.0.11-linux-glibc2.12-x86_64.tar.gz to

/opt/mysql/8.0.11

.........100.........200........289

Renaming directory /opt/mysql/mysql-8.0.11-linux-glibc2.12-x86_64 to

/opt/mysql/8.0.11

We can now use the following command to create a new standalone MySQL sandbox with the newly extracted binary:

# dbdeployer --sandbox-binary=/opt/mysql/ deploy single 8.0.11

And we can observe dbdeployer initializing MySQL:

# dbdeployer --sandbox-binary=/opt/mysql/ deploy single 8.0.11

Creating directory /root/sandboxes

Database installed in $HOME/sandboxes/msb_8_0_11

run 'dbdeployer usage single' for basic instructions'

. sandbox server started

Confirm that MySQL is running with the ps command:

# ps -ef | grep mysql

root      4249     1  0 20:18 pts/0    00:00:00 /bin/sh bin/mysqld_safe

--defaults-file=/root/sandboxes/msb_8_0_11/my.sandbox.cnf

root      4470  4249  1 20:18 pts/0    00:00:00

/opt/mysql/8.0.11/bin/mysqld

--defaults-file=/root/sandboxes/msb_8_0_11/my.sandbox.cnf

--basedir=/opt/mysql/8.0.11 --datadir=/root/sandboxes/msb_8_0_11/data

--plugin-dir=/opt/mysql/8.0.11/lib/plugin --user=root

--log-error=/root/sandboxes/msb_8_0_11/data/msandbox.err

--pid-file=/root/sandboxes/msb_8_0_11/data/mysql_sandbox8011.pid

--socket=/tmp/mysql_sandbox8011.sock --port=8011

root      4527  3836  0 20:18 pts/0    00:00:00 grep --color=auto mysql

And we can already connect to MySQL. Use dbdeployer’s use command to connect:

# cd sandboxes/msb_8_0_11/

# ./use

Or use the default root credentials:

# mysql -uroot -pmsandbox -h 127.0.0.1 -P 8011

NOTE

We got the port information from the previous ps command. Remember that there are two ways to connect to MySQL (check Unix and TCP connections). From the previous ps command, we can get the socket file location and connect with it:

# mysql -uroot -pmsandbox -S/tmp/mysql_sandbox8011.sock

If we want to set a replication environment with master (source) x slave (replica) topology using the previous binaries, we can do it with the following command line:

# dbdeployer --sandbox-binary=/opt/mysql/ deploy replication 8.0.11

And we will have three mysqld processes running:

# ps -ef | grep mysql

root      4673     1  0 20:26 pts/0    00:00:00 /bin/sh bin/mysqld_safe

--defaults-file=/root/sandboxes/rsandbox_8_0_11/master/my.sandbox.cnf

root      4942  4673  1 20:26 pts/0    00:00:00

/opt/mysql/8.0.11/bin/mysqld

...

--pid-file=/root/sandboxes/rsandbox_8_0_11/master/data/mysql_sandbox201

12.pid --socket=/tmp/mysql_sandbox20112.sock --port=20112

root      5051     1  0 20:26 pts/0    00:00:00 /bin/sh bin/mysqld_safe

--defaults-file=/root/sandboxes/rsandbox_8_0_11/node1/my.sandbox.cnf

root      5320  5051  1 20:26 pts/0    00:00:00

/opt/mysql/8.0.11/bin/mysqld

--defaults-file=/root/sandboxes/rsandbox_8_0_11/node1/my.sandbox.cnf

...

--pid-file=/root/sandboxes/rsandbox_8_0_11/node1/data/mysql_sandbox2011

3.pid --socket=/tmp/mysql_sandbox20113.sock --port=20113

root      5415     1  0 20:26 pts/0    00:00:00 /bin/sh bin/mysqld_safe

--defaults-file=/root/sandboxes/rsandbox_8_0_11/node2/my.sandbox.cnf

root      5684  5415  1 20:26 pts/0    00:00:00

/opt/mysql/8.0.11/bin/mysqld

...

--pid-file=/root/sandboxes/rsandbox_8_0_11/node2/data/mysql_sandbox2011

4.pid --socket=/tmp/mysql_sandbox20114.sock --port=20114

Another topology that DBdeployer can configure is the Group Replication. For this example, we will define a base-port. By doing this, we will order DBdeployer to configure our servers starting from port 49007:

# Group Replication

# dbdeployer deploy --topology=group replication  --sandbox-binary=/opt/mysql/ 8.0.11 --base-port=49007

Now let’s see an example of the deployment of Galera Cluster using Percona XtraDB Cluster 5.7.32. We will indicate the base-port, and we want our nodes configured with the log-slave-updates option.

# wget https://downloads.percona.com/downloads/Percona-XtraDB-Cluster-5

7/Percona-XtraDB-Cluster-5.7.32-31.47/binary/tarball/Percona-XtraDB-Clu

ster-5.7.32-rel35-47.1.Linux.x86_64.glibc2.17-debug.tar.gz

# dbdeployer --sandbox-binary=/opt/mysql/ unpack

Percona-XtraDB-Cluster-5.7.32-rel35-47.1.Linux.x86_64.glibc2.17-debug.t

ar.gz

# dbdeployer deploy --topology=pxc replication

--sandbox-binary=/opt/mysql/ 5.7.32  --base-port=45007 -c

log-slave-updates

As we saw, it is possible to customize MySQL parameters. One interesting option is enabling MySQL replication using GTID. We discuss GTID in more detail in Chapter 13.

# Replication using GTID

# dbdeployer deploy replication --sandbox-binary=/opt/mysql/ 5.7.32 --gtid

Our last example shows that it is possible to deploy multiple standalone versions at once:

# Create 5 standalone instances

# dbdeployer deploy multiple --sandbox-binary=/opt/mysql/ 5.7.32 -n 5

The previous examples are just a small sample of DBdeployer’s capabilities. The documentation containing more information is available on GitHub. Another option to understand the universe of possibilities is using --help in the command line:

# dbdeployer --help

dbdeployer makes MySQL server installation an easy task.

Runs single, multiple, and replicated sandboxes.

Usage:

dbdeployer [command]

Available Commands:

admin           sandbox management tasks

cookbook        Shows dbdeployer samples

defaults        tasks related to dbdeployer defaults

delete          delete an installed sandbox

delete-binaries delete an expanded tarball

deploy          deploy sandboxes

downloads       Manages remote tarballs

export          Exports the command structure in JSON format

global          Runs a given command in every sandbox

help            Help about any command

import          imports one or more MySQL servers into a sandbox

info            Shows information about dbdeployer environment samples

sandboxes       List installed sandboxes

unpack          unpack a tarball into the binary directory

update          Gets dbdeployer newest version

usage           Shows usage of installed sandboxes

versions        List available versions

Flags:

--config string           configuration file (default

"/home/vinicius.grippa/.dbdeployer/config.json")

-h, --help                    help for dbdeployer

--sandbox-binary string   Binary repository (default

"/home/vinicius.grippa/opt/mysql")

--sandbox-home string     Sandbox deployment directory (default

"/home/vinicius.grippa/sandboxes")

--shell-path string       Which shell to use for generated

scripts (default "/usr/bin/bash")

--skip-library-check      Skip check for needed libraries (may

cause nasty errors)

--version                 version for dbdeployer

Use "dbdeployer [command] --help" for more information about a command.

Upgrading MySQL Server

If the most common questions are about replication, the second most common are about upgrading a MySQL instance. If the procedure is not well tested before doing it in production, the chances of having a problem are high. When we talk about upgrades, there are two essential terms: major and minor.

A major upgrade in MySQL would be changing versions from 5.6 to 5.7 or 5.7 to 8.0. Such an upgrade is trickier and more complex than a minor upgrade because the changes are more substantial to the architecture. For example, a considerable difference from MySQL 5.7 to MySQL 8.0 involved modifying the data dictionary that is now transactional and encapsulated by InnoDB.

A minor upgrade would be changing from MySQL 5.7.29 to 5.7.30 or MySQL 8.0.22 to MySQL 8.0.23. Most of the time, we need to install the new version using our distribution package manager. A minor upgrade is simpler than a major one because it does not involve any change in the architecture. The modifications are focused on fixing bugs, improving the performance, and optimizing the code.

To start the upgrade planning, first choose between two strategies. These are the recommended strategies according to the documentation and are the ones we use:

In-place ppgrade

This involves shutting down MySQL, replacing the old MySQL binaries or packages with the new ones, restarting MySQL on the existing data directory, and running mysql_upgrade .

NOTE

As of MySQL 8.0.16, the mysql_upgrade binary is deprecated, and the MySQL Server itself executes its functionality. We can call this functionality the “server upgrade.” MySQL added this change alongside the data dictionary upgrade (DD upgrade), which is a process to update the data dictionary table definitions. We can list a few benefits of the new process:

- Faster upgrade

- Simpler process

- Better security

- Significant reduction in upgrade steps and more easily automated

- No restarts

- Plug and play

Logical Upgrade

This involves exporting the data in SQL format from the old MySQL version using a backup or export utility such as mysqldump or mysqlpump, installing the new MySQL version, and applying the SQL data to the new MySQL version. In other words, this process involves rebuilding the entire data dictionary and user data. The logical upgrade usually takes longer than the in-place upgrade.

Regardless of your chosen strategy, it is essential to establish a rollback strategy if something goes wrong. The rollback strategy will vary based on the upgrade plan you choose. Moreover, the database size and the topology present (if you have slaves (replicas) or Galera Cluster, for example) will influence this decision.

Now, upgrading by jumping major versions is not recommended.Even between two consecutive major versions, here are some things to take into consideration (especially the last one):

- Upgrade from MySQL 5.7 to 8.0 is supported. However, the upgrade is only supported between General Availability (GA) releases. For MySQL 8.0, it is required that you upgrade from a MySQL 5.7 GA release (5.7.9 or higher). Upgrades from non-GA releases of MySQL 5.7 are not supported.

- Upgrading to the latest release is recommended before upgrading to the next version. For example, upgrade to the latest MySQL 5.7 release before upgrading to MySQL 8.0.

- Upgrades that skip versions are not supported. For example, upgrading directly from MySQL 5.6 to 8.0 is not supported.

NOTE

Production notes: Based on our experience, upgrading from MySQL 5.6 to MySQL 5.7 is the upgrade that causes the most performance issues. Especially if the application is using derived tables (See Nested queries). MySQL modified the optimizer_switch, enabling the derived_merge, and it can harm the server. We will look at the details in Chapter 5.

The other change is that MySQL 5.7 implements network encryption by default (SSL). Applications that were not using SSL in MySQL 5.6 may suffer a substantial performance hit.

Finally, MySQL 5.7 changed the sync_binlog to synchronous mode. This mode is the safest but can harm performance due to the increased number of disk writes.

Let’s go through an example of upgrading from MySQL 5.7 upstream to MySQL 8 upstream using the in-place method.

1. Stop MySQL service

First, we are going to perform a clean shutdown using systemctl:

# systemctl stop mysqld

2. Now we need to remove the old binaries:

# yum erase mysql-community\ -y*

3. This process only removes the binaries and does not touch the datadir (“The Contents of the MySQL Directory”). With the binaries removed, the next step will follow the regular steps for the installation process (see “Installing MySQL on Linux”). For example, to use the MySQL 8 community version on CentOS 7 using yum:

# yum-config-manager --enable mysql80-community

And installing the new binaries:

# yum install mysql-community-server -y

The last step is starting the MySQL service:

# systemctl start mysqld

We can observe in the logs that MySQL upgraded the data dictionary and is running MySQL 8.0.21:

# tail -f /var/log/mysqld.log

2020-08-09T21:20:10.356938Z 2 [System] [MY-011003] [Server] Finished

populating Data Dictionary tables with data.

2020-08-09T21:20:11.734091Z 5 [System] [MY-013381] [Server] Server

upgrade from '50700' to '80021' started.

2020-08-09T21:20:17.342682Z 5 [System] [MY-013381] [Server] Server

upgrade from '50700' to '80021' completed.

...

2020-08-09T21:20:17.463685Z 0 [System] [MY-010931] [Server]

/usr/sbin/mysqld: ready for connections. Version: '8.0.21'  socket:

'/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server - GPL.

NOTE

We highly recommend before upgrading MySQL that you check the release notes. They contain a summary of the changes made and the bug fixes. MySQL upstream, Percona Server, and MariaDB have release notes available for their products.

And finally, we often hear the following question: Is it safe to upgrade to the latest major release?

The answer to this question is … it depends. As with any new product in the industry, early adopters tend to benefit from the new features, but they are testers as well, and they can potentially hit and discover new bugs. When MySQL 8 was released, our recommendation was to wait for three minor releases before considering moving. In this way, we can still enjoy a new major version’s benefits without having to wait too much. The golden rule of this book is to test everything in advance before executing the next step. If you learn just that from this book, we will consider the mission accomplished.

Chapter 2. Modeling and Designing Databases

When implementing a new database, it’s easy to fall into the trap of quickly getting something up and running without dedicating adequate time and effort to the design. This carelessness frequently leads to costly redesigns and reimplementations down the road. Designing a database is like drafting the blueprints for a house; it’s silly to start building without detailed plans. Notably, good design allows you to extend the original building without pulling everything down and starting from scratch. And as you will see, bad designs are directly related to poor database performance.

How Not to Develop a Database

Database design is probably not the most exciting task in the world, but indeed it is becoming one of the most important ones. Before we describe how to go about the design process, let’s look at an example of database design on the run.

Imagine we want to create a database to store student grades for a university computer science department. We could create a Student_Grades table to store grades for each student and each course. The table would have columns for the given names and the surname of each student and each course they have taken, the course name, and the percentage result (shown as Pctg). We’d have a different row for each student for each of their courses:

+------------+---------+---------------------------+------+

| GivenNames | Surname | CourseName                | Pctg |

+------------+---------+---------------------------+------+

| John Paul  | Bloggs  | Data Science              |   72 |

| Sarah      | Doe     | Programming 1             |   87 |

| John Paul  | Bloggs  | Computing Mathematics     |   43 |

| John Paul  | Bloggs  | Computing Mathematics     |   65 |

| Sarah      | Doe     | Data Science              |   65 |

| Susan      | Smith   | Computing Mathematics     |   75 |

| Susan      | Smith   | Programming 1             |   55 |

| Susan      | Smith   | Computing Mathematics     |   80 |

+------------+---------+---------------------------+------+

The list is nice and compact, we can easily access grades for any student or any course, and it looks similar to a spreadsheet. However, we could have more than one student called Susan Smith; there are two entries for Susan Smith and the Computing Mathematics course in the sample data. Which Susan Smith got an 80? A common way to differentiate duplicate data entries is to assign a unique number to each entry. Here, we can assign a unique Student ID number to each student:

+------------+------------+---------+---------------------------+------+

| StudentID  | GivenNames | Surname | CourseName                | Pctg |

+------------+------------+---------+---------------------------+------+

| 12345678   | John Paul  | Bloggs  | Data Science              |   72 |

| 12345121   | Sarah      | Doe     | Programming 1             |   87 |

| 12345678   | John Paul  | Bloggs  | Computing Mathematics     |   43 |

| 12345678   | John Paul  | Bloggs  | Computing Mathematics     |   65 |

| 12345121   | Sarah      | Doe     | Data Science              |   65 |

| 12345876   | Susan      | Smith   | Computing Mathematics     |   75 |

| 12345876   | Susan      | Smith   | Programming 1             |   55 |

| 12345303   | Susan      | Smith   | Computing Mathematics     |   80 |

+------------+------------+---------+---------------------------+------+

Now we know which Susan Smith got 80; it is the one with the Student ID number 12345303.

There’s another problem. In our table, John Paul Bloggs has failed the Computing Mathematics course once with 43 percent and passed it with 65 percent in his second attempt. In a relational database, the rows form a set, and there is no implicit ordering between them; we might guess that the pass happened after the failure, but we can’t be sure. There’s no guarantee that the newer grade will appear after the older one, so we need to add information about when each grade was awarded, say by adding a year and semester (Sem):

+------------+------------+---------+---------------------------+------+-----+------+

| StudentID  | GivenNames | Surname | CourseName                | Year | Sem | Pctg |

+------------+------------+---------+---------------------------+------+-----+------+

| 12345678   | John Paul  | Bloggs  | Data Science              | 2019 |   2 |   72 |

| 12345121   | Sarah      | Doe     | Programming 1             | 2020 |   1 |   87 |

| 12345678   | John Paul  | Bloggs  | Computing Mathematics     | 2019 |   2 |   43 |

| 12345678   | John Paul  | Bloggs  | Computing Mathematics     | 2020 |   1 |   65 |

| 12345121   | Sarah      | Doe     | Data Science              | 2020 |   1 |   65 |

| 12345876   | Susan      | Smith   | Computing Mathematics     | 2019 |   1 |   75 |

| 12345876   | Susan      | Smith   | Programming 1             | 2019 |   2 |   55 |

| 12345303   | Susan      | Smith   | Computing Mathematics     | 2020 |   1 |   80 |

+------------+------------+---------+---------------------------+------+-----+------+

Notice that the Student_Grades table has become a bit bloated: We repeated the student ID, given names, and surname for every grade. We could split up the information and create a Student_Details table:

+------------+------------+---------+

| StudentID  | GivenNames | Surname |

+------------+------------+---------+

| 12345121   | Sarah      | Doe     |

| 12345303   | Susan      | Smith   |

| 12345678   | John Paul  | Bloggs  |

| 12345876   | Susan      | Smith   |

+------------+------------+---------+

And we could keep less information in the Student_Grades table:

+------------+---------------------------+------+-----+------+

| StudentID  | CourseName                | Year | Sem | Pctg |

+------------+---------------------------+------+-----+------+

| 12345678   | Data Science              | 2019 |   2 |   72 |

| 12345121   | Programming 1             | 2020 |   1 |   87 |

| 12345678   | Computing Mathematics     | 2019 |   2 |   43 |

| 12345678   | Computing Mathematics     | 2020 |   1 |   65 |

| 12345121   | Data Science              | 2020 |   1 |   65 |

| 12345876   | Computing Mathematics     | 2019 |   1 |   75 |

| 12345876   | Programming 1             | 2019 |   2 |   55 |

| 12345303   | Computing Mathematics     | 2020 |   1 |   80 |

+------------+---------------------------+------+-----+------+

To look up a student’s grades, we’d need to first look up her Student ID from the Student_Details table and then read the grades for that Student ID from the Student_Grades table.

There are still issues we haven’t considered. For example, should we keep information on a student’s enrollment date, postal and email addresses, fees, or attendance? Should we store different types of postal addresses? How should we store addresses so that things don’t break when students change their addresses?

Implementing a database in this way is problematic; we keep running into things we hadn’t thought about and have to keep changing our database structure. We can save a lot of reworking by carefully documenting the requirements and then working through them to develop a coherent design.

The Database Design Process

There are three major stages in the database design, each producing a progressively lower-level description:

1. Requirements analysis:: First, we determine and write down what we need from the database, which data we will store, and how the data items relate to each other. In practice, this might involve a detailed study of the application requirements and talk to people in various roles that will interact with the database and application.

2. Conceptual design:: Once we know the database requirements, we distill them into a formal description of the database design. Later in this chapter we’ll see how to use modeling to produce the conceptual design.

3. Logical design:: Finally, we map the database design onto an existing database management system and database tables.

At the end of the chapter, we’ll look at how we can use the open-source MySQL Workbench tool to convert the conceptual design to a MySQL database schema.

The Entity Relationship Model

At a basic level, databases store information about distinct objects, or entities, and the associations, or relationships, between these entities. For example, a university database might store information about students, courses, and enrollment. A student and a course are entities, whereas enrollment is a relationship between a student and a course. Similarly, an inventory and sales database might store information about products, customers, and sales. A product and a customer are entities, and a sale is a relationship between a customer and a product. It is common to get confused between entity and relationships when starting, and we may end up designing relationships as entities and vice-versa. The best way to improve database design is by practicing a lot.

A popular approach to conceptual design uses the Entity Relationship (ER) model, which helps transform the requirements into a formal description of the entities and relationships in the database. We’ll start by looking at how the ER modeling process works and then observe it in “Entity Relationship Modeling Examples” for three sample databases.

Representing Entities

To help visualize the design, the Entity-Relationship Modeling approach involves drawing an ER diagram. In the ER diagram, we represent an entity set by a rectangle containing the entity name. For our sales database example, our ER would show the product and customer entity sets, as shown in Figure 2-1.

Figure 2-1. An entity set is represented by a named rectangle

We typically use the database to store specific characteristics, or attributes, of the entities. We could record the name, email address, postal address, and telephone number of each customer in a sales database. In a more elaborate customer relationship management (CRM) application, we could also store the names of the customer’s spouse and children, the languages the customer speaks, the customer’s history of interaction with our company, and so on. Attributes describe the entity they belong to.

We may form an attribute from smaller parts; for example, we compose a postal address from a street number, city, ZIP code, and country. We classify attributes as composite if they’re composed of smaller parts in this way, and as simple otherwise.

Some attributes can have multiple values for a given entity. For example, a customer could provide several telephone numbers, so the telephone number attribute is multivalued.

Attributes help distinguish one entity from other entities of the same type. We could use the name attribute to differentiate between customers, but this could be an inadequate solution because several customers could have identical names. To tell them apart, we need an attribute (or a minimal combination of attributes) guaranteed to be unique to each customer. The identifying attribute or attributes form a unique key, and in this particular case, we call it a Primary Key.

In our example, we can assume that no two customers have the same email address, so that the email address can be the primary key. However, we need to think carefully about the implications of our choices. For example, if we decide to identify customers by their email addresses, it would be hard to allow a customer to have multiple email addresses. Any applications we build to use this database might treat each email address as a separate person. It might be hard to adapt everything to allow people to have multiple email addresses. Using the email address as the key also means that every customer must have an email address; otherwise, we couldn’t distinguish between customers who don’t have one.

Looking at the other attributes for one that can serve as an alternative key, we see that while it’s possible that two customers would have the same telephone number (and so we cannot use the telephone number as a key), it’s likely that people who have the same telephone number never have the same name so that we can use the combination of the telephone number and the name as a composite key.

Clearly, there may be several possible keys that could be used to identify an entity; we choose one of the alternatives, or candidate, keys to be our main, or primary, key. We usually choose based on how confident we are that the attribute will be non-empty and unique for each entity and how small the key is (shorter keys are faster to maintain and to perform lookup operations).

In the ER diagram, attributes are represented as labeled ovals and connected to their entity, as shown in Figure 2-2. Attributes comprising the primary key are shown underlined. The parts of any composite attributes are drawn connected to the composite attribute’s oval, and multivalued attributes are shown as double-lined ovals.

Figure 2-2. The ER diagram representation of the customer entity

Attribute values are chosen from a domain of legal values; for example, we could specify that a customer’s given names and surname attributes can each be a string of up to 100 characters, while a telephone number can be a string of up to 40 characters. Similarly, a product price could be a positive rational number.

Attributes can be empty; for example, some customers may not provide their telephone numbers. The primary key of an entity (including the components of a multiattribute primary key) must never be unknown (technically, it must be NOT NULL); for example, if it’s possible for a customer to not provide an email address, we cannot use the email address as the key.

You should think carefully when classifying an attribute as multivalued: are all the values equivalent, or do they in fact represent different things? For example, when listing multiple telephone numbers for a customer, would they be more usefully labeled separately as the customer’s business phone number, home phone number, cell phone number, and so on?

Let’s look at another example. The sales database requirements may specify that a product has a name and a price. We can see that the product is an entity because it’s a distinct object. However, the product’s name and price aren’t distinct objects; they’re attributes that describe the product entity. Note that if we want to have different prices for different markets, then the price is no longer just related to the product entity, and we will need to model it differently.

For some applications, no combination of attributes can uniquely identify an entity (or it would be too unwieldy to use a large composite key), so we create an artificial attribute that’s defined to be unique and can therefore be used as a key: student numbers, Social Security numbers, driver’s license numbers, and library card numbers are examples of unique attributes created for various applications. In our inventory and sales application, it’s possible that we could stock different products with the same name and price. For example, we could sell two models of “Four-port USB 2.0 Hub,” both at $4.95 each. To distinguish between products, we can assign a unique product ID number to each item we stock; this would be the primary key. Each product entity would have name, price, and product ID attributes. This is shown in the ER diagram in Figure 2-3.

Figure 2-3. The ER diagram representation of the product entity

Representing Relationships

Entities can participate in relationships with other entities. For example, a customer can buy a product, a student can take a course, an employee can have an address, and so on.

Like entities, relationships can have attributes: we can define a sale to be a relationship between a customer entity (identified by the unique email address) and a given number of the product entity (identified by the unique product ID) that exists at a particular date and time (the timestamp).

Our database could then record each sale and tell us, for example, that at 3:13 p.m. on Wednesday, March 22, Marcos Albe bought one “Raspberry Pi 4”, one “500 GB SSD M.2 NVMe”, and two sets of “2000 Watt 5.1 Channel Sub-Woofer Speakers.”

Different numbers of entities can appear on each side of a relationship. For example, each customer can buy any number of products, and each product can be bought by any number of customers. This is known as a many-to-many relationship. We can also have one-to-many relationships. For example, one person can have several credit cards, but each credit card belongs to just one person. Looking at it the other way, a one-to-many relationship becomes a many-to-one relationship; for example, many credit cards belong to a single person. Finally, the serial number on a car engine is an example of a one-to-one relationship; each engine has just one serial number, and each serial number belongs to just one engine. We often use the shorthand terms 1:1, 1:N, and M:N for one-to-one, one-to-many, and many-to-many relationships, respectively.

The number of entities on either side of a relationship (the cardinality of the relationship) define the key constraints of the relationship. It’s important to think about the cardinality of relationships carefully. There are many relationships that may at first seem to be one-to-one, but turn out to be more complex. For example, people sometimes change their names; in some applications, such as police databases, this is of particular interest, and so it may be necessary to model a many-to-many relationship between a person entity and a name entity. Redesigning a database can be costly and time-consuming if you assume a relationship is simpler than it really is.

In an ER diagram, we represent a relationship set with a named diamond. The cardinality of the relationship is often indicated alongside the relationship diamond; this is the style we use in this book. (Another common style is to have an arrowhead on the line connecting the entity on the “1” side to the relationship diamond.) Figure 2-4 shows the relationship between the customer and product entities, along with the number and timestamp attributes of the sale relationship.

Figure 2-4. The ER diagram representation of the customer and product entities, and the sale relationship between them.

Partial and Total Participation

Relationships between entities can be optional or compulsory. In our example, we could decide that a person is considered to be a customer only if they have bought a product. On the other hand, we could say that a customer is a person whom we know about and whom we hope might buy something—that is, we can have people listed as customers in our database who never buy a product. In the first case, the customer entity has total participation in the bought relationship (all customer\s have bought a product, and we can’t have a customer who hasn’t bought a product), while in the second case it has partial participation (a customer can buy a product). These are referred to as the participation constraints of the relationship. In an ER diagram, we indicate total participation with a double line between the entity box and the relationship diamond.

Entity or Attribute?

From time to time, we encounter cases where we wonder whether an item should be an attribute or an entity on its own. For example, an email address could be modeled as an entity in its own right. When in doubt, consider these rules of thumb:

- Is the item of direct interest to the database?:: Objects of direct interest should be entities, and information that describes them should be stored in attributes. Our inventory and sales database is really interested in customers, and not their email addresses, so the email address would be best modeled as an attribute of the customer entity.

1. Does the item have components of its own?:: If so, we must find a way of representing these components; a separate entity might be the best solution. In the student grades example at the start of the chapter, we stored the course name, year, and semester for each course that a student takes. It would be more compact to treat the course as a separate entity and to create a class ID number to identify each time a course is offered to students (the “offering”).

2. Can the object have multiple instances?:: If so, we must find a way to store data on each instance. The cleanest way to do this is to represent the object as a separate entity. In our sales example, we must ask whether customers are allowed to have more than one email address; if they are, we should model the email address as a separate entity.

- Is the object often nonexistent or unknown?:: If so, it is effectively an attribute of only some of the entities, and it would be better to model it as a separate entity rather than as an attribute that is often empty. Consider a simple example: to store student grades for different courses, we could have an attribute for the student’s grade in every possible course; this is shown in Figure 2-5. Because most students will have grades for only a few of these courses, it’s better to represent the grades as a separate entity set, as in Figure 2-6.

Figure 2-5. The ER diagram representation of student grades as attributes of the student entity

Figure 2-6. The ER diagram representation of student grades as a separate entity

Entity or Relationship?

An easy way to decide whether an object should be an entity or a relationship is to map nouns in the requirements to entities, and map verbs to relations. For example, in the statement “A degree program is made up of one or more courses,” we can identify the entities “program” and “course,” and the relationship “is made up of.” Similarly, in the statement “A student enrolls in one program,” we can identify the entities “student” and “program,” and the relationship “enrolls in.” Of course, we can choose different terms for entities and relationships than those that appear in the relationships, but it’s a good idea not to deviate too far from the naming conventions used in the requirements so that the design can be checked against the requirements. All else being equal, try to keep the design simple, and avoid introducing trivial entities where possible; that is, there’s no need to have a separate entity for the student’s enrollment when we can model it as a relationship between the existing student and program entities.

Intermediate Entities

It is often possible to conceptually simplify many-to-many relationships by replacing the many-to-many relationship with a new intermediate entity (sometimes called an associate entity) and connecting the original entities through a many-to-one and a one-to-many relationship.

Consider this statement: “A passenger can book a seat on a flight.” This is a many-to-many relationship between the entities “passenger” and “flight.” The related ER diagram fragment is shown in Figure 2-7.

Figure 2-7. A passenger participates in an M:N relationship with flight

However, let’s look at this from both sides of the relationship:

- Any given flight can have many passengers with a booking.

- Any given passenger can have bookings on many flights.

Hence, we can consider the many-to-many relationship to be in fact two one-to-many relationships, one each way. This points us to the existence of a hidden intermediate entity, the booking, between the flight and the passenger entities. The requirement could be better worded as: “A passenger can make a booking for a seat on a flight.” The related ER diagram fragment is shown in Figure 2-8.

Figure 2-8. The intermediate booking entity between the passenger and flight entities

Each passenger can be involved in multiple bookings, but each booking belongs to a single passenger, so the cardinality of this relationship is 1:N. Similarly, there can be many bookings for a given flight, but each booking is for a single flight, so this relationship also has cardinality 1:N. Since each booking must be associated with a particular passenger and flight, the booking entity participates totally in the relationships with these entities. This total participation could not be captured effectively in the representation in Figure 2-7. (We described partial and total participation earlier in “Partial and Total Participation”.)

Weak and Strong Entities

Context is very important in our daily interactions; if we know the context, we can work with a much smaller amount of information. For example, we generally call family members by only their first name or nickname. Where ambiguity exists, we add further information such as the surname to clarify our intent. In database design, we can omit some key information for entities that are dependent on other entities. For example, if we wanted to store the names of our customers’ children, we could create a child entity and store only enough key information to identify it in the context of its parent. We could simply list a child’s first name on the assumption that a customer will never have several children with the same first name. Here, the child entity is a weak entity, and its relationship with the customer entity is called an identifying relationship. Weak entities participate totally in the identifying relationship, since they can’t exist in the database independently of their owning entity.

In the ER diagram, we show weak entities and identifying relationships with double lines, and the partial key of a weak entity with a dashed underline, as in Figure 2-9. A weak entity is uniquely identified in the context of its regular (or strong) entity, and so the full key for a weak entity is the combination of its own (partial) key with the key of its owning entity. To uniquely identify a child in our example, we need the first name of the child and the email address of the child’s parent.

Figure 2-9. The ER diagram representation of a weak entity

Figure 2-10 shows a summary of the symbols we’ve explained for ER diagrams.

Figure 2-10. Quick summary of the ER diagram symbols

Database Normalization

Database normalization is an important concept when designing the relational data structure. Dr. Edgar F. Codd, the inventor of the relational database model, proposed the normal forms in the early ’70s, which is widely used by the industry nowadays. Even with the advent of the NoSQL databases, there is no evidence in the short or medium-term that relational databases will disappear, or that the normal forms will fall in disuse.

The main objective of the normal norms is to reduce data redundancy and improve data integrity. It also facilitates the process of redesigning and extending the database structure.

Officially, there are six normal forms, but most database architects only deal with the first three forms. That is because the normalization process is progressive, and we cannot achieve a higher level of database normalization unless the previous levels have been satisfied. Using all the six norms constricts too much of the database model, and in general, they become very complex to implement. In real workloads, usually, there are performance issues. This is one reason for ETL jobs to exist (ETL jobs denormalize the data to process it).

Let’s take a look at the first three normal forms.

1NF: First normal form

- Eliminate repeating groups in individual tables

- Create a separate table for each set of related data

- Identify each set of related data with a primary key

If a relation contains composite or multi-valued attribute, it violates the first normal form, or a relation is in first normal form if it does not contain any composite or multi-valued attribute. A relation is in first normal form if every attribute in that relation is single-value attribute.

2NF: Second normal form

- Create separate tables for sets of values that apply to multiple records

- Relate these tables with a foreign key

Records should not depend on anything other than a table’s primary key (a compound key, if necessary).

3NF: Third normal form

- Eliminate fields that do not depend on the key

Values in a record that are not part of that record’s key do not belong in the table. In general, any time the contents of a group of fields may apply to more than a single record in the table, consider placing those fields in a separate table.

The following table lists the normal forms from the least normalized to the most normalized:

UNF (1970)

1NF (1970)

2NF (1971)

3NF (1971)

EKNF (1982)

BCNF (1974)

4NF (1977)

ETNF (2012)

5NF (1979)

DKNF (1981)

6NF (2003)

Primary key (no duplicate tuples)

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

No repeating groups

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Atomic columns (cells have single value)

No

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Every nontrivial functional dependency either does not begin with a proper subset of a candidate key or ends with a prime attribute (no partial functional dependencies of nonprime attributes on candidate keys)

No

No

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Every nontrivial functional dependency begins with a superkey or ends with a prime attribute (no transitive functional dependencies of nonprime attributes on candidate keys)

No

No

No

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Every nontrivial functional dependency either begins with a superkey or ends with an elementary prime attribute

No

No

No

No

Yes

Yes

Yes

Yes

Yes

Yes

N/A

Every nontrivial functional dependency begins with a superkey

No

No

No

No

No

Yes

Yes

Yes

Yes

Yes

N/A

Every nontrivial multivalued dependency begins with a superkey

No

No

No

No

No

No

Yes

Yes

Yes

Yes

N/A

Every join dependency has a superkey component

No

No

No

No

No

No

No

Yes

Yes

Yes

N/A

Every join dependency has only superkey components

No

No

No

No

No

No

No

No

Yes

Yes

N/A

Every constraint is a consequence of domain constraints and key constraints

No

No

No

No

No

No

No

No

No

Yes

N/A

Every join dependency is trivial

No

No

No

No

No

No

No

No

No

No

Yes

Normalizing an Example Table

Let’s see an example of normalizing a fictional student table.

Unnormalized table:

Student# Advisor Adv-Room Class1 Class2 Class3

1022     Jones    412     101-07 143-01 159-02

4123     Smith    216     201-01 211-02 214-01

First normal form: No repeating groups

Tables should have only two dimensions. Since one student has several classes, these classes should be listed in a separate table. Fields Class1, Class2, and Class3 in the above records are indications of design trouble.

Spreadsheets often use the third dimension, but tables should not. Here’s another way to look at this problem: with a one-to-many relationship, don’t put the one side and the many side in the same table. Instead, create another table in first normal form by eliminating the repeating group (Class#), as shown here:

Student# Advisor Adv-Room Class#

1022     Jones    412     101-07

1022     Jones    412     143-01

1022     Jones    412    159-02

4123     Smith    216     201-01

4123     Smith    216     211-02

4123     Smith    216     214-01

Second normal form: Eliminate redundant data

Note the multiple Class# values for each Student# value in the previous table. Class# is not functionally dependent on Student# (primary key), so this relationship is not in second normal form.

The following two tables demonstrate second normal form:

Students:

Student# Advisor Adv-Room

1022     Jones    412

4123     Smith   216

Registration:

Student# Class#

1022     101-07

1022     143-01

1022     159-02

4123     201-01

4123     211-02

4123     214-01

Third normal form: Eliminate data not dependent on key

In the last example, Adv-Room (the advisor’s office number) is functionally dependent on the Advisor attribute. The solution is to move that attribute from the Students table to the Faculty table, as shown next.

Students:

Student# Advisor

1022     Jones

4123     Smith

Faculty:

Name Room Dept

Jones 412 42

Smith 216 42

Entity Relationship Modeling Examples

Earlier in this chapter, we showed hypothetical cases to design a database and to understand an Entity-Relationship (ER) diagram and the normalization rules. Now we are going to see some ER examples from sample databases available for MySQL. To visualize the ER, we are going to use MySQL Workbench.

MySQL Workbench uses a physical ER representation. Physical ER diagram models are more granular, showing the processes necessary to add information to a database. Rather than using symbols, we use tables in the ER, making it closer to the real database. MySQL Workbench goes one step further and uses an enhanced entity-relationship (EER). EER diagrams are an expanded version of ER diagrams.

We are not going into the details of the EER, but the advantages of an EER diagram is that it provides all the elements of an ER diagram while adding the following:

- Attribute or relationship inheritances

- Category or union types

- Specialization and generalization

- Subclasses and superclasses

Let’s start with the process to download the samples and visualize its EER in MySQL Workbench.

The first one is the Sakila database. Development of the Sakila sample database began in early 2005. Early designs were based on the Dell whitepaper database, which was used in the Dell whitepaper Three Approaches to MySQL Applications on Dell PowerEdge Servers.

Where Dell’s sample database was designed to represent an online DVD store, the Sakila sample database is designed to represent a DVD rental store. The Sakila sample database still borrows film and actor names from the Dell sample database. The following commands will import the Sakila database to our MySQL instance:

# wget https://downloads.mysql.com/docs/sakila-db.tar.gz

# tar -xvf sakila-db.tar.gz

# mysql -uroot -pmsandbox < sakila-db/sakila-schema.sql

# mysql -uroot -pmsandbox < sakila-db/sakila-data.sql

Sakila also provides the EER model (sakila.mwb file). We can open the file with MySQL Workbench as shown in Figure 2-11.

Figure 2-11. Sakila database EER, note the physical representation of the entities instead of using symbols

Next is the World database. Another database available from Oracle, the sample data used in the World database is Copyright Statistics Finland.

The following commands will import the World database to our MySQL instance:

# wget https://downloads.mysql.com/docs/world.sql.gz

# zcat world.sql.gz | mysql -uroot -pmsandbox

The World database does not come with the EER file as Sakila does. But we can create the EER model from the real database using MySQL Workbench using reverse engineering. We need to click the Databases menu and then Reverse Engineer as in

Figure 2-12:

Figure 2-12. Reverse engineering from World database

Then, Workbench will connect to the database (if not connected already) and prompt us to choose the schema we want to reverse to ER as show in Figure 2-13:

Figure 2-13. Choosing the schema we want to see the ER

The last screen is to confirm its execution as shown in Figure 2-14:

Figure 2-14. Press Execute to start

And we have our ER from the World database on Figure 2-15:

Figure 2-15. The ER from the World database

The last database we will import is the Employee database. To give proper credit, we need to mention the creators and supporters of this database. Fusheng Wang and Carlo Zaniolo created the original data at Siemens Corporate Research. Giuseppe Maxia made the relational schema, and Patrick Crews exported the data in relational format.

To import the data, first, we need to clone the Git repository:

# git clone https://github.com/datacharmer/test_db.git

# cd test_db

# cat employees.sql | mysql -uroot -psekret

And we can use the reverse engineering procedure from MySQL Workbench to create the ER model for the Employee database as shown in Figure 2-16:

Figure 2-16. The ER from the Employee database

You’ll find that having an overview of the ER diagrams and the database designs’ explanations is sufficient to work with the material in this chapter. We’ll show you how to create a database on your MySQL server in Chapter 3. Also, during our explanation of the CRUD statements, we will cover again the theory we saw in this chapter, especially when performing JOIN (“Joining Two Tables”) operations.

Using the Entity Relationship Model

This section looks at the steps required to create an ER model and deploy it into database tables. We saw previously that MySQL Workbench lets us reverse engineer an existing database. But how do we model a new database and deploy it? We can automate this process with the MySQL Workbench tool.

Mapping Entities and Relationships to Database Tables

When converting an ER model to a database schema, we work through each entity and then through each relationship according to the rules discussed in the following sections to end up with a set of database tables.

Map the entities to database tables

For each strong entity, create a table comprising its attributes and designate the primary key. The parts of any composite attributes are also included here.

For each weak entity, create a table comprising its attributes and including the primary key of its owning entity. The owning entity’s primary key is a foreign key here because it’s a key not of this table but another table. The table’s primary key for the weak entity is the combination of the foreign key and the partial key of the weak entity. If the relationship with the owning entity has any attributes, add them to this table.

For each entity’s multivalued attribute, create a table comprising the entity’s primary key and the attribute.

Map the relationships to database tables

Each one-to-one relationship between two entities includes the primary key of one entity as a foreign key in the table belonging to the other. If one entity participates totally in the relationship, place the foreign key in its table. If both participate totally in the relationship, consider merging them into a single table.

For each non-identifying one-to-many relationship between two entities, including the entity’s primary key on the “1” side as a foreign key in the table for the entity on the “N” side. Add any attributes of the relationship in the table alongside the foreign key. Note that identifying one-to-many relationships (between a weak entity and its owning entity) are captured as part of the entity-mapping stage.

For each many-to-many relationship between two entities, create a new table containing each entity’s primary key as the primary key and add any attributes of the relationship. This step helps to identify intermediate entities.

For each relationship involving more than two entities, create a table with the primary keys of all the participating entities, and add any relationship attributes.

Creating a Bank Database ER Model

We’ve discussed database models for employees, sales, and CRM. Let’s see how we could model a bank database.

Following the mapping rules as just described, we first map entities to database tables. We collected all the requisites with the stakeholders, we defined our requirements for the online banking system, and we decided we need to have the following entities:

- Employees

- Branches

- Customers

- Accounts

Once we’ve identified the entities, we are going to create the tables and attributes for each table. We established primary keys to ensure every table has a unique identifier column for its records. Following, we need to define the relationships between the tables.

Many to many relationships (N:M):

We’ve established this type of relationship between Branches and Employees, Accounts, and Customers. An employee can work for any number of branches, and a branch could have any number of employees. Similarly, a customer could have many accounts, and an account could be a joint account between more than two customers.

To solve these relationships, we need two more intermediate entities. We create them as follows:

- Account_Customers

- Branch_Employees

Since the source table’s primary key is also the primary key of the join table, the relation was thought to be an identifying relation.

One to many relationship (1:N)

This type of relation was established between Branches and Accounts, and customer and banking transactions. These relationships were non-identifying relationships as the parent table’s primary key is only used as part of the foreign key in the child table.

Because we will work on a physical EER, we are also going to define the Primary Keys. It is common and recommended to use auto increment and unsigned fields for the Primary Key.

The Figure 2-17 shows the final representation of the bank model.

Figure 2-17. The EER model for the bank database

Note that there are items we haven’t considered for this model. For example, our model does not support the inputting of family members of the employees. The model also does not support a customer with multiple addresses (work address and home address, for example). We did this intentionally to emphasize the importance of collecting the requisites prior to the database deployment.

You can download the model at the GitHub repository. The file is bank_model.mwb.

Converting the EER to a MySQL Database Using Workbench

It’s a good idea to use a tool to draw our ER diagrams; this way, we can easily edit diagrams and refine our design until the final diagram is clear and unambiguous. Once we are comfortable with the model, we can deploy it. MySQL workbench allows the conversion of the EER model into DDL statements.

To convert the EER model into DDL statements to create a MySQL database, first, click the Database option, and then Forward Engineer, as shown in Figure 2-18:

Figure 2-18. Finding the Forward Engineer option

We need to enter the credentials to connect to the database, and after that MySQL Workbench will present some options. For this model, we are going to use the standard options as shown in Figure 2-19:

Figure 2-19. Setting the options

The Select Objects` option will ask which elements of the model we want to generate. Since we do not have anything special like triggers, stored procedures, users, and so on, we will only create the table objects and their relation; the rest of the options are zeroed.

MySQL workbench will present us with the SQL statements of our model, as shown in Figure 2-20:

Figure 2-20. We can see all the required DML statements to create the database

And the last step, MySQL Workbench will execute the statements in our MySQL server as demonstrated by Figure 2-21:

Figure 2-21. MySQL Workbench starts running the script

We cover the details of the DDL statements in “Creating Tables”.

Chapter 3. Basic SQL

As mentioned in Chapter 2, Dr. Edgar F. Codd conceived the relational database model in 1969 and its normal forms in the early 1970s. In IBM laboratories in San Jose, a major research project started in the early 1970s called System/R, intending to prove the relational model’s viability. Simultaneously, in 1974, Dr. Donald Chamberlin and his colleagues were also working to define a database language. They developed Structured English Query Language (SEQUEL), which allowed users to query a relational database using clearly defined English-style sentences, which was later renamed to SQL (Structured Query Language or SQL Query Language) for legal reasons.

The first database management systems based on SQL became available commercially by the end of the ’70s. With the growing activity surrounding the development of database languages, standardization emerged to simplify things. And the community elected SQL for standardization. Both the American ANSI and the international ISO took part in the standardization, and in 1986 the first SQL standard was approved. After that, several versions existed. It is common to refer to the SQL standards as “SQL:1999”, “SQL:2003”, “SQL:2008”, and “SQL:2011”, and they refer to the versions of the standard released in the corresponding years, with the last being the most recent version. We will use the phrase the SQL standard or standard SQL to mean the current version of the SQL standard at any time.

MySQL extends the standard SQL providing extra features. For example, MySQL implements the STRAIGHT_JOIN which is a syntax not recognized by other DBMSes.

This chapter introduces MySQL’s SQL implementation, which we often refer to as CRUD operations. CRUD refers to CREATE, READ, UPDATE, and DELETE operations. We will show you how to read data from a database with the SELECT statement and choose what data we can retrieve and in which order it is displayed. We also show you the basics of modifying your databases with the INSERT statement to add data, UPDATE to change, and DELETE to remove it. And we explain how to use the nonstandard SHOW TABLES and SHOW COLUMNS statements to explore your database.

Using the Sakila Database

In Chapter 2, we showed you the principles of how to build a database diagram using the ER model. We also introduced the steps you take to convert an ER model to a format that makes sense for constructing a relational database. This section will show you the structure of the MySQL sakila database for you to start getting familiar with different database relational models. We won’t explain the SQL statements used to create the database here; that’s the subject of Chapter 4.

To begin exploring the sakila database, if you haven’t imported it yet, check “Entity Relationship Modeling Examples” to perform the task.

To choose the sakila database as our current database, we will use the USE statement.

Type the following command:

mysql> USE sakila;

Database changed

mysql>

We can check which is the active database by typing the SELECT DATABASE(); command:

mysql> SELECT DATABASE();

+------------+

| DATABASE() |

+------------+

| sakila     |

+------------+

1 row in set (0.00 sec)

Now, let’s explore what tables make up the sakila database using the SHOW TABLES statement:

mysql> SHOW TABLES;

+----------------------------+

| Tables_in_sakila           |

+----------------------------+

| actor                      |

| actor_info                 |

| address                    |

| category                   |

| city                       |

| country                    |

| customer                   |

| customer_list              |

| film                       |

| film_actor                 |

| film_category              |

| film_list                  |

| film_text                  |

| inventory                  |

| language                   |

| nicer_but_slower_film_list |

| payment                    |

| rental                     |

| sales_by_film_category     |

| sales_by_store             |

| staff                      |

| staff_list                 |

| store                      |

+----------------------------+

23 rows in set (0.00 sec)

So far, there have been no surprises. Let’s find out more about each of the tables that make up the sakila database. First, let’s use the SHOW COLUMNS statement to explore the actor table:

mysql> SHOW COLUMNS FROM actor;

+-------------+-------------------+------+-----+-------------------+---

--------------------------------------------+

| Field       | Type              | Null | Key | Default           |

Extra                                         |

+-------------+-------------------+------+-----+-------------------+---

--------------------------------------------+

| actor_id    | smallint unsigned | NO   | PRI | NULL              |

auto_increment                                |

| first_name  | varchar(45)       | NO   |     | NULL              |

|

| last_name   | varchar(45)       | NO   | MUL | NULL              |

|

| last_update | timestamp         | NO   |     | CURRENT_TIMESTAMP |

DEFAULT_GENERATED on update CURRENT_TIMESTAMP |

+-------------+-------------------+------+-----+-------------------+---

--------------------------------------------+

4 rows in set (0.01 sec)

The DESCRIBE keyword is identical to SHOW COLUMNS FROM, and we can abbreviate it to just DESC, so we can write the previous query as follows:

mysql> DESC actor;

+-------------+-------------------+------+-----+-------------------+---

--------------------------------------------+

| Field       | Type              | Null | Key | Default           |

Extra                                         |

+-------------+-------------------+------+-----+-------------------+---

--------------------------------------------+

| actor_id    | smallint unsigned | NO   | PRI | NULL              |

auto_increment                                |

| first_name  | varchar(45)       | NO   |     | NULL              |

|

| last_name   | varchar(45)       | NO   | MUL | NULL              |

|

| last_update | timestamp         | NO   |     | CURRENT_TIMESTAMP |

DEFAULT_GENERATED on update CURRENT_TIMESTAMP |

+-------------+-------------------+------+-----+-------------------+-----------------------------------------------+

4 rows in set (0.00 sec)

Let’s examine the table structure more closely. The actor table contains four columns, actor_id, first_name, last_name, and last_update. We can also extract the types of the columns — a smallint for actor_id, a varchar(45) for first_name and last_name, and a timestamp for last_update. None of the columns accept to be NULL (empty), actor_id is the primary key and last_name is the first column of a nonunique index. Don’t worry about the details; all that’s important right now are the column names we will use for the SQL commands.

Let’s explore the table city. We will execute the DESC statement:

mysql> DESC city;

+-------------+-------------------+------+-----+-------------------+---

--------------------------------------------+

| Field       | Type              | Null | Key | Default           |

Extra                                         |

+-------------+-------------------+------+-----+-------------------+---

--------------------------------------------+

| city_id     | smallint unsigned | NO   | PRI | NULL              |

auto_increment                                |

| city        | varchar(50)       | NO   |     | NULL              |

|

| country_id  | smallint unsigned | NO   | MUL | NULL              |

|

| last_update | timestamp         | NO   |     | CURRENT_TIMESTAMP |

DEFAULT_GENERATED on update CURRENT_TIMESTAMP |

+-------------+-------------------+------+-----+-------------------+---

--------------------------------------------+

4 rows in set (0.01 sec)

NOTE

The DEFAULT_GENERATED that you see in the Extra column indicates that this particular column uses a default value. This information is a MySQL 8 notation particularity, and it is not present in MariaDB 10.5:

mysql> *DESC city;*

+-------------+----------------------+------+-----+--------------------

-+-------------------------------+

| Field       | Type                 | Null | Key | Default

| Extra                         |

+-------------+----------------------+------+-----+--------------------

-+-------------------------------+

| city_id     | smallint(5) unsigned | NO   | PRI | NULL

| auto_increment                |

| city        | varchar(50)          | NO   |     | NULL

|                               |

| country_id  | smallint(5) unsigned | NO   | MUL | NULL

|                               |

| last_update | timestamp            | NO   |     | current_timestamp()

| on update current_timestamp() |

+-------------+----------------------+------+-----+--------------------

-+-------------------------------+

4 rows in set (0.00 sec)

Again, what’s important is getting familiar with the columns in each table, as we’ll make frequent use of these later when we’re learning about querying.

The next section shows you how to explore the data that MySQL stores in the sakila database and its tables.

The SELECT Statement and Basic Querying Techniques

By now you’ve learned how to install and configure MySQL and how to use the MySQL command line. Now that you understand the ER model, you’re ready to start exploring its data and learning the SQL language that all MySQL clients use. This section introduces the most commonly used SQL keyword: the SELECT keyword. We explain the fundamental elements of style and syntax and the features of the WHERE clause, Boolean operators, and sorting (much of this also applies to our later discussions of INSERT, UPDATE, and DELETE). This isn’t the end of our discussion of SELECT; you’ll find more in Chapter 5, where we show you how to use its advanced features.

Single Table SELECTs

The most basic form of SELECT reads the data in all rows and columns from a table. Connect to MySQL using the command line and choose the sakila database:

mysql> use sakila;

Database changed

Let’s retrieve all of the data in the language table:

mysql> SELECT * FROM language;

+-------------+----------+---------------------+

| language_id | name     | last_update         |

+-------------+----------+---------------------+

|           1 | English  | 2006-02-15 05:02:19 |

|           2 | Italian  | 2006-02-15 05:02:19 |

|           3 | Japanese | 2006-02-15 05:02:19 |

|           4 | Mandarin | 2006-02-15 05:02:19 |

|           5 | French   | 2006-02-15 05:02:19 |

|           6 | German   | 2006-02-15 05:02:19 |

+-------------+----------+---------------------+

6 rows in set (0.00 sec)

The output has six rows, and each row contains the values for all the columns present in the table. We now know that there are six languages, and we can see the languages, identifiers, and the last time we updated these languages.

A simple SELECT statement has four components:

1. The keyword SELECT.

2. The columns to be displayed. In our first example, we asked for all columns using the asterisk (*) symbol as a wildcard character.

3. The keyword FROM.

4. The table name; in this example, the table name is language.

Using everything we saw, we’ve asked for all columns from the language table, and that’s what MySQL has returned to us.

Let’s try another simple SELECT. This time, we’ll retrieve all columns from the city table:

mysql> SELECT * FROM city;

+---------+----------------------------+------------+------------------

---+

| city_id | city                       | country_id | last_update

|

+---------+----------------------------+------------+------------------

---+

|       1 | A Corua (La Corua)         |         87 | 2006-02-15

04:45:25 |

|       2 | Abha                       |         82 | 2006-02-15

04:45:25 |

|       3 | Abu Dhabi                  |        101 | 2006-02-15

04:45:25 |

...

|     599 | Zhoushan                   |         23 | 2006-02-15

04:45:25 |

|     600 | Ziguinchor                 |         83 | 2006-02-15

04:45:25 |

+---------+----------------------------+------------+------------------

---+

600 rows in set (0.00 sec)

There are 600 cities, and the output has the same basic structure as our first example.

The second example gives you an insight into how the relationships between the tables work. Consider the first row of the results. If you observe the column country_id, you will see the value 87. We will see in the next section, but based on this value, we can check on the country table that the country’s name with code 87 is Spain. We’ll discuss how to write queries on relationships between tables later in this chapter in “Joining Two Tables”.

Notice also that we have several different cities with the same country_id. Having multiple country_id values isn’t a problem since we expect a country with many cities (one-to-many relationship).

You should now feel comfortable choosing a database, listing its tables, and retrieving all of the data from a table using the SELECT statement. To practice, you might want to experiment with the other tables from sakila database. Remember that you can use the SHOW TABLES statement to find out the databases’ table names.

Choosing Columns

You’ve so far used the * wildcard character to retrieve all columns in a table. If you don’t want to display all the columns, it’s easy to be more specific by listing the columns you want, in the order you want them, separated by commas. For example, if you want only the city name from the city table, you’d type:

mysql> SELECT city FROM city;

+--------------------+

| city               |

+--------------------+

| A Corua (La Corua) |

| Abha               |

| Abu Dhabi          |

| Acua               |

| Adana              |

+--------------------+

5 rows in set (0.00 sec)

If you want both the city name and the city_id, in that order, you’d use:

mysql> SELECT city, city_id FROM city;

+--------------------+---------+

| city               | city_id |

+--------------------+---------+

| A Corua (La Corua) |       1 |

| Abha               |       2 |

| Abu Dhabi          |       3 |

| Acua               |       4 |

| Adana              |       5 |

+--------------------+---------+

5 rows in set (0.01 sec)

You can even list columns more than once:

mysql> SELECT city, city FROM city;

+--------------------+--------------------+

| city               | city               |

+--------------------+--------------------+

| A Corua (La Corua) | A Corua (La Corua) |

| Abha               | Abha               |

| Abu Dhabi          | Abu Dhabi          |

| Acua               | Acua               |

| Adana              | Adana              |

+--------------------+--------------------+

5 rows in set (0.00 sec)

Even though this appears pointless, it can be useful when combined with aliases in more advanced queries, as we show in Chapter 5.

You can specify databases, tables, and column names in a SELECT statement. This allows you to avoid the USE command and work with any database and table directly with SELECT; it also helps resolve ambiguities, as we show later in “Joining Two Tables”. Consider an example: suppose you want to retrieve the name column from the language table in the sakila database. You can do this with the following command:

mysql> SELECT name FROM sakila.language;

+----------+

| name     |

+----------+

| English  |

| Italian  |

| Japanese |

| Mandarin |

| French   |

| German   |

+----------+

6 rows in set (0.01 sec)

The sakila.language component after the FROM keyword specifies the sakila database and its language table. There’s no need to enter USE sakila; before running this query. This syntax can also be used with other SQL statements, including the UPDATE, DELETE, INSERT, and SHOW statements we discuss later in this chapter.

Selecting Rows with the WHERE Clause

This section introduces the WHERE clause and explains how to use operators to write expressions. You’ll see these in SELECT statements and other statements such as UPDATE and DELETE; we’ll show you examples later in this chapter.

WHERE basics

The WHERE clause is a powerful tool that allows you to filter which rows are returned from a SELECT statement. You use it to return rows that match a condition, such as having a column value that exactly matches a string, a number greater or less than a value, or a string that is a prefix of another. Almost all our examples in this and later chapters contain WHERE clauses, and you’ll become very familiar with them.

The simplest WHERE clause is one that exactly matches a value. Consider an example where we want to find out the English language’s details in the language table. Here’s what you type:

mysql> SELECT * FROM sakila.language WHERE name = 'English';

+-------------+---------+---------------------+

| language_id | name    | last_update         |

+-------------+---------+---------------------+

|           1 | English | 2006-02-15 05:02:19 |

+-------------+---------+---------------------+

1 row in set (0.00 sec)

MySQL returns all rows that match our search criteria — in this case, just the one row and all its columns.

Let’s try another exact-match example. Suppose you want to find out the actor’s first name with an actor_id value of 4. You type:

mysql> SELECT first_name FROM actor WHERE actor_id = 4;

+------------+

| first_name |

+------------+

| JENNIFER   |

+------------+

1 row in set (0.00 sec)

We’ve chosen a column and a row; we included the column first_name after the SELECT keyword and the WHERE actor_id = 4.

If a value matches more than one row, the results will contain all matches. Suppose we ask for all the cities belonging to Brazil, which has the country_id equal 15. You type in:

mysql> SELECT city FROM city WHERE country_id = 15;

+----------------------+

| city                 |

+----------------------+

| Alvorada             |

| Angra dos Reis       |

| Anpolis              |

| Aparecida de Goinia  |

| Araatuba             |

| Bag                  |

| Belm                 |

| Blumenau             |

| Boa Vista            |

| Braslia              |

| Goinia               |

| Guaruj               |

| guas Lindas de Gois  |

| Ibirit               |

| Juazeiro do Norte    |

| Juiz de Fora         |

| Luzinia              |

| Maring               |

| Po                   |

| Poos de Caldas       |

| Rio Claro            |

| Santa Brbara dOeste  |

| Santo Andr           |

| So Bernardo do Campo |

| So Leopoldo          |

| Sorocaba             |

| Vila Velha           |

| Vitria de Santo Anto |

+----------------------+

28 rows in set (0.00 sec)

The results show the names of the 28 cities that belong to Brazil. If we could join the information we get from the city table with information we get from the country table, we could display the cities’ names with their respective country. We’ll see how to perform this type of query in “Joining Two Tables”.

Now let’s retrieve values that belong to a range. Retrieve multiple values is simple for numeric ranges, so let’s start by finding all cities’ names with a city_id less than 5. To do this, execute the following statement:

mysql> SELECT city FROM city WHERE city_id < 5;

+--------------------+

| city               |

+--------------------+

| A Corua (La Corua) |

| Abha               |

| Abu Dhabi          |

| Acua               |

+--------------------+

4 rows in set (0.00 sec)

For numbers, the frequently used operators are equals (=), greater than (>), less than (<), less than or equal (<=), greater than or equal (>=), and not equal (<> or !=).

Consider one more example. If you want to find all languages that don’t have a language_id of 2, you’d type:

mysql> SELECT language_id, name FROM sakila.language

-> WHERE language_id <>2;

+-------------+----------+

| language_id | name     |

+-------------+----------+

|           1 | English  |

|           3 | Japanese |

|           4 | Mandarin |

|           5 | French   |

|           6 | German   |

+-------------+----------+

5 rows in set (0.00 sec)

The previous output shows us the first, third, and all subsequent languages. Note that you can use either <> or != operators for the not-equal condition.

You can use the same operators for strings. By default, string comparisons are not case-sensitive and use the current character set. For example:

mysql> SELECT first_name FROM actor WHERE first_name < 'B';

+------------+

| first_name |

+------------+

| ALEC       |

| AUDREY     |

| ANNE       |

| ANGELA     |

| ADAM       |

| ANGELINA   |

| ALBERT     |

| ADAM       |

| ANGELA     |

| ALBERT     |

| AL         |

| ALAN       |

| AUDREY     |

+------------+

13 rows in set (0.00 sec)

By case-sensitive we mean that B and b will be considered the same filter. So this query will provide the same result:

mysql> SELECT first_name FROM actor WHERE first_name < 'b';

+------------+

| first_name |

+------------+

| ALEC       |

| AUDREY     |

| ANNE       |

| ANGELA     |

| ADAM       |

| ANGELINA   |

| ALBERT     |

| ADAM       |

| ANGELA     |

| ALBERT     |

| AL         |

| ALAN       |

| AUDREY     |

+------------+

13 rows in set (0.00 sec)

Another prevalent task we’ll want to perform with strings is to find matches that begin with a prefix, contain a string, or end in a suffix. For example, we might want to find all album names beginning with the word “Retro.” We can do this with the LIKE operator in a WHERE clause. Let’s see an example where we are searching for a film with a title that contains the word family:

mysql> SELECT title FROM film WHERE title LIKE '%family%';

+----------------+

| title          |

+----------------+

| CYCLONE FAMILY |

| DOGMA FAMILY   |

| FAMILY SWEET   |

+----------------+

3 rows in set (0.00 sec)

Let’s discuss in detail how this works.

The LIKE clause is used with strings and means that a match must meet the pattern in the string that follows. In our example, we’ve used LIKE "%family%", which means the string contains family, and it can be preceded or followed by zero or more characters. Most strings used with LIKE contain the percentage character (%) as a wildcard character that matches all possible strings. You can use it to define a string that ends in a suffix — such as ` “%ing"` — or a string that starts with a particular substring, such as Corruption%.

For example, "John%" would match all strings starting with "John", such as John Smith and John Paul Getty. The pattern "%Paul" matches all strings that have "Paul" at the end. Finally, the pattern "%Paul%" matches all strings that have "Paul" in them, including at the start or at the end.

If you want to match exactly one wildcard character in a LIKE clause, you use the underscore character (_). For example, if you want all movie titles where the actor’s name begins with a three-letter word that starts with 0NAT1, you use:

mysql> SELECT title FROM film_list WHERE actors LIKE 'NAT_%';

+----------------------+

| title                |

+----------------------+

| FANTASY TROOPERS     |

| FOOL MOCKINGBIRD     |

| HOLES BRANNIGAN      |

| KWAI HOMEWARD        |

| LICENSE WEEKEND      |

| NETWORK PEAK         |

| NUTS TIES            |

| TWISTED PIRATES      |

| UNFORGIVEN ZOOLANDER |

+----------------------+

9 rows in set (0.04 sec)

NOTE

Avoid using the percentage (%) wildcard in the beginning of the pattern like in the following example:

mysql> SELECT title FROM film WHERE title LIKE '%family%';

You will get the results, but MySQL will not use the index under this condition. Using the percentage wildcard will force MySQL to read the entire table to retrieve the results and this can cause a severe performance impact if the table has millions of rows.

Combining conditions with AND, OR, NOT, and XOR

So far, we’ve used the WHERE clause to test one condition, returning all rows that meet it. You can combine two or more conditions using the Boolean operators AND, OR, NOT, and XOR.

Let’s start with an example. Suppose you want to find the titles of sci-fi movies that are rated PG. This is straightforward with the AND operator:

mysql> SELECT title FROM film_list WHERE category LIKE 'Sci-Fi'

-> AND rating LIKE 'PG';

+----------------------+

| title                |

+----------------------+

| CHAINSAW UPTOWN      |

| CHARADE DUFFEL       |

| FRISCO FORREST       |

| GOODFELLAS SALUTE    |

| GRAFFITI LOVE        |

| MOURNING PURPLE      |

| OPEN AFRICAN         |

| SILVERADO GOLDFINGER |

| TITANS JERK          |

| TROJAN TOMORROW      |

| UNFORGIVEN ZOOLANDER |

| WONDERLAND CHRISTMAS |

+----------------------+

12 rows in set (0.07 sec)

The AND operation in the WHERE clause restricts the results to those rows that meet both conditions.

The OR operator is used to find rows that meet at least one of several conditions. To illustrate, imagine now that you want a list of children or family movies. You can do this with two OR and three LIKE clauses:

mysql> SELECT title FROM film_list WHERE category LIKE 'Children'

-> OR category LIKE 'Family';

+------------------------+

| title                  |

+------------------------+

| AFRICAN EGG            |

| APACHE DIVINE          |

| ATLANTIS CAUSE         |

...

| WRONG BEHAVIOR         |

| ZOOLANDER FICTION      |

+------------------------+

129 rows in set (0.04 sec)

The OR operations in the WHERE clause restrict the answers to those that meet any of the two conditions. As an aside, we can observe that the results are ordered. This is merely a coincidence; in this case, they’re reported in the order they were added to the database. We’ll return to sorting output later in “ORDER BY Clauses”.

You can combine AND and OR, but you need to make it clear whether you want to first AND the conditions or OR them.

Parentheses cluster parts of a statement together and help make expressions readable; you can use them just as you would in basic math. Let’s say that now you want sci-fi or family movies that are rated PG. We can write our query as follows:

mysql> SELECT title FROM film_list WHERE (category like 'Sci-Fi'

-> OR category LIKE Family) AND rating LIKE 'PG';

+------------------------+

| title                  |

+------------------------+

| BEDAZZLED MARRIED      |

| CHAINSAW UPTOWN        |

| CHARADE DUFFEL         |

| CHASING FIGHT          |

| EFFECT GLADIATOR       |

...

| UNFORGIVEN ZOOLANDER   |

| WONDERLAND CHRISTMAS   |

+------------------------+

30 rows in set (0.07 sec)

The parentheses make the evaluation order clear: we want movies from Sci-Fi or Family category, but all of them need to be PG-rated.

With the use of parentheses, it is possible to change the evaluation order. The easiest way to check is by playing around with calculations:

mysql> SELECT (2+2)*3;

+---------+

| (2+2)*3 |

+---------+

|      12 |

+---------+

1 row in set (0.00 sec)

mysql> SELECT 2+2*3;

+-------+

| 2+2*3 |

+-------+

|     8 |

+-------+

1 row in set (0.00 sec)

TIP

One of the most difficult problems to find is a query that is running with no syntax errors, and it is returning values different from expected. While the AND operator does not influence, the OR operator has a significant influence. For example, this statement will give us the following result:

mysql> SELECT * FROM sakila.city WHERE city_id = 3

-> OR city_id=4 AND country_id = 60;

+---------+-----------+------------+---------------------+

| city_id | city      | country_id | last_update         |

+---------+-----------+------------+---------------------+

|       3 | Abu Dhabi |        101 | 2006-02-15 04:45:25 |

|       4 | Acua      |         60 | 2006-02-15 04:45:25 |

+---------+-----------+------------+---------------------+

2 rows in set (0.00 sec)

If we change the ordering of the operators, we will obtain a different result:

mysql> SELECT * FROM sakila.city WHERE country_id = 60

-> AND city_id = 3 OR city_id=4;

+---------+------+------------+---------------------+

| city_id | city | country_id | last_update         |

+---------+------+------------+---------------------+

|       4 | Acua |         60 | 2006-02-15 04:45:25 |

+---------+------+------------+---------------------+

1 row in set (0.00 sec)

Using parentheses makes the queries much easier to understand. We recommend that you use parentheses whenever there’s a chance MySQL could misinterpret the intention; there’s no good reason to rely on MySQL’s implicit evaluation order.

The unary NOT operator negates a Boolean statement. Suppose you want a list of all languages except the one having a language_id of 2. You’d write the query:

mysql> SELECT language_id, name FROM sakila.language

-> WHERE NOT (language_id =2);

+-------------+----------+

| language_id | name     |

+-------------+----------+

|           1 | English  |

|           3 | Japanese |

|           4 | Mandarin |

|           5 | French   |

|           6 | German   |

+-------------+----------+

5 rows in set (0.01 sec)

The expression in the parentheses says we want:

(language_id = 2)

And the NOT operation negates it, so we get everything but those that meet the parentheses’ condition. There are several other ways you can write a WHERE clause with the same idea. We will see later in Chapter 5 that some have a better performance than others.

Consider another example using NOT and parentheses. Suppose you want to get a list of all movie titles with an FID lesser than 7, but not those numbered 4 or 6:

mysql> SELECT fid,title from film_list where FID < 7 and not (FID=4 OR FID=6);

+------+------------------+

| fid  | title            |

+------+------------------+

|    1 | ACADEMY DINOSAUR |

|    2 | ACE GOLDFINGER   |

|    3 | ADAPTATION HOLES |

|    5 | AFRICAN EGG      |

+------+------------------+

4 rows in set (0.06 sec)

Again, parentheses’ expression lists movies that meet a condition — those that do not have fid equal to 4 or 6 and all movies with the fid lesser than 7.

The operator’s precedence can be a little tricky, and sometimes it takes a lot of time for the DBA to debug a query and identify why the query is not returning the requested values. We show the operator precedences in the following list, from highest priority to the lowest. Operators that are shown together on a line have the same priority.

INTERVAL

BINARY, COLLATE

!

- (unary minus), ~ (unary bit inversion)

^

*, /, DIV, %, MOD

-,+

<<, >>

&

|

= (comparison), <=>, >=, >, <=, <, <>, !=, IS, LIKE, REGEXP, IN, MEMBER OF

BETWEEN, CASE, WHEN, THEN, ELSE

NOT

AND, &&

XOR

OR, ||

= (assignment), :=

It is possible to combine these operators in the most diverse ways to get the desired results. For example, suppose we want the movie titles that have a price range between 2 USD and 4 USD and belong to the Documentary or Horror category, and one of the actors has the name Bob:

mysql> SELECT title

-> FROM film_list

-> WHERE price BETWEEN 2 AND 4

-> AND (category LIKE 'Documentary' OR category LIKE 'Horror')

-> AND actors LIKE '%BOB%';

+------------------+

| title            |

+------------------+

| ADAPTATION HOLES |

+------------------+

1 row in set (0.08 sec)

Finally, before we move to sorting, it is possible to execute queries that do not attend the requisites, and in this case they will return empty:

mysql> SELECT title FROM film_list

-> WHERE price BETWEEN 2 AND 4

-> AND (category LIKE 'Documentary' OR category LIKE Horror)

-> AND actors LIKE '%GRIPPA%';

Empty set (0.04 sec)

ORDER BY Clauses

We’ve so far discussed how to choose the columns, and which rows are returned as part of the query result, but not how to control how the result is displayed. In a relational database, the rows in a table form a set; there is no intrinsic order between the rows, and so we have to ask MySQL to sort the results if we want them in a particular order. This section explains how to use the ORDER BY clause to do this. Sorting does not affect what is returned; it only affects what order the results are returned.

NOTE

MySQL, in particular, InnoDB table, has a special index called clustered index that stores the data. When you define a PRIMARY KEY on a table, InnoDB uses it as the clustered index. Suppose you are executing queries based on the PRIMARY KEY. In that case, the rows will return ordered in ascending order by the PRIMARY KEY. However, we always recommending using the ORDER BY if you want to enforce a particular order.

Suppose you want to return a list of the first ten customers in the sakila database, sorted alphabetically by name. Here’s what you’d type:

mysql> SELECT name FROM customer_list

-> ORDER BY name

-> LIMIT 10;

+-------------------+

| name              |

+-------------------+

| AARON SELBY       |

| ADAM GOOCH        |

| ADRIAN CLARY      |

| AGNES BISHOP      |

| ALAN KAHN         |

| ALBERT CROUSE     |

| ALBERTO HENNING   |

| ALEX GRESHAM      |

| ALEXANDER FENNELL |

| ALFRED CASILLAS   |

+-------------------+

10 rows in set (0.01 sec)

The ORDER BY clause indicates that sorting is required, followed by the column that should be used as the sort key. In this example, we’re sorting by alphabetically-ascending name. The default sort is case-insensitive and in ascending order, and MySQL automatically sorts alphabetically because the columns are character strings. The way strings are sorted is determined by the character set and collation order that are being used. We discuss these in “Collation and Character Sets”. For most of this book, we assume that you’re using the default settings.

Let’s see a second example. This time, let’s sort the output from the address table by ascending the last_update column:

mysql> SELECT address, last_update FROM address

-> *ORDER BY last_update LIMIT 5;

+-----------------------------+---------------------+

| address                     | last_update         |

+-----------------------------+---------------------+

| 1168 Najafabad Parkway      | 2014-09-25 22:29:59 |

| 1031 Daugavpils Parkway     | 2014-09-25 22:29:59 |

| 1924 Shimonoseki Drive      | 2014-09-25 22:29:59 |

| 757 Rustenburg Avenue       | 2014-09-25 22:30:01 |

| 1892 Nabereznyje Telny Lane | 2014-09-25 22:30:02 |

+-----------------------------+---------------------+

5 rows in set (0.00 sec)

As we can see, it is possible to sort different types of columns. Moreover, we can compound the sorting with two or more columns. For example, let’s say you want to sort alphabetically the addresses but for each district.

mysql> SELECT address, district FROM address

-> ORDER BY district, address;

+----------------------------------------+----------------------+

| address                                | district             |

+----------------------------------------+----------------------+

| 1368 Maracabo Boulevard                |                      |

| 18 Duisburg Boulevard                  |                      |

| 962 Tama Loop                          |                      |

| 535 Ahmadnagar Manor                   | Abu Dhabi            |

| 669 Firozabad Loop                     | Abu Dhabi            |

| 1078 Stara Zagora Drive                | Aceh                 |

| 663 Baha Blanca Parkway                | Adana                |

| 842 Salzburg Lane                      | Adana                |

| 614 Pak Kret Street                    | Addis Abeba          |

| 751 Lima Loop                          | Aden                 |

| 1157 Nyeri Loop                        | Adygea               |

| 387 Mwene-Ditu Drive                   | Ahal                 |

| 775 ostka Drive                        | al-Daqahliya         |

...

| 1416 San Juan Bautista Tuxtepec Avenue | Zufar                |

| 138 Caracas Boulevard                  | Zulia                |

+----------------------------------------+----------------------+

603 rows in set (0.00 sec)

You can also sort in descending order, and you can control this behavior for each sort key. Suppose you want to sort the address by descending alphabetical order and the districts in descending order. You type this:

mysql> SELECT address,district FROM address

-> ORDER BY district ASC, address DESC

-> LIMIT 10;

+-------------------------+-------------+

| address                 | district    |

+-------------------------+-------------+

| 962 Tama Loop           |             |

| 18 Duisburg Boulevard   |             |

| 1368 Maracabo Boulevard |             |

| 669 Firozabad Loop      | Abu Dhabi   |

| 535 Ahmadnagar Manor    | Abu Dhabi   |

| 1078 Stara Zagora Drive | Aceh        |

| 842 Salzburg Lane       | Adana       |

| 663 Baha Blanca Parkway | Adana       |

| 614 Pak Kret Street     | Addis Abeba |

| 751 Lima Loop           | Aden        |

+-------------------------+-------------+

10 rows in set (0.01 sec)

If a collision of values occurs and you don’t specify another sort key, the sort order is undefined. This may not be important for you; you may not care about the order in which two customers with the identical name “John A. Smith” appear.

The LIMIT Clause

As you may have noted, a few queries previously mentioned use the LIMIT clause. The LIMIT clause is a useful, nonstandard SQL statement that allows you to control which rows are output. Its basic form allows you to limit the number of rows returned from a SELECT statement, which is useful when you want to limit the amount of data communicated over a network or output to the screen. You might use it, for example, to get a sample of the data from the table as we have been doing. Here’s an example:

mysql> SELECT name FROM customer_list LIMIT 10;

+------------------+

| name             |

+------------------+

| VERA MCCOY       |

| MARIO CHEATHAM   |

| JUDY GRAY        |

| JUNE CARROLL     |

| ANTHONY SCHWAB   |

| CLAUDE HERZOG    |

| MARTIN BALES     |

| BOBBY BOUDREAU   |

| WILLIE MARKHAM   |

| JORDAN ARCHULETA |

+------------------+

The LIMIT clause can have two arguments. With two arguments, the first argument specifies the first row’s offset to return, and the second specifies the maximum number of rows to return. The first argument is known as offset. Suppose you want five rows, but you want to remove the first five rows, which means the result will start at the sixth row. You do this by starting from after the fifth answer:

mysql> SELECT name FROM customer_list LIMIT 5, 5;

+------------------+

| name             |

+------------------+

| CLAUDE HERZOG    |

| MARTIN BALES     |

| BOBBY BOUDREAU   |

| WILLIE MARKHAM   |

| JORDAN ARCHULETA |

+------------------+

5 rows in set (0.00 sec)

The output is rows 6 to 10 from the SELECT query.

There’s an alternative syntax that you might see for the LIMIT keyword: instead of writing LIMIT 10,5, you can write LIMIT 10 OFFSET 5. The OFFSET syntax discards the N values specified in it.

This is an example with no offset:

mysql> SELECT id, name FROM customer_list

-> ORDER BY id LIMIT 10;

+----+------------------+

| ID | name             |

+----+------------------+

|  1 | MARY SMITH       |

|  2 | PATRICIA JOHNSON |

|  3 | LINDA WILLIAMS   |

|  4 | BARBARA JONES    |

|  5 | ELIZABETH BROWN  |

|  6 | JENNIFER DAVIS   |

|  7 | MARIA MILLER     |

|  8 | SUSAN WILSON     |

|  9 | MARGARET MOORE   |

| 10 | DOROTHY TAYLOR   |

+----+------------------+

10 rows in set (0.00 sec)

Now using an offset of 5, we will discard the first five ids:

mysql> SELECT id, name FROM customer_list

-> ORDER BY id LIMIT 10 OFFSET 5;

+----+----------------+

| ID | name           |

+----+----------------+

|  6 | JENNIFER DAVIS |

|  7 | MARIA MILLER   |

|  8 | SUSAN WILSON   |

|  9 | MARGARET MOORE |

| 10 | DOROTHY TAYLOR |

| 11 | LISA ANDERSON  |

| 12 | NANCY THOMAS   |

| 13 | KAREN JACKSON  |

| 14 | BETTY WHITE    |

| 15 | HELEN HARRIS   |

+----+----------------+

10 rows in set (0.01 sec)

Joining Two Tables

We’ve so far worked with just one table in our SELECT queries. However, in the ER model, we saw that a relational database is all about working with the relationships between tables to answer information needs. Indeed, as we’ve explored the tables in the sakila database, it’s become obvious that by using these relationships, we can answer more interesting queries. For example, it’d be useful to know the countries of each city. This section shows you how to answer these queries by joining two tables. We’ll return to this issue as part of a longer, more advanced discussion of joins in Chapter 5.

We use only one join syntax in this chapter. There are several more, and each gives you a different way to bring together data from two or more tables. The syntax we use here is the INNER JOIN, which is the most used in daily activities. Consider an example, and then we’ll explain more about how it works:

mysql> SELECT city, country FROM city INNER JOIN country

-> ON city.country_id = country.country_id

-> WHERE country.country_id < 5

-> ORDER BY country, city;

+----------+----------------+

| city     | country        |

+----------+----------------+

| Kabul    | Afghanistan    |

| Batna    | Algeria        |

| Bchar    | Algeria        |

| Skikda   | Algeria        |

| Tafuna   | American Samoa |

| Benguela | Angola         |

| Namibe   | Angola         |

+----------+----------------+

7 rows in set (0.00 sec)

The output shows the cities and their country. You can see for the first time which city belongs to which country.

How does the INNER JOIN work? The statement has two parts: first, two table names separated by the INNER JOIN keywords; second, the ON keyword that indicates which column (or columns) holds the relationship between the two tables. In our first example, the two tables to be joined are city and country, expressed as city INNER JOIN country (for the basic INNER JOIN, it doesn’t matter what order you list the tables in, and so using country INNER JOIN city would have the same effect). In the example, the ON clause is where we tell MySQL the columns that hold the relationship between the tables; you should recall this from our design and our previous discussion in Chapter 2.

If the join condition uses the equal operator (=) and the column names in both tables used for matching are the same, you can use the USING clause instead:

mysql> SELECT city, country FROM city

-> INNER JOIN country using (country_id)

-> WHERE country.country_id < 5

-> ORDER BY country, city;

+----------+----------------+

| city     | country        |

+----------+----------------+

| Kabul    | Afghanistan    |

| Batna    | Algeria        |

| Bchar    | Algeria        |

| Skikda   | Algeria        |

| Tafuna   | American Samoa |

| Benguela | Angola         |

| Namibe   | Angola         |

+----------+----------------+

7 rows in set (0.01 sec)

The following Venn diagram in Figure 3-1 illustrates the inner join:

Figure 3-1. The Venn diagram representation of the INNER JOIN

As we saw in the previous example, all operators are supported when using INNER JOIN. For example, we used the WHERE condition and the LIMIT clause.

Before we leave SELECT, we’ll give you a taste of one of the functions you can use to aggregate values. Suppose you want to count how many cities Italy has in our database. You can do this by counting the number of rows using the COUNT() function. Here’s how it works:

mysql> SELECT count(1) FROM city INNER JOIN country

-> ON city.country_id = country.country_id

-> WHERE country.country_id = 49

-> ORDER BY country, city;

+----------+

| count(1) |

+----------+

|        7 |

+----------+

1 row in set (0.00 sec)

We explain more features of SELECT and aggregate functions in Chapter 5.

The INSERT Statement

The INSERT statement is used to add new data to tables. This section explains its basic syntax and shows you simple examples that add new rows to the sakila database. In Chapter 4, we’ll discuss how to load data from existing tables or external data sources.

INSERT Basics

Inserting data typically occurs in two situations: when you bulk-load in a large batch as you create your database, and when you add data on an ad hoc basis as you use the database. In MySQL, different optimizations are built into the server for each situation. Importantly, different SQL syntaxes are available to make it easy for you to work with the server in both cases. We explain a basic INSERT syntax in this section and show you examples of using it for bulk and single record insertion.

Let’s start with the basic task of inserting one new row into the language table. To do this, you need to understand the table’s structure. As we explained in Chapter 2, you can discover this with the SHOW COLUMNS statement:

mysql> SHOW COLUMNS FROM language;

+-------------+------------------+------+-----+-------------------+-----------------------------------------------+

| Field       | Type             | Null | Key | Default           | Extra                                         |

+-------------+------------------+------+-----+-------------------+-----------------------------------------------+

| language_id | tinyint unsigned | NO   | PRI | NULL              | auto_increment                                |

| name        | char(20)         | NO   |     | NULL              |                                               |

| last_update | timestamp        | NO   |     | CURRENT_TIMESTAMP | DEFAULT_GENERATED on update CURRENT_TIMESTAMP |

+-------------+------------------+------+-----+-------------------+-----------------------------------------------+

3 rows in set (0.00 sec)

This tells you that the language_id column is auto-generated, and the last_update column is updated every time an UPDATE operation happens. You’ll learn more about the AUTO_INCREMENT shortcut to automatically assign the next available identifier in Chapter 4.

Our new row is for a new language, “Portuguese”. How can we insert the row? There are two ways to do it. The most common is to let MySQL fill the default values for the language_id. It will be like this:

mysql> INSERT INTO language VALUES (NULL, 'Portuguese', NOW());

Query OK, 1 row affected (0.10 sec)

And if we execute a SELECT on the table:

mysql> SELECT * FROM language;

+-------------+------------+---------------------+

| language_id | name       | last_update         |

+-------------+------------+---------------------+

|           1 | English    | 2006-02-15 05:02:19 |

|           2 | Italian    | 2006-02-15 05:02:19 |

|           3 | Japanese   | 2006-02-15 05:02:19 |

|           4 | Mandarin   | 2006-02-15 05:02:19 |

|           5 | French     | 2006-02-15 05:02:19 |

|           6 | German     | 2006-02-15 05:02:19 |

|           7 | Portuguese | 2020-09-26 09:11:36 |

+-------------+------------+---------------------+

7 rows in set (0.00 sec)

We can see MySQL inserted the row. Note that we used the function NOW() in the last_update column. The NOW() function returns the current date and time of the MySQL server.

The second option is inserting the value of the language_id manually. Now that we already have seven languages, we should use 8 for the next value of the language_id. We can check with this SQL instruction:

mysql> SELECT MAX(language_id) FROM language;

+------------------+

| max(language_id) |

+------------------+

|                7 |

+------------------+

1 row in set (0.00 sec)

The MAX() function tells you the maximum value for the column supplied as a parameter. This is cleaner than SELECT artist_id FROM artist, which prints out all rows and requires you to inspect the rows to find the maximum value; adding an ORDER BY makes it easier. Using MAX() is also much more straightforward than SELECT language_id FROM language ORDER BY language_id DESC LIMIT 1, which again returns the correct answer.

We’re now ready to insert the row. In this INSERT, we are going to insert the last_update manually too. Here’s the needed command:

mysql> INSERT INTO language VALUES (8, 'Russian', '2020-09-26 10:35:00');

Query OK, 1 row affected (0.02 sec)

A new row is created — MySQL reports that one row has been affected — and the value. Let’s check again:

mysql> SELECT * FROM language;

+-------------+------------+---------------------+

| language_id | name       | last_update         |

+-------------+------------+---------------------+

|           1 | English    | 2006-02-15 05:02:19 |

|           2 | Italian    | 2006-02-15 05:02:19 |

|           3 | Japanese   | 2006-02-15 05:02:19 |

|           4 | Mandarin   | 2006-02-15 05:02:19 |

|           5 | French     | 2006-02-15 05:02:19 |

|           6 | German     | 2006-02-15 05:02:19 |

|           7 | Portuguese | 2020-09-26 09:11:36 |

|           8 | Russian    | 2020-09-26 10:35:00 |

+-------------+------------+---------------------+

8 rows in set (0.00 sec)

It is also possible to insert multiple values at once:

mysql> INSERT INTO language VALUES (NULL, 'Spanish', NOW()),

-> (NULL, 'Hebrew', NOW());

Query OK, 2 rows affected (0.02 sec)

Records: 2  Duplicates: 0  Warnings: 0

The single row INSERT style detects primary key duplicates: if it finds a duplicate, it’ll stop as soon as it finds a duplicate key. For example, suppose we try to insert the same language again:

mysql> INSERT INTO language VALUES (8, 'Arabic', '2020-09-26 10:35:00');

ERROR 1062 (23000): Duplicate entry '8' for key 'language.PRIMARY'

The INSERT operation stops on the first duplicate key. You can add an IGNORE clause to prevent the error if you want, but note that the row is not going to be inserted:

mysql> INSERT IGNORE INTO language VALUES (8, 'Arabic', '2020-09-26 10:35:00');

Query OK, 0 rows affected, 1 warning (0.00 sec)

However, in most cases, you want to know about possible problems (after all, primary keys are supposed to be unique), so this IGNORE syntax is rarely used.

You’ll notice that MySQL reports the results of bulk insertion differently from single insertion. From our initial bulk insertion, it reports:

mysql> INSERT INTO language VALUES (null, 'Spanish', NOW()),

-> *(NULL, 'Hebrew', NOW());

Query OK, 2 rows affected (0.02 sec)

Records: 2  Duplicates: 0  Warnings: 0

The first line tells you how many rows were inserted, while the first entry in the final line tells you how many rows (or records) were actually processed. If you use INSERT IGNORE and try to insert a duplicate record — for which the primary key matches that of an existing row — then MySQL will quietly skip inserting it and report it as a duplicate in the second entry on the final line:

Query OK, 0 rows affected, 2 warnings (0.00 sec)

Records: 2  Duplicates: 2  Warnings: 2

We discuss causes of warnings — shown as the third entry on the final line — in Chapter 4.

Alternative Syntaxes

There are several alternatives to the VALUES syntax we’ve shown you so far. This section shows you these and explains the advantages and drawbacks of each. If you’re happy with the basic syntax we’ve described so far, and want to move on to a new topic, feel free to skip ahead to “The DELETE Statement”.

There are three disadvantages of the VALUES syntax we’ve shown you: You need to remember the order of the columns. You need to provide a value for each column. It’s closely tied to the underlying table structure: if you change the table’s structure, you need to change the INSERT statements, and the function of the INSERT statement isn’t obvious unless you have the table structure at hand. However, the three advantages of the approach are that it works for both single and bulk inserts, you get an error message if you forget to supply values for all columns, and you don’t have to type in column names. Fortunately, we can avoid the disadvantages by varying the syntax.

Suppose you know that the actor table has four columns, and you recall their names, but you forget their order. You can insert using the following approach:

mysql> INSERT INTO actor (actor_id, first_name, last_name, last_update)

-> VALUES (NULL, 'Vinicius', 'Grippa', NOW());

Query OK, 1 row affected (0.03 sec)

The column names are included in parentheses after the table name, and the values stored in those columns are listed in parentheses after the VALUES keyword. So, in this example, a new row is created and the value 201 is stored as the actor_id (remember actor_id has the auto_increment property), Vinicius is stored as the first_name, Grippa is stored as the last_name, and the last_update is updated with the current timestamp. This syntax’s advantages are that it’s readable and flexible (addressing the third disadvantage we described) and order-independent (addressing the first disadvantage). The burden is that you need to know the column names and type them in.

This new syntax can also address the second disadvantage of the simpler approach — that is, it can allow you to insert values for only some columns. To understand how this might be useful, let’s explore the city table:

mysql> DESC city;

+-------------+----------------------+------+-----+-------------------+-----------------------------+

| Field       | Type                 | Null | Key | Default           | Extra                       |

+-------------+----------------------+------+-----+-------------------+-----------------------------+

| city_id     | smallint(5) unsigned | NO   | PRI | NULL              | auto_increment              |

| city        | varchar(50)          | NO   |     | NULL              |                             |

| country_id  | smallint(5) unsigned | NO   | MUL | NULL              |                             |

| last_update | timestamp            | NO   |     | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |

+-------------+----------------------+------+-----+-------------------+-----------------------------+

4 rows in set (0.00 sec)

Notice that the last_update column has a default value of CURRENT_TIMESTAMP. This means that if you don’t insert a value for the last_update column, it’ll insert the current date and time by default. This is just what we want: when we record a store, we don’t want to bother checking the date and time and typing it in. Here’s how you insert an incomplete played entry:

mysql> INSERT INTO city (city, country_id) VALUES ('Bebedouro', 19);

Query OK, 1 row affected (0.00 sec)

We didn’t set the city_id column, so MySQL defaults it to the next available value (auto_increment property), and last_update stores the current date and time. You can check this with a query:

mysql> SELECT * FROM city where city like 'Bebedouro';

+---------+-----------+------------+---------------------+

| city_id | city      | country_id | last_update         |

+---------+-----------+------------+---------------------+

|     601 | Bebedouro |         19 | 2021-02-27 21:34:08 |

+---------+-----------+------------+---------------------+

1 row in set (0.01 sec)

You can also use this approach for bulk insertion as follows:

mysql> INSERT INTO city (city,country_id) VALUES

-> ('Sao Carlos',19),

-> ('Araraquara',19),

-> ('Ribeirao Preto',19);

Query OK, 3 rows affected (0.00 sec)

Records: 3  Duplicates: 0  Warnings: 0

This approach’s disadvantages are that you can accidentally omit values for columns, and you need to remember and type column names. MySQL will set the omitted columns to the default values.

All columns in a MySQL table have a default value of NULL unless another default value is explicitly assigned when the table is created or modified.

You can set defaults and still use the original INSERT syntax with MySQL 5.7 or later by using the DEFAULT keyword. Here’s an example that adds a country row using DEFAULT:

mysql> INSERT INTO country VALUES (NULL, 'Uruguay', DEFAULT);

Query OK, 1 row affected (0.01 sec)

The keyword DEFAULT tells MySQL to use the default value for that column, and so the current date and time are inserted in our example. This approach’s advantages are that you can use the bulk-insert feature with default values, and you can never accidentally omit a column.

There’s another alternative INSERT syntax. In this approach, you list the column name and value together, giving the advantage that you don’t have to mentally map the list of values to the earlier list of columns. Here’s an example that adds a new row to the country table:

mysql> INSERT INTO country SET country_id=NULL,

-> country='Bahamas', last_update=NOW();

Query OK, 1 row affected (0.01 sec)

The syntax requires you to list a table name, the keyword SET, and then column-equals-value pairs, separated by commas. Columns that aren’t supplied are set to their default values. Again, the disadvantages are that you can accidentally omit values for columns and that you need to remember and type in column names. A significant additional disadvantage is that you can’t use this method for bulk insertion.

You can also insert using values returned from a query. We discuss this in Chapter 7.

The DELETE Statement

The DELETE statement is used to remove one or more rows from a table. We explain single-table deletes here, and discuss multi-table deletes — which remove data from two or more tables through one statement — in Chapter 7.

DELETE Basics

The simplest use of DELETE is to remove all rows in a table. Suppose you want to empty your rental table. You do this with:

mysql> DELETE FROM rental;

Query OK, 16044 rows affected (2.41 sec)

The DELETE syntax doesn’t include column names since it’s used to remove whole rows and not just values from a row. To reset or modify a value in a row, you use the UPDATE statement, described later in this chapter in “The UPDATE Statement”. The DELETE statement doesn’t remove the table itself. For example, having deleted all rows in the rental table, you can still query the table:

mysql> SELECT * FROM rental;

Empty set (0.00 sec)

Of course, you can also continue to explore its structure using DESCRIBE or SHOW CREATE TABLE, and insert new rows using INSERT. To remove a table, you use the DROP statement described in Chapter 4.

Note that if our table has a relationship with another table, the delete will fail because of the foreign key constraint:

mysql> DELETE FROM language;

ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`sakila`.`film`, CONSTRAINT `fk_film_language` FOREIGN KEY (`language_id`) REFERENCES `language` (`language_id`) ON UPDATE CASCADE)

Using WHERE, ORDER BY, and LIMIT

If you’ve deleted rows in the previous section, reload your sakila database now. You need the rows in the rental table restored for the examples in this section.

To remove one or more rows, but not all rows in a table, use a WHERE clause. This works in the same way as it does for SELECT. For example, suppose you want to remove all rows from the rental table with rentail_id less than 10. You do this with:

mysql> DELETE FROM rental WHERE rental_id < 10;

Query OK, 9 rows affected (0.01 sec)

The result is that the nine rental rows that match the criteria are removed.

Suppose we want to remove all the payments from a customer called Mary Smith. First, let’s perform a select with the customer and payment tables using INNER JOIN (“Joining Two Tables”):

mysql> SELECT first_name, last_name, customer.customer_id,

-> amount, payment_date FROM payment INNER JOIN customer

-> ON customer.customer_id=payment.customer_id

-> WHERE first_name like 'Mary'

-> AND last_name like 'Smith';

+------------+-----------+-------------+--------+---------------------+

| first_name | last_name | customer_id | amount | payment_date        |

+------------+-----------+-------------+--------+---------------------+

| MARY       | SMITH     |           1 |   2.99 | 2005-05-25 11:30:37 |

| MARY       | SMITH     |           1 |   0.99 | 2005-05-28 10:35:23 |

| MARY       | SMITH     |           1 |   5.99 | 2005-06-15 00:54:12 |

| MARY       | SMITH     |           1 |   0.99 | 2005-06-15 18:02:53 |

...

| MARY       | SMITH     |           1 |   1.99 | 2005-08-22 01:27:57 |

| MARY       | SMITH     |           1 |   2.99 | 2005-08-22 19:41:37 |

| MARY       | SMITH     |           1 |   5.99 | 2005-08-22 20:03:46 |

+------------+-----------+-------------+--------+---------------------+

32 rows in set (0.00 sec)

Next, to remove the rows from Mary Smith which has the customer_id equal one, the following delete is performed:

mysql> DELETE FROM artist WHERE artist_id = 3;

Query OK, 1 row affected (0.00 sec)

Then do the same thing for the album, track, and played tables:

mysql> DELETE FROM payment where customer_id=1;

Query OK, 32 rows affected (0.01 sec)

You can use the ORDER BY and LIMIT clauses with DELETE. You usually do this when you want to limit the number of rows deleted.

mysql> DELETE FROM payment ORDER BY customer_id LIMIT 10000;

Query OK, 10000 rows affected (0.22 sec)

TIP

We highly recommend using delete and update operations for a small set of rows due to performance issues. This value usually varies depending on the hardware, but a good value is around 20000-40000 rows per batch.

Removing All Rows with TRUNCATE

If you want to remove all rows in a table, there’s a faster method than removing them with DELETE. Using the TRUNCATE TABLE statement, MySQL takes the shortcut of dropping the table, removing the table structures, and then re-creating them. When there are many rows in a table, this is much faster.

NOTE

As a curiosity, there is a bug in MySQL 5.6 version that can stall MySQL when performing TRUNCATE operation when MySQL is configured with a large InnoDB buffer pool (200Gb or more). The bug is reported here:

Concurrent TRUNCATE operations cause stalls

If you want to remove the data in the payment table, you can execute this:

mysql> TRUNCATE TABLE payment;

Query OK, 0 rows affected (0.07 sec)

Notice that the number of rows affected is shown as zero: to quickly delete all the data in the table, MySQL doesn’t count the number of rows that are deleted, so the number shown does not reflect the actual number of rows deleted.

The TRUNCATE TABLE statement differs from DELETE in a lot of ways, but it is worth mentioning a few:

- Truncate operations drop and re-create the table, which is much faster than deleting rows one by one, particularly for large tables.

- Truncate operations cause an implicit commit, so we cannot rollback.

- We cannot perform truncation operations if the session holds an active table lock.

Table types, transactions, and locking are discussed in Chapter 5. None of these limitations affect most applications in practice, and you can use TRUNCATE TABLE to speed up your processing. Of course, it’s not common to delete whole tables during regular operation. An exception is temporary tables used to store query results for a particular user session temporarily and can be deleted without losing the original data.

The UPDATE Statement

The UPDATE statement is used to change data. In this section, we show you how to update one or more rows in a single table. Multitable updates are discussed in “Updates”.

If you’ve deleted rows from your sakila database, reload it by following the instructions in “Entity Relationship Modeling Examples”.

Examples

The simplest use of the UPDATE statement is to change all rows in a table. Imagine the situation where we need to update the amount column of the payment table by 10% for all payments. To change all rows, you need to execute:

mysql> UPDATE payment SET amount=amount*1.1;

Query OK, 16025 rows affected, 16025 warnings (0.41 sec)

Rows matched: 16049  Changed: 16025  Warnings: 16025

Note that we forgot to update the last_update status. To make it coherent with the database model we expect, you can fix running the following statement:

mysql> UPDATE payment SET last_update='2021-02-28 17:53:00';

Query OK, 16049 rows affected (0.27 sec)

Rows matched: 16049  Changed: 16049  Warnings: 0

TIP

You can use the NOW() function to update the last_update column with the current timestamp of the execution. For example:

mysql> UPDATE payment SET last_update=NOW();

The second row reported by an UPDATE statement shows the overall effect of the statement. In our example, you see:

Rows matched: 16049  Changed: 16049  Warnings: 0

The first column reports the number of rows that were retrieved as answers by the statement; in this case, since there’s no WHERE or LIMIT clause, all rows in the table match the query. The second column reports how many rows needed to be changed, and this is always equal to or less than the number of rows that match. If you repeat the statement, you’ll see a different result:

mysql> UPDATE payment SET last_update='2021-02-28 17:53:00';

Query OK, 0 rows affected (0.07 sec)

Rows matched: 16049  Changed: 0  Warnings: 0

This time, since the date is already set to 2021-02-28 17:53:00 and there is no WHERE condition, all the rows still match the statement but none are changed. Note also the number of rows changed is always equal to the number of rows affected, as reported on the first line of the output.

Using WHERE, ORDER BY, and LIMIT

Often, you don’t want to change all rows in a table. Instead, you want to update one or more rows that match a condition. As with SELECT and DELETE, the WHERE clause is used for the task. In addition, in the same way as with DELETE, you can use ORDER BY and LIMIT together to control how many rows are updated from an ordered list.

Let’s try an example that modifies one row in a table. Suppose that in the actor table, the actress Penelope Guiness changed her last name. To update it in the database, you need to execute:

mysql> UPDATE actor SET last_name= UPPER('cruz')

-> WHERE first_name LIKE 'PENELOPE'

-> AND last_name like 'GUINESS';

Query OK, 1 row affected (0.01 sec)

Rows matched: 1  Changed: 1  Warnings: 0

As expected, MySQL matched one row, and MySQL changed one row.

To control how many updates occur, you can use the combination of ORDER BY and LIMIT. As with DELETE, you would do this because you either want to perform the operation in small chunks or modify only some rows.

mysql> UPDATE payment SET last_update=NOW() LIMIT 10;

Query OK, 10 rows affected (0.01 sec)

Rows matched: 10  Changed: 10  Warnings: 0

You can see that 10 rows were matched and were changed.

The previous query also illustrates an important aspect of updates. As you’ve seen, updates have two phases: a matching phase — where rows are found that match the WHERE clause — and a modification phase, where the rows that need changing are updated.

Exploring Databases and Tables with SHOW and mysqlshow

We’ve already explained how you can use the SHOW command to obtain information on the structure of a database, its tables, and the table columns. In this section, we’ll review the most common types of SHOW statement with brief examples using the sakila database. The mysqlshow command-line program performs the same function as several SHOW command variants, but without needing to start the MySQL client.

The SHOW DATABASES statement lists the databases you can access. If you’ve followed our sample database installation steps in “Entity Relationship Modeling Examples” and deployed the bank model in “Creating a Bank Database ER Model”, your output should be as follows:

mysql> SHOW DATABASES;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| bank_model         |

| employees          |

| mysql              |

| performance_schema |

| sakila             |

| sys                |

| world              |

+--------------------+

8 rows in set (0.01 sec)

These are the databases that you can access with the USE command; as we explain in “Using the Sakila Database”, you can’t see databases for which you have no access privileges unless you have the global SHOW DATABASES privilege. You can get the same effect from the command line using the mysqlshow program:

# mysqlshow -uroot -pmsandbox -h 127.0.0.1 -P 3306

You can add a LIKE clause to SHOW DATABASES. This statement is useful only if you have many databases and want a shortlist as output. For example, to see databases beginning with s, run:

mysql> SHOW DATABASES LIKE 's%';

+---------------+

| Database (s%) |

+---------------+

| sakila        |

| sys           |

+---------------+

2 rows in set (0.00 sec)

The LIKE statement’s syntax is identical to that in its use in SELECT.

To see the statement used to create a database, you can use the SHOW CREATE DATABASE statement. For example, to see how you created sakila, type:

mysql> SHOW CREATE DATABASE sakila;

+----------+-----------------------------------------------------------

-----------------------------------------------------------------------+

| Database | Create Database                                                                                                                  |

+----------+-----------------------------------------------------------

-----------------------------------------------------------------------+

| sakila   | CREATE DATABASE `sakila` /*!40100 DEFAULT CHARACTER SET

utf8mb4 COLLATE utf8mb4_0900_ai_ci */ /*!80016 DEFAULT ENCRYPTION='N'

*/ |

+----------+-----------------------------------------------------------

-----------------------------------------------------------------------+

1 row in set (0.00 sec)

This is perhaps the least exciting SHOW statement; it only displays the statement:

CREATE DATABASE `sakila` /*!40100 DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci */ /*!80016 DEFAULT ENCRYPTION='N' */

Some additional keywords are enclosed between the comment symbols 0! and 1:

40100 DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci

80016 DEFAULT ENCRYPTION='N'

These instructions contain MySQL-specific extensions to standard SQL that are unlikely to be understood by other database programs. A database server other than MySQL would ignore this comment text, and so the syntax is usable by both MySQL and other database server software. The optional number 40100 indicates the minimum version of MySQL that can process this particular instruction — in this case, version 4.01.00; older versions of MySQL ignore such instructions. You’ll learn about creating databases in Chapter 4.

The SHOW TABLES statement lists the tables in a database. To check the tables in sakila, type:

mysql> SHOW TABLES FROM sakila;

+----------------------------+

| Tables_in_sakila           |

+----------------------------+

| actor                      |

| actor_info                 |

| address                    |

| category                   |

| city                       |

| country                    |

| customer                   |

| customer_list              |

| film                       |

| film_actor                 |

| film_category              |

| film_list                  |

| film_text                  |

| inventory                  |

| language                   |

| nicer_but_slower_film_list |

| payment                    |

| rental                     |

| sales_by_film_category     |

| sales_by_store             |

| staff                      |

| staff_list                 |

| store                      |

+----------------------------+

23 rows in set (0.01 sec)

If you’ve already selected the sakila database with the USE sakila command, you can use the shortcut:

mysql> SHOW TABLES;

You can get a similar result by specifying the database name to the mysqlshow program:

$ mysqlshow -uroot -pmsandbox -h 127.0.0.1 -P 3306 sakila

As with SHOW DATABASES, you can’t see tables that you don’t have privileges for it. This means you can’t see tables in a database you can’t access, even if you have the SHOW DATABASES global privilege.

The SHOW COLUMNS statement lists the columns in a table. For example, to check the columns of country, type:

mysql> SHOW COLUMNS FROM country;

+-------------+-------------------+------+-----+-------------------+-----------------------------------------------+

| Field       | Type              | Null | Key | Default           | Extra                                         |

+-------------+-------------------+------+-----+-------------------+-----------------------------------------------+

| country_id  | smallint unsigned | NO   | PRI | NULL              | auto_increment                                |

| country     | varchar(50)       | NO   |     | NULL              |                                               |

| last_update | timestamp         | NO   |     | CURRENT_TIMESTAMP | DEFAULT_GENERATED on update CURRENT_TIMESTAMP |

+-------------+-------------------+------+-----+-------------------+-----------------------------------------------+

3 rows in set (0.00 sec)

The output reports all column names, their types, and sizes, whether they can be NULL, whether they are part of a key, their default value, and any extra information. Types, keys, NULL values, and defaults are discussed further in Chapter 4. If you haven’t already chosen the sakila database with the USE command, then you can add the database name before the table name, as in sakila.country. Unlike the previous SHOW statements, you can always see all column names if you have access to a table; it doesn’t matter that you don’t have certain privileges for all columns. You can get a similar result by using mysqlshow with the database and table name:

$ mysqlshow -uroot -pmsandbox -h 127.0.0.1 -P 3306 sakila country

And you will receive:

Database: sakila  Table: country

+-------------+-------------------+--------------------+------+-----+-------------------+-----------------------------------------------+---------------------------------+---------+

| Field       | Type              | Collation          | Null | Key | Default           | Extra                                         | Privileges                      | Comment |

+-------------+-------------------+--------------------+------+-----+-------------------+-----------------------------------------------+---------------------------------+---------+

| country_id  | smallint unsigned |                    | NO   | PRI |                   | auto_increment                                | select,insert,update,references |         |

| country     | varchar(50)       | utf8mb4_0900_ai_ci | NO   |     |                   |                                               | select,insert,update,references |         |

| last_update | timestamp         |                    | NO   |     | CURRENT_TIMESTAMP | DEFAULT_GENERATED on update CURRENT_TIMESTAMP | select,insert,update,references |         |

+-------------+-------------------+--------------------+------+-----+-------------------+-----------------------------------------------+---------------------------------+---------+

You can see the statement used to create a particular table using the SHOW CREATE TABLE statement; creating tables is a subject of Chapter 4. Some users prefer this output to that of SHOW COLUMNS, since it has the familiar format of a CREATE TABLE statement. Here’s an example for the country table:

mysql> SHOW CREATE TABLE country\G

*************************** 1. row ***************************

Table: country

Create Table: CREATE TABLE `country` (

`country_id` smallint unsigned NOT NULL AUTO_INCREMENT,

`country` varchar(50) NOT NULL,

`last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

PRIMARY KEY (`country_id`)

) ENGINE=InnoDB AUTO_INCREMENT=110 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci

1 row in set (0.00 sec)

Chapter 4. Working with Database Structures

This chapter shows you how to create your own databases, add and remove structures such as tables and indexes, and make choices about column types in your tables. It focuses on the syntax and features of SQL, and not the semantics of conceiving, specifying, and refining a database design; you’ll find an introductory description of database design techniques in Chapter 2. To work through this chapter, you need to understand how to work with an existing database and its tables, as discussed in Chapter 3.

This chapter lists the structures in the sample sakila database; detail on how to load the database is presented in “Entity Relationship Modeling Examples”. If you’ve followed those instructions, you’ll already have the database available and know how to restore the database after you’ve modified its structures.

When you finish this chapter, you’ll have all the basics required to create, modify, and delete database structures. Together with the techniques you learned in Chapter 3, you’ll have the skills to carry out a wide range of basic operations. Chapters Chapter 5 and Chapter 7 cover skills that allow you to do more advanced operations with MySQL.

Creating and Using Databases

When you’ve finished designing a database, the first practical step to take with MySQL is to create it. You do this with the CREATE DATABASE statement. Suppose you want to create a database with the name lucy. Here’s the statement you’d type:

mysql> CREATE DATABASE lucy;

Query OK, 1 row affected (0.10 sec)

We assume here that you know how to connect using the MySQL client, as described in Chapter 1. We also assume that you’re able to connect as the root user or as another user who can create, delete, and modify structures (you’ll find a detailed discussion on user privileges in Chapter 8). Note that when you create the database, MySQL says that one row was affected. This isn’t in fact a normal row in any specific database—but a new entry added to the list that you see with SHOW DATABASES command.

Once you’ve created the database, the next step is to use it—that is, choose it as the database you’re working with. You do this with the MySQL command:

mysql> USE lucy;

Database changed

As discussed previously in Chapter 3, this command must be entered on one line and need not be terminated with a semicolon, though we usually do so automatically through habit. Once you’ve used the database, you can start creating tables, indexes, and other structures using the steps discussed in “Creating Tables”.

Before we move on to creating other structures, let’s discuss a few features and limitations of creating databases. First, let’s see what happens if you create a database that already exists:

mysql> CREATE DATABASE lucy;

ERROR 1007 (HY000): Can't create database 'lucy'; database exists

You can avoid this error by adding the IF NOT EXISTS keyword phrase to the statement:

mysql> CREATE DATABASE IF NOT EXISTS lucy;

Query OK, 0 rows affected (0.00 sec)

You can see that MySQL didn’t complain, but it didn’t do anything either: the 0 rows affected message indicates that no data was changed. This addition is useful when you’re adding SQL statements to a script: it prevents the script from aborting on error.

Let’s look at how to choose database names and use character case. Database names define physical directory (or folder) names on disk. On some operating systems, directory names are case-sensitive; on others, case doesn’t matter. For example, Unix-like systems such as Linux and macOS are typically case-sensitive, whereas Windows isn’t. The result is that database names have the same restrictions: when case matters to the operating system, it matters to MySQL. For example, on a Linux machine, LUCY, lucy, and Lucy are different database names; on Windows, they refer to just one database. Using incorrect capitalization under Linux or macOS will cause MySQL to complain:

mysql> select SaKilA.AcTor_id from ACTor;

ERROR 1146 (42S02): Table 'sakila.ACTor' doesn't exist

But under Windows, this will normally work. To make your SQL machine-independent, we recommend that you consistently use lowercase names for databases (and for tables, columns, aliases, and indexes). This behavior is controlled by the lower_case_table_names parameter. If set to 0, table names are stored as specified and comparisons are case-sensitive. If set to 1, table names are stored in lowercase on disk and comparisons are not case-sensitive. If set to 2, table names are stored as given but compared in lowercase. On Windows the default value is 1. On macOS, the default value is 2. On Linux, a value of 2 is not supported; the server forces the value to 0 instead.

There are other restrictions on database names. They can be at most 64 characters in length. You also shouldn’t use MySQL reserved words—such as SELECT, FROM, and USE—as names for structures; these can confuse the MySQL parser, making it impossible to interpret the meaning of your statements. There’s a way around this problem: you can enclose the reserved word with the backtick symbol (`) on either side, but it’s more trouble remembering to do so than it’s worth. In addition, you can’t use selected characters in the names: specifically, you can’t use the forward slash, backward slash, semicolon, and period characters, and a database name can’t end in whitespace. Again, the use of these characters confuses the MySQL parser and can result in unpredictable behavior. For example, here’s what happens when you insert a semicolon into a database name:

mysql> CREATE DATABASE IF NOT EXISTS lu;cy;

Query OK, 1 row affected (0.00 sec)

ERROR 1064 (42000): You have an error in your SQL syntax; check the manual

that corresponds to your MySQL server version for the right syntax to use

near 'cy' at line 1

Since more than one SQL statement can be on a single line, the result is that a database lu is created, and then an error is generated by the very short, unexpected SQL statement cy;.

Creating Tables

This section covers topics on table structures. We show you how to:

- Create tables, through introductory examples

- Choose names for tables and table-related structures

- Understand and choose column types

- Understand and choose keys and indexes

- Use the proprietary MySQL AUTO_INCREMENT feature

When you finish this section, you’ll have completed all of the basic material on creating database structures; the remainder of this chapter covers the sample sakila database, and how to alter and remove existing structures.

Basics

For our examples in this section, we’ll assume that the database sakila hasn’t been created. If you want to follow the examples, and you have already loaded the database, you can drop it for this section and reload it later; dropping it removes the database, tables, and all of the data, but the original is easy to restore by following the steps in “Entity Relationship Modeling Examples”. Here’s how you drop it temporarily:

mysql> DROP DATABASE sakila;

Query OK, 23 rows affected (0.06 sec

The DROP statement is discussed further at the end of this chapter in “Deleting Structures”.

To begin, create the database sakila using the statement:

mysql> CREATE DATABASE sakila;

Query OK, 1 row affected (0.00 sec)

Then select the database with:

mysql> USE sakila;

Database changed

We’re now ready to begin creating the tables that will hold our data. Let’s create a table to hold actor details. For now, we’re going to have a simplified structure, and talk about more complexity later. Here’s the statement we use:

mysql> CREATE TABLE actor (

-> actor_id SMALLINT UNSIGNED NOT NULL DEFAULT 0,

-> first_name VARCHAR(45) DEFAULT NULL,

-> last_name VARCHAR(45),

-> last_update TIMESTAMP,

-> PRIMARY KEY (actor_id)

-> );

Query OK, 0 rows affected (0.01 sec)

Don’t panic: even though MySQL reports that zero rows were affected, it created the table:

mysql> SHOW tables;

+------------------+

| Tables_in_sakila |

+------------------+

| actor            |

+------------------+

1 row in set (0.01 sec)

Let’s consider all this in detail. The CREATE TABLE statement has three major sections:

1. The CREATE TABLE statement, which is followed by the table name to create. In this example, it’s actor.

2. A list of one or more columns to add to the table. In this example, we’ve added quite a few: actor_id SMALLINT UNSIGNED NOT NULL DEFAULT 0, first_name VARCHAR(45) DEFAULT NULL, last_name VARCHAR(45), and last_update TIMESTAMP. We’ll discuss these in a moment.

3. Optional key definitions. In this example, we’ve defined a single key: PRIMARY KEY (actor_id). We’ll discuss keys and indexes in detail later in this section.

Notice that the CREATE TABLE component is followed by an opening parenthesis that’s matched by a closing parenthesis at the end of the statement. Notice also that the other components are separated by commas. There are other elements that you can add to a CREATE TABLE statement, and we’ll discuss some in a moment.

Let’s discuss the column specifications. The basic syntax is as follows: name type [NOT NULL | NULL] [DEFAULT value]. The name field is the column name, and it has the same limitations as database names, as discussed in the previous section. It can be at most 64 characters in length, backward and forward slashes aren’t allowed, periods aren’t allowed, it can’t end in whitespace, and case sensitivity is dependent on the underlying operating system. The type defines how and what is stored in the column; for example, we’ve seen that it can be set to VARCHAR for strings, SMALLINT for numbers, or TIMESTAMP for a date and time.

If you specify NOT NULL, a row isn’t valid without a value for the column; if you specify NULL or omit the clause, a row can exist without a value for the column. If you specify a value with the DEFAULT clause, it’ll be used to populate the column when you don’t otherwise provide data; this is particularly useful when you frequently reuse a default value such as a country name. The value must be a constant (such as 0, "cat", or 20060812045623), except if the column is of the type TIMESTAMP. Types are discussed in detail later in this section.

The NOT NULL and DEFAULT features can be used together. If you specify NOT NULL and add a DEFAULT value, the default is used when you don’t provide a value for the column. Sometimes, this works fine:

mysql> INSERT INTO actor(first_name) VALUES ('John');

Query OK, 1 row affected (0.01 sec)

And sometimes it doesn’t:

mysql> INSERT INTO actor(first_name) VALUES ('Elisabeth');

ERROR 1062 (23000): Duplicate entry '0' for key 'actor.PRIMARY'

Whether it works or not is dependent on the underlying constraints and conditions of the database: in this example, actor_id has a default value of 0, but it’s also the primary key. Having two rows with the same primary-key value isn’t permitted, and so the second attempt to insert a row with no values (and a resulting primary-key value of 0) fails. We discuss primary keys in detail later in this section.

Column names have fewer restrictions than database and table names. What’s more, they’re not dependent on the operating system: the names are case-insensitive and portable across all platforms. All characters are allowed in column names, though if you want terminate them with whitespace or include periods (or other special characters such as the semicolon or the dash sign), you’ll need to enclose the name with a backtick symbol (`) on either side. We recommend that you consistently choose lowercase names for developer-driven choices (such as database, alias, and table names) and avoid characters that require you to remember to use backticks. Naming the columns, as well as other database objects is something of a personal preference when starting anew, or a matter of following standards when working on an existing codebase. We recommend avoiding repeating yourself. Column name actor_first_name is going to look redundant when table name precedes it (e.g. in a complex join query): actor.actor_first_name or actor.first_name. Usually, one exception is done to that: the ubiquitous id column name should either not be used or have the table name prepended for clarity. You may use sakila, world, or employee example databases to get an inspiration. Another good practice is to use the underscore character to separate words; you could use underscores or dashes (but remember to escape dashes and other special symbols with a backtick), or omit the word-separating formatting altogether. However, “CamelCase” is harder to read. As with database and table names, the longest column name is 64 characters in length.

Collation and Character Sets

Because not everyone wants to store English strings, it’s important that a database server be able to manage non-English characters and different ways of sorting characters. When you’re comparing or sorting strings, how MySQL evaluates the result depends on the character set and collation used. Character sets define what characters can be stored; for example, you may need to store non-English characters such as ю or ü. A collation defines how strings are ordered, and there are different collations for different languages: for example, the position of the character ü in the alphabet is different in two German orderings, and different again in Swedish and Finnish.

We understand that discussion of collations and charsets may feel to be too advanced when you’re just starting out learning MySQL. We also think, however, that it’s worth mentioning. Mismatched charsets and collations may result in unexpected situations including loss of data and incorrect query results. You may, however, skip this section, as well as some later discussion, and come back when you want to learn about this specifically. That won’t affect other material in this book.

In our previous string-comparison examples, we ignored the collation and character-set issue, and just let MySQL use its defaults; in versions of MySQL prior to 8.0, the default character set is latin1, and the default collation is latin1_swedish_ci. MySQL 8.0 changed the defaults, and now the default character set is utf8mb4, and the default collation is utf8mb4_0900_ai_ci. MySQL can be configured to use different character sets and collation orders at the connection, database, table, and column levels. Outputs below come from MySQL 8.0.

You can list the character sets available on your server with the SHOW CHARACTER SET command. This shows a short description for each character set, its default collation, and the maximum number of bytes used for each character in that character set:

mysql> SHOW CHARACTER SET;

+----------+---------------------------------+---------------------+--------+

| Charset  | Description                     | Default collation   | Maxlen |

+----------+---------------------------------+---------------------+--------+

| armscii8 | ARMSCII-8 Armenian              | armscii8_general_ci |      1 |

| ascii    | US ASCII                        | ascii_general_ci    |      1 |

| big5     | Big5 Traditional Chinese        | big5_chinese_ci     |      2 |

| binary   | Binary pseudo charset           | binary              |      1 |

| cp1250   | Windows Central European        | cp1250_general_ci   |      1 |

| cp1251   | Windows Cyrillic                | cp1251_general_ci   |      1 |

...

| ujis     | EUC-JP Japanese                 | ujis_japanese_ci    |      3 |

| utf16    | UTF-16 Unicode                  | utf16_general_ci    |      4 |

| utf16le  | UTF-16LE Unicode                | utf16le_general_ci  |      4 |

| utf32    | UTF-32 Unicode                  | utf32_general_ci    |      4 |

| utf8     | UTF-8 Unicode                   | utf8_general_ci     |      3 |

| utf8mb4  | UTF-8 Unicode                   | utf8mb4_0900_ai_ci  |      4 |

+----------+---------------------------------+---------------------+--------+

41 rows in set (0.00 sec)

For example, the latin1 character set is actually the Windows code page 1252 that supports West European languages. The default collation for this character set is latin1_swedish_ci, which follows Swedish conventions to sort accented characters (English is handled as you’d expect). This collation is case-insensitive, as indicated by the letters ci. Finally, each character takes up one byte. By comparison, if you use the default utf8mb4 character set, each character would take up to four bytes of storage. Sometimes, it makes sense to change that default. For example, there’s no reason to store base64-encoded data (which, by definition, is ASCII) in utf8mb4. With a 128 character wide column, at million rows you’re looking at approximately 350MiB of overhead on charset alone in the worst case.

Similarly, you can list the collation orders and the character sets they apply to:

mysql> SHOW COLLATION;

+---------------------+----------+-----+---------+----------+---------+---------------+

| Collation           | Charset  | Id  | Default | Compiled | Sortlen | Pad_attribute |

+---------------------+----------+-----+---------+----------+---------+---------------+

| armscii8_bin        | armscii8 |  64 |         | Yes      |       1 | PAD SPACE     |

| armscii8_general_ci | armscii8 |  32 | Yes     | Yes      |       1 | PAD SPACE     |

| ascii_bin           | ascii    |  65 |         | Yes      |       1 | PAD SPACE     |

| ascii_general_ci    | ascii    |  11 | Yes     | Yes      |       1 | PAD SPACE     |

...

| utf8mb4_0900_ai_ci  | utf8mb4  | 255 | Yes     | Yes      |       0 | NO PAD        |

| utf8mb4_0900_as_ci  | utf8mb4  | 305 |         | Yes      |       0 | NO PAD        |

| utf8mb4_0900_as_cs  | utf8mb4  | 278 |         | Yes      |       0 | NO PAD        |

| utf8mb4_0900_bin    | utf8mb4  | 309 |         | Yes      |       1 | NO PAD        |

...

| utf8_unicode_ci     | utf8     | 192 |         | Yes      |       8 | PAD SPACE     |

| utf8_vietnamese_ci  | utf8     | 215 |         | Yes      |       8 | PAD SPACE     |

+---------------------+----------+-----+---------+----------+---------+---------------+

272 rows in set (0.02 sec)

NOTE

Number of character sets and collations available depends on how the MySQL server was built and packaged. Examples we show come from a default MySQL 8.0 installation, and same numbers can be seen on Linux and Windows. MariaDB 10.5, however, has 322 collations but 40 character sets.

You can see the current defaults on your server as follows:

mysql> SHOW VARIABLES LIKE 'c%';

+--------------------------+--------------------------------+

| Variable_name            | Value                          |

+--------------------------+--------------------------------+

...

| character_set_client     | utf8mb4                        |

| character_set_connection | utf8mb4                        |

| character_set_database   | utf8mb4                        |

| character_set_filesystem | binary                         |

| character_set_results    | utf8mb4                        |

| character_set_server     | utf8mb4                        |

| character_set_system     | utf8                           |

| character_sets_dir       | /usr/share/mysql-8.0/charsets/ |

...

| collation_connection     | utf8mb4_0900_ai_ci             |

| collation_database       | utf8mb4_0900_ai_ci             |

| collation_server         | utf8mb4_0900_ai_ci             |

...

+--------------------------+--------------------------------+

21 rows in set (0.00 sec)

When you’re creating a database, you can set the default character set and sort order for the database and its tables. For example, if you want to use the utf8mb4 character set and the utf8mb4_ru_0900_as_cs (case-sensitive) collation order, you would write:

mysql> CREATE DATABASE rose DEFAULT CHARACTER SET utf8mb4

-> COLLATE utf8mb4_ru_0900_as_cs;

Query OK, 1 row affected (0.00 sec)

Usually, there’s no need to do this if you’ve installed your MySQL correctly for your language and region, and if you’re not planning on internationalizing your application. With utf8mb4 being the default since MySQL 8.0, there’s even less need to change the charset. You can also control the character set and collation for individual tables or columns, but we won’t go into the detail of how to do that here. We will show how collations affect string types in “String types”.

Other Features

This section briefly describes other features of the MySQL CREATE TABLE statement. It includes an example using the IF NOT EXISTS feature, and a list of advanced features and where to find more about them in this book. The statement shown is the full representation of the table taken from sakila database, unlike a previous simplified example.

You can use the IF NOT EXISTS keyword phrase when creating a table, and it works much as it does for databases. Here’s an example that won’t report an error even when the actor table exists:

mysql> CREATE TABLE IF NOT EXISTS actor (

-> actor_id SMALLINT UNSIGNED NOT NULL AUTO_INCREMENT,

-> first_name VARCHAR(45) NOT NULL,

-> last_name VARCHAR(45) NOT NULL,

-> last_update TIMESTAMP NOT NULL DEFAULT

-> CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

-> PRIMARY KEY  (actor_id),

-> KEY idx_actor_last_name (last_name));

Query OK, 0 rows affected, 1 warning (0.01 sec)

You can see that 0 rows are affected, and a warning is reported. Let’s take a look:

mysql> SHOW WARNINGS;

+-------+------+------------------------------+

| Level | Code | Message                      |

+-------+------+------------------------------+

| Note  | 1050 | Table 'actor' already exists |

+-------+------+------------------------------+

1 row in set (0.01 sec)

There are a wide range of additional features you can add to a CREATE TABLE statement, only few of which are present in this more full statement. Many of these are advanced and aren’t discussed in this book, but you can find more information in the MySQL manual under the heading “CREATE TABLE Statement.” These additional features include the following:

The AUTO_INCREMENT feature for numeric columns

This feature allows you to automatically create unique identifiers for a table. We discuss it in detail later in this chapter in “The AUTO_INCREMENT Feature”.

Column comments

You can add a comment to a column; this is displayed when you use the SHOW CREATE TABLE command that we discuss later in this section.

Foreign key constraints

You can tell MySQL to check whether data in one or more columns matches data in another table. For example, sakila database has a foreign key constraint on a city_id column of the address table, referring to the city table’s city_id column. That means, it’s impossible to have an address in a city not present in the city table. We introduced foreign key constraints in Chapter 2, and we’ll take a look at what engines support foreign key constraints in “Alternative Storage Engines”. Not every storage engine in MySQL supports foreign keys.

Creating temporary tables

If you create a table using the keyword phrase CREATE TEMPORARY TABLE, it’ll be removed (dropped) when the connection is closed. This is useful for copying and reformatting data because you don’t have to remember to clean up. Sometimes, they are also used as an optimization to hold some intermediate data.

Advanced table options

You can control a wide range of features of the table using table options. These include the starting value of AUTO_INCREMENT, the way indexes and rows are stored, and options to override the information that the MySQL query optimizer gathers from the table. It’s also possible to specify generated columns, containing data like sum of two other columns, as well as indexes on such columns.

Control over index structures

Some storage engines in MySQL allow you to specify and control what type of internal structure—such as a B-tree or hash table—MySQL uses for its indexes. You can also tell MySQL that you want a full text or spatial data index on a column, allowing special types of search.

Partitioning

MySQL supports different partitioning strategies, which you can select at the table creation time, as well as at a later time. We will not be covering partitioning in this book.

You can check the CREATE TABLE statement for a table using the SHOW CREATE TABLE statement introduced in Chapter 3. This often shows you output that includes some of the advanced features we’ve just discussed; the output rarely matches what you actually typed to create the table. Here’s an example for the actor table:

mysql> SHOW CREATE TABLE actor\G

*************************** 1. row ***************************

Table: actor

Create Table: CREATE TABLE `actor` (

`actor_id` smallint unsigned NOT NULL AUTO_INCREMENT,

`first_name` varchar(45) NOT NULL,

`last_name` varchar(45) NOT NULL,

`last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP

ON UPDATE CURRENT_TIMESTAMP,

PRIMARY KEY (`actor_id`),

KEY `idx_actor_last_name` (`last_name`)

) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4

COLLATE=utf8mb4_0900_ai_ci

1 row in set (0.00 sec)

You’ll notice that the output includes content added by MySQL that wasn’t in our original CREATE TABLE statement:

- The names of the table and columns are enclosed in backticks. This isn’t necessary, but it does avoid any parsing problems that can occur through using reserved words and special characters, as discussed previously

- An additional default ENGINE clause is included, which explicitly states the table type that should be used. The setting in a default installation of MySQL is InnoDB, so it has no effect in this example

- An additional DEFAULT CHARSET=utf8mb4 clause is included, which tells MySQL what character set is used by the columns in the table. Again, this has no effect in a default installation

Column Types

This section describes the column types you can use in MySQL. It explains when each should be used and any limitations it has. The types are grouped by their purpose. We’ll cover widely-used datatypes, and mention more advanced or less used types in passing. That doesn’t mean they have no use, but consider learning about them as an exercise. Most likely, you will not remember each of the data types and its particular intricacies, and that’s okay. It’s worth re-reading this chapter later, and consulting with MySQL documentation on the topic to keep your knowledge up to date.

Integer types

We will start with numeric data types, and more specifically with integer types, or the types holding specific whole number. First, the two most popular integer types.

INT[(width)] [UNSIGNED] [ZEROFILL]

The most commonly used numeric type. Stores integer (whole number) values in the range –2,147,483,648 to 2,147,483,647. If the optional UNSIGNED keyword is added, the range is 0 to 4,294,967,295. The keyword INT is short for INTEGER, and they can be used interchangeably. An INT column requires four bytes of storage space.

INT, as well as other integer types, has two properties specific to MySQL: optional width and ZEROFILL arguments. They are not part of an SQL standard, and as of MySQL 8.0 are deprecated. Still, you will surely notice them in a lot of codebases, so we will briefly cover both of them.

The width parameter specifies the display width, which can be read by applications as part of the column metadata. Contrary to parameters in a similar position for other data types, this parameter has no effect on the storage characteristics of a particular integer type, and does not constrain the usable range of values. INT(4) and INT(32) are same for the purpose of data storage.

ZEROFILL is an additional argument, which is used to left-pad the values with zeros up to the length, specified by the width. If you use ZEROFILL, MySQL automatically adds UNSIGNED to the declaration (since zero filling makes sense only in the context of positive numbers).

In a few applications where ZEROFILL and width are useful, LPAD() function can be used, or numbers can be stored formatted in CHAR column.

BIGINT[(width)] [UNSIGNED] [ZEROFILL]

In the world of growing data sizes, having tables with count of rows in the billions is getting common. Even simple id-type columns might need a wider range than a regular INT provides. BIGINT solves that problem. It is a large integer type with a signed range of -9223372036854775808 to 9223372036854775807. Unsigned BIGINT can store numbers from 0 to 18446744073709551615. Column of this type will require eight bytes of storage.

Internally, all calculations within MySQL are done using signed BIGINT or DOUBLE values. The important consequence of that is that you should be extremely careful when dealing with extremely large numbers. First, unsigned big integers larger than 9223372036854775807 should only be used with bit functions. Second, if a result of a arithmetical operation is larger than 9223372036854775807, unexpected results might be observed.

For example:

mysql> CREATE TABLE test_bigint (id BIGINT UNSIGNED);

Query OK, 0 rows affected (0.01 sec)

mysql> INSERT INTO test_bigint VALUES (18446744073709551615);

Query OK, 1 row affected (0.01 sec)

mysql> INSERT INTO test_bigint VALUES (18446744073709551615-1);

Query OK, 1 row affected (0.01 sec)

mysql> INSERT INTO test_bigint VALUES (184467440737095516*100);

ERROR 1690 (22003): BIGINT value

is out of range in '(184467440737095516 * 100)'

Even though 18446744073709551600 is less than 18446744073709551615, since signed BIGINT is used for multiplication internally, the out of range error is observed.

Data type SERIAL can be used as an alias for BIGINT UNSIGNED NOT NULL AUTO_INCREMENT UNIQUE. Unless you must optimize for data size and performance, consider using SERIAL for your id-like columns. Even UNSIGNED INT runs out of range much quicker than you’d expect, and in the worst possible time.

We will only touch other integer types briefly. However, keep in mind that although it’s possible to store every integer as BIGINT, that’s wasteful in terms of storage space. Moreover, since as we discussed the width parameter doesn’t constrain the range of values, different integer types can be used for that purpose.

SMALLINT[(width)] [UNSIGNED] [ZEROFILL]

As expected from its name, this type holds a small integer, with range from -32768 to 32767 signed, and from 0 to 65535 unsigned. It takes two bytes of storage.

TINYINT[(width)] [UNSIGNED] [ZEROFILL]

Even smaller integer, and the smallest numeric data type. Range of this type is -128 to 127 signed and 0 to 255 unsigned. It only takes one byte of storage.

BOOL[(width)]

Short for BOOLEAN, and a synonym for TINYINT(1). Usually, boolean types only accept two values: true or false. However, since BOOL in MySQL is an integer type, you can store values from -128 to 127 there. 0 will be treated as false, and all nonzero values as true. It’s also possible to use special true and false aliases for 1 and 0 respectively.

mysql> CREATE TABLE test_bool (i BOOL);

Query OK, 0 rows affected (0.04 sec)

mysql> INSERT INTO test_bool VALUES (true),(false);

Query OK, 2 rows affected (0.00 sec)

Records: 2  Duplicates: 0  Warnings: 0

mysql> INSERT INTO test_bool VALUES (1),(0),(-128),(127);

Query OK, 4 rows affected (0.02 sec)

Records: 4  Duplicates: 0  Warnings: 0

mysql> SELECT i, IF(i,'true','false') FROM test_bool;

+------+----------------------+

| i    | IF(i,'true','false') |

+------+----------------------+

|    1 | true                 |

|    0 | false                |

|    1 | true                 |

|    0 | false                |

| -128 | true                 |

|  127 | true                 |

+------+----------------------+

6 rows in set (0.01 sec)

MEDIUMINT[(width)] [UNSIGNED] [ZEROFILL]

Another type of integer. Takes 3 bytes of storage space. Stores values in the signed range of -8388608 to 8388607, and unsigned range of 0 to 16777215.

BIT[(M)]

This is a special type used to store bit values. M specifies number of bits per value and defaults to 1 if omitted. MySQL uses a b'_value_' syntax for the binary values.

Fixed-point types

Both DECIMAL and NUMERIC data types in MySQL are the same. Thus we will only describe DECIMAL here, and everything will apply to NUMERIC. The main difference between fixed-point types (or type, in MySQL case) and floating-point types is precision. For fixed-point types, the value retrieved is identical to the value stored; this isn’t always the case with other types that contain decimal points, such as the FLOAT and DOUBLE types described later. That is the most important property of the DECIMAL data type.

DECIMAL[(width[,decimals])] [UNSIGNED] [ZEROFILL]

A commonly used numeric type. Stores a fixed-point number such as a salary or distance, with a total of width digits of which some smaller number are decimals that follow a decimal point. The maximum value of width is 255. For example, a column declared as price DECIMAL(6,2) should be used to store values in the range –9999.99 to 9999.99. price DECIMAL(10,4) would allow values like 123456.1234.

Prior to MySQL version 5.7, if you tried to store a value that’s outside this range, it would be stored as the closest value in the allowed range. For example, 100 would be stored as 99.99, and –100 would be stored as –99.99. Starting with 5.7, however, special SQL mode is set, which prohibits this, and other unsafe behaviors: STRICT_TRANS_TABLES. Using old behavior is possible, but could result in a data loss.

SQL modes are special settings that control behavior of MySQL when it comes to queries. For example, as you can see above, they can restrict “unsafe” behavior. Other modes might affect how queries are interpreted. For the purpose of learning MySQL, we recommend that you stick to the defaults, as they are safe. Changing SQL modes is usually required for compatibility with legacy applications across MySQL releases.

The width is optional, and a value of 10 is assumed when it is omitted. The number of decimals is optional and, when omitted, a value of 0 is assumed; the maximum value of decimals should be two less than the value of width. The maximum value of width is 65, and decimals is 30.

If you’re storing only positive values, you can use the UNSIGNED keyword as described for INT. If you want zero padding, use the ZEROFILL keyword for the same behavior as described for INT. The keyword DECIMAL has three identical, interchangeable alternatives: DEC, NUMERIC, and FIXED.

Values in DECIMAL column are stored using a binary format. This format uses four bytes for every nine digits.

Floating-point types

In “Fixed-point types”, we discussed the fixed-point DECIMAL type. There are two other types that support decimal points: DOUBLE (also known as REAL) and FLOAT. They’re designed to store approximate numeric values rather than the exact values stored by DECIMAL.

Why would you want approximate values? The answer is that many numbers with a decimal point are approximations of real quantities. For example, suppose you earn $50,000 per annum and you want to store it as a monthly wage. When you convert it to a per-month amount, it’s $4,166 plus 66 and 2/3rds cents. If you store this as $4,166.67, it’s not exact enough to convert to a yearly wage (since 12 multiplied by $4,166.67 is $50,000.04). However, if you store 2/3rds with enough decimal places, it’s a closer approximation. You’ll find that it is accurate enough to correctly multiply to obtain the original value in a high-precision environment such as MySQL, using only a bit of rounding. That’s where DOUBLE and FLOAT are useful: they let you store values such as 2/3rds or pi with a large number of decimal places, allowing accurate approximate representations of exact quantities. You can later use ROUND() function to restore results to a given precision.

Let’s continue the previous example using DOUBLE. Suppose you create a table as follows:

mysql> CREATE TABLE wage (monthly DOUBLE);

Query OK, 0 rows affected (0.09 sec)

You can now insert the monthly wage using:

mysql> INSERT INTO wage VALUES (50000/12);

Query OK, 1 row affected (0.00 sec)

And see what’s stored:

mysql> SELECT * FROM wage;

+----------------+

| monthly        |

+----------------+

| 4166.666666666 |

+----------------+

1 row in set (0.00 sec)

However, when you multiply it to a yearly value, you get a high precision approximation:

mysql> SELECT monthly*12 FROM wage;

+--------------------+

| monthly*12         |

+--------------------+

| 49999.999999992004 |

+--------------------+

1 row in set (0.00 sec)

To get the original value back, you still need to perform a rounding with a desired precision. For example, your business might require precision to five decimal places. In this case, you could restore the original:

mysql> SELECT ROUND(monthly*12,5) FROM wage;

+---------------------+

| ROUND(monthly*12,5) |

+---------------------+

|         50000.00000 |

+---------------------+

1 row in set (0.00 sec)

But precision to eight decimal places would not result in the original value:

mysql> SELECT ROUND(monthly*12,8) FROM wage;

+---------------------+

| ROUND(monthly*12,8) |

+---------------------+

|      49999.99999999 |

+---------------------+

1 row in set (0.00 sec)

It’s important to understand the imprecise and approximate value of floating-point data types.

Here are the details of the DOUBLE and FLOAT types:

FLOAT[(width, decimals)] [UNSIGNED] [ZEROFILL] or FLOAT[(precision)] [UNSIGNED] [ZEROFILL]

Stores floating-point numbers. It has two optional syntaxes: the first allows an optional number of decimals and an optional display width, and the second allows an optional precision that controls the accuracy of the approximation measured in bits. Without parameters, the type stores small, four-byte, single-precision floating-point values; usually, you use it without providing any parameters. When precision is between 0 and 24, the default behavior occurs. When precision is between 25 and 53, the type behaves as for DOUBLE. The width has no effect on what is stored, only on what is displayed. The UNSIGNED and ZEROFILL options behave as for INT.

DOUBLE[(width, decimals)] [UNSIGNED] [ZEROFILL]

Stores floating-point numbers. It has one optional syntax: it allows an optional number of decimals and an optional display width. Without parameters, the type stores normal, eight-byte, double-precision floating point values; usually, you use it without providing any parameters. The width has no effect on what is stored, only on what is displayed. The UNSIGNED and ZEROFILL options behave as for INT. The DOUBLE type has two identical synonyms: REAL and DOUBLE PRECISION.

String types

String data types are used to store text, and, less obviously, binary data. In MySQL, the string data types are CHAR, VARCHAR, BINARY, VARBINARY, BLOB, TEXT, ENUM, and SET.

[NATIONAL] VARCHAR(width) [CHARACTER SET charset_name] [COLLATE collation_name]

Probably, the single most commonly used string type. VARCHAR stores variable-length strings up to a maximum width. The maximum value of width is 65,535 characters. Most of the information applicable to this type will apply to other string types as well.

CHAR and VARCHAR types are very similar, but have a few important distinctions. VARCHAR incurs one or two extra bytes of overhead to store the value of the string, depending on whether the value is smaller than or larger than 255 bytes. Note that this is different from string length in characters, as certain character might require up to 4 bytes of space. It might seem obvious then, that VARCHAR is less efficient. However, that is not always true. As VARCHAR stores strings of arbitrary length (up to the width definied), shorter strings stored will require less storage than CHAR of similar width.

Another difference betwee CHAR and VARCHAR is their handling of trailing spaces. VARCHAR retains trailing spaces up to the specified column length, and will truncate the excess, producing a warning. As will be shown later, CHAR values are right-padded to the column length, and the trailing spaces aren’t preserved. For VARCHAR, trailing spaces are significant, unless they are trimmed, and will count as unique values. Let’s demonstrate:

mysql> CREATE TABLE test_varchar_trailing(d VARCHAR(2) UNIQUE);

Query OK, 0 rows affected (0.02 sec)

mysql> INSERT INTO test_varchar_trailing VALUES ('a'), (\'a ');

Query OK, 2 rows affected (0.01 sec)

Records: 2  Duplicates: 0  Warnings: 0

mysql> SELECT d, LEGNTH(d) FROM test_varchar_trailing;

+------+-----------+

| d    | LENGTH(d) |

+------+-----------+

| a    |         1 |

| a    |         2 |

+------+-----------+

2 rows in set (0.00 sec)

The second row we inserted has a trailing space, but since width for column d is 2, that space counted towards uniqueness of a row. If we try inserting a row with two trailing spaces, however:

mysql> INSERT INTO test_varchar_trailing VALUES (\'a  ');

ERROR 1062 (23000): Duplicate entry 'a '

for key 'test_varchar_trailing.d'

MySQL refuses to accept the new row. VARCHAR(2) implicitly truncates trailing spaces beyond the set width, and so the value stored changes from “a " to “a “. Since we already have a row with such value, a duplicate entry error is reported. This behavior for VARCHAR and TEXT can be controlled by changing the column collation. Some collations have PAD SPACE attribute, meaning that upon retreival they are padded to the width with spaces. This doesn’t affect storage, but does affect uniqueness checks as well as how GROUP BY and DISTINCT operators work, which we’ll discuss in Chapter 5. Let’s see the effect in action by creating a table with a PAD SPACE collation:

mysql> CREATE TABLE test_varchar_pad_collation(

-> data VARCHAR(5) CHARACTER SET latin1

-> COLLATE latin1_bin UNIQUE);

Query OK, 0 rows affected (0.02 sec)

mysql> INSERT INTO test_varchar_pad_collation VALUES ('a');

Query OK, 1 row affected (0.00 sec)

mysql> INSERT INTO test_varchar_pad_collation VALUES (\'a ');

ERROR 1062 (23000): Duplicate entry 'a '

for key 'test_varchar_pad_collation.data'

The NO PAD collations is a new addition of MySQL 8. In prior releases of MySQL, which you may still often see, every collation implicitly has the PAD SPACE attribute. Therefore, in MySQL 5.7 and prior releases, your only option to preserve trailing spaces is to use a binary type: VARBINARY or BLOB.

Both CHAR and VARCHAR data types disallow storage of values longer than width, unless strict SQL mode is disabled. With the protection disabled, values longer than width are truncated and a warning is shown. We don’t recommend enabling legacy behavior, as it might result in an unaccounted data loss.

Sorting and comparison of VARCHAR, CHAR, and TEXT types happens according to the collation of the character set assigned. You can see that it is possible to specify character set, as well as collation for each individual string type column. It’s also possible to specify binary character set, which effectively converts VARCHAR into VARBINARY. Don’t mistake binary charset for a BINARY attribute for a charset. The latter is a MySQL-only shorthand to specify a binary (_bin) collation.

What’s more, it’s possible to specify a collation directly in the ORDER BY clause. Available collations will depend on the character set of the column. Continuing with the test_varchar_pad_collation table, it’s possible to store an ä symbol there, and then see the effect collations make on the string ordering:

mysql> INSERT INTO test_varchar_pad_collation VALUES ('ä'), ('z');

Query OK, 2 rows affected (0.01 sec)

Records: 2  Duplicates: 0  Warnings: 0

mysql> SELECT * FROM test_varchar_pad_collation

-> ORDER BY data COLLATE latin1_german1_ci;

+------+

| data |

+------+

| a    |

| ä    |

| z    |

+------+

3 rows in set (0.00 sec)

mysql> SELECT * FROM test_varchar_pad_collation

-> ORDER BY data COLLATE latin1_swedish_ci;

+------+

| data |

+------+

| a    |

| z    |

| ä    |

+------+

3 rows in set (0.00 sec)

NATIONAL (or its equivalent short form, NCHAR) attribute is a standard SQL way to specify that a string type column must use a predefined character set. MySQL uses utf8 as such charset. It’s important to note that MySQL versions 5.7 and 8.0 disagree on what is utf8 exactly: former using it as an alias for utf8mb3, and latter — for utf8mb4. Thus, it is best to not use the NATIONAL attribute, as well as ambiguous aliases. The best practice with any text-related columns and data is to be as unambiguous and specific as possible.

[NATIONAL] CHAR(width) [CHARACTER SET charset_name] [COLLATE collation_name]

CHAR stores a fixed-length string (such as a name, address, or city) of length width. If a width is not provided, CHAR(1) is assumed. The maximum value of width is 255. As we discussed in the VARCHAR section above, values in CHAR columns are always stored at the specified length. Single letter stored in a CHAR(255) column will take 255 bytes (in latin1 charset), and will be padded with spaces. The padding is removed when reading the data, unless PAD_CHAR_TO_FULL_LENGTH SQL mode is enabled. Worth mentioning again that it means that strings stored in CHAR will lose all of their trailing spaces.

Note that earlier, width of a CHAR column was often associated with bytes. That’s not always the case now, and it’s definitely not the case by default. Multi-byte character sets, such as default utf8mb4 can result in a much larger values. InnoDB will actually encode fixed-length columns as variable-length columns, if their maximum size exceeds 768 bytes. Thus, in MySQL 8, by default, InnoDB will store CHAR(255) as it would VARCHAR. Let’s see a small example:

mysql> CREATE TABLE test_char_length(

->   utf8char CHAR(10) CHARACTER SET utf8mb4

-> , asciichar CHAR(10) CHARACTER SET binary

-> );

Query OK, 0 rows affected (0.04 sec)

mysql> INSERT INTO test_char_length VALUES ('Plain text', 'Plain text');

Query OK, 1 row affected (0.01 sec)

mysql> INSERT INTO test_char_length VALUES ('的開源軟體', 'Plain text');

Query OK, 1 row affected (0.00 sec)

mysql> SELECT LENGTH(utf8char), LENGTH(asciichar) FROM test_char_length;

+------------------+-------------------+

| LENGTH(utf8char) | LENGTH(asciichar) |

+------------------+-------------------+

|               10 |                10 |

|               15 |                10 |

+------------------+-------------------+

2 rows in set (0.00 sec)

As the values are left-aligned and right-padded with spaces, and any trailing spaces aren’t considered for CHAR at all, it’s impossible to compare strings consisting of spaces alone. If you find yourself in a situation when that’s important, VARCHAR is the data type to use.

BINARY[(width)] and VARBINARY(width)

These types are very similar to CHAR and VARCHAR but store binary strings. Binary strings have the special binary character set and collaction, and sorting them is dependent on the numeric values of the bytes in values stored. Instead of character strings, byte strings are stored. Remember that in VARCHAR we described the binary charset and BINARY attribute. Only the binary charset “converts” VARCHAR or CHAR into a respective BINARY form. BINARY attribute to a charset will not change the fact that character strings are stored. Unlike VARCHAR and CHAR, width here is exactly the number of bytes. When width is omitted for BINARY, it defaults to 1.

Similar to CHAR, data in the BINARY column is padded on the right. However, that being a binary data, it’s padded using zero bytes, usually written as 0x00 or \0. BINARY treats spaces as a significant character, not padding. If you need to store data which might end in zero bytes which are significant to you, VARBINARY or BLOB types should be used.

It is important to keep the concept of binary string in mind when working with both of these data types. Even though they’ll accept strings, they aren’t a synonym for data types using text strings. For example, you cannot change the case of the letters stored, as that concept doesn’t really apply to binary data. That becomes quite clear when you consider the actual data stored. Let’s see an example:

mysql> CREATE TABLE test_binary_data (

->   d1 BINARY(16)

-> , d2 VARBINARY(16)

-> , d3 CHAR(16)

-> , d4 VARCHAR(16)

-> );

Query OK, 0 rows affected (0.03 sec)

mysql> INSERT INTO test_binary_data VALUES (

->   'something'

-> , 'something'

-> , 'something'

-> , 'something');

Query OK, 1 row affected (0.00 sec)

mysql> SELECT d1, d2, d3, d4 FROM test_binary_data;

*************************** 1. row ***************************

d1: 0x736F6D657468696E6700000000000000

d2: 0x736F6D657468696E67

d3: something

d4: something

1 row in set (0.00 sec)

mysql> SELECT UPPER(d2), UPPER(d4) FROM test_binary_data;

*************************** 1. row ***************************

UPPER(d2): 0x736F6D657468696E67

UPPER(d4): SOMETHING

1 row in set (0.01 sec)

Note how MySQL command-line client actually shows values of binary types in a hex format. We believe that this is much better than silent conversions that were performed prior to MySQL 8.0, which might’ve resulted in misunderstanding. To get the actual text data back, you have to explicitly cast the binary data to text:

mysql> SELECT CAST(d1 AS CHAR) d1t, CAST(d2 AS CHAR) d2t

-> FROM test_binary_data;

+------------------+-----------+

| d1t              | d2t       |

+------------------+-----------+

| something        | something |

+------------------+-----------+

1 row in set (0.00 sec)

You can also notice that BINARY padding was converted to spaces when casting was performed.

BLOB[(width)] and TEXT[(width)] [CHARACTER SET charset_name] [COLLATE collation_name]

BLOB and TEXT are commonly used data types for storing large data. You may think of BLOB as a VARBINARY holding as many data as you like, and similarly of TEXT for VARCHAR. The BLOB and TEXT types itself can store up to 65,535 bytes or characters respectively. As usual, note that multi-byte charsets do exist. width attribute is optional, and when it is specified, MySQL actually will change BLOB or TEXT data type to whatever is the smallest type capable of holding that amount of data. For example, BLOB(128) will result in TINYBLOB being used:

mysql> CREATE TABLE test_blob(data BLOB(128));

Query OK, 0 rows affected (0.07 sec)

mysql> DESC test_blob;

+-------+----------+------+-----+---------+-------+

| Field | Type     | Null | Key | Default | Extra |

+-------+----------+------+-----+---------+-------+

| data  | tinyblob | YES  |     | NULL    |       |

+-------+----------+------+-----+---------+-------+

1 row in set (0.00 sec)

For BLOB type and related types, data is treated exactly as it would be in the case of VARBINARY. That is, no character set is assumed, and comparison and sorting is based on the numeric values of actual bytes stored. For TEXT, you may specify exact desired charset and collation. For both types and their variants, no padding is performed on insert, and no trimming is performed on select, making them ideal for storing data exactly as it is. In addition, a DEFAULT clause is not permitted, and you must take a prefix of the value when using it in an index (this is discussed in the next section).

One potential difference between BLOB and TEXT is their handling of trailing spaces. As we’ve shown already, VARCHAR and TEXT may pad strings depending on the collation used. BLOB and VARBINARY both use a binary character set with a single binary collation with no padding, and are impervious to collation mixups and related issues. Sometimes, it can be a good choice to use these types for additional safety. In addition to that, prior to MySQL 8.0, these were the only types that preserved the trailing spaces.

TINYBLOB and TINYTEXT [CHARACTER SET charset_name] [COLLATE collation_name]

Identical to BLOB and TEXT, respectively, except that a maximum of 255 bytes or characters can be stored.

MEDIUMBLOB and MEDIUMTEXT [CHARACTER SET charset_name] [COLLATE collation_name]

Identical to BLOB and TEXT, respectively, except that a maximum of 16,777,215 bytes or characters can be stored. Types LONG and LONG VARCHAR map to MEDIUMTEXT data type for compatibility.

LONGBLOB` and `LONGTEXT [CHARACTER SET charset_name] [COLLATE collation_name]

Identical to BLOB and TEXT, respectively, except that a maximum of four gigabytes of data can be stored. Note that this is a hard limit even in case of LONGTEXT, and thus number of characters in multi-byte charsets can be less than 4,294,967,295. The effective maximum size of the data that can be stored by a client will be limited by the amount of available memory as well as the value of the max_packet_size variable, which defaults to 64MiB.

ENUM(value1[,value2[, …]]) [CHARACTER SET charset_name] [COLLATE collation_name]

A list, or enumeration of string values. A column of type ENUM can be set to a value from the list value1, value2, and so on, up to a maximum of 65,535 different values. While the values are stored and retrieved as strings, what’s stored in the database is an integer representation. The enumerated column can contain NULL (stored as NULL), the empty string 0 (stored as 0), or any of the valid elements (stored as 1, 2, 3, and so on). You can prevent NULL values from being accepted by declaring the column as NOT NULL when creating the table.

This type is a compact way of storing values from a list of predefined values, such as state or country names. Consider this example using fruit names; the name can be any one of the predefined values Apple, Orange, or Pear (in addition to NULL and the empty string):

mysql> CREATE TABLE fruits_enum

-> (fruit_name ENUM('Apple', 'Orange', 'Pear'));

Query OK, 0 rows affected (0.00 sec)

mysql> INSERT INTO fruits_enum VALUES ('Apple');

Query OK, 1 row affected (0.00 sec)

If you try inserting a value that’s not in the list, MySQL produces an error to tell you that it didn’t store the data you asked:

mysql> INSERT INTO fruits_enum VALUES ('Banana');

ERROR 1265 (01000): Data truncated for column 'fruit_name' at row 1

Similarly, a list of several allowed values isn’t accepted either:

mysql> INSERT INTO fruits_enum VALUES ('Apple,Orange');

ERROR 1265 (01000): Data truncated for column 'fruit_name' at row 1

Displaying the contents of the table, you can see that no wrong values were stored:

mysql> SELECT * FROM fruits_enum;

+------------+

| fruit_name |

+------------+

| Apple      |

+------------+

1 row in set (0.00 sec)

Earlier, MySQL produced a warning instead of error, and stored empty string in place of a wrong value. That behavior can be used by unsetting the default strict SQL mode. It’s also possible specify a default value other than the empty string:

mysql> CREATE TABLE new_fruits_enum

-> (fruit_name ENUM('Apple', 'Orange', 'Pear')

-> DEFAULT 'Pear');

Query OK, 0 rows affected (0.01 sec)

mysql> INSERT INTO new_fruits_enum VALUES();

Query OK, 1 row affected (0.02 sec)

mysql> SELECT * FROM new_fruits_enum;

+------------+

| fruit_name |

+------------+

| Pear       |

+------------+

1 row in set (0.00 sec)

Here, not specifying a value results in the default value Pear being stored.

SET('_value1_'[,'_value2_'[, ...]]) [CHARACTER SET charset_name] [COLLATE collation_name]

A set of string values. A column of type SET can be set to zero or more values from the list value1, value2, and so on, up to a maximum of 64 different values. While the values are strings, what’s stored in the database is an integer representation. SET differs from ENUM in that each row can store only one ENUM value in a column, but can store multiple SET values. This type is useful for storing a selection of choices from a list, such as user preferences. Consider this example using fruit names; the name can be any combination of the predefined values:

mysql> CREATE TABLE fruits_set ( fruit_name SET('Apple', 'Orange', 'Pear') );

Query OK, 0 rows affected (0.08 sec)

mysql> INSERT INTO fruits_set VALUES ('Apple');

Query OK, 1 row affected (0.00 sec)

mysql> INSERT INTO fruits_set VALUES ('Banana');

ERROR 1265 (01000): Data truncated for column 'fruit_name' at row 1

mysql> INSERT INTO fruits_set VALUES ('Apple,Orange');

Query OK, 1 row affected (0.00 sec)

mysql> SELECT * FROM fruits_set;

+--------------+

| fruit_name   |

+--------------+

| Apple        |

| Apple,Orange |

+--------------+

2 rows in set (0.00 sec)

Again, note that we can store multiple values from the set in a single field, and that an empty string is stored for invalid input.

As with numeric types, we recommend that you always choose the smallest possible type to store values. For example, if you’re storing a city name, use CHAR or VARCHAR, rather than, say, the TEXT type. Having shorter columns helps keep your table size down, which in turns helps performance when the server has to search through a table.

Using a fixed size with the CHAR type is often faster than using a variable size with VARCHAR, since the MySQL server knows where each row starts and ends, and can quickly skip over rows to find the one it needs. However, with fixed-length fields, any space that you don’t use is wasted. For example, if you allow up to 40 characters in a city name, then CHAR(40) will always use up 40 characters, no matter how long the city name actually is. If you declare the city name to be VARCHAR(40), then you’ll use up only as much space as you need, plus one byte to store the name length. If the average city name is 10 characters long, this means that using a variable length field will take up 29 fewer bytes per entry; this can make a big difference if you’re storing millions of addresses.

In general, if storage space is at a premium or you expect large variations in the length of strings that are to be stored, use a variable-length field; if performance is a priority, use a fixed length.

Date and time types

These types serve the purpose of storing particular timestamps, dates, or time ranges. Particular care should be taken when dealing with timezones. We will try to explain details, but it’s worth re-reading this section and documentation later when you’ll need to actually work with timezones.

DATE

Stores and displays a date in the format YYYY-MM-DD for the range 1000-01-01 to 9999-12-31. Dates must always be input as year, month, and day triples, but the format of the input can vary, as shown in the following examples:

YYYY-MM-DD or YY-MM-DD

It’s optional whether you provide two-digit or four-digit years. We strongly recommend that you use the four-digit version to avoid confusion about the century. In practice, if you use the two-digit version, you’ll find that 70 to 99 are interpreted as 1970 to 1999, and 00 to 69 are interpreted as 2000 to 2069.

YYYY/MM/DD, YYYY:MM:DD, YY-MM-DD, or other punctuated formats

MySQL allows any punctuation characters to separate the components of a date. We recommend using dashes and, again, avoiding the two-digit years.

YYYY-M-D, YYYY-MM-D, or YYYY-M-DD

When punctuation is used (again, any punctuation character is allowed), single-digit days and months can be specified as such. For example, February 2, 2006, can be specified as 2006-2-2. The two-digit year equivalent is available, but not recommended.

YYYYMMDD or YYMMDD

Punctuation can be omitted in both date styles, but the digit sequences must be six or eight digits in length.

You can also input a date by providing both a date and time in the formats described later for DATETIME and TIMESTAMP, but only the date component is stored in a DATE type column. Regardless of the input type, the storage and display type is always YYYY-MM-DD. The zero date 0000-00-00 is allowed in all versions and can be used to represent an unknown or dummy value. If an input date is out of range, the zero date 0000-00-00 is stored. However, only MySQL versions up to and including 5.6 allow that by default. Both 5.7 and 8.0 set SQL modes that prohibit this behavior: STRICT_TRANS_TABLES, NO_ZERO_DATE, and NO_ZERO_IN_DATE. If you’re using an older version of a MySQL server, we recommend that you add these modes to your current session:

mysql> SET sql_mode=CONCAT(@@sql_mode, ,STRICT_TRANS_TABLES,

-> ,NO_ZERO_DATE, ,NO_ZERO_IN_DATE);

You can also set the sql_mode variable on a global server level and in the configuration file. This variable must list every mode you want to be enabled.

Here are some examples of inserting dates on a MySQL 8.0 server with default settings:

mysql> CREATE TABLE testdate (mydate DATE);

Query OK, 0 rows affected (0.00 sec)

mysql> INSERT INTO testdate VALUES ('2020/02/0');

ERROR 1292 (22007): Incorrect date value: '2020/02/0'

for column 'mydate' at row 1

mysql> INSERT INTO testdate VALUES ('2020/02/1');

Query OK, 1 row affected (0.00 sec)

mysql> INSERT INTO testdate VALUES ('2020/02/31');

ERROR 1292 (22007): Incorrect date value: '2020/02/31'

for column 'mydate' at row 1

mysql> INSERT INTO testdate VALUES ('2020/02/100');

ERROR 1292 (22007): Incorrect date value: '2020/02/100'

for column 'mydate' at row 1

Once INSERTs are executed, table will have the following data:

mysql> SELECT * FROM testdate;

+------------+

| mydate     |

+------------+

| 2020-02-01 |

+------------+

1 row in set (0.00 sec)

MySQL protected you from having “bad” data stored. Sometimes you may need to preserve the actual input and manually process it later. That is possible by unsetting aforementioned SQL modes. Doing so, you could end up with the following data:

mysql> SELECT * FROM testdate;

+------------+

| mydate     |

+------------+

| 2020-02-00 |

| 2020-02-01 |

| 0000-00-00 |

| 0000-00-00 |

+------------+

4 rows in set (0.01 sec)

Note also that the date is displayed in the YYYY-MM-DD format, regardless of how it was input.

TIME [fraction]

Stores a time in the format HHH:MM:SS for the range -838:59:59 to 838:59:59. Useful for storing duration of some activity. The values that can be stored are outside the range of the 24-hour clock to allow large differences between time values (up to 34 days, 22 hours, 59 minutes, and 59 seconds) to be computed and stored. fraction in TIME and other related data types specifies the fractional seconds precision in the range from 0 to 6. The default value is 0, meaning that no fractional seconds are preserved.

Times must always be input in the order days, hours, minutes, and seconds, using the following formats:

DD HH:MM:SS[.fraction], HH:MM:SS[.fraction], DD HH:MM, HH:MM, DD HH, or SS[.fraction]

The DD represents a one-digit or two-digit value of days in the range 0 to 34. The DD value is separated from the hour value, HH, by a space, while the other components are separated by a colon. Note that MM:SS is not a valid combination, since it cannot be disambiguated from HH:MM. If TIME definition doesn’t specify the fraction or sets it to 0, inserting fractional seconds will result in values being rounded to the nearest second.

For example, if you insert 2 13:25:58.999999 into a TIME type column with fraction of 0, the value 61:25:59 is stored, since the sum of 2 days (48 hours) and 13 hours is 61 hours. Starting with MySQL 5.7, default SQL mode set prohibits insertion of incorrect values. However, it is possible to enable the older behavior. Then, if you try inserting a value that’s out of bounds, a warning is generated, and the value is limited to the maximum time available. Similarly, if you try inserting an incorrect value, a warning is generated and the value is set to zero. You can use the SHOW WARNINGS command to reports the details of the warning generated by the previous SQL statement. Our recommendation is to stick to the default STRICT SQL mode. Unlike DATE type, there’s seemingly no benefit in allowing incorrect TIME entries, apart from easier error management on application side and maintaiting legacy behaviors.

Let’s try all these out in practice:

mysql> CREATE TABLE test_time(id SMALLINT, mytime TIME);

Query OK, 0 rows affected (0.00 sec)

mysql> INSERT INTO test_time VALUES(1, "2 13:25:59");

Query OK, 1 row affected (0.00 sec)

mysql> INSERT INTO test_time VALUES(2, "35 13:25:59");

ERROR 1292 (22007): Incorrect time value: '35 13:25:59'

for column 'mytime' at row 1

mysql> INSERT INTO test_time VALUES(3, "900.32");

Query OK, 1 row affected (0.00 sec)

mysql> SELECT * FROM test_time;

+------+----------+

| id   | mytime   |

+------+----------+

|    1 | 61:25:59 |

|    3 | 00:09:00 |

+------+----------+

2 rows in set (0.00 sec)

H:M:S, and single-, double-, and triple-digit combinations

You can use different combinations of digits when inserting or updating data; MySQL converts them into the internal time format and displays them consistently. For example, 1:1:3 is equivalent to 01:01:03. Different numbers of digits can be mixed; for example, 1:12:3 is equivalent to 01:12:03. Consider these examples:

mysql> CREATE TABLE mytime (testtime TIME);

Query OK, 0 rows affected (0.12 sec)

mysql> INSERT INTO mytime VALUES

-> ('-1:1:1'), ('1:1:1'),

-> ('1:23:45'), ('123:4:5'),

-> ('123:45:6'), ('-123:45:6');

Query OK, 4 rows affected (0.00 sec)

Records: 4  Duplicates: 0  Warnings: 0

mysql> SELECT * FROM mytime;

+------------+

| testtime   |

+------------+

| -01:01:01  |

| 01:01:01   |

| 01:23:45   |

| 123:04:05  |

| 123:45:06  |

| -123:45:06 |

+------------+

5 rows in set (0.01 sec)

Note that hours are shown with two digits for values within the range –99 to +99.

HHMMSS, MMSS, and SS

Punctuation can be omitted, but the digit sequences must be two, four, or six digits in length. Note that the rightmost pair of digits is always interpreted as a SS (seconds) value, the second next rightmost pair (if present) as MM (minutes), and the third rightmost pair (if present) as HH (hours). The result is that a value such as 1222 is interpreted as 12 minutes and 22 seconds, not 12 hours and 22 minutes.

You can also input a time by providing both a date and time in the formats described for DATETIME and TIMESTAMP, but only the time component is stored in a TIME type column. Regardless of the input type, the storage and display type is always HH:MM:SS. The zero time 00:00:00 can be used to represent an unknown or dummy value.

TIMESTAMP[(fraction)]

Stores and displays a date and time pair in the format YYYY-MM-DD HH:MM:SS[.fraction][time zone offset] for the range 1970-01-01 00:00:01.000000 to 2038-01-19 03:14:07.999999. This type is very similar to DATETIME type, but there are few differences. Both types accept a time zone modifier to the input value in the most recent MySQL version. Both types will store and present the data equally to the client in the same time zone. However, the values in TIMESTAMP columns are internally always stored in the UTC time zone, making it possible to get a local time zone automatically for clients in different time zones. That on its own is a very important distinction to remember. Arguably, TIMESTAMP is more convenient to use when dealing with time zones.

In earlier MySQL versions, preceding 5.6, only TIMESTAMP type supported automatic initialization and update. Moreover, only a single such column per a given table could do that. However, starting with 5.6, both types support the behaviors, and any number of columns can do so.

The value stored always matches the template YYYY-MM-DD HH:MM:SS[.fraction][time zone offset], but the value can be provided in a wide range of formats.

YYYY-MM-DD HH:MM:SS or YY-MM-DD HH:MM:SS

The date and time components follow the same relaxed restrictions as the DATE and TIME components described previously. This includes allowance for any punctuation characters, including (unlike TIME) flexibility in the punctuation used in the time component. For example, 0 is valid.

YYYYMMDDHHMMSS or YYMMDDHHMMSS

Punctuation can be omitted, but the string should be either 12 or 14 digits in length. We recommend only the unambiguous 14-digit version, for the reasons discussed for the DATE type. You can specify values with other lengths without providing separators, but we don’t recommend doing so.

Let’s look at the automatic-update feature in detail. You control them by adding the following attributes to the column definition when creating a table, or later, as we’ll explain in “Altering Structures”:

1. If you want the timestamp to be set only when a new row is inserted into the table, add DEFAULT CURRENT_TIMESTAMP to the end of the column declaration.

2. If you don’t want a default timestamp but want the current time to be used whenever the data in a row is updated, add ON UPDATE CURRENT_TIMESTAMP to the end of the column declaration.

3. If you want both of the above—that is, you want the timestamp to be set to the current time in each new row or whenever an existing row is modified— add DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP to the end of the column declaration.

If you do not specify DEFAULT NULL or NULL for a TIMESTAMP column, it will have 0 as the default value.

YEAR[(4)]

Stores a four-digit year. The four-digit version stores values in the range 1901 to 2155, as well as the zero year, 0000. Illegal values are converted to the zero date. You can input year values as either strings (such as '2005') or integers (such as 2005). The YEAR type requires one byte of storage space. In the earlier versions of MySQL, it was possible to specify the digits parameter. Depending on whether 2 or 4 is passed as the optional digits parameter. The two-digit version stored values from 70 to 69, representing 1970 to 2069. MySQL 8.0 doesn’t support the two-digit YEAR type, and specifying the digits parameter for display purposes is deprecated.

DATETIME[(fraction)]

Stores and displays a date and time pair in the format YYYY-MM-DD HH:MM:SS[.fraction][time zone offset] for the range 1000-01-01 00:00:00 to 9999-12-31 23:59:59. As for TIMESTAMP, the value stored always matches the template YYYY-MM-DD HH:MM:SS, but the value can be input in the same formats listed for the TIMESTAMP description. If you assign only a date to a DATETIME column, the zero time 00:00:00 is assumed. If you assign only a time to a DATETIME column, the zero date 0000-00-00 is assumed. This type has the sa,e automatic update features as TIMESTAMP has. Unless NOT NULL attribute is specified for a DATETIME column, a NULL value is the default, otherwise the default is 0. Unlike TIMESTAMP, DATETIME values aren’t converted to UTC time zone for storage.

Other types

Currently, as of MySQL 8, spatial and JSON data types fall under this broad category. Using both is a quite advanced topic, and we won’t cover them in-depth.

Spatial data types are concerned with storing geometrical objects, and MySQL has types corresponding to OpenGIS classes. Working with these types is a topic worth writing a book on its own.

JSON data type allows a native storage of valid JSON documents. Before MySQL version 5.7, JSON was usually stored in a TEXT or a similar column. However, that has a lot of disadvantages, for example, documents aren’t validated. Moreover, there’s no storage optimization performed, and all JSON is just stored in its text form. With native JSON, it’s stored in binary format. If we were to summarize in one sentence: use JSON data type for JSON, dear reader.

Keys and Indexes

You’ll find that almost all tables you use will have a PRIMARY KEY clause declared in their CREATE TABLE statement, and some times multiple KEY clauses. The reasons why you need a primary key and secondary keys are discussed in Chapter 2. This section discusses how primary keys are declared, what happens behind the scenes when you do so, and why you might want to also create other keys and indexes on your data.

A primary key uniquely identifies each row in a table. Even more importantly, for the default InnoDB storage engine, a primary key is also used as a clustered index. That means that the all of the actual table data is stored in an index structure. That is different to MyISAM, which stores data and indexes separately. When a table is using a clustered index, it’s called a clustered table. As we said, in a clustered table, each row is stored within an index, compared to being stored in what’s usually called a heap. Clustering a table results in that its rows will be sorted according to the clustered index ordering, and actually physically storead within the leaf pages of that index. There can’t be more than one clustered index per table. For such tables, secondary indexes refer to records in the clustered index instead of the actual table rows. That results in a generally improved query performance, though can be detrimental to writes. InnoDB does not allow you to choose between clustered and non-clustered tables, thus this is a design decision that you cannot change.

Primary keys are generally a recommended part of any database design, but for InnoDB they are necessary. In fact, if you do not specify a PRIMARY KEY clause when creating an InnoDB table, MySQL will use the first UNIQUE NOT NULL column as a base for the clustered index. If even that is not available, a hidden clustered index is created, based on ID values assigned by InnoDB to each row.

Given that InnoDB is a default storage engine, and a de-facto standard nowadays, we will concentrate on its behavior in this chapter. Alternative storage engines like MyISAM, MEMORY, or MyRocks will be discussed in the “Alternative Storage Engines”.

When primary key is defined, as we mentioned, it becomes a clustered index, and all data in the table is stored in the leaf blocks of that index. InnoDB uses B-tree indexes (more specifically, B+tree variant), with the exception of indexes on spatial data types, which use R-tree. Other storage engines might implement different index types, however, when table’s storage engine is not specified, you can assume that all indexes are B-tree.

Having a clustered index, or in other words having index-organized tables, speeds up queries and sorts involving the primary key columns. However, a downside is that modifying columns in a primary key is expensive. Thus, a good design will require a primary key based on columns which are frequently used for filtering in queries but are rarely modified. Remember that having no primary key at all will result in InnoDB using an implicit cluster index, thus if you’re not sure what columns to pick for a primary key, consider using a synthetic id-like column. For example, the SERIAL data type might fit well in that case.

Stepping away from the InnoDB internal details, when you declare a PRIMARY KEY for a table in MySQL, it creates a structure that stores information about where the data from each row in the table is stored. This information is called an index, and its purpose is to speed up searches that use the primary key. For example, when you declare PRIMARY KEY (actor_id) in the actor table in the sakila database, MySQL creates a structure that allows it to find rows that match a specific actor_id (or a range of identifiers) extremely quickly.

This is very useful to match actors to films, or films to categories for example. You can display the indexes available on a table using the SHOW INDEX (or SHOW INDEXES) command:

mysql> SHOW INDEX FROM category\G

*************************** 1. row ***************************

Table: category

Non_unique: 0

Key_name: PRIMARY

Seq_in_index: 1

Column_name: category_id

Collation: A

Cardinality: 16

Sub_part: NULL

Packed: NULL

Null:

Index_type: BTREE

Comment:

Index_comment:

Visible: YES

Expression: NULL

1 row in set (0.00 sec)

The cardinality is the number of unique values in the index; for an index on a primary key, this is the same as the number of rows in the table.

Note that all columns that are part of a primary key must be declared as NOT NULL, since they must have a value for the row to be valid. Without the index, the only way to find rows in the table is to read each one from disk and check whether it matches the category_id you’re searching for. For tables with many rows, this exhaustive, sequential searching is extremely slow. However, you can’t just index everything; we’ll come back to this point at the end of this section.

You can create other indexes on the data in a table. You do this so that other searches—on other columns or combinations of columns—are extremely fast and in order to avoid sequential scans. For example, take the table actor again. Apart from having a primary key on actor_id, it also has a secondary key on last_name, to improve searching by actor’s last name.

mysql> SHOW CREATE TABLE actor\G

*************************** 1. row ***************************

Table: actor

Create Table: CREATE TABLE `actor` (

`actor_id` smallint unsigned NOT NULL AUTO_INCREMENT,

...

`last_name` varchar(45) NOT NULL,

...

PRIMARY KEY (`actor_id`),

KEY `idx_actor_last_name` (`last_name`)

) ...

1 row in set (0.00 sec)

You can see the keyword KEY is used to tell MySQL that an extra index is needed; you can use the word INDEX in place of KEY. Following that keyword is an index name, and then the column to index is included in parentheses. You can also add indexes after tables are created—in fact, you can pretty much change anything about a table after its creation—and this is discussed in “Altering Structures”.

You can build an index on more than one column. For example, consider the following table, which is a modified table from sakila:

mysql> CREATE TABLE customer_mod (

-> customer_id smallint unsigned NOT NULL AUTO_INCREMENT,

-> first_name varchar(45) NOT NULL,

-> last_name varchar(45) NOT NULL,

-> email varchar(50) DEFAULT NULL,

-> PRIMARY KEY (customer_id),

-> KEY idx_names_email (first_name, last_name, email));

Query OK, 0 rows affected (0.06 sec)

You can see that we’ve added a primary key index on the customer_id identifier column, and we’ve also added another index—called idx_names_email—that includes the first_name, last_name, and email columns in this order. Let’s now consider how you can use that extra index.

You can use the idx_names_email index for fast searching by combinations of the three name columns. For example, it’s useful in the following query:

mysql> SELECT * FROM customer_mod WHERE

-> first_name = 'Rose' AND

-> last_name = 'Williams' AND

-> email = 'rose.w@nonexistent.edu';

We know it helps the search, because all columns listed in the index are used in the query. You can use the EXPLAIN statement to check whether what you think should happen is in fact happening:

mysql> EXPLAIN SELECT * FROM customer_mod WHERE

-> first_name = 'Rose' AND

-> last_name = 'Williams' AND

-> email = 'rose.w@nonexistent.edu'\G

*************************** 1. row ***************************

id: 1

select_type: SIMPLE

table: customer_mod

partitions: NULL

type: ref

possible_keys: idx_names_email

key: idx_names_email

key_len: 567

ref: const,const,const

rows: 1

filtered: 100.00

Extra: Using index

1 row in set, 1 warning (0.00 sec)

You can see that MySQL reports that the possible_keys are idx_names_email (meaning that the index could be used for this query) and that the key that it’s decided to use is idx_names_email. So, what you expect and what is happening are the same, and that’s good news! You’ll find out more about the EXPLAIN statement in Chapter 7.

The index we’ve created is also useful for queries on only the first_name column. For example, it can be used by the following query:

mysql> SELECT * FROM customer_mod WHERE

-> first_name = Rose;

You can use EXPLAIN to check whether the index is being used. The reason it can be used is because the first_name column is the first listed in the index. In practice, this means that the index clusters, or stores together, information about rows for all people with the same first name, and so the index can be used to find anyone with a matching first name.

The index can also be used for searches involving combinations of first name and last name, for exactly the same reasons we’ve just discussed. The index clusters together people with the same first name, and within that it clusters people with identical first names ordered by last name. So, it can be used for this query:

mysql> SELECT * FROM customer_mod WHERE

-> first_name = Rose AND

-> last_name = Williams;

However, the index can’t be used for this query because the leftmost column in the index, first_name, does not appear in the query:

mysql> SELECT * FROM customer_mod WHERE

-> last_name = Williams AND

-> email = rose.w@nonexistent.edu;

The index should help narrow down the set of rows to a smaller set of possible answers. For MySQL to be able to use an index, the query needs to meet both the following conditions:

1. The leftmost column listed in the KEY (or PRIMARY KEY) clause must be in the query.

2. The query must contain no OR clauses for columns that aren’t indexed.

Again, you can always use the EXPLAIN statement to check whether an index can be used for a particular query.

Before we finish this section, here are a few ideas on how to choose and design indexes. When you’re considering adding an index, think about the following:

- Indexes cost space on disk, and they need to be updated whenever data changes. If your data changes frequently, or lots of data changes when you do make a change, indexes will slow the process down. However, in practice, since SELECT statements (data reads) are usually much more common than other statements (data modifications), indexes are usually beneficial.

- Only add an index that’ll be used frequently. Don’t bother indexing columns before you see what queries your users and your applications need. You can always add indexes afterward.

- If all columns in an index are used in all queries, list the column with the highest number of duplicates at the left of the KEY clause. This minimizes index size.

- The smaller the index, the faster it’ll be. If you index large columns, you’ll get a larger index. This is a good reason to ensure your columns are as small as possible when you design your tables.

- For long columns, you can use only a prefix of the values from a column to create the index. You can do this by adding a value in parentheses after the column definition, such as KEY idx_names_email (first_name(3), last_name(2), email(10)). This means that only the first three characters of first_name are indexed, then the first two characters of last_name, and then 10 characters from email. This is a significant saving over indexing 140 characters from the three columns! When you do this, your index will be less able to uniquely identify rows, but it’ll be much smaller and still reasonably good at finding matching rows. That is also mandatory for long types like TEXT.

To finish this section, we need to also discuss some peculiarities regarding secondary keys in InnoDB. Remember that all the table data is stored in the leafs of the clustered index. That means, using the actor example, that if we need to get the first_name data when filtering by last_name, even though we can use the idx_actor_last_name for quick filterting, we will need to access the data by the primary key. As a consequence, each secondary key in InnoDB has all of the primary key columns appended to its definition implicitly. Having unnecessarily long primary keys in InnoDB results in significantly bloated secondary keys.

This can also be seen in the EXPLAIN output, note the Extra: Using index in the first output:

mysql> EXPLAIN SELECT actor_id, last_name FROM actor WHERE last_name = 'Smith'\G

*************************** 1. row ***************************

id: 1

select_type: SIMPLE

table: actor

partitions: NULL

type: ref

possible_keys: idx_actor_last_name

key: idx_actor_last_name

key_len: 182

ref: const

rows: 1

filtered: 100.00

Extra: Using index

1 row in set, 1 warning (0.00 sec)

mysql> EXPLAIN SELECT first_name FROM actor WHERE last_name = 'Smith'\G

*************************** 1. row ***************************

id: 1

select_type: SIMPLE

table: actor

partitions: NULL

type: ref

possible_keys: idx_actor_last_name

key: idx_actor_last_name

key_len: 182

ref: const

rows: 1

filtered: 100.00

Extra: NULL

1 row in set, 1 warning (0.00 sec)

Effectively, idx_actor_last_name is a covering index for the first query, meaning that InnoDB can extract all the required data from that index alone. However, for the second query, InnoDB will have to do an additional lookup of a clustered index to get the value for first_name column.

The AUTO_INCREMENT Feature

MySQL’s proprietary AUTO_INCREMENT feature allows you to create a unique identifier for a row without running a SELECT query. Here’s how it works. Let’s take the simplified actor table again:

mysql> CREATE TABLE actor (

-> actor_id smallint unsigned NOT NULL AUTO_INCREMENT,

-> first_name varchar(45) NOT NULL,

-> last_name varchar(45) NOT NULL,

-> PRIMARY KEY (actor_id)

-> );

Query OK, 0 rows affected (0.03 sec)

It’s possible to insert rows into that table without specifying the actor_id:

mysql> INSERT INTO actor VALUES (NULL, 'Alexander', 'Kaidanovsky');

Query OK, 1 row affected (0.01 sec)

mysql> INSERT INTO actor VALUES (NULL, 'Anatoly', 'Solonitsyn');

Query OK, 1 row affected (0.01 sec)

mysql> INSERT INTO actor VALUES (NULL, 'Nikolai', 'Grinko');

Query OK, 1 row affected (0.00 sec)

When you view the data in this table you can see that each row has a value assigned for the actor_id column:

mysql> SELECT * FROM actor;

+----------+------------+-------------+

| actor_id | first_name | last_name   |

+----------+------------+-------------+

|        1 | Alexander  | Kaidanovsky |

|        2 | Anatoly    | Solonitsyn  |

|        3 | Nikolai    | Grinko      |

+----------+------------+-------------+

3 rows in set (0.00 sec)

Each time a new row is inserted, a unique value for the actor_id column is created for that new row.

Let’s consider how the new feature works. You can see that the actor_id column is declared as an integer with the clauses NOT NULL AUTO_INCREMENT. The AUTO_INCREMENT keyword tells MySQL that when a value isn’t provided for this column, the value allocated should be one more than the maximum currently stored in the table. The AUTO_INCREMENT sequence begins at 1 for an empty table.

The NOT NULL is required for AUTO_INCREMENT columns; when you insert NULL (or 0, though this isn’t recommended), the MySQL server automatically finds the next available identifier and assigns it to the new row. You can manually insert negative values if the column was not defined as UNSIGNED; however, for the next automatic increment, MySQL will simply use the largest (most positive) value in the column, or start from 1 if there are no positive values.

The AUTO_INCREMENT feature has the following requirements:

- The column it is used on must be indexed.

- The column that is it used on cannot have a DEFAULT value.

- There can be only one AUTO_INCREMENT column per table.

MySQL supports different storage engines; we’ll learn more about these in “Alternative Storage Engines” in Chapter 7. When using the non-default MyISAM table type, you can use the AUTO_INCREMENT feature on keys that comprise multiple columns. In effect, you could have multiple independent counters within a single AUTO_INCREMENT column. However, it’s not possible with InnoDB.

While the AUTO_INCREMENT feature is useful, it isn’t portable to other database environments, and it hides the logical steps to creating new identifiers. It can also lead to ambiguity; for example, dropping or truncating a table will reset the counter, but deleting selected rows (with a WHERE clause) doesn’t reset the counter. Moreover, if a row is inserted inside a transaction, but then transaction is rolled back, an identifier would be used up anyway. Consider an example; let’s create the table count that contains an auto-incrementing field counter:

mysql> CREATE TABLE count (counter INT AUTO_INCREMENT KEY);

Query OK, 0 rows affected (0.13 sec)

mysql> INSERT INTO count VALUES (),(),(),(),(),();

Query OK, 6 rows affected (0.01 sec)

Records: 6  Duplicates: 0  Warnings: 0

mysql> SELECT * FROM count;

+---------+

| counter |

+---------+

| 1       |

| 2       |

| 3       |

| 4       |

| 5       |

| 6       |

+---------+

6 rows in set (0.00 sec)

Inserting several values works as expected. Now, let’s delete a few rows and then add six new rows:

mysql> DELETE FROM count WHERE counter > 4;

Query OK, 2 rows affected (0.00 sec)

mysql> INSERT INTO count VALUES (),(),(),(),(),();

Query OK, 6 rows affected (0.00 sec)

Records: 6  Duplicates: 0  Warnings: 0

mysql> SELECT * FROM count;

+---------+

| counter |

+---------+

| 1       |

| 2       |

| 3       |

| 4       |

| 7       |

| 8       |

| 9       |

| 10      |

| 11      |

| 12      |

+---------+

10 rows in set (0.00 sec)

Here, we see that the counter is not reset, and continues from 7. If, however, we truncate the table, thus removing all of the data, the counter is reset to 1:

mysql> TRUNCATE TABLE count;

Query OK, 0 rows affected (0.00 sec)

mysql> INSERT INTO count VALUES (),(),(),(),(),();

Query OK, 6 rows affected (0.01 sec)

Records: 6  Duplicates: 0  Warnings: 0

mysql> SELECT * FROM count;

+---------+

| counter |

+---------+

| 1       |

| 2       |

| 3       |

| 4       |

| 5       |

| 6       |

+---------+

6 rows in set (0.00 sec)

To summarize: AUTO_INCREMENT guarantees a sequence of transactional and monotonically-increasing numbers. However, it does not in any way guarantee that each individual identifier provided will exactly follow the previous one. Usually, this behavior of AUTO_INCREMENT is clear enough and should not be a problem. However, if your particular use case requires a counter that guarantees no gaps, you should consider using some kind of a workaround. Unfortunately, it’ll likely be implemented on the application side.

Altering Structures

We’ve shown you all the basics you need for creating databases, tables, indexes, and columns. In this section, you’ll learn how to add, remove, and change columns, databases, tables, and indexes in structures that already exist.

Adding, Removing, and Changing Columns

You can use the ALTER TABLE statement to add new columns to a table, remove existing columns, and change column names, types, and lengths.

Let’s begin by considering how you modify existing columns. Consider an example in which we rename a table column. The language table has a column called last_update that contains the time the record was modified. To change the name of this column to last_updated_time, you would write:

mysql> ALTER TABLE language RENAME COLUMN last_update TO last_updated_time;

Query OK, 0 rows affected (0.03 sec)

Records: 0  Duplicates: 0  Warnings: 0

This particular example utilizes online DDL feature of MySQL. What actually happens behind the scenes is that MySQL only modifies metadata, and doesn’t need to actually re-write the table in any way. You can notice that by lack of affected rows. Not all of the DDL statements can be performed online, and that’s not going to be the case with a lot of other changes. DDL stands for Data Definition Language, and in the context of SQL it’s a subset of syntax and statements used to create, modify, and delete schema objects such as databases, tables, indexes, and columns. CREATE TABLE and ALTER TABLE are both DDL operations, for example. Executing statements like that requires special internal mechanisms, including special locking. You probably wouldn’t like tables changing while your queries are running! Special locks are called Metadata Locks in MySQL, and we give a detailed overview of how they work in “Metadata Locks”.

Note that ALTER and other statements that execute through online DDL still require metadata locks to be obtained. In that sense, they are not so online, but they won’t lock the target table entirely while they are running. Executing DDL statements on a running system under load is a risky venture: even a statement that should execute momentarily may wreak havoc. We recommend that you read carefully about the metadata locking in our book and in the MySQL documentation, and experiment with running different DDL statements with and without concurrent load. That may not be too important while you’re learning MySQL, but we think that it’s worth cautioning you. With that covered, let’s get back to our ALTER of the language table.

You can check the result with the SHOW COLUMNS statement:

mysql> SHOW COLUMNS FROM language;

+-------------------+------------------+------+-----+-------------------+...

| Field             | Type             | Null | Key | Default           |...

+-------------------+------------------+------+-----+-------------------+...

| language_id       | tinyint unsigned | NO   | PRI | NULL              |...

| name              | char(20)         | NO   |     | NULL              |...

| last_updated_time | timestamp        | NO   |     | CURRENT_TIMESTAMP |...

+-------------------+------------------+------+-----+-------------------+...

...+-----------------------------------------------+

...| Extra                                         |

...+-----------------------------------------------+

...| auto_increment                                |

...|                                               |

...| DEFAULT_GENERATED on update CURRENT_TIMESTAMP |

...+-----------------------------------------------+

3 rows in set (0.01 sec)

In the previos example we used the ALTER TABLE statement with RENAME COLUMN keyword. That is a MySQL 8 feature. We could alternatively use ALTER TABLE with the CHANGE keyword for compatibility.

mysql> ALTER TABLE language CHANGE last_update last_updated_time TIMESTAMP

-> NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP;

Query OK, 0 rows affected (0.04 sec)

Records: 0  Duplicates: 0  Warnings: 0

In this example, you can see that we provided four parameters to the ALTER TABLE statement with the CHANGE keyword:

1. The table name, language

2. The original column name, last_update

3. The new column name, last_updated_time

4. The column type, TIMESTAMP, with a lot of extra attributes, which are necessary to not change the original definition

You must provide all four; that means you need to respecify the type and any clauses that go with it. In this example, as we’re using MySQL 8 with default settings, TIMESTAMP no longer has explicit defaults. As you can see, using RENAME COLUMN is much easier than CHANGE.

If you want to modify the type and clauses of a column, but not its name, you can use the MODIFY keyword:

mysql> ALTER TABLE language MODIFY name CHAR(20) DEFAULT 'n/a';

Query OK, 0 rows affected (0.14 sec)

Records: 0  Duplicates: 0  Warnings: 0

You can also do this with the CHANGE keyword, but by specifying the same column name twice:

mysql> ALTER TABLE language CHANGE name name CHAR(20) DEFAULT 'n/a';

Query OK, 0 rows affected (0.03 sec)

Records: 0  Duplicates: 0  Warnings: 0

Be careful when you’re modifying types:

- Don’t use incompatible types, since you’re relying on MySQL to successfully convert data from one format to another (for example, converting an INT column to a DATETIME column isn’t likely to do what you hoped).

- Don’t truncate the data unless that’s what you want. If you reduce the size of a type, the values will be edited to match the new width, and you can lose data.

Suppose you want to add an extra column to an existing table. Here’s how to do it with the ALTER TABLE statement:

mysql> ALTER TABLE language ADD native_name CHAR(20);

Query OK, 0 rows affected (0.04 sec)

Records: 0  Duplicates: 0  Warnings: 0

You must supply the ADD keyword, the new column name, and the column type and clauses. This example adds the new column, native_name, as the last column in the table, as shown with the SHOW COLUMNS statement:

mysql> SHOW COLUMNS FROM artist;

+-------------------+------------------+------+-----+-------------------+...

| Field             | Type             | Null | Key | Default           |...

+-------------------+------------------+------+-----+-------------------+...

| language_id       | tinyint unsigned | NO   | PRI | NULL              |...

| name              | char(20)         | YES  |     | n/a               |...

| last_updated_time | timestamp        | NO   |     | CURRENT_TIMESTAMP |...

| native_name       | char(20)         | YES  |     | NULL              |...

+-------------------+------------------+------+-----+-------------------+...

4 rows in set (0.00 sec)

If you want it to instead be the first column, use the FIRST keyword as follows:

mysql> ALTER TABLE language ADD native_name CHAR(20) FIRST;

Query OK, 0 rows affected (0.08 sec)

Records: 0  Duplicates: 0  Warnings: 0

mysql> SHOW COLUMNS FROM language;

+-------------------+------------------+------+-----+-------------------+...

| Field             | Type             | Null | Key | Default           |...

+-------------------+------------------+------+-----+-------------------+...

| native_name       | char(20)         | YES  |     | NULL              |...

| language_id       | tinyint unsigned | NO   | PRI | NULL              |...

| name              | char(20)         | YES  |     | n/a               |...

| last_updated_time | timestamp        | NO   |     | CURRENT_TIMESTAMP |...

+-------------------+------------------+------+-----+-------------------+...

4 rows in set (0.01 sec)

If you want it added in a specific position, use the AFTER keyword:

mysql> ALTER TABLE language ADD native_name CHAR(20) AFTER name;

Query OK, 0 rows affected (0.08 sec)

Records: 0  Duplicates: 0  Warnings: 0

mysql> SHOW COLUMNS FROM language;

+-------------------+------------------+------+-----+-------------------+...

| Field             | Type             | Null | Key | Default           |...

+-------------------+------------------+------+-----+-------------------+...

| language_id       | tinyint unsigned | NO   | PRI | NULL              |...

| name              | char(20)         | YES  |     | n/a               |...

| native_name       | char(20)         | YES  |     | NULL              |...

| last_updated_time | timestamp        | NO   |     | CURRENT_TIMESTAMP |...

+-------------------+------------------+------+-----+-------------------+...

4 rows in set (0.00 sec)

To remove a column, use the DROP keyword followed by the column name. Here’s how to get rid of the newly added formed column:

mysql> ALTER TABLE language DROP native_name;

Query OK, 0 rows affected (0.07 sec)

Records: 0  Duplicates: 0  Warnings: 0

This removes both the column structure and any data contained in that column. It also removes the column from any index it was in; if it’s the only column in the index, the index is dropped, too. You can’t remove a column if it’s the only one in a table; to do this, you drop the table instead as explained later in “Deleting Structures”. Be careful when dropping columns; you discard both the data and the structure of your table. When the structure of a table changes, you will generally have to modify any INSERT statements that you use to insert values in a particular order. We described INSERT statements in “The INSERT Statement” in Chapter 3.

MySQL allows you to specify multiple alterations in a single ALTER TABLE statement by separating them with commas. Here’s an example that adds a new column and adjusts another:

mysql> ALTER TABLE language ADD native_name CHAR(255), MODIFY name CHAR(255);

Query OK, 6 rows affected (0.06 sec)

Records: 6  Duplicates: 0  Warnings: 0

Note that this time, you can see that 6 records were change. If you haven’t noticed yet, note that we didn’t see any records affected when altering tables previously. The difference is that this time, we’re not performing an online DDL, because changing any column’s type will always result in a table being rebuilt. We recommend reading through the documentation section titled “Online DDL Operations” when planning your changes. Combining online and “offline” operations will result in an “offline” operation.

When not using online DDL, or when any of the modifications is “offline”, it’s very efficient to join multiple modifications in a single operation. That potentially saves the cost of creating a new table, copying data from the old table to the new table, dropping the old table, and renaming the new table with the name of the old table for each modification individually.

Adding, Removing, and Changing Indexes

As we discussed previously, it’s often hard to know what indexes are useful before the application you’re building is used. You might find that a particular feature of the application is much more popular than you expected, causing you to evaluate how to improve performance for the associated queries. You’ll therefore find it useful to be able to add, alter, and remove indexes on the fly after your application is deployed. This section shows you how. Modifying indexes does not affect the data stored in the table.

We’ll start with adding a new index. Imagine that the language table is frequently queried using a WHERE clause that specifies the name. To speed this query, you’ve decided to add a new index, which you’ve named idx_name. Here’s how you add it after the table is created:

mysql> ALTER TABLE language ADD INDEX idx_name (name);

Query OK, 0 rows affected (0.05 sec)

Records: 0  Duplicates: 0  Warnings: 0

Again, you can use the terms KEY and INDEX interchangeably. You can check the results with the SHOW CREATE TABLE statement:

mysql> SHOW CREATE TABLE language\G

*************************** 1. row ***************************

Table: language

Create Table: CREATE TABLE `language` (

`language_id` tinyint unsigned NOT NULL AUTO_INCREMENT,

`name` char(255) DEFAULT NULL,

`last_updated_time` timestamp NOT NULL

DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

PRIMARY KEY (`language_id`),

KEY `idx_name` (`name`)

) ENGINE=InnoDB AUTO_INCREMENT=8

DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci

As expected, the new index forms part of the table structure. You can also specify a primary key for a table after it’s created:

mysql> CREATE TABLE no_pk (id INT);

Query OK, 0 rows affected (0.02 sec)

mysql> INSERT INTO no_pk VALUES (1),(2),(3);

Query OK, 3 rows affected (0.01 sec)

Records: 3  Duplicates: 0  Warnings: 0

mysql> ALTER TABLE no_pk ADD PRIMARY KEY (id);

Query OK, 0 rows affected (0.13 sec)

Records: 0  Duplicates: 0  Warnings: 0

Now let’s consider how to remove an index. To remove a non-primary-key index, you do the following:

mysql> ALTER TABLE language DROP INDEX idx_name;

Query OK, 0 rows affected (0.08 sec)

Records: 0  Duplicates: 0  Warnings: 0

You can drop a primary-key index as follows:

mysql> ALTER TABLE no_pk DROP PRIMARY KEY;

Query OK, 3 rows affected (0.07 sec)

Records: 3  Duplicates: 0  Warnings: 0

MySQL won’t allow you to have multiple primary keys in a table. If you want to change the primary key, you’ll have to remove the existing index before adding the new one. However, we know that it’s possible to group DDL operations. Consider this example:

mysql> ALTER TABLE language DROP PRIMARY KEY,

-> ADD PRIMARY KEY (language_id, name);

Query OK, 0 rows affected (0.09 sec)

Records: 0  Duplicates: 0  Warnings: 0

You can’t modify an index once it’s been created. However, sometimes you’ll want to; for example, you might want to reduce the number of characters indexed from a column or add another column to the index. The best method to do this is to drop the index and then create it again with the new specification. For example, suppose you decide that you want the idx_name index to include only the first 10 characters of the artist_name. Simply do the following:

mysql> ALTER TABLE language DROP INDEX idx_name,

-> ADD INDEX idx_name (name(10));

Query OK, 0 rows affected (0.05 sec)

Records: 0  Duplicates: 0  Warnings: 0

Renaming Tables and Altering Other Structures

We’ve seen how to modify columns and indexes in a table; now let’s see how to modify tables themselves. It’s easy to rename a table. Suppose you want to rename language to languages. Use the following command:

mysql> ALTER TABLE language RENAME TO languages;

Query OK, 0 rows affected (0.04 sec)

The TO keyword is optional.

There are several other things you can do with ALTER statements:

- Change the default character set and collation order for a database, a table, or a column.

- Manage and change constraints. For example, you can add and remove foreign keys.

- Add partitioning to a table, or alter the current partitioning definition.

- Change the storage engine of a table.

You can find more about these operations in the MySQL manual under the “ALTER DATABASE” and “ALTER TABLE” headings. An alternative shorter notation for the same statement is RENAME TABLE:

mysql> RENAME TABLE languages TO language;

Query OK, 0 rows affected (0.04 sec)

One thing that is not possible to alter is a name of a particular database. However, if you’re using InnoDB engine, you can use RENAME to move tables between databases:

mysql> CREATE DATABASE sakila_new;

Query OK, 1 row affected (0.05 sec)

mysql> RENAME TABLE sakila.language TO sakila_new.language;

Query OK, 0 rows affected (0.05 sec)

mysql> USE sakila;

Database changed

mysql> SHOW TABLES LIKE 'lang%';

Empty set (0.00 sec)

mysql> USE sakila_new;

Database changed

mysql> SHOW TABLES LIKE 'lang%';

+------------------------------+

| Tables_in_sakila_new (lang%) |

+------------------------------+

| language                     |

+------------------------------+

1 row in set (0.00 sec)

Deleting Structures

In the previous section, we showed how you can delete columns and rows from a database; now we’ll describe how to remove databases and tables.

Dropping Databases

Removing, or dropping, a database is straightforward. Here’s how you drop the sakila database:

mysql> DROP DATABASE sakila;

Query OK, 25 rows affected (0.16 sec)

The number of rows returned in the response is the number of tables removed. You should take care when dropping a database, since all its tables, indexes, and columns are deleted, as are all the associated disk-based files and directories that MySQL uses to maintain them.

If a database doesn’t exist, trying to drop it causes MySQL to report an error. Let’s try dropping the sakila database again:

mysql> DROP DATABASE sakila;

ERROR 1008 (HY000): Can't drop database 'sakila'; database doesn't exist

You can avoid the error, which is useful when including the statement in a script, by using the IF EXISTS phrase:

mysql> DROP DATABASE IF EXISTS sakila;

Query OK, 0 rows affected, 1 warning (0.00 sec)

You can see that a warning is reported, since the sakila database has already been dropped.

Removing Tables

Removing tables is as easy as removing a database. Let’s create and remove a table from the sakila database:

mysql> CREATE TABLE temp (id SERIAL PRIMARY KEY);

Query OK, 0 rows affected (0.05 sec)

mysql> DROP TABLE temp;

Query OK, 0 rows affected (0.03 sec)

Don’t worry: the 0 rows affected message is misleading. You’ll find the table is definitely gone.

You can use the IF EXISTS phrase to prevent errors. Let’s try dropping the temp table again:

mysql> DROP TABLE IF EXISTS temp;

Query OK, 0 rows affected, 1 warning (0.01 sec)

Again, you can investigate the warning indicates with the SHOW WARNINGS statement:

mysql> SHOW WARNINGS;

+-------+------+-----------------------------+

| Level | Code | Message                     |

+-------+------+-----------------------------+

| Note  | 1051 | Unknown table 'sakila.temp' |

+-------+------+-----------------------------+

1 row in set (0.00 sec)

You can drop more than one table in a single statement by separating table names with commas:

mysql> DROP TABLE IF EXISTS temp, temp1, temp2;

Query OK, 0 rows affected, 3 warnings (0.00 sec)

You can see three warnings because none of these tables existed.

Chapter 5. Advanced Querying

Over the previous two chapters, you’ve completed an introduction to the basic features of querying and modifying databases with SQL. You should now be able to create, modify, and remove database structures, as well as work with data as you read, insert, delete, and update entries. Over this and the next two chapters, we’ll look at more advanced concepts and then will proceed to more administrative and operations-oriented content. You can skim these chapters and return to read them thoroughly when you’re comfortable using MySQL.

This chapter teaches you more about querying, giving you skills to answer complex information needs. You’ll learn how to do the following:

- Use nicknames, or aliases, in queries to save typing and allow a table to be used more than once in a query

- Aggregate data into groups so you can discover sums, averages, and counts

- Join tables in different ways

- Use nested queries

- Save query results in variables so they can be reused in other queries

- Understand why MySQL supports several table types

Aliases

Aliases are nicknames. They give you a shorthand way of expressing a column, table, or function name, allowing you to:

- Write shorter queries

- Express your queries more clearly

- Use one table in two or more ways in a single query

- Access data more easily from programs

- Use special types of nested queries; these are the subject of “Nested Queries”, discussed later in this chapter

Column Aliases

Column aliases are useful for improving the expression of your queries, reducing the number of characters you need to type, and making it easier to work with programming languages such as Python or PHP. Consider a simple, not-very-useful example:

mysql> SELECT first_name AS 'First Name', last_name AS 'Last Name'

-> FROM actor LIMIT 5;

+------------+--------------+

| First Name | Last Name    |

+------------+--------------+

| PENELOPE   | GUINESS      |

| NICK       | WAHLBERG     |

| ED         | CHASE        |

| JENNIFER   | DAVIS        |

| JOHNNY     | LOLLOBRIGIDA |

+------------+--------------+

5 rows in set (0.00 sec)

The column first_name is aliased as First Name, and column last_name as Last Name . You can see that in the output, the usual column headings, first_name and last_name, are replaced by the aliases First Name and Last Name. The advantage is that the aliases might be more meaningful to users. In this case, at the very least, they are more human-readable. Other than that, it’s not very useful, but it does illustrate the idea that for a column, you add the keyword AS and then a string that represents what you’d like the column to be known as. Specifying the AS keyword is not required but makes things much clearer.

NOTE

We’ll be using the LIMIT clause extensively throughout this chapter, otherwise almost every output will be unwieldy and long. Sometimes, we’ll mention that explicitly, sometimes not. You can experiment on your own by removing LIMIT from the queries we give. More information about the LIMIT clause can be found in “The LIMIT Clause”.

Now let’s see column aliases doing something useful. Here’s an example that uses a MySQL function and an ORDER BY clause:

mysql> SELECT CONCAT(first_name, ' ', last_name, ' played in ', title) AS movie

-> FROM actor JOIN film_actor USING (actor_id)

-> JOIN film USING (film_id)

-> ORDER BY movie LIMIT 20;

+--------------------------------------------+

| movie                                      |

+--------------------------------------------+

| ADAM GRANT played in ANNIE IDENTITY        |

| ADAM GRANT played in BALLROOM MOCKINGBIRD  |

...

| ADAM GRANT played in TWISTED PIRATES       |

| ADAM GRANT played in WANDA CHAMBER         |

| ADAM HOPPER played in BLINDNESS GUN        |

| ADAM HOPPER played in BLOOD ARGONAUTS      |

+--------------------------------------------+

20 rows in set (0.03 sec)

The MySQL function CONCAT ( ) concatenates together the strings that are parameters — in this case, the first_name, a constant string with a space, the last_name, the constant string played in, and the title to give output such as ZERO CAGE played in CANYON STOCK. We’ve added an alias to the function, AS movie, so that we can refer to it easily as movie throughout the query. You can see that we do this in the ORDER BY clause, where we ask MySQL to sort the output by ascending movie value. This is much better than the unaliased alternative, which requires you to write out the CONCAT ( ) function again:

mysql> SELECT CONCAT(first_name, " ", last_name, " played in ", title) AS movie

-> FROM actor JOIN film_actor USING (actor_id)

-> JOIN film USING (film_id)

-> ORDER BY CONCAT(first_name, " ", last_name, " played in ", title)

-> LIMIT 20;

+--------------------------------------------+

| movie                                      |

+--------------------------------------------+

| ADAM GRANT played in ANNIE IDENTITY        |

| ADAM GRANT played in BALLROOM MOCKINGBIRD  |

...

| ADAM GRANT played in TWISTED PIRATES       |

| ADAM GRANT played in WANDA CHAMBER         |

| ADAM HOPPER played in BLINDNESS GUN        |

| ADAM HOPPER played in BLOOD ARGONAUTS      |

+--------------------------------------------+

20 rows in set (0.03 sec)

The alternative is unwieldy, and worse, you risk mistyping some part of the ORDER BY clause and getting a result different from what you expect. (Note that we’ve used AS movie on the first line so that the displayed column has the label movie.)

There are restrictions on where you can use column aliases. You can’t use them in a WHERE clause, or in the USING and ON clauses that we discuss later in this chapter. This means you can’t write a query like this:

mysql> SELECT first_name AS name FROM actor WHERE name = 'ZERO CAGE';

ERROR 1054 (42S22): Unknown column 'name' in 'where clause'

You can’t do that because MySQL doesn’t always know the column values before it executes the WHERE clause. However, you can use column aliases in the ORDER BY clause, and in the GROUP BY and HAVING clauses discussed later in this chapter.

The AS keyword is optional, as we’ve mentioned. Because of this, the following two queries are equivalent:

mysql> SELECT actor_id AS id FROM actor WHERE first_name = 'ZERO';

+----+

| id |

+----+

| 11 |

+----+

1 row in set (0.00 sec)

mysql> SELECT actor_id id FROM actor WHERE first_name = 'ZERO';

+----+

| id |

+----+

| 11 |

+----+

1 row in set (0.00 sec)

We recommend using the AS keyword, since it helps to clearly distinguish an aliased column, especially where you’re selecting multiple columns from a list of columns separated by commas.

Alias names have a few restrictions. They can be at most 255 characters in length and can contain any character. Aliases don’t always need to be quoted, and they follow the same rules as table and column names do, which we described in Chapter 4. If an alias is a single word and doesn’t include special symbols—like a dash, or a plus sign, or a space, for example—and is not a keyword—like USE,--then you don’t need to put quotes around it. Otherwise, you need to quote that alias, and can use double quotes, single quotes, or backticks. We recommend using lowercase alphanumeric strings for alias names and using a consistent character choice—such as an underscore—to separate words. Aliases are case-insensitive on all platforms.

Table Aliases

Table aliases are useful for the same reasons as column aliases, but they are also sometimes the only way to express a query. This section shows you how to use table aliases, and “Nested Queries”, later in this chapter, shows you other sample queries where table aliases are essential.

Here’s a basic table-alias example that shows you how to save some typing:

mysql> SELECT ac.actor_id, ac.first_name, ac.last_name, fl.title FROM

-> actor AS ac INNER JOIN film_actor AS fla USING (actor_id)

-> INNER JOIN film AS fl USING (film_id)

-> WHERE fl.title = 'AFFAIR PREJUDICE';

+----------+------------+-----------+------------------+

| actor_id | first_name | last_name | title            |

+----------+------------+-----------+------------------+

|       41 | JODIE      | DEGENERES | AFFAIR PREJUDICE |

|       81 | SCARLETT   | DAMON     | AFFAIR PREJUDICE |

|       88 | KENNETH    | PESCI     | AFFAIR PREJUDICE |

|      147 | FAY        | WINSLET   | AFFAIR PREJUDICE |

|      162 | OPRAH      | KILMER    | AFFAIR PREJUDICE |

+----------+------------+-----------+------------------+

5 rows in set (0.00 sec)

You can see that the film and actor tables are aliased as fl and ac, respectively, using the AS keyword. This allows you to express column names more compactly, such as fl.title. Notice also that you can use table aliases in the WHERE clause; unlike column aliases, there are no restrictions on where table aliases can be used in queries. From our example, you can see that we’re referring to the table aliases in SELECT before they have been defined in FROM. There’s, however, a catch with table aliases: if an alias has been used for a table, it’s impossible to refer to that table without using its new alias. For example, the following statement will error out, as it would if we’d mention film in SELECT, or in ORDER BY:

mysql> SELECT ac.actor_id, ac.first_name, ac.last_name, fl.title FROM

-> actor AS ac INNER JOIN film_actor AS fla USING (actor_id)

-> INNER JOIN film AS fl USING (film_id)

-> WHERE film.title = 'AFFAIR PREJUDICE';

ERROR 1054 (42S22): Unknown column 'film.title' in 'where clause'

As with column aliases, the AS keyword is optional. This means that:

actor AS ac INNER JOIN film_actor AS fla

is the same as

actor ac INNER JOIN film_actor fla

Again, we prefer the AS style because it’s clearer to anyone looking at your queries than the alternative. The restrictions on table-alias-name characters and lengths are the same as column aliases, and our recommendations on choosing them are the same, too.

As discussed in the introduction to this section, table aliases allow you to write queries that you can’t otherwise easily express. Consider an example: suppose you want to know whether two or more films in our collection have the same title, and if so, what are those films. Let’s think about the basic requirement: you want to know if two movies have the same name. To do get that, you might try a query like this:

mysql> SELECT * FROM film WHERE title = title;

But that doesn’t make sense: every film has the same title as itself, and so the query just produces all films as output:

+---------+------------------...

| film_id | title            ...

+---------+------------------...

|       1 | ACADEMY DINOSAUR ...

|       2 | ACE GOLDFINGER   ...

|       3 | ADAPTATION HOLES ...

...

|    1000 | ZORRO ARK        ...

+---------+------------------...

1000 rows in set (0.01 sec)

What you really want is to know whether two different films from the film table have the same name. But how can you do that in a single query? The answer is to give the table two different aliases; you then check to see whether one row in the first aliased table matches a row in the second:

mysql> SELECT m1.film_id, m2.title

-> FROM film AS m1, film AS m2

-> WHERE m1.title = m2.title;

+---------+-------------------+

| film_id | title             |

+---------+-------------------+

|       1 | ACADEMY DINOSAUR  |

|       2 | ACE GOLDFINGER    |

|       3 | ADAPTATION HOLES  |

...

|     999 | ZOOLANDER FICTION |

|    1000 | ZORRO ARK         |

+---------+-------------------+

1000 rows in set (0.02 sec)

But it still doesn’t work! We get all 1000 movies as answers. The reason is that a film still matches itself because it occurs in both aliased tables.

To get the query to work, we need to make sure a movie from one aliased table doesn’t match itself in the other aliased table. The way to do that is to specify that the movies in each table shouldn’t have the same id:

mysql> SELECT m1.film_id, m2.title

-> FROM film AS m1, film AS m2

-> WHERE m1.title = m2.title

-> AND m1.film_id <> m2.film_id;

Empty set (0.00 sec)

You can now see that there aren’t two films in the database with the same name. The additional AND m1.film_id != m2.film_id stops answers from being reported where the movie id is the same in both tables.

Table aliases are also useful in nested queries that use the EXISTS and ON clauses. We show you examples later in this chapter when we introduce nested techniques.

Aggregating Data

Aggregate functions allow you to discover the properties of a group of rows. You use them for purposes such as discovering how many rows there are in a table, how many rows in a table share a property (such as having the same name or date of birth), finding averages (such as the average temperature in November), or finding the maximum or minimum values of rows that meet some condition (such as finding the coldest day in August).

This section explains the GROUP BY and HAVING clauses, the two most commonly used SQL statements for aggregation. But first it explains the DISTINCT clause, which is used to report unique results for the output of a query. When neither the DISTINCT nor the GROUP BY clause is specified, the returned raw data can still be processed using the aggregate functions that we describe in this section.

The DISTINCT Clause

To begin our discussion on aggregate functions, we’ll focus on the DISTINCT clause. This isn’t really an aggregate function, but more of a post-processing filter that allows you to remove duplicates. We’ve added it into this section because, like aggregate functions, it’s concerned with picking examples from the output of a query, rather than processing individual rows.

An example is the best way to understand DISTINCT. Consider this query:

mysql> SELECT DISTINCT first_name

-> FROM actor JOIN film_actor USING (actor_id);

+-------------+

| first_name  |

+-------------+

| PENELOPE    |

| NICK        |

...

| GREGORY     |

| JOHN        |

| BELA        |

| THORA       |

+-------------+

128 rows in set (0.00 sec)

The query finds all first names of all the actors listed in our database that have participated in a film and reports one example of each name. If you remove the DISTINCT clause, you get one row of output for each role in every film we have in our database, or 5462 rows. That’s a lot of output, so we’re limiting to five rows, but you can spot the difference immediately with names being repeated:

mysql> SELECT first_name

-> FROM actor JOIN film_actor USING (actor_id)

-> LIMIT 5;

+------------+

| first_name |

+------------+

| PENELOPE   |

| PENELOPE   |

| PENELOPE   |

| PENELOPE   |

| PENELOPE   |

+------------+

5 rows in set (0.00 sec)

So, the DISTINCT clause helps get a summary.

The DISTINCT clause applies to the query output and removes rows that have identical values in the columns selected for output in the query. If you rephrase the previous query to output both first_name and last_name (but otherwise don’t change the JOIN clause and still use DISTINCT), you’ll get 199 rows in the output (that’s why we use last names):

mysql> SELECT DISTINCT first_name, last_name

-> FROM actor JOIN film_actor USING (actor_id);

+-------------+--------------+

| first_name  | last_name    |

+-------------+--------------+

| PENELOPE    | GUINESS      |

| NICK        | WAHLBERG     |

...

| JULIA       | FAWCETT      |

| THORA       | TEMPLE       |

+-------------+--------------+

199 rows in set (0.00 sec)

Unfortunately, people’s names even when last names are added still make for a bad unique key. There are 200 rows in the actor table in sakila database, and we’re missing one of them. You should remember this issue, as using DISTINCT indiscriminately may result in wrong query results.

To remove duplicates, MySQL needs to sort the output. If indexes are available that are in the same order as required for the sort — or the data itself is in an order that’s useful — this process has very little overhead. However, for large tables and without an easy way of accessing the data in the right order, sorting can be very slow. You should use DISTINCT (and other aggregate functions) with caution on large data sets. If you do use it, you can check its behavior using the EXPLAIN statement discussed in Chapter 7.

The GROUP BY Clause

The GROUP BY clause groups output data for the purpose of aggregation. Particularly, that allows us to use aggregate functions (covered later in “Aggregate functions”) on our data, when our projection (that is, contents of the SELECT clause) contains columns other than those within an aggregate function. GROUP BY is similar to ORDER BY in that it takes a list of columns as an argument. However, these clauses are evaluated at different times and are only similar in how they look, not how they operate.

Let’s take a look at a few GROUP BY examples that will demonstrate what it can be used for. In its most basic form, when we list every column we SELECT in GROUP BY, we end up with a DISTINCT equivalent. We’ve already established that a first name is not a unique identifier for an actor.

mysql> SELECT first_name FROM actor

-> WHERE first_name IN ('GENE', 'MERYL');

+------------+

| first_name |

+------------+

| GENE       |

| GENE       |

| MERYL      |

| GENE       |

| MERYL      |

+------------+

5 rows in set (0.00 sec)

We can tell MySQL to group the output by a given column. In this case, we only have one, so let’s do that:

mysql> SELECT first_name FROM actor

-> WHERE first_name IN ('GENE', 'MERYL')

-> GROUP BY first_name;

+------------+

| first_name |

+------------+

| GENE       |

| MERYL      |

+------------+

2 rows in set (0.00 sec)

You can see that the original five rows were “folded”, or, more accurately, grouped into just two resulting rows. Not very helpful, as DISTINCT could do the same. It’s worth mentioning, however, that this is not always going to be the case. DISTINCT and GROUP BY are evaluated and executed at the very different stages of query execution, so you should not confuse them, even if sometimes the effects are similar.

By the SQL standard, every column projected in the SELECT clause, which is not part of an aggregate function, should be listed in the GROUP BY clause. The only time this rule may be violated is when the resulting groups have only one row each. If you think about it, that’s logical: when you select first name and last name from the actor table, and group only by first name, how should the database behave? It cannot output a few rows with the same first name, as that goes against the grouping rules, but there may be more last names for a given name than one. For a long time, MySQL extended the standard by allowing to GROUP BY based on fewer columns than defined in SELECT. What did it do with the extra columns? Well, it output some value in a non-deterministic way. For example, when you grouped the first name, but didn’t the last name, from two rows GENE, WILLIS and GENE, HOPKINS you could either get first or second. That’s a non-standard and dangerous behavior. Imagine that for a year, you got the last name as if it was ordered by the alphabetic order, and came to rely on that. Then the table was re-organized and the order changed. Your authors firmly believe that SQL standard is correct to limit such behaviors.

We may also note here that while every column in SELECT must either be used in GROUP BY or in an aggregate function, you can GROUP BY columns that are not part of the SELECT. You’ll see some examples of that later.

Now let’s construct a more useful example. An actor usually takes part in many films throughout their career. We may want to find out just how many films a particular actor played in, or do a calculation for each actor we know of and get an actor rating by productivity. To start, we can use the techniques we’ve learned so far and perform an INNER JOIN between actor and film_actor tables. We don’t need the film table as we’re not looking for any details. We can then order the output by actor’s name making it easier to count what we want.

mysql> SELECT first_name, last_name, film_id

-> FROM actor INNER JOIN film_actor USING (actor_id)

-> ORDER BY first_name, last_name LIMIT 20;

+------------+-----------+---------+

| first_name | last_name | film_id |

+------------+-----------+---------+

| ADAM       | GRANT     |      26 |

| ADAM       | GRANT     |      52 |

| ADAM       | GRANT     |     233 |

| ADAM       | GRANT     |     317 |

| ADAM       | GRANT     |     359 |

| ADAM       | GRANT     |     362 |

| ADAM       | GRANT     |     385 |

| ADAM       | GRANT     |     399 |

| ADAM       | GRANT     |     450 |

| ADAM       | GRANT     |     532 |

| ADAM       | GRANT     |     560 |

| ADAM       | GRANT     |     574 |

| ADAM       | GRANT     |     638 |

| ADAM       | GRANT     |     773 |

| ADAM       | GRANT     |     833 |

| ADAM       | GRANT     |     874 |

| ADAM       | GRANT     |     918 |

| ADAM       | GRANT     |     956 |

| ADAM       | HOPPER    |      81 |

| ADAM       | HOPPER    |      82 |

+------------+-----------+---------+

20 rows in set (0.01 sec)

By running down the list, it’s easy to count off how many films we’ve got for each actor, or at least for Adam Grant. Without a LIMIT, however, the query would return 5462 distinct rows and calculating our counts would take a lot of time. The GROUP BY clause can help automate this process by grouping the movies by actor; we can then use the COUNT() function to count off the number of films in each group. Finally, we can use ORDER BY and LIMIT to get the top ten actors by the number of films they appeared in. Here’s the query that does what we want:

mysql> SELECT first_name, last_name, COUNT(film_id) AS num_films FROM

-> actor INNER JOIN film_actor USING (actor_id)

-> GROUP BY first_name, last_name

-> ORDER BY num_films DESC LIMIT 5;

+------------+-------------+-----------+

| first_name | last_name   | num_films |

+------------+-------------+-----------+

| SUSAN      | DAVIS       |        54 |

| GINA       | DEGENERES   |        42 |

| WALTER     | TORN        |        41 |

| MARY       | KEITEL      |        40 |

| MATTHEW    | CARREY      |        39 |

| SANDRA     | KILMER      |        37 |

| SCARLETT   | DAMON       |        36 |

| VAL        | BOLGER      |        35 |

| ANGELA     | WITHERSPOON |        35 |

| UMA        | WOOD        |        35 |

+------------+-------------+-----------+

10 rows in set (0.01 sec)

You can see that the output we’ve asked for is first_name, last_name, COUNT(film_id) as num_films, and this tells us exactly what we wanted to know. We group our data by first_name and last_name columns, running the COUNT() aggregate function in the process. For each “bucket” of rows we got in the previous query, we now get only a single row, albeit giving the information we want. Notice how we’ve combined GROUP BY and ORDER BY to get the ordering we wanted, that is by the number of films from more to fewer. GROUP BY doesn’t guarantee ordering, only grouping. Finally, we LIMIT the output to ten rows representing our most productive actors, otherwise we’d get 199 rows of output.

Let’s consider the query further. We’ll start with the GROUP BY clause. This tells us how to put rows together into groups: in this example, we’re telling MySQL that the way to group rows is by first_name, last_name. The result is that rows for actors with the same name form a cluster (or a bucket) — that is, each distinct name becomes one group. Once the rows are grouped, they’re treated in the rest of the query as if they’re one row. So, for example, when we write SELECT first_name, last_name, we get just one row for each group. This is exactly the same as DISTINCT, as we’ve already discussed. The COUNT() function tells us about the properties of the group. More specifically, it tells us the number of rows that form each group; you can count any column in a group, and you’ll get the same answer, so COUNT(film_id) is almost always the same as COUNT(*) or COUNT(first_name). See “Aggregate functions” for more details on why almost. We could also just do COUNT(1), which in some databases is a useful optimization technique, although not in MySQL. Of course, you can use a column alias for the COUNT() column.

Let’s try another example. Suppose you want to know how many different actors played in each movie, along with the film name and its category, and get five films with the largest crew. Here’s the query:

mysql> SELECT title, name AS category_name, COUNT(*) AS cnt

-> FROM film INNER JOIN film_actor USING (film_id)

-> INNER JOIN film_category USING (film_id)

-> INNER JOIN category USING (category_id)

-> GROUP BY film_id, category_id

-> ORDER BY cnt DESC LIMIT 5;

+------------------+---------------+-----+

| title            | category_name | cnt |

+------------------+---------------+-----+

| LAMBS CINCINATTI | Games         |  15 |

| CRAZY HOME       | Comedy        |  13 |

| CHITTY LOCK      | Drama         |  13 |

| RANDOM GO        | Sci-Fi        |  13 |

| DRACULA CRYSTAL  | Classics      |  13 |

+------------------+---------------+-----+

5 rows in set (0.03 sec)

Before we discuss what’s new, think about the general function of the query. We join four tables together using INNER JOIN: film, film_actor, film_category, and category using the identifier columns. Forgetting the aggregation for a moment, the output of this query is one row per a combination of movie and actor.

The GROUP BY clause puts the rows together into clusters. In this query, we want the films grouped together with their category. So, the GROUP BY clause uses film_id and category_id to do that. You can use the film_id from any of the three tables; film.film_id, film_actor.film_id, or film_category.film_id are the same for this purpose. It doesn’t matter since the INNER JOIN makes sure they match anyway. The same applies to category_id.

Even though it’s required to list every non-aggregated column in GROUP BY, you can GROUP BY on columns outside of the SELECT.

As in the previous example query, we’re using the COUNT() function to tell us how many rows are in each group. For example, you can see that COUNT(*) tells us that there are 15 actors in the LAMBS CINCINATTI game show. Again, it doesn’t matter what column or columns you count in the query: for example, COUNT(*) has the same effect as COUNT(film.film_id) or COUNT(category.name).

We’re then ordering the output by the COUNT(*) column aliased cnt in a descending order and pick the first five rows. Note how there are multiple rows with cnt equal to thirteen. In fact, there are even more of those—six total—in the database, making this ordering a bit unfair, as movies having the same number of actors will be sorted randomly. You may add another column to the ORDER BY like title, to make sorting more predictable.

Let’s try another example. Sakila is not only about movies and actors: it’s a video rental place database, after all. We have, among other things, the customer information, as well as data on what films they rented. Say we want to know five customers that rent movies from the same category most. For example, we might want to adjust our ads based on whether a person likes different categories or sticks to a single one most of the time. We need to carefully think about our grouping: we don’t need to group by movie, as that’d just give us the number of times a customer rented it. The resulting query is quite complex, although it’s still a variation on INNER JOIN and GROUP BY.

mysql> SELECT email, name AS category_name, COUNT(category_id) AS cnt

-> FROM customer cs INNER JOIN rental USING (customer_id)

-> INNER JOIN inventory USING (inventory_id)

-> INNER JOIN film_category USING (film_id)

-> INNER JOIN category cat USING (category_id)

-> GROUP BY email, category_name

-> ORDER BY cnt DESC LIMIT 5;

+----------------------------------+---------------+-----+

| email                            | category_name | cnt |

+----------------------------------+---------------+-----+

| WESLEY.BULL@sakilacustomer.org   | Games         |   9 |

| ALMA.AUSTIN@sakilacustomer.org   | Animation     |   8 |

| KARL.SEAL@sakilacustomer.org     | Animation     |   8 |

| LYDIA.BURKE@sakilacustomer.org   | Documentary   |   8 |

| NATHAN.RUNYON@sakilacustomer.org | Animation     |   7 |

+----------------------------------+---------------+-----+

5 rows in set (0.08 sec)

You can see that some customers do enjoy renting films from the same category multiple times. What we don’t know is if any of them rented the same movie multiple times, or if those were all different movies within a category. The GROUP BY clause hides the details. Again, we use COUNT(*) to do the counting of rows in the groups, and you can see the INNER JOIN spread over lines 2 to 5 in the query.

The interesting thing about this query is that we didn’t specify column names for GROUP BY or ORDER BY clauses explicitly. Instead, we used the columns’ position numbers (counted from 1) as they appear in the SELECT clause. This technique saves on typing, but can be problematic if you later decide to add another column in the SELECT, which would break the ordering.

We should talk about the danger of GROUP BY, as we’ve previously mentioned for DISTINCT. Consider the following query:

mysql> SELECT COUNT(*) FROM actor GROUP BY first_name, last_name;

+----------+

| COUNT(*) |

+----------+

|        1 |

|        1 |

...

|        1 |

|        1 |

+----------+

199 rows in set (0.00 sec)

Looks simple enough, and it produces the number of times a combination of a given first name and last name was found in the actor table. You could even assume that it just outputs 199 rows of digit 1. However, if we do a COUNT(*) on the actor table we get 200 rows. What’s the catch? Apparently, two actors have the same first name and last name. These things happen, and you have to be mindful of them. When you group based on columns which do not form a unique identifier, you may accidentally group together unrelated rows, resulting in misleading data. To find the duplicates, we can modify a query that we constructed in “Table Aliases” to look for films with the same name:

mysql> SELECT a1.actor_id, a1.first_name, a1.last_name

-> FROM actor AS a1, actor AS a2

-> WHERE a1.first_name = a2.first_name

-> AND a1.last_name = a2.last_name

-> AND a1.actor_id <> a2.actor_id;

+----------+------------+-----------+

| actor_id | first_name | last_name |

+----------+------------+-----------+

|      101 | SUSAN      | DAVIS     |

|      110 | SUSAN      | DAVIS     |

+----------+------------+-----------+

2 rows in set (0.00 sec)

Before we end this section, let’s again touch on how MySQL extends the SQL standard around the GROUP BY clause. Before MySQL 5.7, it was possible by default to specify an incomplete column list in the GROUP BY, and, as we’ve explained, that resulted in a random rows output within groups for non-grouped dependent columns. For reasons of supporting legacy sofware, both MySQL 5.7 and 8.0 continue providing this behavior, though it has to be explicitly enabled. The behavior is controlled by the ONLY_FULL_GROUP_BY SQL mode, which is set by default. If you find yourself in a situation where you need to port a program relying on the legacy GROUP BY behavior, we recommend that you do not resort to changing the SQL mode. There are generally two ways to handle this problem. The first is to understand whether the query logic requires incomplete grouping at all — that is hardly the case. The second is to support the random data behavior for non-grouped columns by using either an aggregate function like MIN() or MAX(), or by using the special ANY_VALUE() aggregate function, which, unsurprisingly, just produces a random value from within a group.

Aggregate functions

We’ve seen examples of how the COUNT() function can be used to tell how many rows are in a group. Here we will cover some other functions commonly used to explore the properties of aggregated rows. But first, we should expand a bit on COUNT() as it’s frequently used.

COUNT()

Returns the number of rows or the number of values in a column. Remember we mentioned that COUNT(*) is almost always the equivalent of COUNT(<column>). The problem is NULL. COUNT(*) will do a count of rows returned, regardless of whether the column in those rows is NULL or not. However, when you do a COUNT(<column>), only non-NULL values will be counted. In our sakila database, a customer’s email may be NULL, and we can observe the impact.

mysql> SELECT COUNT(*) FROM customer;

+----------+

| count(*) |

+----------+

|      599 |

+----------+

1 row in set (0.00 sec)

mysql> SELECT COUNT(email) FROM customer;

+--------------+

| count(email) |

+--------------+

|          598 |

+--------------+

1 row in set (0.00 sec)

We should also add that COUNT() can be run with an internal DISTINCT clause like this — COUNT(DISTINCT <column>) and will return a number of distinct values instead of all values.

AVG()

Returns the average (mean) of the values in the specified column for all rows in a group. For example, you could use it to find the average cost of a house in a city, when the houses are grouped by city:

SELECT AVG(cost) FROM house_prices GROUP BY city;

MAX()

Returns the maximum value from rows in a group. For example, you could use it to find the warmest day in a month, when the rows are grouped by month.

MIN()

Returns the minimum value from rows in a group. For example, you could use it to find the youngest student in a class, when the rows are grouped by class.

STD() or STDDEV() or STDDEV_POP()

Returns the standard deviation of values from rows in a group. For example, you could use it to understand the spread of test scores, when rows are grouped by university course. All three of these are synonyms. STD() is a MySQL extension, STDDEV() is added for compatibility with Oracle, and STDDEV_POP() is an SQL standard function.

SUM()

Returns the sum of values from rows in a group. For example, you could use it to compute the dollar amount of sales in a given month, when rows are grouped by month.

There are other functions available for use with GROUP BY; they’re less frequently used than the ones we’ve introduced. You can find more details on them in the MySQL manual under the heading “Aggregate Function Descriptions.”

The HAVING Clause

You’re now familiar with the GROUP BY clause, which allows you to sort and cluster data. You should now be able to use it to find out about counts, averages, minimums, and maximums. This section shows how you can use the HAVING clause to add additional control to the aggregation of rows in a GROUP BY operation.

Suppose you want to know how many popular actors there are in our database. You’ve decided to define an actor as popular if they’ve taken part in at least forty movies. In the previous section, we tried an almost identical query but without the popularity limitation. We also grouped the actors by first and last name, losing one record, so we’ll add grouping on the actor_id, which we know to be unique. Here’s the new query, with an additional HAVING clause that adds the constraint:

mysql> SELECT first_name, last_name, COUNT(film_id)

-> FROM actor INNER JOIN film_actor USING (actor_id)

-> GROUP BY actor_id, first_name, last_name

-> HAVING COUNT(film_id) > 40

-> ORDER BY COUNT(film_id) DESC;

+------------+-----------+----------------+

| first_name | last_name | COUNT(film_id) |

+------------+-----------+----------------+

| GINA       | DEGENERES |             42 |

| WALTER     | TORN      |             41 |

+------------+-----------+----------------+

2 rows in set (0.01 sec)

You can see there are only two actors that meet the new criteria.

The HAVING clause must contain an expression or column that’s listed in the SELECT clause. In this example, we’ve used HAVING COUNT(film_id) >= 40, and you can see that COUNT(film_id) is part of the SELECT clause. Typically, the expression in the HAVING clause uses an aggregate function such as COUNT(), SUM(), MIN(), or MAX(). If you find yourself wanting to write a HAVING clause that uses a column or expression that isn’t in the SELECT clause, chances are you should be using a WHERE clause instead. The HAVING clause is only for deciding how to form each group or cluster, not for choosing rows in the output. We’ll show you an example later that illustrates when not to use HAVING.

Let’s try another example. Suppose you want a list of top five movies that were rented more than 30 times, together with the number of times they were rented ordered by popularity in reverse. Here’s the query you’d use:

mysql> SELECT title, COUNT(rental_id) AS num_rented FROM

-> film INNER JOIN inventory USING (film_id)

-> INNER JOIN rental USING (inventory_id)

-> GROUP BY title

-> HAVING num_rented > 30

-> ORDER BY num_rented DESC LIMIT 5;

+--------------------+------------+

| title              | num_rented |

+--------------------+------------+

| BUCKET BROTHERHOOD |         34 |

| ROCKETEER MOTHER   |         33 |

| FORWARD TEMPLE     |         32 |

| GRIT CLOCKWORK     |         32 |

| JUGGLER HARDLY     |         32 |

+--------------------+------------+

5 rows in set (0.04 sec)

You can again see that the expression COUNT() is used in both the SELECT and HAVING clauses. This time, though, we aliased the COUNT(rental_id) to num_rented, and used the alias in both HAVING and ORDER BY clauses.

Now let’s consider an example where you shouldn’t use HAVING. You want to know how many films a particular actor played in. Here’s the query you shouldn’t use:

mysql> SELECT first_name, last_name, COUNT(film_id) AS film_cnt FROM

-> actor INNER JOIN film_actor USING (actor_id)

-> GROUP BY actor_id, first_name, last_name

-> HAVING first_name = 'EMILY' AND last_name = 'DEE';

+------------+-----------+----------+

| first_name | last_name | film_cnt |

+------------+-----------+----------+

| EMILY      | DEE       |       14 |

+------------+-----------+----------+

1 row in set (0.02 sec)

It gets the right answer, but in the wrong — and, for large amounts of data, a much slower — way. It’s not the correct way to write the query because the HAVING clause isn’t being used to decide what rows should form each group, but is instead being incorrectly used to filter the answers to display. For this query, we should really use a WHERE clause as follows:

mysql> SELECT first_name, last_name, COUNT(film_id) AS film_cnt FROM

-> actor INNER JOIN film_actor USING (actor_id)

-> WHERE first_name = 'EMILY' AND last_name = 'DEE'

-> GROUP BY actor_id, first_name, last_name;

+------------+-----------+----------+

| first_name | last_name | film_cnt |

+------------+-----------+----------+

| EMILY      | DEE       |       14 |

+------------+-----------+----------+

1 row in set (0.00 sec)

This correct query forms the groups, and then picks which groups to display based on the WHERE clause.

Advanced Joins

So far in the book, we’ve used the INNER JOIN clause to bring together rows from two or more tables. We’ll explain the inner join in more detail in this section, contrasting it with the other join types we discuss: the union, left and right joins, and natural joins. At the conclusion of this section, you’ll be able to answer difficult information needs and be familiar with the correct choice of join for the task.

The Inner Join

The INNER JOIN clause matches rows between two tables based on the criteria you provide in the USING clause. For example, you’re very familiar now with an inner join of the actor and film_actor tables:

mysql> SELECT first_name, last_name, film_id FROM

-> actor INNER JOIN film_actor USING (actor_id)

-> LIMIT 20;

+------------+-----------+---------+

| first_name | last_name | film_id |

+------------+-----------+---------+

| PENELOPE   | GUINESS   |       1 |

| PENELOPE   | GUINESS   |      23 |

...

| PENELOPE   | GUINESS   |     980 |

| NICK       | WAHLBERG  |       3 |

+------------+-----------+---------+

20 rows in set (0.00 sec)

Let’s review the key features of an INNER JOIN:

- Two tables (or results of a previous join) are listed on either side of the INNER JOIN keyphrase.

- The USING clause defines one or more columns that are in both tables or results, and used to join or match rows.

- Rows that don’t match aren’t returned. For example, if you have a row in the actor table that doesn’t have any matching films in the film_actor table, it won’t be included in the output.

You can actually write inner-join queries with the WHERE clause without using the INNER JOIN keyphrase. Here’s a rewritten version of the previous query that produces the same result:

mysql> SELECT first_name, last_name, film_id

-> FROM actor, film_actor

-> WHERE actor.actor_id = film_actor.actor_id

-> LIMIT 20;

+------------+-----------+---------+

| first_name | last_name | film_id |

+------------+-----------+---------+

| PENELOPE   | GUINESS   |       1 |

| PENELOPE   | GUINESS   |      23 |

...

| PENELOPE   | GUINESS   |     980 |

| NICK       | WAHLBERG  |       3 |

+------------+-----------+---------+

20 rows in set (0.00 sec)

You can see that we didn’t spell out the inner join: we’re selecting from the actor and film_actor tables the rows where the identifiers match between the tables.

You can modify the INNER JOIN syntax to express the join criteria in a way that’s similar to using a WHERE clause. This is useful if the names of the identifiers don’t match between the tables, although that’s not the case in this example. Here’s the previous query, rewritten in this style:

mysql> SELECT first_name, last_name, film_id FROM

-> actor INNER JOIN film_actor

-> ON actor.actor_id = film_actor.actor_id

-> LIMIT 20;

+------------+-----------+---------+

| first_name | last_name | film_id |

+------------+-----------+---------+

| PENELOPE   | GUINESS   |       1 |

| PENELOPE   | GUINESS   |      23 |

...

| PENELOPE   | GUINESS   |     980 |

| NICK       | WAHLBERG  |       3 |

+------------+-----------+---------+

20 rows in set (0.00 sec)

You can see that the ON clause replaces the USING clause, and that the columns that follow are fully specified to include the table and column names. If the columns were named differently and uniquely between two tables, you may have skipped the table name there. There’s no real advantage or disadvantage in using ON or a WHERE clause; it’s just a matter of taste. Typically, these days, you’ll find most SQL professionals use the INNER JOIN with ON clause in preference to WHERE, but it’s not universal.

Before we move on, let’s consider what purpose the WHERE, ON, and USING clauses serve. If you omit the WHERE clause from the query we showed you, you get a very different result. Here’s the query, and the first few lines of output:

mysql> SELECT first_name, last_name, film_id

-> FROM actor, film_actor LIMIT 20;

+------------+-------------+---------+

| first_name | last_name   | film_id |

+------------+-------------+---------+

| THORA      | TEMPLE      |       1 |

| JULIA      | FAWCETT     |       1 |

...

| DEBBIE     | AKROYD      |       1 |

| MATTHEW    | CARREY      |       1 |

+------------+-------------+---------+

20 rows in set (0.00 sec)

The output is nonsensical: what’s happened is that each row from the actor table has been output alongside each row from the film_actor table, for all possible combinations. Since there are 200 actors and 5462 records in film_actor table, there are 200 × 5462 = 1092400 rows of output, and we know that only 5462 of those combinations actually make sense (there are only 5462 records for actors that played in films). We can see the number of rows we’d get without a LIMIT:

mysql> SELECT COUNT(*) FROM actor, film_actor;

+----------+

| COUNT(*) |

+----------+

|  1092400 |

+----------+

1 row in set (0.00 sec)

This type of query, without a clause that matches rows, is known as a Cartesian product. Incidentally, you also get the Cartesian product if you perform an inner join without specifying a column with a USING or ON clause, as in the query:

SELECT first_name, last_name, film_id

FROM actor INNER JOIN film_actor;

Later in “The Natural Join”, we’ll introduce the natural join, which is an inner join on identically named columns. While the natural join doesn’t use explicitly specified columns, it still produces an inner join, rather than a Cartesian product.

The keyphrase INNER JOIN can be replaced with JOIN or STRAIGHT JOIN; they all do the same thing. However, STRAIGHT JOIN forces MySQL to always read the table on the left before it reads the table on the right. We’ll have a look at how MySQL processes queries behind the scenes in Chapter 7. The keyphrase JOIN is the one you’ll see most commonly used: it’s a standard shorthand for INNER JOIN used by many other database systems besides MySQL, and we will use it in most of our inner-join examples.

The Union

The UNION statement isn’t really a join operator. Rather, it allows you to combine the output of more than one SELECT statement to give a consolidated result set. It’s useful in cases where you want to produce a single list from more than one source, or you want to create lists from a single source that are difficult to express in a single query.

Let’s look at an example. If you wanted to output all actor and movie and customer names in the sakila database, you could do this with a UNION statement. It’s a contrived example, but you might want to do this just to list all of the text fragments, rather than to meaningfully present the relationships between the data. There’s text in the actor.first_name, film.title, and customer.first_name columns in the actor, film, and customer tables, respectively. Here’s how to display it:

mysql> SELECT first_name FROM actor

-> UNION

-> SELECT first_name FROM customer

-> UNION

-> SELECT title FROM film;

+-----------------------------+

| first_name                  |

+-----------------------------+

| PENELOPE                    |

| NICK                        |

| ED                          |

...

| ZHIVAGO CORE                |

| ZOOLANDER FICTION           |

| ZORRO ARK                   |

+-----------------------------+

1647 rows in set (0.00 sec)

We’ve only shown a few of the 1647 rows. The UNION statement outputs all results from all queries together, under a heading appropriate to the first query.

A slightly less contrived example is to create a list of five top rented and five least rented movies in our database. You can do this easily with the UNION operator:

mysql> (SELECT title, COUNT(rental_id) AS num_rented

-> FROM film JOIN inventory USING (film_id)

-> JOIN rental USING (inventory_id)

-> GROUP BY title ORDER BY num_rented DESC LIMIT 5)

-> UNION

-> (SELECT title, COUNT(rental_id) AS num_rented

-> FROM film JOIN inventory USING (film_id)

-> JOIN rental USING (inventory_id)

-> GROUP BY title ORDER BY num_rented ASC LIMIT 5);

+--------------------+------------+

| title              | num_rented |

+--------------------+------------+

| BUCKET BROTHERHOOD |         34 |

| ROCKETEER MOTHER   |         33 |

| FORWARD TEMPLE     |         32 |

| GRIT CLOCKWORK     |         32 |

| JUGGLER HARDLY     |         32 |

| TRAIN BUNCH        |          4 |

| HARDLY ROBBERS     |          4 |

| MIXED DOORS        |          4 |

| BUNCH MINDS        |          5 |

| BRAVEHEART HUMAN   |          5 |

+--------------------+------------+

10 rows in set (0.04 sec)

The first query uses ORDER BY with the DESC (descending) modifier and a LIMIT 5 clause to find the top five movies rented. The second query uses ORDER BY with the ASC (ascending) modifier and a LIMIT 5 clause to find the five movies least rented. The UNION combines the result sets. Because there are multiple titles with the same num_rented value, ordering of titles with the same value in num_rented is not guaranteed to be determined. You may see different titles listed for num_rented of 32 and 5 on your end.

The UNION operator has several limitations:

- The output is labeled with the names of the columns or expressions from the first query. Use column aliases to change this behavior.

- The queries should output the same number of columns. If you try using different numbers of columns, MySQL will report an error.

- All matching columns should have the same type. So, for example, if the first column output from the first query is a date, the first column output from any other query must be a date.

- The results returned are unique, as if you’d applied a DISTINCT to the overall result set. To see this in action, let’s try a pretty simple example. Remember we had issues with actor’s names — first name is a bad unique identifier. If we select two actors with the same first name, and UNION the two queries, we will end up with just one row. The implicit DISTINCT operation hides the duplicate (for UNION) rows:

mysql> SELECT first_name FROM actor WHERE actor_id = 88

-> UNION

-> SELECT first_name FROM actor WHERE actor_id = 169;

+------------+

| first_name |

+------------+

| KENNETH    |

+------------+

1 row in set (0.01 sec)

If you want to show any duplicates, replace UNION with UNION ALL:

mysql> SELECT first_name FROM actor WHERE actor_id = 88

-> UNION ALL

-> SELECT first_name FROM actor WHERE actor_id = 169;

+------------+

| first_name |

+------------+

| KENNETH    |

| KENNETH    |

+------------+

2 rows in set (0.00 sec)

Here, the first name KENNETH appears twice.

The implicit DISTINCT that UNION performs has a non-zero cost from the performance side of things. Whenever you use UNION, see whether UNION ALL fits logically, and if it can improve query performance.

- If you want to apply LIMIT or ORDER BY to an individual query that is part of a UNION statement, enclose that query in parentheses (as shown in the previous example). It’s useful to use parentheses anyway to keep the query easy to understand.

The UNION operation simply concatenates the results of the component queries with no attention to order, so there’s not much point in using ORDER BY within one of the subqueries. The only time that it makes sense to order a subquery in a UNION operation is when you want to select a subset of results. In our example, we’ve ordered the movies by the number of times they were rented, and then selected only the top five (in the first subquery) and the bottom five (in the second subquery).

For efficiency, MySQL will actually ignore an ORDER BY clause within a subquery if it’s used without LIMIT. Let’s look at some examples to see exactly how this works.

First, let’s run a simple query to list the rental information for a particular movie, along with the time the rent happened. We’ve enclosed the query in parentheses for consistency with our other examples — the parentheses don’t actually have any effect here — and haven’t used an ORDER BY or LIMIT clause:

mysql> (SELECT title, rental_date, return_date

-> FROM film JOIN inventory USING (film_id)

-> JOIN rental USING (inventory_id)

-> WHERE film_id = 998);

+--------------+---------------------+---------------------+

| title        | rental_date         | return_date         |

+--------------+---------------------+---------------------+

| ZHIVAGO CORE | 2005-06-17 03:19:20 | 2005-06-21 00:19:20 |

| ZHIVAGO CORE | 2005-07-07 12:18:57 | 2005-07-12 09:47:57 |

| ZHIVAGO CORE | 2005-07-27 14:53:55 | 2005-07-31 19:48:55 |

| ZHIVAGO CORE | 2005-08-20 17:18:48 | 2005-08-26 15:31:48 |

| ZHIVAGO CORE | 2005-05-30 05:15:20 | 2005-06-07 00:49:20 |

| ZHIVAGO CORE | 2005-06-18 06:46:54 | 2005-06-26 09:48:54 |

| ZHIVAGO CORE | 2005-07-12 05:24:02 | 2005-07-16 03:43:02 |

| ZHIVAGO CORE | 2005-08-02 02:05:04 | 2005-08-10 21:58:04 |

| ZHIVAGO CORE | 2006-02-14 15:16:03 | NULL                |

+--------------+---------------------+---------------------+

9 rows in set (0.00 sec)

The query returns all the times the movie was rented, in no particular order (see the fourth and fifth entries).

Now, let’s add an ORDER BY clause to this query:

mysql> (SELECT title, rental_date, return_date

-> FROM film JOIN inventory USING (film_id)

-> JOIN rental USING (inventory_id)

-> WHERE film_id = 998

-> ORDER BY rental_date ASC);

+--------------+---------------------+---------------------+

| title        | rental_date         | return_date         |

+--------------+---------------------+---------------------+

| ZHIVAGO CORE | 2005-05-30 05:15:20 | 2005-06-07 00:49:20 |

| ZHIVAGO CORE | 2005-06-17 03:19:20 | 2005-06-21 00:19:20 |

| ZHIVAGO CORE | 2005-06-18 06:46:54 | 2005-06-26 09:48:54 |

| ZHIVAGO CORE | 2005-07-07 12:18:57 | 2005-07-12 09:47:57 |

| ZHIVAGO CORE | 2005-07-12 05:24:02 | 2005-07-16 03:43:02 |

| ZHIVAGO CORE | 2005-07-27 14:53:55 | 2005-07-31 19:48:55 |

| ZHIVAGO CORE | 2005-08-02 02:05:04 | 2005-08-10 21:58:04 |

| ZHIVAGO CORE | 2005-08-20 17:18:48 | 2005-08-26 15:31:48 |

| ZHIVAGO CORE | 2006-02-14 15:16:03 | NULL                |

+--------------+---------------------+---------------------+

9 rows in set (0.00 sec)

As expected, we get all the times the movie was rented, in the order of the rent date.

Adding a LIMIT clause to the previous query selects the first five rents, in chronological order — no surprises here:

mysql> (SELECT title, rental_date, return_date

-> FROM film JOIN inventory USING (film_id)

-> JOIN rental USING (inventory_id)

-> WHERE film_id = 998

-> ORDER BY rental_date ASC LIMIT 5);

+--------------+---------------------+---------------------+

| title        | rental_date         | return_date         |

+--------------+---------------------+---------------------+

| ZHIVAGO CORE | 2005-05-30 05:15:20 | 2005-06-07 00:49:20 |

| ZHIVAGO CORE | 2005-06-17 03:19:20 | 2005-06-21 00:19:20 |

| ZHIVAGO CORE | 2005-06-18 06:46:54 | 2005-06-26 09:48:54 |

| ZHIVAGO CORE | 2005-07-07 12:18:57 | 2005-07-12 09:47:57 |

| ZHIVAGO CORE | 2005-07-12 05:24:02 | 2005-07-16 03:43:02 |

+--------------+---------------------+---------------------+

5 rows in set (0.01 sec)

Now, let’s see what happens when we perform a UNION operation. In this example, we’re using two subqueries, each with an ORDER BY clause. We’ve used a LIMIT clause for the second subquery, but not for the first:

mysql> (SELECT title, rental_date, return_date

-> FROM film JOIN inventory USING (film_id)

-> JOIN rental USING (inventory_id)

-> WHERE film_id = 998

-> ORDER BY rental_date ASC)

-> UNION ALL

-> (SELECT title, rental_date, return_date

-> FROM film JOIN inventory USING (film_id)

-> JOIN rental USING (inventory_id)

-> WHERE film_id = 998

-> ORDER BY rental_date ASC LIMIT 5);

+--------------+---------------------+---------------------+

| title        | rental_date         | return_date         |

+--------------+---------------------+---------------------+

| ZHIVAGO CORE | 2005-06-17 03:19:20 | 2005-06-21 00:19:20 |

| ZHIVAGO CORE | 2005-07-07 12:18:57 | 2005-07-12 09:47:57 |

| ZHIVAGO CORE | 2005-07-27 14:53:55 | 2005-07-31 19:48:55 |

| ZHIVAGO CORE | 2005-08-20 17:18:48 | 2005-08-26 15:31:48 |

| ZHIVAGO CORE | 2005-05-30 05:15:20 | 2005-06-07 00:49:20 |

| ZHIVAGO CORE | 2005-06-18 06:46:54 | 2005-06-26 09:48:54 |

| ZHIVAGO CORE | 2005-07-12 05:24:02 | 2005-07-16 03:43:02 |

| ZHIVAGO CORE | 2005-08-02 02:05:04 | 2005-08-10 21:58:04 |

| ZHIVAGO CORE | 2006-02-14 15:16:03 | NULL                |

| ZHIVAGO CORE | 2005-05-30 05:15:20 | 2005-06-07 00:49:20 |

| ZHIVAGO CORE | 2005-06-17 03:19:20 | 2005-06-21 00:19:20 |

| ZHIVAGO CORE | 2005-06-18 06:46:54 | 2005-06-26 09:48:54 |

| ZHIVAGO CORE | 2005-07-07 12:18:57 | 2005-07-12 09:47:57 |

| ZHIVAGO CORE | 2005-07-12 05:24:02 | 2005-07-16 03:43:02 |

+--------------+---------------------+---------------------+

14 rows in set (0.01 sec)

As expected, the first subquery returns all the times the movie was rented (the first 9 rows of this output), and the second subquery returns the first 5 rentals (the last 5 rows of this output). Notice how the first 9 rows are not in order (see the fourth and fifth rows), even though the first subquery does have a ORDER BY clause. Since we’re performing a UNION operation, the MySQL server has decided that there’s no point sorting the result of the subquery. The second subquery includes a LIMIT operation, so the results of that subquery are sorted.

The output of a UNION operation isn’t guaranteed to be ordered, even if the subqueries are ordered, so if you want the final output to be ordered, you should add an ORDER BY clause at the end of the whole query. Note that it can be in another order from the subqueries. See the following:

mysql> (SELECT title, rental_date, return_date

-> FROM film JOIN inventory USING (film_id)

-> JOIN rental USING (inventory_id)

-> WHERE film_id = 998

-> ORDER BY rental_date ASC)

-> UNION ALL

-> (SELECT title, rental_date, return_date

-> FROM film JOIN inventory USING (film_id)

-> JOIN rental USING (inventory_id)

-> WHERE film_id = 998

-> ORDER BY rental_date ASC LIMIT 5)

-> ORDER BY rental_date DESC;

+--------------+---------------------+---------------------+

| title        | rental_date         | return_date         |

+--------------+---------------------+---------------------+

| ZHIVAGO CORE | 2006-02-14 15:16:03 | NULL                |

| ZHIVAGO CORE | 2005-08-20 17:18:48 | 2005-08-26 15:31:48 |

| ZHIVAGO CORE | 2005-08-02 02:05:04 | 2005-08-10 21:58:04 |

| ZHIVAGO CORE | 2005-07-27 14:53:55 | 2005-07-31 19:48:55 |

| ZHIVAGO CORE | 2005-07-12 05:24:02 | 2005-07-16 03:43:02 |

| ZHIVAGO CORE | 2005-07-12 05:24:02 | 2005-07-16 03:43:02 |

| ZHIVAGO CORE | 2005-07-07 12:18:57 | 2005-07-12 09:47:57 |

| ZHIVAGO CORE | 2005-07-07 12:18:57 | 2005-07-12 09:47:57 |

| ZHIVAGO CORE | 2005-06-18 06:46:54 | 2005-06-26 09:48:54 |

| ZHIVAGO CORE | 2005-06-18 06:46:54 | 2005-06-26 09:48:54 |

| ZHIVAGO CORE | 2005-06-17 03:19:20 | 2005-06-21 00:19:20 |

| ZHIVAGO CORE | 2005-06-17 03:19:20 | 2005-06-21 00:19:20 |

| ZHIVAGO CORE | 2005-05-30 05:15:20 | 2005-06-07 00:49:20 |

| ZHIVAGO CORE | 2005-05-30 05:15:20 | 2005-06-07 00:49:20 |

+--------------+---------------------+---------------------+

14 rows in set (0.00 sec)

Here’s another example of sorting the final results, including a limit on the number of returned results:

mysql> (SELECT first_name, last_name FROM actor WHERE actor_id < 5)

-> UNION

-> (SELECT first_name, last_name FROM actor WHERE actor_id > 190)

-> ORDER BY first_name LIMIT 4;

+------------+-----------+

| first_name | last_name |

+------------+-----------+

| BELA       | WALKEN    |

| BURT       | TEMPLE    |

| ED         | CHASE     |

| GREGORY    | GOODING   |

+------------+-----------+

4 rows in set (0.00 sec)

The UNION operation is somewhat unwieldy, and there are generally alternative ways of getting the same result. For example, the previous query could have been written more simply like this:

mysql> SELECT first_name, last_name FROM actor

-> WHERE actor_id < 5 OR actor_id > 190

-> ORDER BY first_name LIMIT 4;

+------------+-----------+

| first_name | last_name |

+------------+-----------+

| BELA       | WALKEN    |

| BURT       | TEMPLE    |

| ED         | CHASE     |

| GREGORY    | GOODING   |

+------------+-----------+

4 rows in set (0.00 sec)

The Left and Right Joins

The joins we’ve discussed so far output only rows that match between tables. For example, when you join the film and rental tables through the inventory table, you see only the films that were rented. Therefore, rows for films that haven’t been rented are ignored. This makes sense in many cases, but it isn’t the only way to join data. This section explains other options you have.

Suppose you did want a comprehensive list of all films and the number of times they’ve been rented. Unlike the example earlier in this chapter, included in the list you want to see a zero next to movies that haven’t been rented. You can do this with a left join, a different type of join that’s driven by one of the two tables participating in the join. A left join works like this: each row in the left table — the one that’s doing the driving — is processed and output, with the matching data from the second table if it exists and NULL values if there is no matching data in the second table. We’ll show you how to write this type of query later in this section, but we’ll start with a simpler example.

Here’s a simple LEFT JOIN example. You want to list all movies, and next to each movie you want to show when it was rented. If a movie has never been rented, you want to see that. If it’s been rented many times, you want to see that too. Here’s the query:

mysql> SELECT title, rental_date

-> FROM film LEFT JOIN inventory USING (film_id)

-> LEFT JOIN rental USING (inventory_id);

+-----------------------------+---------------------+

| title                       | rental_date         |

+-----------------------------+---------------------+

| ACADEMY DINOSAUR            | 2005-07-08 19:03:15 |

| ACADEMY DINOSAUR            | 2005-08-02 20:13:10 |

| ACADEMY DINOSAUR            | 2005-08-21 21:27:43 |

...

| WAKE JAWS                   | NULL                |

| WALLS ARTIST                | NULL                |

...

| ZORRO ARK                   | 2005-07-31 07:32:21 |

| ZORRO ARK                   | 2005-08-19 03:49:28 |

+-----------------------------+---------------------+

16087 rows in set (0.06 sec)

You can see what happens: movies that have been rented have dates and times, and those that haven’t don’t (the rental_date value is NULL). Let’s also discuss the fact that we LEFT JOIN twice in this example. First, we join film and inventory, and we want to make sure that even if a movie is not in our inventory (and thus cannot be rented by definition), we still output it. Then we join the rental table with the dataset resulting from prior join. We use the LEFT JOIN again, as we may have films that are not in our inventory, and those won’t have any row in rental table, too. However, we may also have films listed in our inventory that just haven’t been rented. That’s why we need to LEFT JOIN both tables here.

The order of the tables in the LEFT JOIN is important. If you reverse the order in the previous query, you get very different output:

mysql> SELECT title, rental_date

-> FROM rental LEFT JOIN inventory USING (inventory_id)

-> LEFT JOIN film USING (film_id)

-> ORDER BY rental_date DESC;

+-----------------------------+---------------------+

| title                       | rental_date         |

+-----------------------------+---------------------+

...

| LOVE SUICIDES               | 2005-05-24 23:04:41 |

| GRADUATE LORD               | 2005-05-24 23:03:39 |

| FREAKY POCUS                | 2005-05-24 22:54:33 |

| BLANKET BEVERLY             | 2005-05-24 22:53:30 |

+-----------------------------+---------------------+

16044 rows in set (0.06 sec)

In this, the query is driven by the rental table, so all rows from it are matched against the inventory, and then against film. Since all rows in rental table by definition are based on the inventory, which is linked to film, we have no NULL values in the output. There can be no rental for a film that doesn’t exist. We adjusted the query with ORDER BY rental_date DESC to show that we really didn’t get any NULL values (these would have been last).

By now you can see that left joins are useful when we’re sure that our left table has some important data, but not sure whether right table even has any. We want to get the rows from the left one with or without corresponding rows from the right one. Let’s try to apply this to a query we wrote in “The GROUP BY Clause”, which showed customers renting a lot from the same category. Here’s the query:

mysql> SELECT email, name AS category_name, COUNT(cat.category_id) AS cnt

-> FROM customer cs INNER JOIN rental USING (customer_id)

-> INNER JOIN inventory USING (inventory_id)

-> INNER JOIN film_category USING (film_id)

-> INNER JOIN category cat USING (category_id)

-> GROUP BY email, category_name

-> ORDER BY cnt DESC LIMIT 5;

+----------------------------------+---------------+-----+

| email                            | category_name | cnt |

+----------------------------------+---------------+-----+

| WESLEY.BULL@sakilacustomer.org   | Games         |   9 |

| ALMA.AUSTIN@sakilacustomer.org   | Animation     |   8 |

| KARL.SEAL@sakilacustomer.org     | Animation     |   8 |

| LYDIA.BURKE@sakilacustomer.org   | Documentary   |   8 |

| NATHAN.RUNYON@sakilacustomer.org | Animation     |   7 |

+----------------------------------+---------------+-----+

5 rows in set (0.06 sec)

What if we want now to see whether a customer we found this way is looking at anything but her favorite category. Well, it turns out that it’s actually pretty difficult!

Let’s consider this task. We need to start with a category table, as that will have all the categories we may have for our films. We then need to start constructing a whole chain of LEFT JOIN. category left joins to film_category as we may have categories having no films. Then we left join an inventory, as some movies we know about may not be in our catalog. We then left join the rental, as we may not have rented some of the films in a category. Finally, we need to left join our customer table. Even though there can be no associated customer record without a rent, omitting the left join will cause MySQL to discard rows for categories that end up with no customer records.

Now, after this whole long explanation, can we finally go ahead and filter by email and get our data? No! Unfortunately, by adding a WHERE condition on the table which is not left in our left join relationship, we break the idea of this join. See what happens:

mysql> SELECT COUNT(*) FROM category;

+----------+

| COUNT(*) |

+----------+

|       16 |

+----------+

1 row in set (0.00 sec)

mysql> SELECT email, name AS category_name, COUNT(category_id) AS cnt

-> FROM category cat LEFT JOIN film_category USING (category_id)

-> LEFT JOIN inventory USING (film_id)

-> LEFT JOIN rental USING (inventory_id)

-> JOIN customer cs ON rental.customer_id = cs.customer_id

-> WHERE cs.email = 'WESLEY.BULL@sakilacustomer.org'

-> GROUP BY email, category_name

-> ORDER BY cnt DESC;

+--------------------------------+---------------+-----+

| email                          | category_name | cnt |

+--------------------------------+---------------+-----+

| WESLEY.BULL@sakilacustomer.org | Games         |   9 |

| WESLEY.BULL@sakilacustomer.org | Foreign       |   6 |

...

| WESLEY.BULL@sakilacustomer.org | Comedy        |   1 |

| WESLEY.BULL@sakilacustomer.org | Sports        |   1 |

+--------------------------------+---------------+-----+

14 rows in set (0.00 sec)

We got 14 categories for our customer, while there are 16 in total. In fact, MySQL will optimize away all the left joins in this query, as it understands they are meaningless when put like this. There’s no easy way to answer the question we have with just joins, and we’ll need to add some more knowledge. We’ll get back to this example in “Nested Queries in JOINs”.

The query that we’ve written is still useful. While by default sakila does not have a film category that has no film rented, if we were to expand our database slightly, we can see the effectiveness of left joins.

mysql> INSERT INTO category(name) VALUES (Thriller);

Query OK, 1 row affected (0.01 sec)

mysql> SELECT cat.name, COUNT(rental_id) cnt

-> FROM category cat LEFT JOIN film_category USING (category_id)

-> LEFT JOIN inventory USING (film_id)

-> LEFT JOIN rental USING (inventory_id)

-> LEFT JOIN customer cs ON rental.customer_id = cs.customer_id

-> GROUP BY 1

-> ORDER BY 2 DESC;

+---------------+------+

| category_name | cnt  |

+---------------+------+

| Sports        | 1179 |

| Animation     | 1166 |

...

| Music         |  830 |

| Thriller      |    0 |

+---------------+------+

17 rows in set (0.07 sec)

If we were to use regular INNER JOIN (or just JOIN, its synonym), we’d not get information for “Thriller” category, and might’ve had different counts for other categories. As the category is our leftmost table, it drives the process of query, and every row from that table is present in the output.

We’ve shown you that it matters what comes before and after the LEFT JOIN statement. Whatever is on the left drives the process, hence the name “left join.” If you really don’t want to reorganize your query so it matches that template, you can use RIGHT JOIN. It’s exactly the same, except whatever is on the right drives the process. Earlier we showed the importantce of order of the tables in the LEFT JOIN using two queries for film rental info. Let’s rewrite the second of them (which showed incorrect data) using a right join:

mysql> SELECT title, rental_date

-> FROM rental RIGHT JOIN inventory USING (inventory_id)

-> RIGHT JOIN film USING (film_id)

-> ORDER BY rental_date DESC;

...

| SUICIDES SILENCE            | NULL                |

| TADPOLE PARK                | NULL                |

| TREASURE COMMAND            | NULL                |

| VILLAIN DESPERATE           | NULL                |

| VOLUME HOUSE                | NULL                |

| WAKE JAWS                   | NULL                |

| WALLS ARTIST                | NULL                |

+-----------------------------+---------------------+

16087 rows in set (0.06 sec)

We got the same amount of rows and see the NULL values are the same as the “correct” query gave us. The right join is useful sometimes because it allows you to write a query more naturally, expressing it in a way that’s more intuitive. However, you won’t often see it used, and we’d recommend avoiding it where possible.

Both the LEFT JOIN and RIGHT JOIN can use either the USING or ON clauses discussed for the INNER JOIN earlier in this chapter in “The Inner Join”. You should use one or the other: without them, you’ll get the Cartesian product also discussed in “The Inner Join”.

There’s an extra OUTER keyword that you can optionally use in left and right joins, to make them read as LEFT OUTER JOIN and RIGHT OUTER JOIN. It’s just an alternative syntax that doesn’t do anything different, and you won’t often see it used. We stick to the basic versions in this book.

The Natural Join

We’re not big fans of the natural join that we’re about to describe in this section. It’s in here only for completeness and because you’ll see it used sometimes in SQL statements you’ll encounter. Our advice is to avoid using it where possible.

A natural join is, well, supposed to be magically natural. This means that you tell MySQL what tables you want to join, and it figures out how to do it and gives you an INNER JOIN result set. Here’s an example for the actor_info and film_actor tables:

mysql> SELECT first_name, last_name, film_id

-> FROM actor_info NATURAL JOIN film_actor

-> LIMIT 20;

+------------+-----------+---------+

| first_name | last_name | film_id |

+------------+-----------+---------+

| PENELOPE   | GUINESS   |       1 |

| PENELOPE   | GUINESS   |      23 |

...

| PENELOPE   | GUINESS   |     980 |

| NICK       | WAHLBERG  |       3 |

+------------+-----------+---------+

20 rows in set (0.28 sec)

In reality, it’s not quite magical: all MySQL does is look for columns with the same names and, behind the scenes, adds these silently into an inner join with join conditions written into the where clause. So, the above query is actually translated into something like this:

mysql> SELECT first_name, last_name, film_id FROM

-> actor_info JOIN film_actor

-> WHERE (actor_info.actor_id = film_actor.actor_id)

-> LIMIT 20;

If identifier columns don’t share the same name, natural joins won’t work. Also, more dangerously, if columns that do share the same names aren’t identifiers, they’ll get thrown into the behind-the-scenes USING clause anyway. You can very easily see this in the sakila database. In fact, that’s why we resorted to showing the above example with actor_info, which isn’t even a table: it’s a view. Let’s see what would have happened if we used regular actor and film_actor tables.

mysql> SELECT first_name, last_name, film_id FROM actor NATURAL JOIN film_actor;

Empty set (0.01 sec)

But how? The problem is: NATURAL JOIN really does take ALL of the columns into consideration. With the sakila database, that’s a huge roadblock, as every table has a last_update column. If you were to run an EXPLAIN statement on the above query, and then execute SHOW WARNINGS, you’d see that the resulting query is meaningless. See below:

mysql> SHOW WARNINGS\G

*************************** 1. row ***************************

Level: Note

Code: 1003

Message: /* select#1 */ select `sakila`.`customer`.`email` AS `email`,

`sakila`.`rental`.`rental_date` AS `rental_date`

from `sakila`.`customer` join `sakila`.`rental`

where ((`sakila`.`rental`.`last_update` = `sakila`.`customer`.`last_update`)

and (`sakila`.`rental`.`customer_id` = `sakila`.`customer`.`customer_id`))

1 row in set (0.00 sec)

You’ll sometimes see the natural join mixed with left and right joins. The following are valid join syntaxes: NATURAL LEFT JOIN, NATURAL LEFT OUTER JOIN, NATURAL RIGHT JOIN, and NATURAL RIGHT OUTER JOIN. The former two are left joins without ON or USING clauses, and the latter two are right joins. Again, avoid writing them when you can, but you should understand what they mean if you see them used.

Constant expressions in joins

In all of the examples of the joins we gave you so far, we always used column identifiers to define the join condition. When you’re using the USING clause, that’s the only possible way to go. When you’re defininig the join conditions in WHERE, that’s also the only thing that will work. However, when you’re using the ON clause, you can actually add constant expressions.

Let’s consider an example with our films and actors again, listing all films for a particular actor.

mysql> SELECT first_name, last_name, title

-> FROM actor JOIN film_actor USING (actor_id)

-> JOIN film USING (film_id)

-> WHERE actor_id = 11;

+------------+-----------+--------------------+

| first_name | last_name | title              |

+------------+-----------+--------------------+

| ZERO       | CAGE      | CANYON STOCK       |

| ZERO       | CAGE      | DANCES NONE        |

...

| ZERO       | CAGE      | WEST LION          |

| ZERO       | CAGE      | WORKER TARZAN      |

+------------+-----------+--------------------+

25 rows in set (0.00 sec)

What we can do, however, is move the actor_id clause to the join like this:

mysql> SELECT first_name, last_name, title

-> FROM actor JOIN film_actor

->   ON actor.actor_id = film_actor.actor_id

->   AND actor.actor_id = 11

-> JOIN film USING (film_id);

+------------+-----------+--------------------+

| first_name | last_name | title              |

+------------+-----------+--------------------+

| ZERO       | CAGE      | CANYON STOCK       |

| ZERO       | CAGE      | DANCES NONE        |

...

| ZERO       | CAGE      | WEST LION          |

| ZERO       | CAGE      | WORKER TARZAN      |

+------------+-----------+--------------------+

25 rows in set (0.00 sec)

Well, that’s neat, of course, but why? Is this any more expressive than having the proper WHERE clause? The answer to both questions is that constant conditions in joins are evaluated and resolved differently than the conditions in WHERE clause are. It’s easier to show this with an example, but the query above is a poor one. The impact of constant conditions in joins is best shown with a left join.

Remember this query from our left join chapter:

mysql> SELECT email, name AS category_name, COUNT(rental_id) AS cnt

-> FROM category cat LEFT JOIN film_category USING (category_id)

-> LEFT JOIN inventory USING (film_id)

-> LEFT JOIN rental USING (inventory_id)

-> LEFT JOIN customer cs USING (customer_id)

-> WHERE cs.email = 'WESLEY.BULL@sakilacustomer.org'

-> GROUP BY email, category_name

-> ORDER BY cnt DESC;

+--------------------------------+---------------+-----+

| email                          | category_name | cnt |

+--------------------------------+---------------+-----+

| WESLEY.BULL@sakilacustomer.org | Games         |   9 |

| WESLEY.BULL@sakilacustomer.org | Foreign       |   6 |

...

| WESLEY.BULL@sakilacustomer.org | Comedy        |   1 |

| WESLEY.BULL@sakilacustomer.org | Sports        |   1 |

+--------------------------------+---------------+-----+

14 rows in set (0.01 sec)

If we go ahead and move the cs.email clause to the LEFT JOIN customer cs part, we’ll see completely different results:

mysql> SELECT email, name AS category_name, COUNT(rental_id) AS cnt

-> FROM category cat LEFT JOIN film_category USING (category_id)

-> LEFT JOIN inventory USING (film_id)

-> LEFT JOIN rental USING (inventory_id)

-> LEFT JOIN customer cs ON rental.customer_id = cs.customer_id

-> AND cs.email = 'WESLEY.BULL@sakilacustomer.org'

-> GROUP BY email, category_name

-> ORDER BY cnt DESC;

+--------------------------------+-------------+------+

| email                          | name        | cnt  |

+--------------------------------+-------------+------+

| NULL                           | Sports      | 1178 |

| NULL                           | Animation   | 1164 |

...

| NULL                           | Travel      |  834 |

| NULL                           | Music       |  829 |

| WESLEY.BULL@sakilacustomer.org | Games       |    9 |

| WESLEY.BULL@sakilacustomer.org | Foreign     |    6 |

...

| WESLEY.BULL@sakilacustomer.org | Comedy      |    1 |

| NULL                           | Thriller    |    0 |

+--------------------------------+-------------+------+

31 rows in set (0.07 sec)

That’s interesting! Instead of getting only Wesley’s rental counts per category, we also get rental counts for everyone else broken down by category. That even includes our new and so far empty “Thriller” category. Let’s try to understand what happens here.

The WHERE clause contents are applied logically after the joins were resolved and executed. We tell MySQL we only need rows from whatever we join where the cs.email column equals 'WESLEY.BULL@sakilacustomer.org'. In reality, MySQL is smart enough to optimize this situation and will actually start the plan execution as if regular inner joins were used. When we have the cs.email condition within the LEFT JOIN customer clause, we tell MySQL that we want to add columns from customer table to our resultset so far (which includes category, inventory, and rental tables), but only when the certain value of email column is met. Since this is a LEFT JOIN, we get NULL in every column of customer in rows which didn’t match.

It’s important to mind this behavior.

Nested Queries

Nested queries — supported by MySQL since version 4.1 — are the most difficult to learn. However, they provide a powerful, useful, and concise way of expressing difficult information needs in short SQL statements. This section explains them, beginning with simple examples and leading to the more complex features of the EXISTS and IN statements. At the conclusion of this section, you’ll have completed everything this book contains about querying data, and you should be comfortable understanding almost any SQL query you encounter.

Nested Query Basics

You know how to find names of all actors who played in a particular movie using an INNER JOIN:

mysql> SELECT first_name, last_name FROM

-> actor JOIN film_actor USING (actor_id)

-> JOIN film USING (film_id)

-> WHERE title = 'ZHIVAGO CORE';

+------------+-----------+

| first_name | last_name |

+------------+-----------+

| UMA        | WOOD      |

| NICK       | STALLONE  |

| GARY       | PENN      |

| SALMA      | NOLTE     |

| KENNETH    | HOFFMAN   |

| WILLIAM    | HACKMAN   |

+------------+-----------+

6 rows in set (0.00 sec)

But there’s another way, using a nested query:

mysql> SELECT first_name, last_name FROM

-> actor JOIN film_actor USING (actor_id)

-> WHERE film_id = (SELECT film_id FROM film

-> WHERE title = 'ZHIVAGO CORE');

+------------+-----------+

| first_name | last_name |

+------------+-----------+

| UMA        | WOOD      |

| NICK       | STALLONE  |

| GARY       | PENN      |

| SALMA      | NOLTE     |

| KENNETH    | HOFFMAN   |

| WILLIAM    | HACKMAN   |

+------------+-----------+

6 rows in set (0.00 sec)

It’s called a nested query because one query is inside another. The inner query, or subquery — the one that is nested — is written in parentheses, and you can see that it determines the film_id for the film with the title ZHIVAGO CORE. The parentheses are required for inner queries. The outer query is the one that’s listed first and isn’t parenthesized here: you can see that it finds the first_name and last_name of the the actors from a JOIN with film_actor with an film_id that matches the result of the subquery. So, overall, the inner query finds the film_id, and the outer query uses it to find actor’s names. Whenever nested queries are used, it’s possible to rewrite them as a few separate queries, and we’ll break down the example above, as it may help understand what is going on:

mysql> SELECT film_id FROM film WHERE title = 'ZHIVAGO CORE';

+---------+

| film_id |

+---------+

|     998 |

+---------+

1 row in set (0.03 sec)

mysql> SELECT first_name, last_name

-> FROM actor JOIN film_actor USING (actor_id)

-> WHERE film_id = 998;

+------------+-----------+

| first_name | last_name |

+------------+-----------+

| UMA        | WOOD      |

| NICK       | STALLONE  |

| GARY       | PENN      |

| SALMA      | NOLTE     |

| KENNETH    | HOFFMAN   |

| WILLIAM    | HACKMAN   |

+------------+-----------+

6 rows in set (0.00 sec)

So, which approach is preferable: nested or not nested? The answer isn’t easy. In terms of performance, the answer is usually not: nested queries are hard to optimize, and so they’re almost always slower to run than the unnested alternative.

Does this mean you should avoid nesting? The answer is no: sometimes it’s your only choice if you want to write a single query, and sometimes nested queries can answer information needs that can’t be easily solved otherwise. What’s more, nested queries are expressive. Once you’re comfortable with the idea, they’re a very readable way to show how a query is evaluated. In fact, many SQL designers advocate teaching nested queries before the join-based alternatives we’ve shown you in the past few chapters. We’ll show you examples of where nesting is readable and powerful throughout this section.

Before we begin to cover the keywords that can be used in nested queries, let’s visit an example that can’t be done easily in a single query — at least, not without MySQL’s non-standard, although ubiquitous, LIMIT clause! Suppose you want to know what movie a customer rented most recently. To do this, following the methods we’ve learned previously, you could find the date and time of the most recently stored row in the rental table:

mysql> SELECT MAX(rental_date) FROM rental

-> JOIN customer USING (customer_id)

-> WHERE email = 'WESLEY.BULL@sakilacustomer.org';

+---------------------+

| MAX(rental_date)    |

+---------------------+

| 2005-08-23 15:46:33 |

+---------------------+

1 row in set (0.01 sec)

You can then use the output as input to another query to find the film title:

mysql> SELECT title FROM film

-> JOIN inventory USING (film_id)

-> JOIN rental USING (inventory_id)

-> JOIN customer USING (customer_id)

-> WHERE email = 'WESLEY.BULL@sakilacustomer.org'

-> AND rental_date = '2005-08-23 15:46:33';

+-------------+

| title       |

+-------------+

| KARATE MOON |

+-------------+

1 row in set (0.00 sec)

In “User Variables”, later in this chapter, we’ll show how you can use variables to avoid having to type in the value in the second query.

With a nested query, you can do both steps in one shot:

mysql> SELECT title FROM film JOIN inventory USING (film_id)

-> JOIN rental USING (inventory_id)

-> WHERE rental_date = (SELECT MAX(rental_date) FROM rental

-> JOIN customer USING (customer_id)

-> WHERE email = 'WESLEY.BULL@sakilacustomer.org');

+-------------+

| title       |

+-------------+

| KARATE MOON |

+-------------+

1 row in set (0.01 sec)

You can see the nested query combines the two previous queries. Rather than using the constant date and time value discovered from a previous query, it executes the query directly as a subquery. This is the simplest type of nested query, one that returns a scalar operand — that is, a single value.

The previous example used the equality operator, the equals sign, =. You can use all types of comparison operators: < (less than), <= (less than or equal to), > (greater than), >= (greater than or equal to), and != (not equals) or <> (not equals).

The ANY, SOME, ALL, IN, and NOT IN Clauses

Before we start to show some more advanced features of nested queries, we need to switch to a new database in our examples. Unfortunately, our sakila database is a little too well normalized to effectively demonstrate the full power of nested querying. So, let’s add a new database to give us something to play with.

The database we’ll install is the employees sample database. You can find instructions for installation in the MySQL documentation https://dev.mysql.com/doc/employee/en/employees-installation.html or in the database’s github repo over at https://github.com/datacharmer/test_db Either clone the repository using git, or download the latest release (1.0.7 at the time of writing https://github.com/datacharmer/test_db/releases/tag/v1.0.7). Once you have the necessary files ready, we need to run two commands.

First command creates necessary structures and loads the data.

$ mysql -uroot -p < employees.sql

INFO

CREATING DATABASE STRUCTURE

INFO

storage engine: InnoDB

INFO

LOADING departments

INFO

LOADING employees

INFO

LOADING dept_emp

INFO

LOADING dept_manager

INFO

LOADING titles

INFO

LOADING salaries

data_load_time_diff

00:00:28

Second command verifies the installation.

$ mysql -uroot -p < test_employees_md5.sql

INFO

TESTING INSTALLATION

table_name      expected_records        expected_crc

departments     9       d1af5e170d2d1591d776d5638d71fc5f

dept_emp        331603  ccf6fe516f990bdaa49713fc478701b7

dept_manager    24      8720e2f0853ac9096b689c14664f847e

employees       300024  4ec56ab5ba37218d187cf6ab09ce1aa1

salaries        2844047 fd220654e95aea1b169624ffe3fca934

titles  443308  bfa016c472df68e70a03facafa1bc0a8

table_name      found_records           found_crc

departments     9       d1af5e170d2d1591d776d5638d71fc5f

dept_emp        331603  ccf6fe516f990bdaa49713fc478701b7

dept_manager    24      8720e2f0853ac9096b689c14664f847e

employees       300024  4ec56ab5ba37218d187cf6ab09ce1aa1

salaries        2844047 fd220654e95aea1b169624ffe3fca934

titles  443308  bfa016c472df68e70a03facafa1bc0a8

table_name      records_match   crc_match

departments     OK      ok

dept_emp        OK      ok

dept_manager    OK      ok

employees       OK      ok

salaries        OK      ok

titles  OK      ok

computation_time

00:00:25

summary result

CRC     OK

count   OK

Once this is done, you can proceed to work through the examples we’ll be providing next.

To connect to the new database, either run mysql from the command line like this (or specify employees as a target for your MySQL client of choice):

$ mysql employees

Or execute the following in a mysql prompt to change the default database:

mysql> use employees

Now you’re ready to move forward.

Using ANY and IN

Now that you’ve created the sample tables, you can try an example using ANY. Suppose you’re looking to find assistant engineers who’ve been working longer than the least experienced manager. You can express this information need as follows:

mysql> SELECT emp_no, first_name, last_name, hire_date

-> FROM employees JOIN titles USING (emp_no)

-> WHERE title = 'Assistant Engineer'

-> AND hire_date < ANY (SELECT hire_date FROM

-> employees JOIN titles USING (emp_no)

-> WHERE title = 'Manager');

+--------+----------------+------------------+------------+

| emp_no | first_name     | last_name        | hire_date  |

+--------+----------------+------------------+------------+

|  10009 | Sumant         | Peac             | 1985-02-18 |

|  10066 | Kwee           | Schusler         | 1986-02-26 |

...

...

| 499958 | Srinidhi       | Theuretzbacher   | 1989-12-17 |

| 499974 | Shuichi        | Piazza           | 1989-09-16 |

+--------+----------------+------------------+------------+

10747 rows in set (0.20 sec)

Turns out, there are a lot of people falling under this criteria! The subquery finds the dates on which managers were hired:

mysql> SELECT hire_date FROM

-> employees JOIN titles USING (emp_no)

-> WHERE title = 'Manager';

+------------+

| hire_date  |

+------------+

| 1985-01-01 |

| 1986-04-12 |

...

| 1991-08-17 |

| 1989-07-10 |

+------------+

24 rows in set (0.10 sec)

The outer query goes through each employee with the title Associate Engineer, returning the engineer if their hire date is lower (older) than any of the values in the set returned by the subquery. So, for example, Sumant Peac is output because 1985-02-18 is older than at least one value in the set. For example, the second hire date returned for managers is 1986-04-12. The ANY keyword means just that: it’s true if the column or expression preceding it is true for any of the values in the set returned by the subquery. Used in this way, ANY has the alias SOME, which was included so that some queries can be read more clearly as English expressions; it doesn’t do anything different and you’ll rarely see it used.

The ANY keyword gives you more power in expressing nested queries. Indeed, the previous query is the first nested query in this section with a column subquery — that is, the results returned by the subquery are one or more values from a column, instead of a single scalar value as in the previous section. With this, you can now compare a column value from an outer query to a set of values returned from a subquery.

Consider another example using ANY. Suppose you want to know the managers who also have some other title. You can do this with the following nested query:

mysql> SELECT emp_no, first_name, last_name

-> FROM employees JOIN titles USING (emp_no)

-> WHERE title = 'Manager'

-> AND emp_no = ANY (SELECT emp_no FROM employees

-> JOIN titles USING (emp_no) WHERE

-> title <> 'Manager');

+--------+-------------+--------------+

| emp_no | first_name  | last_name    |

+--------+-------------+--------------+

| 110022 | Margareta   | Markovitch   |

| 110039 | Vishwani    | Minakawa     |

...

| 111877 | Xiaobin     | Spinelli     |

| 111939 | Yuchang     | Weedman      |

+--------+-------------+--------------+

24 rows in set (0.11 sec)

The = ANY causes the outer query to return a manager when the emp_no is equal to any of the engineer employee numbers returned by the subquery. The = ANY keyphrase has the alias IN, which you’ll see commonly used in nested queries. Using IN, the previous example can be rewritten:

mysql> SELECT emp_no, first_name, last_name

-> FROM employees JOIN titles USING (emp_no)

-> WHERE title = 'Manager'

-> AND emp_no IN (SELECT emp_no FROM employees

-> JOIN titles USING (emp_no) WHERE

-> title <> 'Manager');

+--------+-------------+--------------+

| emp_no | first_name  | last_name    |

+--------+-------------+--------------+

| 110022 | Margareta   | Markovitch   |

| 110039 | Vishwani    | Minakawa     |

...

| 111877 | Xiaobin     | Spinelli     |

| 111939 | Yuchang     | Weedman      |

+--------+-------------+--------------+

24 rows in set (0.11 sec)

Of course, for this particular example, you could also have used a join query. Note that we have to use DISTINCT here, because otherwise we get 30 rows returned. Some people hold more than one non-engineer title.

mysql> SELECT DISTINCT emp_no, first_name, last_name

-> FROM employees JOIN titles mgr USING (emp_no)

-> JOIN titles nonmgr USING (emp_no)

-> WHERE mgr.title = 'Manager'

-> AND nonmgr.title <> 'Manager';

+--------+-------------+--------------+

| emp_no | first_name  | last_name    |

+--------+-------------+--------------+

| 110022 | Margareta   | Markovitch   |

| 110039 | Vishwani    | Minakawa     |

...

| 111877 | Xiaobin     | Spinelli     |

| 111939 | Yuchang     | Weedman      |

+--------+-------------+--------------+

24 rows in set (0.11 sec)

Again, nested queries are expressive but typically slow in MySQL, so use a join where you can.

Using ALL

Suppose you want to find assistant engineers who are more experienced than all of the managers — that is, more experienced than the most experienced manager. You can do this with the ALL keyword in place of ANY:

mysql> SELECT emp_no, first_name, last_name, hire_date

-> FROM employees JOIN titles USING (emp_no)

-> WHERE title = 'Assistant Engineer'

-> AND hire_date < ALL (SELECT hire_date FROM

-> employees JOIN titles USING (emp_no)

-> WHERE title = 'Manager');

Empty set (0.18 sec)

You can see that there are no answers. We can inspect the data further to check what is the oldest hire date of a manager, and of an assistant engineer:

mysql> (SELECT 'Assistant Engineer' AS title,

-> MIN(hire_date) AS mhd FROM employees

-> JOIN titles USING (emp_no)

-> WHERE title = 'Assistant Engineer')

-> UNION

-> (SELECT 'Manager' title, MIN(hire_date) mhd FROM employees

-> JOIN titles USING (emp_no)

-> WHERE title = 'Manager');

+--------------------+------------+

| title              | mhd        |

+--------------------+------------+

| Assistant Engineer | 1985-02-01 |

| Manager            | 1985-01-01 |

+--------------------+------------+

2 rows in set (0.26 sec)

Looking at the data, we see that the first manager was hired on January first, 1985, and the first assistant engineer only on February first, same year. While the ANY keyword returns values that satisfy at least one condition (Boolean OR), the ALL keyword returns values when all the conditions are satisfied (Boolean AND).

We can use the alias NOT IN in place of <> ANY or != ANY. Let’s find all the managers who aren’t senior staff:

mysql> SELECT emp_no, first_name, last_name

-> FROM employees JOIN titles USING (emp_no)

-> WHERE title = 'Manager' AND emp_no NOT IN

-> (SELECT emp_no FROM titles

-> WHERE title = 'Senior Staff');

+--------+-------------+--------------+

| emp_no | first_name  | last_name    |

+--------+-------------+--------------+

| 110183 | Shirish     | Ossenbruggen |

| 110303 | Krassimir   | Wegerle      |

...

| 111400 | Arie        | Staelin      |

| 111692 | Tonny       | Butterworth  |

+--------+-------------+--------------+

15 rows in set (0.09 sec)

As an exercise, try writing the above query using the ANY syntax and as a join query.

The ALL keyword has a few tricks and traps:

- If it’s false for any value, it’s false. Suppose that table a contains a row with the value 14. Suppose table b contains the values 16, 1, and NULL. If you check whether the value in a is greater than ALL values in b, you’ll get false, since 14 isn’t greater than 16. It doesn’t matter that the other values are 1 and NULL.

- If it isn’t false for any value, it isn’t true unless it’s true for all values. Suppose that table a again contains 14, and suppose b contains 1 and NULL. If you check whether the value in a is greater than ALL values in b, you’ll get UNKNOWN (neither true or false) because it can’t be determined whether NULL is greater than or less than 14.

- If the table in the subquery is empty, the result is always true. Hence, if a contains 14 and b is empty, you’ll get true when you check if the value in a is greater than ALL values in b.

When using the ALL keyword, be very careful with tables that can have NULL values in columns; consider disallowing NULL values in such cases. Also, be careful with empty tables.

Writing row subqueries

In the previous examples, the subquery returned a single, scalar value (such as an actor_id) or a set of values from one column (such as all of the emp_no values). This section describes another type of subquery, the row subquery that works with multiple columns from multiple rows.

Suppose you’re interested in whether a manager had another position within the same calendar year. To answer this need, you must match both employee number and the title assignment date, or, more precisely, year. You can write this as a join:

mysql> SELECT mgr.emp_no, YEAR(mgr.from_date) AS fd

-> FROM titles AS mgr, titles AS other

-> WHERE mgr.emp_no = other.emp_no

-> AND mgr.title = 'Manager'

-> AND mgr.title <> other.title

-> AND YEAR(mgr.from_date) = YEAR(other.from_date);

+--------+------+

| emp_no | fd   |

+--------+------+

| 110765 | 1989 |

| 111784 | 1988 |

+--------+------+

2 rows in set (0.11 sec)

But you can also write it as a nested query:

mysql> SELECT emp_no, YEAR(from_date) AS fd

-> FROM titles WHERE title = 'Manager' AND

-> (emp_no, YEAR(from_date)) IN

-> (SELECT emp_no, YEAR(from_date)

-> FROM titles WHERE title <> 'Manager');

+--------+------+

| emp_no | fd   |

+--------+------+

| 110765 | 1989 |

| 111784 | 1988 |

+--------+------+

2 rows in set (0.12 sec)

You can see there’s a different syntax being used in this nested query: a list of two column names in parentheses follows the WHERE statement, and the inner query returns two columns. We’ll explain this syntax next.

The row subquery syntax allows you to compare multiple values per row. The expression (emp_no, YEAR(from_date)) means two values per row are compared to the output of the subquery. You can see following the IN keyword that the subquery returns two values, emp_no and YEAR(from_date). So, the fragment:

(emp_no, YEAR(from_date)) IN (SELECT emp_no, YEAR(from_date)

FROM titles WHERE title <> 'Manager')

matches manager numbers and starting years to non managers numbers and starting years, and returns a true value when a match is found. The result is that if a matching pair is found, the overall query outputs a result. This is a typical row subquery: it finds rows that exist in two tables.

To explain the syntax further, let’s consider another example. Suppose you want to see if a particular employee is a senior staff member. You can do this with the following query:

mysql> SELECT first_name, last_name

-> FROM employees, titles

-> WHERE (employees.emp_no, first_name, last_name, title) =

-> (titles.emp_no, 'Marjo', 'Giarratana', 'Senior Staff');

+------------+------------+

| first_name | last_name  |

+------------+------------+

| Marjo      | Giarratana |

+------------+------------+

1 row in set (0.09 sec)

It’s not a nested query, but it shows you how the new row subquery syntax works. You can see that the query matches the list of columns before the equals sign, (employees.emp_no, first_name, last_name, title), to the list of columns and values after the equals sign, (titles.emp_no, 'Marjo', 'Giarratana', 'Senior Staff'). So, when the emp_no values match, the employee’s full name is Marjo Giarratana, and the title is Senior Staff, we get output from the query. We don’t recommend writing queries like this—use a regular WHERE clause with multiple AND conditions instead—but it does illustrate exactly what’s going on. For an exercise, try writing this query using a join.

Row subqueries require that the number, order, and type of values in the columns match. So, for example, our previous example matches an INT to an INT, and two character strings to two character strings.

The EXISTS and NOT EXISTS Clauses

You’ve now seen three types of subquery: scalar subqueries, column subqueries, and row subqueries. In this section, you’ll learn about a fourth type, the correlated subquery, where a table used in the outer query is referenced in the subquery. Correlated subqueries are often used with the IN statement we’ve already discussed, and almost always used with the EXISTS and NOT EXISTS clauses that are the focus of this section.

EXISTS and NOT EXISTS basics

Before we start on our discussion of correlated subqueries, let’s investigate what the EXISTS clause does. We’ll need a simple but strange example to introduce the clause, since we’re not discussing correlated subqueries just yet. So, here goes: suppose you want to find a count of all films in the database, but only if the database is active (which you’ve defined to mean only if at least one movie from any branch has been rented). Here’s the query that does it. Do not forget to connect to sakila database again (hint: use the use <db> command).

mysql> SELECT COUNT(*) FROM film

-> WHERE EXISTS (SELECT * FROM rental);

+----------+

| COUNT(*) |

+----------+

|     1000 |

+----------+

1 row in set (0.01 sec)

The subquery returns all rows from the rental table. However, what’s important is that it returns at least one row; it doesn’t matter what’s in the row, how many rows there are, or whether the row contains only NULL values. So, you can think of the subquery as being true or false, and in this case it’s true because it produces some output. When the subquery is true, the outer query that uses the EXISTS clause returns a row. The overall result is that all rows in the film table are counted because, for each one, the subquery is true.

Let’s try a query where the subquery isn’t true. Again, let’s contrive a query: this time, we’ll output the names of all films in the database, but only if a particular film exists. Here’s the query:

mysql> SELECT title FROM film

-> WHERE EXISTS (SELECT * FROM film

-> WHERE title = 'IS THIS A MOVIE?');

Empty set (0.00 sec)

Since the subquery isn’t true — no rows are returned because IS THIS A MOVIE? isn’t in our database — no results are returned by the outer query.

The NOT EXISTS clause does the opposite. Imagine you want a list of all actors if you don’t have a particular movie in the database. Here it is:

mysql> SELECT * FROM actor WHERE NOT EXISTS

-> (SELECT * FROM film WHERE title = 'ZHIVAGO CORE');

Empty set (0.00 sec)

This time, the inner query is true but the NOT EXISTS clause negates it to give false. Since it’s false, the outer query doesn’t produce results.

You’ll notice that the subquery begins with SELECT * FROM film. It doesn’t actually matter what you select in an inner query when you’re using the EXISTS clause, since it’s not used by the outer query anyway. You can select one column, everything, or even a constant (as in SELECT 'cat' from film), and it’ll have the same effect. Traditionally, though, you’ll see most SQL authors write SELECT * by convention.

Correlated subqueries

So far, it’s difficult to imagine what you’d do with the EXISTS or NOT EXISTS clauses. This section shows you how they’re really used, illustrating the most advanced type of nested query that you’ll typically see in action.

Let’s think about the realistic information you might want to answer from the sakila database. Suppose you want a list of all employees who’ve rented something from our company, or are just a customer. You can do this easily with a join query, which we recommend you try to think about before you continue. You can also do it with the following nested query that uses a correlated subquery:

mysql> SELECT first_name, last_name FROM staff

-> WHERE EXISTS (SELECT * FROM customer

-> WHERE customer.first_name = staff.first_name

-> AND customer.last_name = staff.last_name);

Empty set (0.01 sec)

There’s no output because nobody from the staff is also a customer (or that’s forbidden, but we’ll bend rules). Let’s add a customer with the same details as one of the staff members.

mysql> INSERT INTO customer(store_id, first_name, last_name,

-> email, address_id, create_date)

-> VALUES (1, 'Mike', 'Hillyer',

-> 'Mike.Hillyer@sakilastaff.com', 3, NOW());

Query OK, 1 row affected (0.02 sec)

Now the query:

mysql> SELECT first_name, last_name FROM staff

-> WHERE EXISTS (SELECT * FROM customer

-> WHERE customer.first_name = staff.first_name

-> AND customer.last_name = staff.last_name);

+------------+-----------+

| first_name | last_name |

+------------+-----------+

| Mike       | Hillyer   |

+------------+-----------+

1 row in set (0.00 sec)

So, the query works; now, we just need to understand how!

Let’s examine the subquery in our previous example. You can see that it lists only the customer table in the FROM clause, but it uses a column from the staff table in the WHERE clause. If you run it in isolation, you’ll see this isn’t allowed:

mysql> SELECT * FROM customer WHERE customer.first_name = staff.first_name;

ERROR 1054 (42S22): Unknown column 'staff.first_name' in 'where clause'

However, it’s legal when executed as a subquery because tables listed in the outer query are allowed to be accessed in the subquery. So, in this example, the current value of staff.first_name and staff.last_name in the outer query is supplied to the subquery as a constant, scalar value and compared to the customer’s first and last names. If the customer’s name matches the staff member’s name, the subquery is true, and so the outer query outputs a row. Consider two cases that illustrate this more clearly:

- When the first_name and last_name being processed by the outer query are Jon and Stephens, the subquery is false because SELECT * FROM customer WHERE first_name = 'Jon' and last_name = 'Stephens'; doesn’t return any rows, and so the staff row for Jon Stephens isn’t output as an answer.

- When the first_name and last_name being processed by the outer query are Mike and Hillyer, the subquery is true because SELECT * FROM customer WHERE first_name = 'Mike' and last_name = 'Hillyer'; returns at least one row. Overall, the staff row for Mike is returned.

Can you see the power of correlated subqueries? You can use values from the outer query in the inner query to evaluate complex information needs.

We’ll now explore another example using EXISTS. Let’s try to find a count of all films from whom we own at least two copies. To do this with EXISTS, we need to think through what the inner and outer queries should do. The inner query should produce a result only when the condition we’re checking is true; in this case, it should produce output when the are at least two rows in the inventory for the same film. The outer query should increment the counter whenever the inner query is true. Here’s the query:

mysql> SELECT COUNT(*) FROM film WHERE EXISTS

-> (SELECT film_id FROM inventory

-> WHERE inventory.film_id = film.film_id

-> GROUP BY film_id HAVING COUNT(*) >= 2);

+----------+

| COUNT(*) |

+----------+

|      958 |

+----------+

1 row in set (0.00 sec)

This is yet another query where nesting isn’t necessary and a join would suffice, but let’s stick with this version for the purpose of explanation. Have a look at the inner query: you can see that the WHERE clause ensures that films match by the unique film_id, and only matching rows — for the current film — are considered by the subquery. The GROUP BY clause clusters the rows for that film, but only if there are at least two entries in the inventory. Therefore, the inner query only produces output when there are at least two rows for the current film in our inventory. The outer query is straightforward: it can be thought of as incrementing a counter when the subquery produces output.

Here’s one more example before we move on and discuss other issues. This example will be in the employees database, so switch your client. We’ve already shown you a query that uses IN and finds managers who also had some other position:

mysql> SELECT emp_no, first_name, last_name

-> FROM employees JOIN titles USING (emp_no)

-> WHERE title = 'Manager'

-> AND emp_no IN (SELECT emp_no FROM employees

-> JOIN titles USING (emp_no) WHERE

-> title <> 'Manager');

+--------+-------------+--------------+

| emp_no | first_name  | last_name    |

+--------+-------------+--------------+

| 110022 | Margareta   | Markovitch   |

| 110039 | Vishwani    | Minakawa     |

...

| 111877 | Xiaobin     | Spinelli     |

| 111939 | Yuchang     | Weedman      |

+--------+-------------+--------------+

24 rows in set (0.11 sec)

Let’s rewrite the query to use EXISTS. First, think about the subquery: it should produce output when there’s a title record for an employee with the same name as a manager.

Second, think about the outer query: it should return the employee’s name when the inner query produces output. Here’s the rewritten query:

mysql> SELECT emp_no, first_name, last_name

-> FROM employees JOIN titles USING (emp_no)

-> WHERE title = 'Manager'

-> AND EXISTS (SELECT emp_no FROM titles

-> WHERE titles.emp_no = employees.emp_no

-> AND title <> 'Manager');

+--------+-------------+--------------+

| emp_no | first_name  | last_name    |

+--------+-------------+--------------+

| 110022 | Margareta   | Markovitch   |

| 110039 | Vishwani    | Minakawa     |

...

| 111877 | Xiaobin     | Spinelli     |

| 111939 | Yuchang     | Weedman      |

+--------+-------------+--------------+

24 rows in set (0.09 sec)

Again, you can see that the subquery references the emp_no column, which comes from the outer query.

Correlated subqueries can be used with any nested query type. Here’s the previous IN query rewritten with an outer reference:

mysql> SELECT emp_no, first_name, last_name

-> FROM employees JOIN titles USING (emp_no)

-> WHERE title = 'Manager'

-> AND emp_no IN (SELECT emp_no FROM titles

-> WHERE titles.emp_no = employees.emp_no

-> AND title <> 'Manager');

+--------+-------------+--------------+

| emp_no | first_name  | last_name    |

+--------+-------------+--------------+

| 110022 | Margareta   | Markovitch   |

| 110039 | Vishwani    | Minakawa     |

...

| 111877 | Xiaobin     | Spinelli     |

| 111939 | Yuchang     | Weedman      |

+--------+-------------+--------------+

24 rows in set (0.09 sec)

The query is more convoluted than it needs to be, but it illustrates the idea. You can see that the emp_no in the subquery references the employees table from the outer query.

If the query would return a single row, it could also be rewritten to use an equals instead of IN:

mysql> SELECT emp_no, first_name, last_name

-> FROM employees JOIN titles USING (emp_no)

-> WHERE title = 'Manager'

-> AND emp_no = (SELECT emp_no FROM titles

-> WHERE titles.emp_no = employees.emp_no

-> AND title <> 'Manager');

ERROR 1242 (21000): Subquery returns more than 1 row

This doesn’t work because the subquery returns more than one scalar value. Let’s narrow down.

mysql> SELECT emp_no, first_name, last_name

-> FROM employees JOIN titles USING (emp_no)

-> WHERE title = 'Manager'

-> AND emp_no = (SELECT emp_no FROM titles

-> WHERE titles.emp_no = employees.emp_no

-> AND title = 'Senior Engineer');

+--------+------------+-----------+

| emp_no | first_name | last_name |

+--------+------------+-----------+

| 110344 | Rosine     | Cools     |

| 110420 | Oscar      | Ghazalie  |

| 110800 | Sanjoy     | Quadeer   |

+--------+------------+-----------+

3 rows in set (0.10 sec)

Works now — there’s only one manager and senior engineer title with each name — and so the column subquery operator IN isn’t necessary. Of course, if titles are duplicated (person switches between positions back and forth), you’d need to use IN, ANY, or ALL instead.

Nested Queries in the FROM Clause

The techniques we’ve shown all use nested queries in the WHERE clause. This section shows you how they can alternatively be used in the FROM clause. This is useful when you want to manipulate the source of the data you’re using in a query.

In the employees database, the salaries table stores the annual wage alongside the employee id. If you want to find the monthly rate, for example, you can do some math in the query; one option in this class is to do it with a subquery:

mysql> SELECT emp_no, monthly_salary FROM

-> (SELECT emp_no, salary/12 AS monthly_salary FROM salaries) AS ms

-> LIMIT 5;

+--------+----------------+

| emp_no | monthly_salary |

+--------+----------------+

|  10001 |      5009.7500 |

|  10001 |      5175.1667 |

|  10001 |      5506.1667 |

|  10001 |      5549.6667 |

|  10001 |      5580.0833 |

+--------+----------------+

5 rows in set (0.00 sec)

Focus on what follows the FROM clause: the subquery uses the salaries table and returns two columns. The first column is the emp_no; the second column is aliased as monthly_salary, and is the salary value divided by 12. The outer query is straightforward: it just returns the emp_no and the monthly_salary value created through the subquery. Note that we’ve added the table alias AS ms for the subquery. When we use a subquery as a table, that is, we use a SELECT FROM operation on it — this “derived table” must have an alias — even if we don’t use the alias in our query. MySQL complains if we omit the alias:

mysql> SELECT emp_no, monthly_salary FROM

-> (SELECT emp_no, salary/12 AS monthly_salary FROM salaries)

-> LIMIT 5;

ERROR 1248 (42000): Every derived table must have its own alias

Here’s another example, now in the sakila database. We’ll find out the average sum a film brings us through rentals, or the average gross, as we’ll call it. Let’s begin by thinking through the subquery. It should return the sum of payments that we have for each film. Then, the outer query should average the values to give the answer. Here’s the query:

mysql> SELECT AVG(gross) FROM

-> (SELECT SUM(amount) AS gross

-> FROM payment JOIN rental USING (rental_id)

-> JOIN inventory USING (inventory_id)

-> JOIN film USING (film_id)

-> GROUP BY film_id) AS gross_amount;

+------------+

| AVG(gross) |

+------------+

|  70.361754 |

+------------+

1 row in set (0.05 sec)

You can see that the inner query joins together payment, rental, inventory, and film, and groups the sales together by film so you can get a sum for each film. If you run it in isolation, here’s what happens:

mysql> SELECT SUM(amount) AS gross

-> FROM payment JOIN rental USING (rental_id)

-> JOIN inventory USING (inventory_id)

-> JOIN film USING (film_id)

-> GROUP BY film_id;

+--------+

| gross  |

+--------+

|  36.77 |

|  52.93 |

|  37.88 |

...

|  14.91 |

|  73.83 |

| 214.69 |

+--------+

958 rows in set (0.08 sec)

Now, the outer query takes these sums — which are aliased as gross — and averages them to give the final result. This query is the typical way that you apply two aggregate functions to one set of data. You can’t apply aggregate functions in cascade, as in AVG(SUM(amount)); it won’t work:

mysql> SELECT AVG(SUM(amount)) AS avg_gross

-> FROM payment JOIN rental USING (rental_id)

-> JOIN inventory USING (inventory_id)

-> JOIN film USING (film_id) GROUP BY film_id;

ERROR 1111 (HY000): Invalid use of group function

With subqueries in FROM clauses, you can return a scalar value, a set of column values, more than one row, or even a whole table. However, you can’t use correlated subqueries, meaning that you can’t reference tables or columns from tables that aren’t explicitly listed in the subquery. Note also that you must alias the whole subquery using the AS keyword and give it a name, even if you don’t use that name anywhere in the query.

Nested Queries in JOINs

The last use of nested queries we’ll show, but not the least useful, is using them in joins. In this use case, the results of the subquery basically form a new table and can be used in any of the JOIN types we have discussed.

For the example of this, let’s go back to the query which listed the number of films from each of the categories a particular customer has rented. Remember, we had an issue writing that query using just joins: we didn’t get a 0 count for categories from which our customer didn’t rent. This was the query:

mysql> SELECT cat.name AS category_name, COUNT(cat.category_id) AS cnt

-> FROM category AS cat LEFT JOIN film_category USING (category_id)

-> LEFT JOIN inventory USING (film_id)

-> LEFT JOIN rental USING (inventory_id)

-> JOIN customer AS cs ON rental.customer_id = cs.customer_id

-> WHERE cs.email = 'WESLEY.BULL@sakilacustomer.org'

-> GROUP BY category_name ORDER BY cnt DESC;

+-------------+-----+

| name        | cnt |

+-------------+-----+

| Games       |   9 |

| Foreign     |   6 |

...

...

| Comedy      |   1 |

| Sports      |   1 |

+-------------+-----+

14 rows in set (0.00 sec)

Now that we know about subqueries and joins, and the subqueries can be used in joins, we can easily finish the task. This is our new query:

mysql> SELECT cat.name AS category_name, cnt

-> FROM category AS cat

-> LEFT JOIN (SELECT cat.name, COUNT(cat.category_id) AS cnt

->    FROM category AS cat

->    LEFT JOIN film_category USING (category_id)

->    LEFT JOIN inventory USING (film_id)

->    LEFT JOIN rental USING (inventory_id)

->    JOIN customer cs ON rental.customer_id = cs.customer_id

->    WHERE cs.email = 'WESLEY.BULL@sakilacustomer.org'

->    GROUP BY cat.name) customer_cat USING (name)

-> ORDER BY cnt DESC;

+-------------+------+

| name        | cnt  |

+-------------+------+

| Games       |    9 |

| Foreign     |    6 |

...

| Children    |    1 |

| Sports      |    1 |

| Sci-Fi      | NULL |

| Action      | NULL |

| Thriller    | NULL |

+-------------+------+

17 rows in set (0.01 sec)

Finally, we get all the categories displayed, and we get NULL values for those where no rentals were made. Let’s review what’s going on in our new query. The subquery, which we aliased customer_cat, is our previous query sans the ORDER BY clause. Thus, we know what it will return: 14 rows for categories Wesley rented something and the number of rentals. What we do next is we use LEFT JOIN to concatenate that information to the full list of categories from the category table. The category table is driving the join, so it’ll have every row selected. We join the subquery using the name column that matches between the subquery’s output and the category table’s column.

The technique we showed here is a very powerful one, however, as always with subqueries, it comes at a cost. MySQL cannot optimize the whole query as efficiently, when a subquery is present in the join clause.

User Variables

Often you’ll want to save values that are returned from queries. You might want to do this so that you can easily use a value in a later query. You might also simply want to save a result for later display. In both cases, user variables solve the problem: they allow you to store a result and use it later.

Let’s illustrate user variables with a simple example. The following query finds the title of a film and saves the result in a user variable:

mysql> SELECT @film:=title FROM film WHERE film_id = 1;

+------------------+

| @film:=title     |

+------------------+

| ACADEMY DINOSAUR |

+------------------+

1 row in set, 1 warning (0.00 sec)

The user variable is named film, and it’s denoted as a user variable by the @ character that precedes it. The value is assigned using the := operator. You can print out the contents of the user variable with the following very short query:

mysql> SELECT @film;

+------------------+

| @film            |

+------------------+

| ACADEMY DINOSAUR |

+------------------+

1 row in set (0.00 sec)

You could notice a warning, what was that about?

mysql> SET @film := (SELECT title FROM film WHERE film_id = 1);

mysql> SHOW WARNINGS\G

*************************** 1. row ***************************

Level: Warning

Code: 1287

Message: Setting user variables within expressions is deprecated

and will be removed in a future release. Consider alternatives:

'SET variable=expression, ...', or

'SELECT expression(s) INTO variables(s)'.

1 row in set (0.00 sec)

Let’s cover the two alternatives proposed. First, we can still execute a nested query within a SET statement:

mysql> SET @film := (SELECT title FROM film WHERE film_id = 1);

Query OK, 0 rows affected (0.00 sec)

mysql> SELECT @film;

+------------------+

| @film            |

+------------------+

| ACADEMY DINOSAUR |

+------------------+

1 row in set (0.00 sec)

Second, we can do the SELECT INTO statement:

mysql> SELECT title INTO @film FROM film WHERE film_id = 1;

Query OK, 1 row affected (0.00 sec)

mysql> SELECT @film;

+------------------+

| @film            |

+------------------+

| ACADEMY DINOSAUR |

+------------------+

1 row in set (0.00 sec)

You can explicitly set a variable using the SET statement without a SELECT. Suppose you want to initialize a counter to 0:

mysql> SET @counter := 0;

Query OK, 0 rows affected (0.00 sec)

The := is optional, and you can write = instead, and mix them up. You should separate several assignments with a comma, or put each in a statement of its own:

mysql> SET @counter = 0, @age := 23;

Query OK, 0 rows affected (0.00 sec)

The alternative syntax for SET is SELECT INTO. You can initialize a single variable:

mysql> SELECT 0 INTO @counter;

Query OK, 1 row affected (0.00 sec)

Or multiple at once:

mysql> SELECT 0, 23 INTO @counter, @age;

Query OK, 1 row affected (0.00 sec)

The most common use of user variables is to save a result and use it later. You’ll recall the following example from earlier in the chapter, which we used to motivate nested queries (which are certainly a better solution for this problem). We want to find the name of the film that was rented most recently:

mysql> SELECT MAX(rental_date) FROM rental

-> JOIN customer USING (customer_id)

-> WHERE email = 'WESLEY.BULL@sakilacustomer.org';

+---------------------+

| MAX(rental_date)    |

+---------------------+

| 2005-08-23 15:46:33 |

+---------------------+

1 row in set (0.01 sec)

mysql> SELECT title FROM film

-> JOIN inventory USING (film_id)

-> JOIN rental USING (inventory_id)

-> JOIN customer USING (customer_id)

-> WHERE email = 'WESLEY.BULL@sakilacustomer.org'

-> AND rental_date = '2005-08-23 15:46:33';

+-------------+

| title       |

+-------------+

| KARATE MOON |

+-------------+

1 row in set (0.00 sec)

You can use a user variable to save the result for input into the following query. Here’s the same query pair rewritten using this approach:

mysql> SELECT MAX(rental_date) INTO @recent FROM rental

-> JOIN customer USING (customer_id)

-> WHERE email = 'WESLEY.BULL@sakilacustomer.org';

1 row in set (0.01 sec)

mysql> SELECT title FROM film

-> JOIN inventory USING (film_id)

-> JOIN rental USING (inventory_id)

-> JOIN customer USING (customer_id)

-> WHERE email = 'WESLEY.BULL@sakilacustomer.org'

-> AND rental_date = @recent;

+-------------+

| title       |

+-------------+

| KARATE MOON |

+-------------+

1 row in set (0.00 sec)

This can save you cutting and pasting, and it certainly helps you avoid typing errors.

Here are some guidelines on using user variables:

- User variables are unique to a connection: variables that you create can’t be seen by anyone else, and two different connections can have two different variables with the same name.

- The variable names can be alphanumeric strings and can also include the period (.), underscore (_), and dollar ($) characters.

- Variable names are case-sensitive in MySQL versions earlier than version 5, and case-insensitive from version 5 onward.

- Any variable that isn’t initialized has the value NULL; you can also manually set a variable to be NULL.

- Variables are destroyed when a connection closes.

- You should avoid trying to both assign a value to a variable and use the variable as part of a SELECT query. Two reasons for this are that the new value may not be available for use immediately in the same statement, and a variable’s type is set when it’s first assigned in a query; trying to use it later as a different type in the same SQL statement can lead to unexpected results.

Let’s look at the first issue in more detail using the new variable @fid. Since we haven’t used this variable before, it’s empty. Now, let’s show the film_id for movies that have an entry in the inventory table. Instead of showing it directly, we’ll assign the film_id to the @fid variable. Our query will show the variable twice: once before the assignment operation, once as part of the assignment operation, and once afterwards:

mysql> SELECT @fid, @fid:=film.film_id, @fid FROM film, inventory

-> WHERE inventory.film_id = @fid;

Empty set, 1 warning (0.16 sec)

This returns nothing apart from a deprecation warning; since there’s nothing in the variable to start with, the WHERE clause tries to look for empty inventory.film_id values. If we modify the query to use film.film_id as part of the WHERE clause, things work as expected:

mysql> SELECT @fid, @fid:=film.film_id, @fid FROM film, inventory

-> WHERE inventory.film_id = film.film_id LIMIT 20;

+------+--------------------+------+

| @fid | @fid:=film.film_id | @fid |

+------+--------------------+------+

| NULL |                  1 | 1    |

| 1    |                  1 | 1    |

| 1    |                  1 | 1    |

...

| 4    |                  4 | 4    |

| 4    |                  4 | 4    |

+------+--------------------+------+

20 rows in set, 1 warning (0.00 sec)

Now that if @fid isn’t empty, the initial query will produce some results:

mysql> SELECT @fid, @fid:=film.film_id, @fid FROM film, inventory

-> WHERE inventory.film_id = film.film_id;

+------+--------------------+------+

| @fid | @fid:=film.film_id | @fid |

+------+--------------------+------+

|    4 |                  1 |    1 |

|    1 |                  1 |    1 |

...

|    4 |                  4 |    4 |

|    4 |                  4 |    4 |

+------+--------------------+------+

20 rows in set, 1 warning (0.00 sec)

It’s best to avoid such circumstances where the behavior is not guaranteed and is hence unpredictable.

Chapter 6. Transactions and Locking

One of the SQL databases’ pillars is the transaction isolation concept, locks, and the (in)famous deadlocks. Developers often think that lock issues belong to the DBA realm and that locking is a database issue. The DBAs believe this is a problem that belongs to the application and, consequently, the developers. This chapter will shed light on the nature of database locking and its variations among different isolation levels.

First, let’s define what a transaction is. A transaction is an operation performed (using one or more SQL statements) on a database as a single logical unit of work. All the SQL statements’ modifications in a transaction are all committed (applied to the database) or rolled back (undone from the database) but never partially completed or rolled back. A database transaction must be atomic, consistent, isolated, and durable (the famous acronym ACID).

The locks are part of InnoDB’s gears behind the curtains to ensure data integrity when the application and users access data. We will see that there are different lock types, and some of its types are more restrictive than others.

The databases would not need transactions and locks if the application serialized the requests arriving in the database. By this, imagine only one request from the application at a single unit of time. One SELECT, after this one INSERT, then UPDATE, and so on. We illustrate this behavior in Figure 6-1.

Figure 6-1. SQL statements executed serialized

However, the reality (for good) is that MySQL can handle thousands of requests per second. This chapter discusses what MySQL does to achieve this parallelism; for example, when a request to SELECT and UPDATE in the same row arrives simultaneously or while one of them is still executing. The figure Figure 6-2 shows MySQL running parallel queries:

Figure 6-2. SQL statements executed in parallel

For this chapter, we are particularly interested in how MySQL isolates the transactions (isolation is the I of ACID). We will show daily situations where locks are happening, investigate them, and discuss MySQL parameters that control how much time a transaction can wait to have a lock to be granted.

Isolation Levels

Isolation level is the setting that balances performance, reliability, consistency, and reproducibility of results when multiple transactions are making changes and performing queries simultaneously.

The SQL:1992 standard defines four classic isolation levels, and MySQL supports all of them. InnoDB supports each of the transaction isolation levels described here using different locking strategies. A user can also change the isolation level for a single session or all subsequent connections with the SET [GLOBAL/SESSION] TRANSACTION statement.

We can enforce a high degree of consistency with the default REPEATABLE READ isolation level for operations on crucial data where ACID compliance is essential. Or we can relax the consistency rules with READ COMMITTED or even READ UNCOMMITTED isolation in situations such as bulk reporting where precise consistency and repeatable results are less important than minimizing the amount of overhead for locking. SERIALIZABLE isolation enforces even stricter rules than REPEATABLE READ and is used mainly for special situations such as troubleshooting. Before diving into the details, let’s see some terminology commonly used:

- Dirty reads

Dirty reads occur when a transaction can read data from a row modified by another transaction that has not executed COMMIT yet.

- Non-repeatable reads

A non-repeatable read occurs when, during a transaction, we execute a SELECT twice, and the values within the SELECT differ between the readings (the difference for the dirty read is that in the case there is a COMMIT).

- Phantom reads

A phantom read occurs when a transaction is running, and another transaction adds new rows to the records being read (again, in this case, there is a commit of the transaction modifying the data). Also, the problem is that there is no range lock guaranteeing the consistency of the data.

Now let’s see each isolation level in detail.

REPEATABLE READ

The REPEATABLE READ is the default isolation level for InnoDB. It ensures consistent reads within the same transaction read the snapshot established by the first read. In this mode, InnoDB locks the index range scanned, using gap locks or next-key locks (we will cover these details in the Locking section) to block insertions by other sessions into the gaps covered by the range. Check the example below:

In one session (let’s call it session 1), we execute the following SELECT:

session1 > SELECT * FROM person WHERE i BETWEEN 1 AND 4;

+---+----------+

| i | name     |

+---+----------+

| 1 | Vinicius |

| 2 | Sergey   |

| 3 | Iwo      |

| 4 | Peter    |

+---+----------+

4 rows in set (0.00 sec)

In another session (session 2), we will run an update in the second row:

session2 > UPDATE person SET name = 'Kuzmichev' WHERE i=2;

Query OK, 1 row affected (0.00 sec)

Rows matched: 1  Changed: 1  Warnings: 0

session2> COMMIT;

Query OK, 0 rows affected (0.00 sec)

We can confirm the change in session 2:

session2 > SELECT * FROM person WHERE i BETWEEN 1 AND 4;

+---+-----------+

| i | name      |

+---+-----------+

| 1 | Vinicius  |

| 2 | Kuzmichev |

| 3 | Iwo       |

| 4 | Peter     |

+---+-----------+

4 rows in set (0.00 sec)

But session 1 remains with the snapshot value (old value):

session1> SELECT * FROM person WHERE i BETWEEN 1 AND 4;

+---+----------+

| i | name     |

+---+----------+

| 1 | Vinicius |

| 2 | Sergey   |

| 3 | Iwo      |

| 4 | Peter    |

+---+----------+

If we see the example, using the REPEATABLE READ level, there are no dirty reads and either non-repeatable reads. Each session has its particular snapshot of the data from the moment the transaction started.

READ COMMITTED

As a curiosity, the READ COMMITTED isolation level is the default for many databases, like Postgres, Oracle, and SQL Server, but not MySQL. So for those who are migrating to MySQL, it will notice this difference in the default behavior.

The main difference between READ COMMITTED and REPEATABLE READ is that with READ COMMITTED, each consistent read, even within the same transaction, sets and reads its fresh snapshot. This behavior can lead to phantom reads when executing multiple queries inside a transaction. Let’s see an example. In session 1, we can see the following rows:

session1 > SELECT * FROM person WHERE i = 1;

+---+----------+

| i | name     |

+---+----------+

| 1 | Vinicius |

+---+----------+

1 row in set (0.00 sec)

In session 2, we will update the first row and commit the transaction:

session2 > UPDATE person SET name = 'Grippa' WHERE i = 1;

Query OK, 1 row affected (0.00 sec)

Rows matched: 1  Changed: 1  Warnings: 0

session2 > COMMIT;

Query OK, 0 rows affected (0.00 sec)

And if we check session 1, the first-row value changed:

session1 > SELECT * FROM person WHERE i = 1;

+---+--------+

| i | name   |

+---+--------+

| 1 | Grippa |

+---+--------+

The significant advantage of READ COMMITTED is that there are no gap locks, allowing the free insertion of new records next to locked records.

READ UNCOMMITTED

MySQL performs SELECT statements in a non-locking fashion, but the SELECT statement might use a possible earlier version of a row. Thus, using this isolation level, such reads are not consistent. This phenomenon is also called a dirty read. The main difference from the previous example is that session two can see the results before the commit. Let’s see the example. In session one, we execute the SELECT statement:

session1 > SELECT * FROM person WHERE i = 5;

+---+---------+

| i | name    |

+---+---------+

| 5 | Marcelo |

+---+---------+

1 row in set (0.00 sec)

And on session 2, we perform the update without committing:

session2 > UPDATE person SET name = 'Altmann' WHERE i = 5;

Query OK, 1 row affected (0.00 sec)

Rows matched: 1  Changed: 1  Warnings: 0

And if we observe session 1:

session1 > SELECT * FROM person WHERE i = 5;

+---+---------+

| i | name    |

+---+---------+

| 5 | Altmann |

+---+---------+

1 row in set (0.00 sec)

We can see that session 1 can read the modified data even though it is in a transient state.

SERIALIZABLE

This level is like REPEATABLE READ, but InnoDB implicitly converts all explicit SELECT statements to SELECT … FOR SHARE if auto-commit is disabled. If auto-commit is enabled, the SELECT is its transaction. Therefore, it is known to be read-only and can be serialized if performed as a consistent (non-locking) read and need not block for other transactions. (To force a direct SELECT to block if other transactions have modified the selected rows, disable auto-commit.)

NOTE

A final observation regarding isolation level: the MyISAM storage engine does not support transactions. This means that if a write operation comes, all the reads will halt waiting for the DML statement’s execution. All locks happen at the table level and not at row level.

Locking

Now that we’ve looked at how each isolation level works, let’s see how the InnoDB engine uses its different locking strategies to make them work.

Locks are used in the databases to protect shared resources or objects. They can act at different levels, such as:

- Table locking

- Metadata locking

- Row locks

- Application-level locks

MySQL uses table locking (instead of a page, row, or column locking) for all storage engines except InnoDB, like MyISAM.

MySQL uses metadata locking to manage concurrent access to database objects and to ensure data consistency. We use it mostly when performing DDL operations and during trigger creation. Metadata locking applies to tables and schemas, stored programs (procedures, functions, triggers, scheduled events), tablespaces, and user locks acquired with the GET_LOCK() function. We will see that we can execute certain DDLs online while the table is processing inserts, updates, and deletes, but there are cases where the table is not available for DML.

To support simultaneous write access by multiple sessions, InnoDB supports row-level locking.

Application-level or user-level locks, such as those provided by GET_LOCK() and RELEASE_LOCK(), can be used to implement application locks or simulate record locks.

This book focuses on the metadata and the row locks since they are the ones that affect the majority of the users and are the most common.

Metadata Locks

The MySQL documentation provides the best definition of metadata locks:

“To ensure transaction serializability, the server must not permit one session to perform a data definition language (DDL) statement on a table that is used in an uncompleted explicitly or implicitly started transaction in another session. The server achieves this by acquiring metadata locks on tables used within a transaction and deferring the locks’ release until the transaction ends. A metadata lock on a table prevents changes to the table’s structure. This locking approach implies that a table that is being used by a transaction within one session cannot be used in DDL statements by other sessions until the transaction ends.”

With definition in-place, let’s see a case of metadata lock happening. First, we will create a dummy table and load some rows into it:

USE test;

DROP TABLE IF EXISTS `joinit`;

CREATE TABLE `joinit` (

`i` int(11) NOT NULL AUTO_INCREMENT,

`s` varchar(64) DEFAULT NULL,

`t` time NOT NULL,

`g` int(11) NOT NULL,

PRIMARY KEY (`i`)

) ENGINE=InnoDB  DEFAULT CHARSET=latin1;

INSERT INTO joinit VALUES (NULL, uuid(), time(now()),  (FLOOR( 1 + RAND( ) *60 )));

INSERT INTO joinit SELECT NULL, uuid(), time(now()),  (FLOOR( 1 + RAND( ) *60 )) FROM joinit;

INSERT INTO joinit SELECT NULL, uuid(), time(now()),  (FLOOR( 1 + RAND( ) *60 )) FROM joinit;

INSERT INTO joinit SELECT NULL, uuid(), time(now()),  (FLOOR( 1 + RAND( ) *60 )) FROM joinit;

INSERT INTO joinit SELECT NULL, uuid(), time(now()),  (FLOOR( 1 + RAND( ) *60 )) FROM joinit;

INSERT INTO joinit SELECT NULL, uuid(), time(now()),  (FLOOR( 1 + RAND( ) *60 )) FROM joinit;

INSERT INTO joinit SELECT NULL, uuid(), time(now()),  (FLOOR( 1 + RAND( ) *60 )) FROM joinit;

INSERT INTO joinit SELECT NULL, uuid(), time(now()),  (FLOOR( 1 + RAND( ) *60 )) FROM joinit;

INSERT INTO joinit SELECT NULL, uuid(), time(now()),  (FLOOR( 1 + RAND( ) *60 )) FROM joinit;

Now that we have some dummy data, we will open one session(session 1) and execute an UPDATE:

session1> UPDATE joinit SET t=now();

In session 2, we will try to add a new column to this table while the UPDATE is still running:

session 2> ALTER TABLE joinit ADD COLUMN b INT;

Running another session (session 3), we can execute the SHOW PROCESSLIST to visualize the metadata lock:

session 3> SHOW PROCESSLIST;

+----+----------+-----------+------+---------+------+---------------------------------+-------------------------------------+-----------+---------------+

| Id | User     | Host      | db   | Command | Time | State                           | Info                                | Rows_sent | Rows_examined |

+----+----------+-----------+------+---------+------+---------------------------------+-------------------------------------+-----------+---------------+

| 70 | msandbox | localhost | test | Sleep   |   39 |                                 | NULL                                |   4194304 |       4194304 |

| 71 | msandbox | localhost | test | Query   |    5 | Waiting for table metadata lock | alter table joinit add column b int |         0 |             0 |

| 72 | msandbox | localhost | NULL | Query   |    0 | starting                        | show processlist                    |         0 |             0 |

+----+----------+-----------+------+---------+------+---------------------------------+-------------------------------------+-----------+---------------+

3 rows in set (0.00 sec)

Metadata locks are also related to the table cache. Even if a connection already has access to a table in the table cache, any query that executes a DDL to this table will invalidate the table cache entry. This invalidation forces all queries that access the table to reopen the modified table and load the new entries in the table cache.

NOTE

MySQL is multithreaded, so there may be many clients issuing queries for a given table simultaneously. To minimize the problem with multiple client sessions having different states on the same table, the table is opened independently by each concurrent session. The table cache uses additional memory but typically increases performance.

The following example illustrates the metadata lock. In session 1 we will execute a DDL operation:

session1> ALTER TABLE joinit ADD COLUMN c CHAR(32) DEFAULT 'dummy_text';

And in session 2 we will run a SELECT statement:

session2> > SELECT * FROM joinit ORDER BY i;

And to observe the phenomenon, we will open a session 3:

session3> SELECT * FROM sys.schema_table_lock_waits;

+---------------+-------------+-------------------+-------------+-------

-------------+-------------------+-----------------------+--------------

--------------------------------------------------+--------------------+

-----------------------------+-----------------------------+------------

--------+--------------+--------------------+--------------------+------

------------------+-------------------------+---------------------------

---+

| object_schema | object_name | waiting_thread_id | waiting_pid |

waiting_account    | waiting_lock_type | waiting_lock_duration |

waiting_query                                                  |

waiting_query_secs | waiting_query_rows_affected |

waiting_query_rows_examined | blocking_thread_id | blocking_pid |

blocking_account   | blocking_lock_type | blocking_lock_duration |

sql_kill_blocking_query | sql_kill_blocking_connection |

+---------------+-------------+-------------------+-------------+-------

-------------+-------------------+-----------------------+--------------

--------------------------------------------------+--------------------+

-----------------------------+-----------------------------+------------

--------+--------------+--------------------+--------------------+------

------------------+-------------------------+---------------------------

---+

| test          | joinit      |               103 |          77 |

msandbox@localhost | EXCLUSIVE         | TRANSACTION           | alter

table joinit  add column c char(32) default 'dummy_text' |

9 |                           0 |                           0 |

103 |           77 | msandbox@localhost | SHARED_UPGRADABLE  |

TRANSACTION            | KILL QUERY 77           | KILL 77

|

| test          | joinit      |               103 |          77 |

msandbox@localhost | EXCLUSIVE         | TRANSACTION           | alter

table joinit  add column c char(32) default 'dummy_text' |

9 |                           0 |                           0 |

104 |           78 | msandbox@localhost | SHARED_READ        |

TRANSACTION            | KILL QUERY 78           | KILL 78

|

+---------------+-------------+-------------------+-------------+-------

-------------+-------------------+-----------------------+--------------

--------------------------------------------------+--------------------+

-----------------------------+-----------------------------+------------

--------+--------------+--------------------+--------------------+------

------------------+-------------------------+---------------------------

---+

2 rows in set (0.01 sec)

NOTE

The MySQL sys schema is a set of objects that helps DBAs and developers interpret data collected by the Performance Schema. It is available for MySQL 5.7 and MySQL 8.0. In case you want to use the sys schema in MySQL 5.6, it is possible to install using the sys project available in GitHub:

# git clone https://github.com/mysql/mysql-sys.git

# cd mysql-sys/

# mysql -u root -p < ./sys_56.sql

Even if MySQL already opened the table and the table object is in the table_cache, this does not change anything for metadata locks.

And how can you troubleshoot metadata locks? It is possible to use the Performance Schema to visualize metadata locks. First of all, it is necessary to enable instrumentation for metadata locks:

mysql> UPDATE performance_schema.setup_instruments SET enabled = 'YES'

-> WHERE NAME = 'wait/lock/metadata/sql/mdl';

Query OK, 0 rows affected (0.00 sec)

Rows matched: 1 Changed: 0 Warnings: 0

Going back to our previous example, let’s query the metadata_locks table:

mysql> SELECT * FROM performance_schema.metadata_locks\G

*************************** 1. row ***************************

OBJECT_TYPE: GLOBAL

OBJECT_SCHEMA: NULL

OBJECT_NAME: NULL

OBJECT_INSTANCE_BEGIN: 140089691017472

LOCK_TYPE: INTENTION_EXCLUSIVE

LOCK_DURATION: STATEMENT

LOCK_STATUS: GRANTED

SOURCE:

OWNER_THREAD_ID: 97

OWNER_EVENT_ID: 34

*************************** 2. row ***************************

OBJECT_TYPE: SCHEMA

OBJECT_SCHEMA: test

OBJECT_NAME: NULL

OBJECT_INSTANCE_BEGIN: 140089645108128

LOCK_TYPE: INTENTION_EXCLUSIVE

LOCK_DURATION: TRANSACTION

LOCK_STATUS: GRANTED

SOURCE:

OWNER_THREAD_ID: 97

OWNER_EVENT_ID: 34

*************************** 3. row ***************************

OBJECT_TYPE: TABLE

OBJECT_SCHEMA: test

OBJECT_NAME: joinit

OBJECT_INSTANCE_BEGIN: 140089605952464

LOCK_TYPE: SHARED_UPGRADABLE

LOCK_DURATION: TRANSACTION

LOCK_STATUS: GRANTED

SOURCE:

OWNER_THREAD_ID: 97

OWNER_EVENT_ID: 34

*************************** 4. row ***************************

OBJECT_TYPE: BACKUP

OBJECT_SCHEMA: NULL

OBJECT_NAME: NULL

OBJECT_INSTANCE_BEGIN: 140089653496336

LOCK_TYPE: INTENTION_EXCLUSIVE

LOCK_DURATION: STATEMENT

LOCK_STATUS: GRANTED

SOURCE:

OWNER_THREAD_ID: 97

OWNER_EVENT_ID: 34

*************************** 5. row ***************************

OBJECT_TYPE: TABLE

OBJECT_SCHEMA: test

OBJECT_NAME: joinit

OBJECT_INSTANCE_BEGIN: 140089649301152

LOCK_TYPE: EXCLUSIVE

LOCK_DURATION: TRANSACTION

LOCK_STATUS: PENDING

SOURCE:

OWNER_THREAD_ID: 97

OWNER_EVENT_ID: 34

*************************** 6. row ***************************

OBJECT_TYPE: TABLE

OBJECT_SCHEMA: performance_schema

OBJECT_NAME: metadata_locks

OBJECT_INSTANCE_BEGIN: 140089640911984

LOCK_TYPE: SHARED_READ

LOCK_DURATION: TRANSACTION

LOCK_STATUS: GRANTED

SOURCE:

OWNER_THREAD_ID: 98

OWNER_EVENT_ID: 10

6 rows in set (0.00 sec)

Note that SHARED_UPGRADABLE lock set on table joinit and EXCLUSIVE lock is pending on the same table above.

We can get a nice view of all metadata locks from other sessions, excluding our current one with the following query:

mysql> SELECT object_type, object_schema, object_name, lock_type,

-> lock_status, thread_id, processlist_id, processlist_info FROM

-> performance_schema.metadata_locks INNER JOIN performance_schema.threads

-> ON thread_id = owner_thread_id WHERE processlist_id <> connection_id();

+-------------+---------------+-------------+---------------------+-----

--------+-----------+----------------+----------------------------------

---+

| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | LOCK_TYPE           |

LOCK_STATUS | THREAD_ID | PROCESSLIST_ID | PROCESSLIST_INFO

|

+-------------+---------------+-------------+---------------------+-----

--------+-----------+----------------+----------------------------------

---+

| GLOBAL      | NULL          | NULL        | INTENTION_EXCLUSIVE |

GRANTED     |        97 |             71 | alter table joinit add column

b int |

| SCHEMA      | test          | NULL        | INTENTION_EXCLUSIVE |

GRANTED     |        97 |             71 | alter table joinit add column

b int |

| TABLE       | test          | joinit      | SHARED_UPGRADABLE   |

GRANTED     |        97 |             71 | alter table joinit add column

b int |

| BACKUP      | NULL          | NULL        | INTENTION_EXCLUSIVE |

GRANTED     |        97 |             71 | alter table joinit add column

b int |

| TABLE       | test          | joinit      | EXCLUSIVE           |

PENDING     |        97 |             71 | alter table joinit add column

b int |

+-------------+---------------+-------------+---------------------+-----

--------+-----------+----------------+----------------------------------

---+

5 rows in set (0.00 sec)

Another alternative is to use the sys schema:

mysql> SELECT * FROM sys.schema_table_lock_waits;

+---------------+-------------+-------------------+-------------+--------------------+-------------------+-----------------------+-------------------------------------+--------------------+-----------------------------+-----------------------------+--------------------+--------------+--------------------+--------------------+------------------------+-------------------------+------------------------------+

| object_schema | object_name | waiting_thread_id | waiting_pid | waiting_account    | waiting_lock_type | waiting_lock_duration | waiting_query                       | waiting_query_secs | waiting_query_rows_affected | waiting_query_rows_examined | blocking_thread_id | blocking_pid | blocking_account   | blocking_lock_type | blocking_lock_duration | sql_kill_blocking_query | sql_kill_blocking_connection |

+---------------+-------------+-------------------+-------------+--------------------+-------------------+-----------------------+-------------------------------------+--------------------+-----------------------------+-----------------------------+--------------------+--------------+--------------------+--------------------+------------------------+-------------------------+------------------------------+

| test          | joinit      |                97 |          71 | msandbox@localhost | EXCLUSIVE         | TRANSACTION           | alter table joinit add column b int |                767 |                           0 |                           0 |                 97 |           71 | msandbox@localhost | SHARED_UPGRADABLE  | TRANSACTION            | KILL QUERY 71           | KILL 71                      |

+---------------+-------------+-------------------+-------------+--------------------+-------------------+-----------------------+-------------------------------------+--------------------+-----------------------------+-----------------------------+--------------------+--------------+--------------------+--------------------+------------------------+-------------------------+------------------------------+

1 row in set (0.00 sec)

To conclude, it is vital to keep in mind that we cannot have long-running transactions whenever we want to modify table structure; otherwise, we risk having multiple queries waiting for the metadata locks.

Row Locks

InnoDB implements a standard row-level locking. The standard row-level locking means that, in general terms, there are two types of locks: shared (S) locks and exclusive (X) locks.

- A shared (S) lock permits the transaction that holds the lock to read a row.

- An exclusive (X) lock permits the transaction that holds the lock to update or delete a row.

The names are self-explanatory: exclusive locks don’t allow multiple transactions to acquire an exclusive lock(X) in the same row while sharing a shared lock(S). That is why it is possible to have parallel reads for the same row, and writes are not allowed.

Also, InnoDB supports multiple granularity locking, which permits the coexistence of row locks and table locks. Granular locking is possible due to the existence of intention locks. Intention locks are table-level locks that indicate which type of lock (shared or exclusive) a transaction requires later for a row in a table. There are two types of intention locks:

- An intention shared lock (IS) indicates that a transaction intends to set a shared lock on individual rows in a table.

- An intention exclusive lock (IX) indicates that a transaction intends to set an exclusive lock on individual rows in a table.

Before a transaction can acquire a shared or an exclusive lock, it is necessary to obtain the respective intention lock (IS or IX).

To make things a bit easier to understand, look at the following matrix:

X

IX

S

IS

X

Conflict

Conflict

Conflict

Conflict

IX

Conflict

Compatible

Conflict

Compatible

S

Conflict

Conflict

Compatible

Compatible

IS

Conflict

Compatible

Compatible

Compatible

Now, another important concept is the gap lock. A gap lock is a lock on the gap between index records. Thanks to the gap lock, when you run the same query twice, you get the same result, regardless of other session modifications on that table. The gap lock makes the reads consistent and therefore makes the replication between servers consistent. If you execute SELECT * FROM example_table WHERE id > 1000 FOR UPDATE twice, you expect to get the same value twice. To accomplish that, InnoDB locks all index records found by the WHERE clause with an exclusive lock and the gaps between them with a shared gap lock. Let’s see an example of a gap lock in action.

First, we will execute a SELECT stament in the person table:

mysql> SELECT * FROM PERSON;

+----+-----------+

| i  | name      |

+----+-----------+

|  1 | Vinicius  |

|  2 | Kuzmichev |

|  3 | Iwo       |

|  4 | Peter     |

|  5 | Marcelo   |

|  6 | Guli      |

|  7 | Nando     |

| 10 | Jobin     |

| 15 | Rafa      |

| 18 | Leo       |

+----+-----------+

10 rows in set (0.00 sec)

Now, on session 1, we will perform a delete operation, but we will not commit:

session 1> DELETE FROM person WHERE name LIKE 'Jobin';

Query OK, 1 row affected (0.00 sec)

And if we check session 2, we can still see the row with Jobin:

session 2> SELECT * FROM person;

+----+-----------+

| i  | name      |

+----+-----------+

|  1 | Vinicius  |

|  2 | Kuzmichev |

|  3 | Iwo       |

|  4 | Peter     |

|  5 | Marcelo   |

|  6 | Guli      |

|  7 | Nando     |

| 10 | Jobin     |

| 15 | Rafa      |

| 18 | Leo       |

+----+-----------+

10 rows in set (0.00 sec)

Now if we try to insert a new row with the value of 11 (the gap between 10 and 15), the insert will be locked and will fail:

transaction2 > INSERT INTO person VALUES (11, 'Bennie');

ERROR 1205 (HY000): Lockwait timeout exceeded; try restarting transaction

If we run SHOW ENGINE INNODB STATUS we will see the locked transaction in the TRANSACTIONS section:

------- TRX HAS BEEN WAITING 17 SEC FOR THIS LOCK TO BE GRANTED:

RECORD LOCKS space id 28 page no 3 n bits 80 index PRIMARY of table

`test`.`person` trx id 4773 lock_mode X locks gap before rec insert

intention waiting

Note that MySQL does not need Gap locking for statements that lock rows using a unique index to search for a unique row. (This does not include the case that the search condition includes only some columns of a multiple-column unique index; in that case, gap locking does occur.) For example, if the name column has a unique index, the following statement uses only an index-record lock:

mysql> CREATE UNIQUE INDEX idx ON PERSON (name);

Query OK, 0 rows affected (0.01 sec)

Records: 0  Duplicates: 0  Warnings: 0

mysql> DELETE FROM person WHERE name LIKE 'Jobin';

Deadlocks

A deadlock is a situation where two or more competing actions are waiting for the other to finish. As a consequence, neither ever does. In computer science, deadlock refers to a specific condition when two or more processes are each waiting for each other to release a resource. In this part, we will talk specifically about transaction deadlocks.

For a deadlock to happen, four conditions (Coffman conditions) should exist:

1. Mutual exclusion: The process must hold at least one resource in a non-shareable mode. Otherwise, MySQL would not prevent the process from using the resource when necessary. Only one process can use the resource at any given instant of time.

2. Hold and wait or resource holding: A process is currently holding at least one resource and requesting additional resources held by other processes.

3. No preemption: A resource can be released only voluntarily by the process holding it.

4. Circular wait: Each process must be waiting for a resource held by another process, which in turn is waiting for the first process to release the resource.

Before moving with an example, there are some misconceptions that you might hear, and it is essential to clarify. They are:

- Transaction isolation levels are responsible for deadlocks: The possibility of deadlocks is not affected by the isolation level. The READ COMMITTED can set fewer locks; hence it can help you avoid certain lock types (e.g., gap locking that we mentioned before), but it won’t prevent the deadlock.

- Small transactions are not affected by deadlocks. The fact is those small transactions are less prone to deadlocks because they run fast, and the chance of conflict is smaller than more prolonged operations, but it can still happen if transactions do not use the same order of operations.

- Deadlocks are terrible things. They are mechanisms that exist to keep our data consistent. On the opposite, they are allies in the sense that if it were not for them, the processes could hold the resources for a long time, slowing or stalling the database completely until the query gets canceled by the innodb_lock_wait_timeout setting.

Let’s see a deadlock example with the World database. If you do not have it, you can check “Entity Relationship Modeling Examples” to load the world database.

Once we’ve imported, let’s see an example of a deadlock. Here we have a list of Italian cities from the province of Toscana:

mysql> SELECT * FROM city WHERE CountryCode = 'ITA' AND District='Toscana';

+------+---------+-------------+----------+------------+

| ID   | Name    | CountryCode | District | Population |

+------+---------+-------------+----------+------------+

| 1471 | Firenze | ITA         | Toscana  |     376662 |

| 1483 | Prato   | ITA         | Toscana  |     172473 |

| 1486 | Livorno | ITA         | Toscana  |     161673 |

| 1516 | Pisa    | ITA         | Toscana  |      92379 |

| 1518 | Arezzo  | ITA         | Toscana  |      91729 |

+------+---------+-------------+----------+------------+

5 rows in set (0.00 sec)

Now let’s say we have two transactions trying to update the population of two same cities in Toscana at the same time, but into different orders:

session1> UPDATE city SET Population=Population + 1 WHERE ID = 1471;

Query OK, 1 row affected (0.00 sec)

Rows matched: 1  Changed: 1  Warnings: 0

session1> UPDATE city SET Population=Population + 1 WHERE ID =1516;

ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

session2> UPDATE city SET Population=Population + 1 WHERE ID =1516;

Query OK, 1 row affected (0.00 sec)

Rows matched: 1  Changed: 1  Warnings: 0

session2> UPDATE city SET Population=Population + 1 WHERE ID = 1471;

Query OK, 1 row affected (5.15 sec)

Rows matched: 1  Changed: 1  Warnings: 0

And we had a deadlock in session 1. It is important to note that it is not always the first transaction that will fail. In this example, session 1 was the one that MySQL aborted. We can observe the latest deadlock that happened in the MySQL database by running SHOW SLAVE STATUS\G:

mysql> SHOW ENGINE INNODB STATUS\G

...

`------------------------

LATEST DETECTED DEADLOCK

`------------------------

2020-12-05 16:08:19 0x7f6949359700

*** (1) TRANSACTION:

TRANSACTION 10502342, ACTIVE 34 sec starting index read

mysql tables in use 1, locked 1

LOCK WAIT 3 lock struct(s), heap size 1136, 2 row lock(s), undo log

entries 1

MySQL thread id 71, OS thread handle 140090386671360, query id 5979282

localhost msandbox updating

update city set Population=Population + 1 where ID = 1471

*** (1) WAITING FOR THIS LOCK TO BE GRANTED:

RECORD LOCKS space id 6041 page no 15 n bits 248 index PRIMARY of table

`world`.`city` trx id 10502342 lock_mode X locks rec but not gap waiting

*** (2) TRANSACTION:

TRANSACTION 10502341, ACTIVE 62 sec starting index read

mysql tables in use 1, locked 1

3 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 1

MySQL thread id 75, OS thread handle 140090176542464, query id 5979283

localhost msandbox updating

update city set Population=Population + 1 where ID =1516

*** (2) HOLDS THE LOCK(S):

RECORD LOCKS space id 6041 page no 15 n bits 248 index PRIMARY of table

`world`.`city` trx id 10502341 lock_mode X locks rec but not gap

*** (2) WAITING FOR THIS LOCK TO BE GRANTED:

RECORD LOCKS space id 6041 page no 16 n bits 248 index PRIMARY of table

`world`.`city` trx id 10502341 lock_mode X locks rec but not gap waiting

*** WE ROLL BACK TRANSACTION (2)

...

If you want, you can to log all the deadlocks that happens in MySQL in the MySQL error log. Using the innodb_print_all_deadlocks parameter, MySQL records all information about deadlocks from InnoDB user transactions in the error log. Otherwise, you see information about only the last deadlock using the SHOW ENGINE INNODB STATUS command.

MySQL Parameters Related to Isolation and Locks

To summarize this chapter, let’s see a few MySQL parameters related to this topic that can change isolation behavior and lock duration:

transaction_isolation

The transaction_isolation parameter controls the transaction isolation level. It can change the behavior at GLOBAL, SESSION or NEXT_TRANSACTION level:

mysql> SET SESSION transaction_isolation='READ-COMMITTED';

Query OK, 0 rows affected (0.00 sec)

mysql> SHOW SESSION VARIABLES LIKE '%isol%';

+-----------------------+----------------+

| Variable_name         | Value          |

+-----------------------+----------------+

| transaction_isolation | READ-COMMITTED |

| tx_isolation          | READ-COMMITTED |

+-----------------------+----------------+

NOTE

transaction_isolation was added in MySQL 5.7.20 as a synonym for tx_isolation, which is now deprecated and is removed in MySQL 8.0. Applications should be adjusted to use transaction_isolation in preference to tx_isolation.

innodb_lock_wait_timeout

The amount of time in seconds an InnoDB transaction waits for a row lock before giving up. The default value is 50 seconds. The transaction raises the following error if the time waiting for the lock exceeds the innodb_lock_wait_timeout value:

ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

innodb_print_all_deadlocks

MySQL records all information about deadlocks from InnoDB user transactions in the MySQL error log when this option is enabled. We can do it dynamically with the following command:

mysql> SET GLOBAL innodb_print_all_deadlocks = 1;

lock_wait_timeout

This variable specifies the timeout in seconds for attempts to acquire metadata locks. To avoid long metadata locks stalling the database, we can set lock_wait_timeout=1 at the session-level before executing the DDL. In this case, if the DDL can’t acquire the lock, the DDL will give up letting the other requests execute. For example:

mysql> SET SESSION lock_wait_timeout=1;

mysql> CREATE TABLE t1(i INT NOT NULL AUTO_INCREMENT PRIMARY KEY) ENGINE=InnoDB;

innodb_deadlock_detect

Yes, it is possible to disable deadlock monitoring. However, this only means that MySQL will not kill a query to undo the deadlock knot. Disabling the deadlock will not prevent deadlocks from happening. Disabling the deadlock detection will make MySQL rely on the innodb_lock_wait_timeout setting for transaction rollback when a deadlock occurs.

Chapter 10. Backups and Recovery

The most important task for any DBA is backing up the data. Correct and tested backup and recovery procedures can save a company, and thus a job. Mistakes happen, disasters happen, and errors happen. MySQL is a robust piece of software, but it’s not completely free of bugs or crashes. Thus it is crucial to understand why you need to perform backups, as well as know various methods of doing so.

Apart from preserving database contents, most backup methods can also be used for another important purpose: copying the contents of the database between separate systems. Though probably not as important as saving the day is when corruption happens, this copying is a routine operation for the vast majority of database operators. Developers will face a need to use downstream environments which should be similar to production. QA staff may need a volatile environment with a lifespan of an hour. Analytics may be run on a dedicated host. Some of these tasks can be solved by replication, but any replica first starts from a restored backup.

This chapter first briefly reviews two major types of backups and discusses their fundamental properties. It then looks at some of the tools available in MySQL world for the purpose of backup and recovery. Covering each and every tool and their parameter is neyond the scope of this book, but by the end of the chapter you should know your way around backing up and recovering MySQL data. We’ll also see some basic scenarios of data transfer. Finally, the chapter outlines a robust backup architecture that you can use as a foundation for your work.

An overview of what we think is a good backup strategy is given in the end of this chapter in “Database Backup Strategy Primer”. We think that it’s important to understand the tools and moving parts before deciding on the strategy, therefore that section comes last.

Physical and Logical Backups

Broadly speaking, most if not all of the backup tools fit into just two wide categories: logical and physical. Logical backups operate on the internal structures: databases (schemas), tables, views, users, and other objects.Physical backups are concerned with the OS-side representation of the database structures: data files, transaction journals, and so on.

It might be easier to explain with an example. Imagine backing up a single MyISAM table in MySQL database. As you will see later in this chapter, InnoDB storage engine is more complex to back up correctly. Knowing that MyISAM is not transactional, and that there are no ongoing writes to this table, we may go ahead and copy the files related to it. In doing so, we create a physical backup of the table. We could instead go ahead and run SELECT * and SHOW CREATE TABLE statements against this table, and preserve the outputs of those statements somehwere. That’s a very basic form of a logical backup. Of course, these are just simple examples, and in reality the process of obtaining both types of backup will be more complex and nuanced. The conceptual differences between these imaginary backups can, however, be transferred and applied to any logical and physical backups.

Logical Backups

Logical backups are concerned with the actual data, and not its physical representation. As you’ve already seen, such backups don’t copy any existing database files and instead rely on queries or other means to obtain needed database contents. The result is usually some textual representation, though that’s not granted, and a logical backup’s output may well be binary-encoded. Let’s see some more examples of how such backups might look and then discuss their properties.

Here are some examples of logical backups:

- Table data queried and saved into an external .csv file using the SELECT ... INTO OUTFILE statement that we cover in “Writing Data into Comma-Delimited Files”.

- A table or any other object’s definition saved as a SQL statement.

- One or more INSERT SQL statements that, run against a database and an empty table, would populate that table up to a preserved state.

- Recording of all statements that modify data or schema objects ever run that touched a particular table or database. What we mean are DML—Data Modification Language—and DDL—Data Definition Language—commands. By now you should be familiar with both types covered in Chapter 3 and Chapter 4.

NOTE

That last example actually represents a way by which both replication and point-in-time recovery work in MySQL. We tackle those topics later, and you’ll see that the term logical doesn’t just apply to backups.

Recovery of a logical backup is usually an execution of one or more SQL statements. Continuing with our earlier examples, let’s review recovery for them:

- Data from a .csv file can be loaded into a table using the LOAD DATA INFILE command.

- Table can be created or re-created by running a DDL SQL statement.

- INSERT SQL statements can be executed using mysql CLI or any other client.

- Replay of all statements run in a database will restore it to a state after the last statement.

Logical backups have some interesting properties making them extremely useful in some situations. More often than not, logical backup is some form of text file, mostly a bunch of SQL statements. That is not necessary, however, and not a defining property, albeit a usually useful one. The process of creating logical backups is also usually an execution of some queries. These are important features because they allow for a great degree of flexibility and portability.

Logical backups are flexible because they make it very easy to back up a part of a database. For example, you can back up schema objects without their contents, or easily back up only a few of the database’s tables. Even more, you can back up a part of a table’s data, which is usually impossible with physical backups. Once the backup file is ready, some tools can review and modify it either manually or automatically, which is something not easily done with copies of database files.

Portability comes from the fact that logical backups can be loaded easily into different versions of MySQL running on different operating systems and architectures. With some modification, you can actually load logical backups taken from one RDBMS into an absolutely different one. Most database migration tools use logical replication internally due to this fact. This property also makes this backup type suitable for backing up cloud managed databases off-site, and for migrations between them.

Logical backups have an interesting property in that they are effective in combating corruption. Corruption here means any sort of actual physical corruption of a physical data file. Errors in data can be introduced, for example, by bugs in software, or by gradual degradation of storage media. The topic of corruption and its counterpart — integrity — is very wide, but this brief explanation should be enough for us.

Once a data file becomes corrupted, a database might not be able to read data from it and serve the queries. Since corruptions tend to happen silently, you might not know when it occurred. However, if a logical backup was generated without error, that means it’s sound and has good data. Corruption might happen in a secondary index (any non-primary index, see Chapter 4, Working with Database Structures for more details), so logical backup doing a full table scan might not face an error and generate normally. In short, a logical backup will either help you detect a corruption early (as it scans all tables) or actually help you save the data (as the last successful logical backup will have the sound data copy).

The inherent problem with all logical backups comes from the fact that they are created and restored by executing SQL statements against a running database system. While that allows for flexibility and portability, it also means that these backups result in a load on the database, and are generally quite slow. DBAs always frown when someone runs a query that reads all data from a table indiscriminately, and that’s exactly what logical backup tools usually do. The restore operation for a logical backup usually results in the interpreting and running of each statement as if it came from a regular client. This doesn’t mean that logical backups are bad or shouldn’t be used, but it’s a trade-off that must be remembered.

Physical Backups

Whereas logical backups are all about data, as in database contents, physical backups are all about data as in operating system files and internal RDBMS workings. Remember, in an example with MyISAM table being backed up, a physical backup was a copy of the files representing that table. Most of the backups and tools of this type are concerned with copying and transferring all or parts of database files.

Some examples of physical backups include the following:

- Cold database directory copy, meaning taken when the database is shut down, as opposed to hot — done while database is running.

- Storage snapshot of volumes and filesystems used by database.

- Copy of table data files.

- Stream of changes to database data files some form, most RDBMSes use a stream like this for crash recovery, and sometimes for replication. InnoDB’s redo log is a similar concept.

Recovery of a physical backup is usually done by copying back the files and making them consistent. Continuing with our examples, let’s review recovery for them:

- Cold copy can be moved to a desired location or server, and then used as a data directory by a MySQL instance, old or new.

- Snapshot can be restored in place or on another volume, then used by MySQL.

- Table files can be put in place of existing one.

- Replay of the changes stream against data files will recover their state to the last point in time.

Among the examples, we’ve shown the simplest physical backup that can be performed: a cold database directory backup. Yes, it’s simple and basic, but it’s a very powerful tool.

Physical backups, unlike logical ones, are very rigid, giving little leeway in control of what can be backed up and where the backup can be used. Generally speaking, most physical backups can only be used to restore the exact same state of a database or a table. Usually, these backups also put constraints on a target database software version and operating system. With some work, you can restore a logical backup from MySQL to PostgreSQL. However, a cold copy of MySQL data directory done on Linux may not work if restored on Windows, although it actually may work. You cannot take a physical backup if you don’t have physical access to the database server. This means that performing such backup on a managed database in the cloud is impossible: the vendor might be performing physical backups in the background, but you may not have a way to get them out.

Since physical backups are by nature a copy of all or a subset of original backup pages, any corruptions present in the original will be included in the backup. It’s important to remember that, because this property makes physical backups ill-suited for combating corruption.

You may wonder why you would use such a seemingly inconvenient way of backing up. The reason is that physical backups are fast. Operating on OS or even storage level, physical backup methods are sometimes the only possible way to actually back up a database. By way of example, a storage snapshot of a multi-terabyte volume might take few seconds or minutes, whereas querying and streaming that data for a logical backup might take hours or days. The same goes for recovery.

Overview of Logical and Physical Backups

We’ve now covered two categories of backups and are ready to move forward to the actual tools used for these backups in MySQL world. Before we do that, though, let’s summarize the differences between logical and physical backup tools by listing their properties.

Properties of logical backups:

- Description and contents of the logical structures

- Relatively slow to take and restore

- Very flexible, allows to rename objects, combine separate sources, perform partial restores, and more

- Human-readable and editable

- Not usually bound to a specific database version or platform

- Can extract data from corrupted tables and safeguard from corruption

- Suitable for backing up remote databases (for example, cloud)

Properties of physical backups:

- Byte-by-byte copy of parts of data, files, or entire filesystems/volumes

- Fast to take and restore

- Cumbersome to operate

- Offers little flexibility, will always result in the same structure upon restore

- Usually don’t allow for an easy cross-platform or even cross-version portability

- Backup will include corrupted pages

- Cannot back up remote databases without OS access

TIP

These are not conflicting approaches. In fact, a generally good idea is to perform both types of backups on a regular basis. They serve different purposes and satisfy different requirements.

Replication as a Backup Tool

Replication is a very wide topic that upcoming chapters cover in detail. In this section, we briefly discuss how replication relates to the concept of backing up and recovering the database.

NOTE

In MySQL world, replication is a type of logical backup. That’s because it’s based on transfering logical SQL statements.

In brief, replication is not a substitute for taking backups. The specifics of replications are such that they result in a full or partial copy of a target database. This lets you use replication in a lot of, but not all, possible failure scenarios involving MySQL. Let’s review two examples. They will be helpful later in the chapter as well.

Infrastructure Failure

Infrastructure is prone to failure: drives go bad, power goes out, fires happen. Almost no system can provide 100% uptime, and only vastly distributed ones can even get close. What that means is that eventually, any database will crash due to its host server failure. In a good case, a restart might be enough. In a bad case, part or all of the data may be gone.

Restoring and recovering a backup is by no means an instanteneous operation. In a replicated environment, a special operation called switchover can be performed to put a replica in place of the failed database. In a lot of cases, switchover saves a lot of time and allows for work on a failed system to proceed without too much rush.

Imagine a setup with two identical servers running MySQL. One is a dedicated primary, which receives all the connections and serves all queries. The other one is a replica. There’s a mechanism to redirect connections to the replica, with switchover resulting in 5 minutes of downtime.

One day, a hard disk drive goes bad in the primary server. It’s a simple server, so that alone results in crash and downtime. Monitoring catches the issue, and the DBA immediately understands that to restore the database on that server, they’ll need to install a new disk and then restore and recover the recent backup. The whole operation will take a couple of hours.

Switching over to a replica is a good idea in this case, because it saves a lot of valuable uptime.

Deployment bug

Software bugs are a fact of life that has to be accepted. The more complex the system, the higher the incidence of possible logical errors. While we all strive to limit and reduce bugs, we must understand that they will happen, and plan accordingly.

Imagine that a new version of an application is released that includes a database migration script. Even though both the new version and the script were tested in downstream environments, there’s a bug. Migration irrecoverably corrupts all customers’ last names that have “special” non-ascii symbols. The corruption is silent, since the script finishes successfully, and the issue is only noticed a week later by an angry customer, whose name is now incorrect.

Even though there’s a replica of the production database, it has the same data, and the same logical corruption. Switching over to a replica won’t help in this case, and backup taken prior to the migration must be restored to obtain a list of correct last names.

NOTE

There’s a way to make delayed replicas protect you from such situations, but the longer the delay, the less practical it is to operate such a replica. You can create a replica with a delay of a week, but you may need data for an hour ago. Usually, replica delays are measured in minutes and hours.

The two failure scenarios just discussed cover two distinct domains: physical and logical. Replication is a good fit for protection in case of physical issues, whereas it provides no (or little) protection from logical issues. Replication is a useful tool, but it’s no substitute for backups.

The mysqldump Program

The paramount logical backup type, and possibly the simplest way to back up a database online is to dump its contents as SQL statements. Dumping in computing usually means to output the contents of some system or its part, and the dump is a result. In the database world, a dump is usually the logical backup, and dumping is an action of obtaining such backup. The restoration of such a backup consists of applying these statements to a database. You can generate dumps manually using, for example, SHOW CREATE TABLE and some CONCAT to get INSERT statements from data rows in the tables, like this:

mysql> SHOW CREATE TABLE sakila.actor\G

*************************** 1. row ***************************

Table: actor

Create Table: CREATE TABLE `actor` (

`actor_id` smallint unsigned NOT NULL AUTO_INCREMENT,

`first_name` varchar(45) NOT NULL,

`last_name` varchar(45) NOT NULL,

`last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP

ON UPDATE CURRENT_TIMESTAMP,

PRIMARY KEY (`actor_id`),

KEY `idx_actor_last_name` (`last_name`)

) ENGINE=InnoDB AUTO_INCREMENT=201 DEFAULT CHARSET=utf8mb4

COLLATE=utf8mb4_0900_ai_ci

1 row in set (0.00 sec)

mysql> SELECT CONCAT("INSERT INTO actor VALUES",

-> "(",actor_id,",'",first_name,"','",

-> last_name,"','",last_update,"');")

-> AS insert_statement FROM actor LIMIT 1\G

*************************** 1. row ***************************

insert_statement: INSERT INTO actor VALUES

(1,'PENELOPE','GUINESS','2006-02-15 04:34:33');

1 row in set (0.00 sec)

That, however, becomes extremely impractical very fast. Moreover, there are more things to consider: order of statements, so that upon restore INSERT doesn’t run before the table is created, and ownership and consistency. Even though generating logical backups manually is good for understanding, it is tedious and error-prone. MySQL is bundled with a powerful logical backup tool called mysqldump, which hides most of this complexity.

The mysqldump program bundled with MySQL allows you to produce dumps from running database instances. The output of mysqldump is a number of SQL statements that can later be applied to the same or another instance of MySQL. mysqldump is a cross-platform tool, available on all the operating systems on which the MySQL server itself is available. As the resulting backup file is just a lot of text, it’s also platform-independent.

Command-line arguments to mysqldump are numerous, so it is always wise to review the MySQL Reference Manual before jumping into using the tool. However, the most basic scenario requires just one argument: target database name.

NOTE

Outputs of mysqldump are lengthy and ill-suited for printing in books. Here and elsewhere, the outputs are truncated to only include lines we’re interested in.

TIP

We recommend that you set up a client login path following instructions in “Login Path Configuration File” to root user and password. You then won’t need to specify an account and give its credentials to any of the commands we show in this chapter.

In the following example, mysqldump is invoked without output redirection, and the tool will print all the statements to standard output:

$ mysqldump sakila

...

--

-- Table structure for table `actor`

--

DROP TABLE IF EXISTS `actor`;

/*!40101 SET @saved_cs_client     = @@character_set_client */;

/*!50503 SET character_set_client = utf8mb4 */;

CREATE TABLE `actor` (

`actor_id` smallint unsigned NOT NULL AUTO_INCREMENT,

`first_name` varchar(45) NOT NULL,

`last_name` varchar(45) NOT NULL,

`last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP

ON UPDATE CURRENT_TIMESTAMP,

PRIMARY KEY (`actor_id`),

KEY `idx_actor_last_name` (`last_name`)

) ENGINE=InnoDB AUTO_INCREMENT=201 DEFAULT CHARSET=utf8mb4

COLLATE=utf8mb4_0900_ai_ci;

/*!40101 SET character_set_client = @saved_cs_client */;

--

-- Dumping data for table `actor`

--

LOCK TABLES `actor` WRITE;

/*!40000 ALTER TABLE `actor` DISABLE KEYS */;

INSERT INTO `actor` VALUES

(1,'PENELOPE','GUINESS','2006-02-15 01:34:33'),

(2,'NICK','WAHLBERG','2006-02-15 01:34:33'),

...

(200,'THORA','TEMPLE','2006-02-15 01:34:33');

/*!40000 ALTER TABLE `actor` ENABLE KEYS */;

UNLOCK TABLES;

...

You may notice that this output is probably more nuanced than the one you would expect. For example, there’s a DROP TABLE IF EXISTS statement, which prevents an error for the following CREATE when the table does exist on the target. LOCK and UNLOCK tables statements will improve data insertion performance, and so on.

Speaking of schema structure, it is possible to generate a dump that has no data. This can be useful to create a logical clone of the database, for example, for a development environment. Flexibility like this is one of the key features of logical backups and mysqldump:

$ mysqldump --no-data sakila

...

--

-- Table structure for table `actor`

--

DROP TABLE IF EXISTS `actor`;

/*!40101 SET @saved_cs_client     = @@character_set_client */;

/*!50503 SET character_set_client = utf8mb4 */;

CREATE TABLE `actor` (

`actor_id` smallint unsigned NOT NULL AUTO_INCREMENT,

`first_name` varchar(45) NOT NULL,

`last_name` varchar(45) NOT NULL,

`last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP

ON UPDATE CURRENT_TIMESTAMP,

PRIMARY KEY (`actor_id`),

KEY `idx_actor_last_name` (`last_name`)

) ENGINE=InnoDB AUTO_INCREMENT=201 DEFAULT CHARSET=utf8mb4

COLLATE=utf8mb4_0900_ai_ci;

/*!40101 SET character_set_client = @saved_cs_client */;

--

-- Temporary view structure for view `actor_info`

--

...

It’s also possible to create a dump of a single table in a database. In the next example, sakila is a database, whereas category is a target table:

$ mysqldump sakila category

Turning flexibility up a notch, you can dump just a few rows from the table by specifying a --where= or -w argument. As the name suggests, the syntax for the clause is same as for the WHERE in an SQL:

$ mysqldump sakila actor --where="actor_id > 195"

...

--

-- Table structure for table `actor`

--

DROP TABLE IF EXISTS `actor`;

CREATE TABLE `actor` (

...

--

-- Dumping data for table `actor`

--

-- WHERE:  actor_id > 195

LOCK TABLES `actor` WRITE;

/*!40000 ALTER TABLE `actor` DISABLE KEYS */;

INSERT INTO `actor` VALUES

(196,'BELA','WALKEN','2006-02-15 09:34:33'),

(197,'REESE','WEST','2006-02-15 09:34:33'),

(198,'MARY','KEITEL','2006-02-15 09:34:33'),

(199,'JULIA','FAWCETT','2006-02-15 09:34:33'),

(200,'THORA','TEMPLE','2006-02-15 09:34:33');

/*!40000 ALTER TABLE `actor` ENABLE KEYS */;

UNLOCK TABLES;

/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;

Examples so far cover only dumping the specific database: sakila. Sometimes it’s necessary to output every database, every object, and even every user. mysqldump is capable of that. The following command will effectively create a full and complete logical backup of a database instance:

$ mysqldump --all-databases --triggers \

--routines --events > dump.sql

Triggers are dumped by default, so this option won’t appear in future outputs. In the event you don’t want to dump triggers, you can use --no-triggers.

There are still a couple of problems with this command, however. First, even though we have redirected the output of the command to a file, the resulting file can be huge. However, its contents are likely going to be well suited for compression, though this depends on the actual data. Still, it’s a good idea to compress the output:

$ mysqldump --all-databases \

--routines --events | gzip > dump.sql.gz

On Windows, compressing an output through pipe is difficult, so just compress the dump.sql received by running a previous command. On a system that is is CPU-choked, like the little VM we’re using here, compression might add significant time to the backup process. That’s a trade-off that will have to be made for a particular system:

$ time mysqldump --all-databases \

--routines --events > dump.sql

real    0m24.608s

user    0m15.201s

sys     0m2.691s

$ time mysqldump --all-databases \

--routines --events | gzip > dump.sql.gz

real    2m2.769s

user    2m4.400s

sys     0m3.115s

$ ls -lh dump.sql*

-rw... 2.0G ... dump.sql

-rw... 794M ... dump.sql.gz

The second problem is that to ensure consistency, locks will be placed on tables, preventing writes while a database is being dumped (writes to other databases can continue). This is bad both for performance and backup consistency. The resulting dump is only consistent within the database, not across the whole instance. This default behavior is necessary because some of the storage engines that MySQL uses are non-transactional, mainly the older MyISAM. The default InnoDB storage engine, on the other hand, has an MVCC model that allows maintenance of a read snapshot. We cover different storage engines in more depth in “Alternative Storage Engines”, and locking in Chapter 6.

Utilizing InnoDB’s transaction capabilities is possible by passing a --single-transaction command-line argument to mysqldump. However, that removes table locking, thus making non-transactional tables prone to inconsistencies during the dump. If your system uses, for example, both InnoDB and MyISAM tables, it may be necessary to dump them separately, if no interruption of writes and consistency are required.

NOTE

Although --single-transaction ensures that writes can continue while mysqldump is running, there are still some caveats: DDL statements that are run concurrently might cause inconsistencies. And long-running transactions, such as one initiated by mysqldump, can have a negative impact on the overall instance performance.

The basic command to make a dump of a system using mainly InnoDB tables, which guarantees limited impact on concurrent writes is as follows:

$ mysqldump --single-transaction --all-databases \

--routines --events | gzip > dump.sql.gz

In the real world, you will probably have some more arguments to specify connection options. You might also script around the mysqldump statement to catch any issues and notify if anything went wrong.

Dumping with --all-databases includes internal MySQL databases such as mysql, sys, information_schema. That information is not always needed to restore your data, and might be harmful when restoring into an instance that already has some databases. However, you should remember that MySQL user details will only be dumped as part of the mysql database.

In general, using mysqldump and the logical backups it produces allows for the following:

- Easy transfer of the data between environments

- Editing of the data in place both by humans and programs. For example, you can delete personal or unnecessary data from the dump

- Finding certain data file corruptions

- Transfer of the data between major database versions, different platforms, and even databases

Bootstrapping Replication with mysqldump

The mysqldump program can be used to create a replica instance either empty or with data. To facilitate that, multiple command-line arguments are available. For example, when --master-data is specified, the resulting output will contain an SQL statement (CHANGE MASTER TO) that will set replication coordinates correctly on a target instance. When replication is later started using these coordinates on a target instance, there will be no gaps in data. In GTID-based replication topology, --set-gtid-purged can be used to achieve the same result. However, mysqldump will detect that gtid_mode=ON and include the necessary output even without any additional command-line argument.

An example of setting up replication with mysqldump is provided in “Create a Replica Using mysqldump”.

Loading Data from an SQL Dump File

When performing a backup, it’s always important to keep in mind that you’re doing that to be able to later restore the data. With logical backups, the restoration process is as simple as piping contents of the backup file to the mysql cli. As discussed earlier, the fact that MySQL has to be up for a logical backup restore makes for both good and bad consequences:

- You can restore a single object while other parts of your system are working normally, which is a plus.

- The process of restoration is inefficient and will load a system just like any regular client would if it decided to insert a large amount of data. That’s a minus.

Let’s take a look at a simple example with a single database backup and restore. As we’ve seen before, mysqldump will include necessary DROP statements into the dump, so even if the objects are present, they will be successfully restored:

$ mysqldump sakila > /tmp/sakila.sql

$ mysql -e "CREATE DATABASE sakila_mod"

$ mysql sakila_mod < /tmp/sakila.sql

$ mysql sakila_mod -e "SHOW TABLES"

+----------------------------+

| Tables_in_sakila_mod       |

+----------------------------+

| actor                      |

| actor_info                 |

...

| store                      |

+----------------------------+

Restoring SQL dumps like the one produced by mysqldump or mysqlpump (discussed in the next section) is a resource-heavy operation. By default, it’s also a serial process, which might take a significant amount of time. There are a couple of tricks you can use to make this process faster, but keep in mind that mistakes can lead to missing or incorrectly-restored data:

1. Parallel restore per-schema/per-database

2. Parallel restore of objects within schema

The first one is esily done if the dumping with mysqldump is done on a per-database basis. The backup process can also be parallelized if consistency across databases isn’t required (it won’t be guaranteed). The following example uses the & modifier, which instructs the shell to execute preceding command in the background:

$ mysqldump sakila > /tmp/sakila.sql &

$ mysqldump nasa > /tmp/nasa.sql &

Resulting dumps are independent. mysqldump doesn’t process users and grants unless the mysql database is dumped, so you need to take care of that. Restoration is just as straightforward:

$ mysql sakila < /tmp/sakila.sql &

$ mysql nasa < /tmp/nasa.sql &

On Windows, it’s also possible to send command execution to the background using PowerShell Start-Process or, in later versions, the same &.

The second option is a bit more involved. You need to either dump on a per-table basis (mysqldump sakila artists > sakila.artists.sql), which results in a straightforward restore, or you’ll need to go ahead and edit the dump file to split it into multiple ones. Taken to the extreme, you can even parallelize data insertion on the table level, although that’s probably not going to be practical.

Although this is doable, it’s preferable to use tools that are purpose-built for this task.

mysqlpump

mysqlpump is a utility program that comes bundled with MySQL versions 5.7 and later, which improves mysqldump in several areas, mainly the performance and usability. The key differentiators are as follows:

- Parallel dump capability

- Built-in dump compression

- Improved restore performance by delayed creation of secondary indexes

- Easier control over what objects are dumped

- Modified behavior of dumping user accounts

Using the program is very similar to using mysqldump. The main immediate difference is that when no arguments are passed, mysqlpump will default to dumping all of the databases (excluding INFORMATION_SCHEMA, performance_schema, ndbinfo, and sys schema). The other notable things are that there’s a progress indicator and that mysqlpump defaults to parallel dump with two threads:

$ mysqlpump > pump.out

Dump progress: 1/2 tables, 0/530419 rows

Dump progress: 80/184 tables, 2574413/646260694 rows

...

Dump progress: 183/184 tables, 16297773/646260694 rows

Dump completed in 10680

The concept of parallelism in mysqlpump is slightly complicated. You can use concurrency between different databases and between different objects within a given database. By default, when no other parallel options are specified, mysqlpump will use a single queue with two parallel threads to process all databases and user definitions (if requested). You can control the level of parallelism of the default queue using the --default-parallelism=N argument. To further fine-tune concurrency, you can set up multiple parallel queues to process separate databases. You should pay attention when choosing your desired concurrency level, since you could end up using most of the database resources for the backup run.

An important distinction to mysqldump when using mysqlpump lies in how the latter handles user accounts. mysqldump managed users just by dumping mysql.user and other relevant tables. If mysql database wasn’t included in the dump, no user information was preserved. mysqlpump improves on that by introducing the command-line arguments --users and --include-users. The first one tells the utility to add user-related commands to the dump for all users, and the second accepts a list of usernames. This is a great improvement on the old way of doing things.

Let’s combine all the new features to produce a compressed dump of non-system databases, user definitions, and use concurrency in the process:

$ mysqlpump --compress-output=zlib --include-users=bob,kate \

--include-databases=sakila,nasa,employees \

--parallel-schemas=2:employees \

--parallel-schemas=sakila,nasa > pump.out

Dump progress: 1/2 tables, 0/331579 rows

Dump progress: 19/23 tables, 357923/3959313 rows

...

Dump progress: 22/23 tables, 3755358/3959313 rows

Dump completed in 10098

NOTE

mysqlpump output can be compressed with the ZLIB or LZ4 algorithms. When OS-level commands lz and openssl zlib aren’t available, you can use the lz4_decompress and zlib_decompress utilities included in MySQL distribution.

A dump resulting from mysqlpump run is not suitable for parallel restore because data inside of it is interleaved. For example, the following is a result of a mysqlpump execution showing table creation amidst inserts to tables in different databases:

...,(294975,"1955-07-31","Lucian","Rosis","M","1986-12-08");

CREATE TABLE `sakila`.`store` (

`store_id` tinyint unsigned NOT NULL AUTO_INCREMENT,

`manager_staff_id` tinyint unsigned NOT NULL,

`address_id` smallint unsigned NOT NULL,

`last_update` timestamp NOT NULL DEFAULT

CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

PRIMARY KEY (`store_id`)

) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT

CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci

;

INSERT INTO `employees`.`employees` VALUES

(294976,"1961-03-19","Rayond","Khalid","F","1989-11-03"),...

mysqlpump is an improvement over mysqldump and adds important concurrency, compression, and object control features. However, the tool doesn’t allow parallel restore of the dump, and in fact makes it impossible. The only improvement to the restore performance is that secondary indexes are added after the main load is complete.

mydumper and myloader

mydumper and myloader are both part of the open source project mydumper. This set of tools attempts to make logical backups more performant, easier to manage, and more faced towards humans. We’re only going to mention this tool briefly — otherwise, we could easily run out of space in the book covering each and every possible MySQL backup variety.

These programs can either be installed by taking the freshest release from the project’s github page or by compiling the source. At the time of writing, the latest release is somewhat behind the current development in master. Step-by-step installation instructions are available in “Setting up the mydumper and myloader utilities”.

We previously showed how mysqlpump improves dumping performance but mentioned that its intertwined outputs don’t help with restoration. mydumper combines the parallel dumping approach with setting ground for parallel restore with myloader. That’s achieved by dumping every table into a separate file.

The default invocation of mydumper is very simple. The tool tries to connect to the database, initiates a consistent dumping, and creates a directory under the current one for the export files. Note how each table has its own file. By default, mydumper will also dump mysql and sys databases. Default parallelism of the dump operation is 4, meaning 4 separate tables will be read simultaneously. myloader invoked on this directory will be able to restore the tables in parallel.

$ mydumper

$ ls -ld export*

drwx... export-20210613-204512

$ ls -la export-20210613-204512

...

-rw... sakila.actor.sql

-rw... sakila.address-schema.sql

-rw... sakila.address.sql

-rw... sakila.category-schema.sql

-rw... sakila.category.sql

-rw... sakila.city-schema.sql

-rw... sakila.city.sql

...

Apart from parallel dumping and restore capabilities, mydumper has some more advanced features:

- Lightweight backup locks support. Percona Server for MySQL implements some additional lightweight locking that’s used by Percona XtraBackup. mydumper utilizes these locks by default when possible.

- When faced with prolonged metadata locking, mydumper may either fail or kill long-running queries, preventing its operation.

- Use of savepoints. mydumper uses a trick with transaction savepoints to minimize metadata locking.

mydumper and myloader are advanced tools taking logical backup capabilities to the maximum. However, being a community project, they lack the documentation and polish that other tools provide. Another major downside is the lack of any support or guarantees. Still, they can be a useful addition to database operators toolbelt.

Cold Backup and Filesystem Snapshots

The cornerstone of physical backups, cold backup is really just a copy of the data directory and other necessary files, done while the database instance is down. Cold backup isn’t frequently used but it can save the day when you need to create a consistent backup quickly. With databases now regularly approaching the multi-terabyte size range, just copying the files can take a very long time. However, the cold backup still has its good points:

- Very fast, arguably the fastest backup method (apart from snapshots)

- Straightforward

- Easy to use, hard to do wrong

- Requires complete downtime

Modern storage systems and some filesystems have readily available snapshot capabilities. They allow you to create near-instantaneous copies of volumes of arbitrary size by utilizing internal mechanisms. The properties of each and every snapshot-capable system will vary widely, making it impossible for us to cover all of them. However, we can still talk a bit about them from the database perspective.

Most of the snapshots will be Copy-on-Write (or COW), and will be internally consistent to some point in time. However, we already know that database files aren’t consistent on disk, especially with transactional storage engines like InnoDB. This makes it somewhat difficult to get the snapshot backup right. There are two options:

Cold backup snapshot

When the database is shut down, its data files may still not be perfectly consistent. But if you do a snapshot of all of the database files (including InnoDB redo logs, for example), they all together will allow for the database to start. That’s only natural, because otherwise the database would lose data on every restart. Don’t forget that you may have database files split among many volumes. You will need to have all of them. This method will work for all storage engines.

Hot backup snapshot

With a running database, taking a snapshot correctly is a greater challenge than when the database is down. If your database files are located over multiple volumes, you cannot guarantee that snapshots, even initiated simultaneously, will be consistent to the same point in time, which can lead to disastrous results. Moreover, non-transactional storage engines like MyISAM don’t guarantee consistency for files on disk while the database is running. That’s actually true for InnoDB as well, but InnoDB’s redo logs are always consistent (unless safeguards are disabled), and MyISAM lacks this functionality.

The recommended way to do a hot backup snapshot would therefore be to utilize some amount of locking. Since the snapshot-taking process is usually a quick one, the resulting downtime shouldn’t be significant:

1. Create a new session, and lock all of the tables with the FLUSH TABLES WITH READ LOCK; command. This session cannot be closed, or else locks will be released.

2. Optionally, record the current binlog position by running the SHOW MASTER STATUS; command.

3. Create snapshots of all volumes where MySQL’s database files are located according to the storage system’s manual.

4. Unlock tables with the UNLOCK TABLES; command in the session opened initially.

This general approach should be suitable for most if not all of the current storage and filesystems capable of doing snapshots. Note that they all differ subtly in the actual procedure and requirements. Some cloud vendors require you to additionally perform an fsfreeze on the filesystems.

Always test your backups thoroughly before implementing them in production and trusting them with your data. You can only trust a solution that you tested and are comfortable using. Copying arbitrary backup strategy suggestions is not a very good idea.

Percona XtraBackup

The logical step forward in physical backups is implementing so-called “hot” backups. That is, making a copy of database files while the database is running. We’ve already mentioned that MyISAM tables can be copied, but that doesn’t work for InnoDB and other transactional storage engines like MyRocks. The problem therefore is that you can’t just copy the files because the database is constantly undergoing changes. For example, InnoDB might flush some dirty pages in the background even if no writes are hitting the database right now. You can test your luck and copy the database directory under a running system and try to restore that directory and start a MySQL server using it. Chances are, it may not work. While it can also work, we strongly recommend against taking chances with database backups.

The capability to perform hot backups is built-in to three main MySQL backup tools: Percona XtraBackup, MySQL Enterprise Backup, and mariabackup. We’ll briefly talk about all of them but will mainly concentrate on the XtraBackup utility. It’s important to understand that all the tools share properties, thus knowing how to use one tool will help using the others.

Percona XtraBackup is a free and open source software, maintained by Percona and the wider MySQL community. This tool is capable of performing online backups of MySQL instances with InnoDB, MyISAM, and MyRocks tables. This program is only available on Linux. Note that it’s impossible to use XtraBackup with recent versions of MariaDB: only MySQL and Percona Server are supported. For MariaDB, use the mariabackup utility that we mention in “mariabackup”.

Here is XtraBackup’s operation in essence:

1. Records current LSN--Log Sequence Number—an internal number of the operation.

2. Starts accumulating InnoDB redo data--the type of data InnoDB stores for crash recovery.

3. Locks tables in the least intrusive way possible.

4. Copies InnoDB tables.

5. Locks non-transactional engines tables completely.

6. Copies MyISAM tables.

7. Unlocks all tables.

8. Processes MyRocks if present.

9. Puts accumulated redo data alongside the copied database files.

The main idea between XtraBackup and hot backups in general is combining the no-downtime nature of logical backups with the performance and relative lack of performance impact of cold backup. XtraBackup doesn’t guarantee no disruption of service, but it’s a great step forward compared with regular cold backup. The lack of performance impact means that XtraBackup will use some CPU and IO, but only that needed to copy the database files. Logical backups, on the other hand, must pass each row through all of the database internals, making them inherently slow.

NOTE

XtraBackup requires physical access to the database files and cannot be run remotely. This makes it unsuitable for doing off-site backups of managed databases (DBaaS), for example. Some cloud vendors, however, allow to import databases using backups made by this tool.

XtraBackup utility is widely available in various Linux distributions’ repositories, and thus can easily be installed using a package manager. Alternatively, you can download packages and binary distributions directly from the XtraBackup Downloads page on Percona’s website.

WARNING

To back up MySQL 8, XtraBackup 8.0 must be used. Versions of XtraBackup and MySQL ideally should also match: XtraBackup 8.0.25 is guaranteed to work with MySQL 8.0.25. For MySQL 5.7 and older releases use XtraBackup 2.4.

Backing up and Recovering

Unlike other tools we’ve mentioned previously, XtraBackup, by the nature of it being a physical backup tool, requires not only access to MySQL server, but also read access to the database files. On most MySQL installations, that usually means that xtrabackup program should be run from the root user, or sudo must be used. We’ll be using the root user throughout this section, and we set up a login path using steps from “Login Path Configuration File”.

First, we need to run the basic xtrabackup command:

# xtrabackup --host=127.0.0.1 \

--target-dir=/tmp/backup --backup

...

Using server version 8.0.25

210613 22:23:06 Executing LOCK INSTANCE FOR BACKUP...

...

210613 22:23:07 [01] Copying ./sakila/film.ibd

to /tmp/backup/sakila/film.ibd

210613 22:23:07 [01]        ...done

...

210613 22:23:10 [00] Writing /tmp/backup/xtrabackup_info

210613 22:23:10 [00]        ...done

xtrabackup: Transaction log of lsn (6438976119)

to (6438976129) was copied.

210613 22:23:11 completed OK!

In case the login path didn’t work, you should pass root user’s credentials to xtrabackup using --user and --password command-line arguments. XtraBackup will usually recognize the target server’s data directory by reading the default option files. However, if that didn’t work, or you have multiple installations of MySQL, you may need to specify --datadir option, too.

TIP

While we use /tmp/backup as the backup’s destination path for our example, you should avoid storing important files under /tmp. That’s especially true for backups.

The result of that xtrabackup --backup invication is a bunch of database files, which are actually not consistent to any point in time, and a chunk of redo data that InnoDB won’t be able to apply.

# ls -l /tmp/backup/

...

drwxr-x---.  2 root root       160 Jun 13 22:23 mysql

-rw-r-----.  1 root root  46137344 Jun 13 22:23 mysql.ibd

drwxr-x---.  2 root root        60 Jun 13 22:23 nasa

drwxr-x---.  2 root root       580 Jun 13 22:23 sakila

drwxr-x---.  2 root root       580 Jun 13 22:23 sakila_mod

drwxr-x---.  2 root root        80 Jun 13 22:23 sakila_new

drwxr-x---.  2 root root        60 Jun 13 22:23 sys

...

To make the backup ready for future restore, another phase must be performed — preparation. There’s no need to connect to a MySQL server for that:

# xtrabackup --target-dir=/tmp/backup --prepare

...

xtrabackup: cd to /tmp/backup/

xtrabackup: This target seems to be not prepared yet.

...

Shutdown completed; log sequence number 6438976524

210613 22:32:23 completed OK!

The resulting data directory is actually perfectly ready to be used. You can start up a MySQL instance pointing directly to this directory, and it will work. A very common mistake done here is trying to start MySQL Server under mysql user while the restored and prepared backup is owned by root or another OS user. Make sure to incorporate chown and chmod as required into your backup recovery procedure. However, there’s a useful user experience feature of --copy-back available. xtrabackup preserves the original database file layout locations, and invoked with --copy-back will restore all files to their original locations:

# xtrabackup --target-dir=/tmp/backup --copy-back

...

Original data directory /var/lib/mysql is not empty!

That didn’t work, because our original MySQL server is still running, and its data directory is not empty. XtraBackup will refuse to restore a backup unless the target data directory is empty. That should protect you from accidentally restoring a backup. Let’s shut down the running MySQL server, remove or move its data directory, and restore the backup:

# systemctl stop mysqld

# mv /var/lib/mysql /var/lib/mysql_old

# xtrabackup --target-dir=/tmp/backup --copy-back

...

210613 22:39:01 [01] Copying ./sakila/actor.ibd

to /var/lib/mysql/sakila/actor.ibd

210613 22:39:01 [01]        ...done

...

210613 22:39:01 completed OK!

After that, files are in their correct locations, but owned by root:

# ls -l /var/lib/mysql/

drwxr-x---. 2 root root      4096 Jun 13 22:39 sakila

drwxr-x---. 2 root root      4096 Jun 13 22:38 sakila_mod

drwxr-x---. 2 root root      4096 Jun 13 22:39 sakila_new

We need to change the owner of the files back to mysql (or the user used in your system) and fix directory permissions. Once that’s done, MySQL can be started and data can be verified:

# chown -R mysql:mysql /var/lib/mysql/

# chmod o+rx /var/lib/mysql/

# systemctl start mysqld

# mysql sakila -e "SHOW TABLES;"

+----------------------------+

| Tables_in_sakila           |

+----------------------------+

| actor                      |

...

| store                      |

+----------------------------+

TIP

The best practice is to do both backup and prepare phases during the backup phase, minimizing the amount of possible surprises later. Imagine having the prepare phase fail when you’re trying to recover some financial data.

Advanced Features

In this section we discuss some of XtraBackup’s more advanced features. They are not required to use the tool, and we give them just as a brief overview.

Database files verification

While performing the backup, XtraBackup will verify the checksums of all of the pages of the datafiles it’s processing. This is an attempt to alleviate the inherent problem of physical backups that they will contain corruptions of the source database. We recommend augmenting this check with other steps shown in “Testing and Verifying Your Backups”.

Compression

Even though copying physical files is much faster than querying the database, you may still want to improve performance, especially if you have spare capacity. XtraBackup uses the qpress tool for compression. This tool is available from the percona-release package.

# xtrabackup --host=127.0.0.1 \

--target-dir=/tmp/backup_compressed/ \

--backup --compress

Parallelism

It’s possible to make both backup and copy-back processes parallel by using the --parallel command-line argument.

Encryption

In addition to being able to work with encrypted databases, it’s also possible for XtraBackup to create encrypted backups.

Streaming

Instead of creating a directory full of backed up files, XtraBackup can stream the resulting backup in an xbstream format. This results in a more portable backups, and allows integration with xbcloud. You can stream backups over SSH, for example.

Cloud upload

Backups taken with XtraBackup can be uploaded to any S3-compatible storage using xbcloud. S3 is the Amazon’s object storage facility and an API that is widely adopted by many companies. This tool only works with backups streamed through the xbstream, discussed earlier.

Incremental Backups with XtraBackup

As described, and as XtraBackup works by default, hot backup is still a copy of every byte of information in the database. But in a lot of cases, databases undergo change at an irregular rate — new data is being added while old data doesn’t change that much or at all. A stream of financial records is added, and accounts get modified, but in a given week, only a few percent of accounts are changed. Thus, the next logical step improving the hot backups is the ability to perform so-called incremental backups, or a backup of the changed data. That will allow you to perform backups more frequently by decreasing the need for space.

For incremental backup to work, you need first to have a full backup of the database, called a base backup — otherwise there’s nothing to increment from. Once your base backup is ready, you can perform any number of incremental backups, each consisting of changes since the previous one (or from the base backup in case of the first incremental backup). Taken to the extremes, you could create an incremental backup every minute, achieving something called Point-in-time Recovery (PITR), but this is not very practical, and as you will soon learn there are better ways to do that.

Here’s an example of XtraBackup commands to create a base backup and then an incremental backup. Notice how the incremental backup points to the base backup via the --incremental-basedir argument:

# xtrabackup --host=127.0.0.1 \

--target-dir=/tmp/base_backup --backup

# xtrabackup --host=127.0.0.1 --backup \

--incremental-basedir=/tmp/base_backup \

--target-dir=/tmp/inc_backup1

If you check backup sizes, you’ll see that incremental backup is very small compared to the base backup:

# du -sh /tmp/base_backup

2.2G    /tmp/base_backup

6.0M    /tmp/inc_backup1

Let’s create another incremental backup. In this case, we’ll pass previous incremental backup’s directory as a base directory:

Once that’s done, we can create yet another incremental backup. We’ll make it incremental from the first such backup:

# xtrabackup --host=127.0.0.1 --backup \

--incremental-basedir=/tmp/inc_backup1 \

--target-dir=/tmp/inc_backup2

210613 23:32:20 completed OK!

You may notice that it’s possible to specify the original base backup’s directory as an --incremental-basedir for each new incremental backup. In fact, that results in a completely valid backup, which is a variation of an incremental backup (or the other way around). Such incremental backups that contain changes not since the previous incremental backup but since the base backup are usually called cumulative backups. Incremental backups targeting any previous backup are differential backups. Cumulative incremental backups usually consume more space, but can considerably decrease the time needed for the prepare phase when backup is restored.

Importantly, the prepare process for incremental backups differs from what it is for regular backups. Let’s prepare the backups we’ve just taken, starting with the base backup:

# xtrabackup --prepare --apply-log-only \

--target-dir=/tmp/base_backup

The --apply-log-only argument tells xtrabackup to not finalize the prepare process, as we still need to apply changes from the incremental backups. Let’s do that:

# xtrabackup --prepare --apply-log-only \

--target-dir=/tmp/base_backup \

--incremental-dir=/tmp/inc_backup1

# xtrabackup --prepare --apply-log-only \

--target-dir=/tmp/base_backup \

--incremental-dir=/tmp/inc_backup2

All commands should report completed OK! in the end. Once the --prepare --apply-log-only is run, base backup advances to the point of the incremental backup, making Point-in-time Recovery to an earlier time impossible. So, it’s not a good idea to prepare immediately when performing incremental backups. To finalize the prepare process, the base backup with the changes applied from incremental backups, must be prepared normally:

# xtrabackup --prepare --target-dir=/tmp/base_backup

Once the base backup is “fully” prepared, attempts to apply incremental backups will fail with the following message:

xtrabackup: This target seems to be already prepared.

xtrabackup: error: applying incremental backup needs

target prepared with --apply-log-only.

Incremental backups are inefficient when the relative amount of changes in the database is high. In the worst case, where every row in the database was changed between full backup and incremental backup, the latter will actually just be a full backup, storing 100% of the data. Incremental backups are most efficient when most of the data is appended, and the relative amount of old data being changed is low. There are no rules regarding this, but if 50% of your data changes between your base backup and incremental backup, consider not using incremental backups.

Other Physical Backup Tools

XtraBackup isn’t the only tool available that’s capable of performing hot MySQL physical backups. Our choice for explaining the concepts using this particular tool were driven by our experience with it. However, that doesn’t mean that other tools are worse in any way. They may well be better. This book’s volume is limited, and the yopic of backing up is very wide. We could write a Backing Up MySQL book of a considerable volume!

That said, let’s take a look at two other physical backup tools readily-available.

MySQL Enterprise Backup

Called MEB for short, this tool is a available as part of the MySQL Enterprise Edition product provided by Oracle. It’s a closed-source proprietary tool, which is similar in functionality to XtraBackup. You’ll find comprehensive documentation for it at the mysql.com site. Two tools are currently at feature parity, so almost everything that was covered for XtraBackup will be true for MEB as well.

MEB’s standout property is that it’s truly a cross-platform solution. XtraBackup only works on Linux, whereas MEB does so also on Solaris, Windows, macOS, and FreeBSD. MEB doesn’t support flavors of MySQL other than Oracle’s standard one.

Some additional features that MEB has, which are not available in XtraBackup, include the following:

- Backup progress reporting

- Offline backups

- Tape backups through Oracle Secure Backups

- Binary and Relay log backups

- Table rename at restore time

mariabackup

mariabackup is a tool by MariaDB for backing up MySQL databases. Originally forked from XtraBackup, this is a free open source tool that is available on Linux and Windows. The standout property of mariabackup is its seamless work with the MariaDB fork of MySQL, which continues to diverge significantly from both the mainstream MySQL and Percona Server. Since this is a direct fork of XtraBackup, you may find many similarities in how the tools are used and in their properties. Some of XtraBackup’s newer features like backup encryption and secondary index omission are not present in mariabackup. However, using XtraBackup to back up MariaDB is currently impossible.

Point-in-Time Recovery

Now that you’re accustomed to the concept of hot backups, you have almost everything you need to complete the backup toolkit. So far all the backup types that we’ve discussed share a similar trait — a deficiency. They allow restore only at the point in time where they were taken. If you have two backups, one done at 23:00 on Monday, and the second at 23:00 on Tuesday, you cannot restore to 17:00 on Tuesday.

Remember the infrastructure failure example given in the beginning of the chapter? Now, let’s make it worse and say that the data is gone, all the drives failed, and there’s no replication. The event happened on Wednesday at 21:00. Without Point-in-Time Recovery, or PITR, and with daily backups taken at 23:00, this means that you’ve just lost a full day’s worth of data irrevocably. Arguably, incremental backups done with XtraBackup allow you to make that problem somewhat less pronounced, but they still leave some room for data loss, and it’s less than practical to be running them very often.

MySQL maintans a journal of transactions called Binary Log. By combining any of the backup methods we’ve discussed so far with binary logs, we get the ability to restore to an arbitrary point in time with an up to a transaction resolution. It’s very important to understand that you need both a backup and binary logs after the backup for this to work. You also cannot go back in time, so you cannot recover the data to a point in time before your oldest base backup or dump was created.

Binary logs contain both timestamps and transaction identifiers, but you have to rely on transaction ids for recovery. It’s not possible to tell MySQL to recover to a certain timestamp, so you will need to identify a transaction closest to that. This is not a problem when you want to recover to the latest point in time, but can be extremely important and helpful when trying to perform a restore to fix a logical inconsistency, like the one described in the Deployment bug example in “Deployment bug”.

One interesting peculiarity of MySQL is that it allows for PITR for logical backups. Section “Loading Data from an SQL Dump File” discusses storing binlog position for replica provisioning using mysqldump. The same binlog position can be used as a starting point for PITR. Every backup type in MySQL is suitable for PITR, unlike other databases. To facilitate this property, make sure to note the binlog position when taking your backup. Some backup tools do that for you. When using those that don’t, you can run SHOW MASTER STATUS; to get that data.

Binary Logs Technical Background

MySQL differs from a lot of other mainstream RDBMS due to the fact that it supports multiple storage engines covered in “Alternative Storage Engines”. Not only that, but it supports multiple storage engines for tables inside a single database. The result of that is that some concepts in MySQL are different from other systems.

Binary logs in MySQL are essentially transaction logs. When binary logging is enabled, every transaction (excluding read-only transactions) will be reflected in the binary logs. There are tree ways to write transactions to binary logs:

- Statement: In this mode, statements are logged to the binary logs as they were, which might cause indeterministic execution in replication scenarios.

- Row: In this mode, statements are broken down into minimal DML operations, each modifying a single specific row. Although it guarantees deterministic execution, this mode is the most verbose and results in the largest files and thus IO overhead.

- Mixed: In this mode, “safe” statements are logged as is, while others are broken down.

Usually, the transaction log is used for crash recovery, replication, and for PITR. However, in the case of MySQL, its multi-engined nature doesn’t allow the use of binlog for crash recovery. Each engine maintains its own crash recovery mechanism. For example, MyISAM is not crash-safe, whereas InnoDB has its own Redo logs. Every transaction in MySQL is a distributed transaction with a two-phase commit, to allow for this multi-engined nature. Each committed transaction is guaranteed to be reflected in storage engine’s redo logs, if the engine is transactional, as well as in MySQL’s own transaction log: binary logs.

NOTE

Binary logging has to be enabled in your MySQL instance for PITR to be possible. You should also default to having sync_binlog=1, which guarantees the durability of each write. Refer to the MySQL documentation to understand tradeoffs of disabling the binlog syncing.

We talk more about binary logs in Chapter 13.

Preserving Binary Logs

To allow Point-in-Time Recovery you must preserve binary logs starting from the binlog position of the oldest backup. There are few ways to do this:

- Copy or sync binary logs “manually” using some readily available tool like rsync. Remember that MySQL continues to write to the current binary log file. If you’re copying files instead of continuously syncing them, do not copy the current binary log file. Continuously syncing files will take care of this problem by overwriting the partial file once it becomes non-current.

- Use mysqlbinlog to copy individual files or stream binlog continuously. Steps are readily available in the documentation.

- Use MySQL Enterprise Backup, which has a built-in binlog copy feature. Note that it’s not a continuous copying, but relies on incremental backups to have binlog copies. This allows for PITR between two backups.

- Allow MySQL server to store enough binary logs in its data directory by seting a high value for binlog_expire_logs_seconds or expire_logs_days variables. This option should ideally not be used, or can be used in addition to any of those described above. If anything happens to the data directory, like a filesystem corruption, binary logs stored there may also get lost.

Identifying a PITR Target

You may use PITR technique to achieve two objectives:

1. Recover to the latest point in time.

2. Recover to an arbitrary point in time.

The first one, as discussed earlier, is useful to recover a completely lost database to the latest available state. The second is useful to get data as it was before. An example of a case when this can be useful is the deployment bug example given in “Deployment bug”. To recover lost or incorrectly modified data, you can restore a backup, and then recover it to a point in time just before the deployment was executed.

Identifying the actual specific time when an issue happened can be a challenge. More often than not, the only way for you to find the desired point in time is by inspecting binary logs written around the issue time frame. For example, if you suspect that a table was dropped, you may look for the table name, then for any DDL, or specifically for a DROP TABLE statement.

Let’s illustrate that example. First, we need to actually drop the table, and we’ll drop the facilities table we created in “Loading Data from Comma-Delimited Files”. However, before that we’ll insert a record that’s for sure missing in the original backup:

mysql> INSERT INTO facilities(center)

-> VALUES ('this row was not here before');

Query OK, 1 row affected (0.01 sec)

mysql> DROP TABLE nasa.facilities;

Query OK, 0 rows affected (0.02 sec)

We could now go back and restore one of the backups we’ve taken throughout this chapter, but then we lose any changes made to the database until the DROP. Instead, we’ll use mysqlbinlog command to inspect binary logs contents and find the recovery target just before the DROP was run. To find the list of binary logs available in the data directory, you can run the following command:

mysql> SHOW BINARY LOGS;

+---------------+-----------+-----------+

| Log_name      | File_size | Encrypted |

+---------------+-----------+-----------+

| binlog.000291 |       156 | No        |

| binlog.000292 |       711 | No        |

+---------------+-----------+-----------+

2 rows in set (0.00 sec)

WARNING

MySQL won’t keep binary logs in its data directory forever. Binary logs are removed automatically by the MySQL server when they are older than duration specified under binlog_expire_logs_seconds or expire_log_days, and also can be removed manually by running PURGE BINARY LOGS. If you want to make sure binary logs are available, you should preserve them outside of the data directory as described in “Preserving Binary Logs”.

Now that the list of binary logs is available, you can either try to search in them from the last one to the oldest one, or you can just dump all their contents together. In our example, files are small, so we can do the latter approach. In any case, mysqlbinlog command is used:

# cd /var/lib/mysql

# mysqlbinlog binlog.000291 binlog.000292 \

-vvv --base64-output=decode-rows > /tmp/mybinlog.sql

Inspecting the output file, we can find the problematic statement:

...

#210613 23:32:19 server id 1  end_log_pos 200 ... Rotate to binlog.000291

...

# at 499

#210614  0:46:08 server id 1  end_log_pos 576 ...

# original_commit_timestamp=1623620769019544 (2021-06-14 00:46:09.019544 MSK)

# immediate_commit_timestamp=1623620769019544 (2021-06-14 00:46:09.019544 MSK)

/*!80001 SET @@session.original_commit_timestamp=1623620769019544*//*!*/;

/*!80014 SET @@session.original_server_version=80025*//*!*/;

/*!80014 SET @@session.immediate_server_version=80025*//*!*/;

SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/;

# at 576

#210614  0:46:08 server id 1  end_log_pos 711 ... Xid = 25

use `nasa`/*!*/;

SET TIMESTAMP=1623620768/*!*/;

DROP TABLE `facilities` /* generated by server */

/*!*/;

SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;

DELIMITER ;

...

We should stop our recovery before 2021-06-14 00:46:09.019544 Moscow time, or at the binary log position 499. We’ll also need all binary logs from the latest backup until and including binlog.00291. Using this information we can proceed to backup restoration and recovery.

Example Point-in-Time-Recovery: XtraBackup

On its own, XtraBackup doesn’t provide PITR capabilities. You need to add the additional step of running mysqlbinlog to replay the binlog contents on the restored database.

1. Restore the backup. See “Backing up and Recovering” for exact steps.

2. Start MySQL server. If you are restoring on the source instance directly, it is recommended to use skip-networking option to prevent non-local clients from accessing the database. Otherwise, some clients may change the database before you’ve actually finished the recovery.

3. Locate the backup’s binary log position. It’s available in the xtrabackup_binlog_info file in the backup directory.

# cat /tmp/base_backup/xtrabackup_binlog_info

binlog.000291   156

4. Find the desired timestamp or binlog position to which you want to recover. For example, immediately before a DROP TABLE was executed, as discussed before.

5. Replay the binlogs up to a desired timestamp. For this example, we’ve preserved binary log binlog.000291 separately, but you would use your centralized binlog storage for the source of binary logs. mysqlbinlog command is used:

# mysqlbinlog /opt/mysql/binlog.000291 \

/opt/mysql/binlog.000292 --start-position=156 \

--stop-datetime="2021-06-14 00:46:00" | mysql

6. Make sure that recovery was successful and data is not missing. In our case, we’ll look for the record we added to the facilities table before dropping it:

mysql> SELECT center FROM facilities

-> WHERE center LIKE '%before%';

+------------------------------+

| center                       |

+------------------------------+

| this row was not here before |

+------------------------------+

1 row in set (0.00 sec)

Example Point-in-Time-Recovery: mysqldump

The steps necessary for PITR with mysqldump are analogous to the steps taken earlier for XtraBackup. We’re only showing this for completeness and so that you can see that PITR is going to be very similar to each and every backup type in MySQL.

1. Restore the SQL dump. Again, if your recovery target server is the backup source, you probably want to make it inaccessible to clients.

2. Locate the binary log position from the mysqldump backup file.

CHANGE MASTER TO MASTER_LOG_FILE='binlog.000010',

MASTER_LOG_POS=191098797;

3. Find the desired timestamp or binlog position to which you want to recover. For example, immediately before a DROP TABLE was executed, as discussed before.

4. Replay the binlogs up to a desired timestamp.

# mysqlbinlog /path/to/datadir/mysql-bin.000010 \

/path/to/datadir/mysql-bin.000011 \

--start-position=191098797 \

--stop-datetime="20-05-25 13:00:00" | mysql

Export and Import InnoDB Tablespaces

One of the major downsides of physical backups is that they usually require a significant portion of your database files to be copied at the same time. Although a storage engine like MyISAM allows for the copying of idle tables’ data files, you cannot guarantee consistency of InnoDB files. There are situations, though, where you need to transfer only a few tables, or just one table. So far the only option we’ve seen for that would be to utilize logical backups, which can be unacceptably slow. The Export and Import Tablespaces feature, officially called the Transportable Tablespace feature is a way to get best of both worlds. We will also call this feature export/import for brevity.

The transportable tablespace feature lets you combine the performance of online physical backup with the granularity of a logical one. This feature, in essence, offers the ability to do an online copy of an InnoDB table’s data files to be used for import into the same or a different table. Such a copy can serve as a backup, or as media for data transfer between separate MySQL installations.

Why use export/import when a logical dump achieves the same thing? Export/import is much faster and, apart from the table being locked, doesn’t impact the server significantly. This is especially true for import. With table sizes in multi-gigabyte range, this is one of the few feasible options for data transfer.

Technical Background

To help you understand how this feature works, we’ll look at two concepts: one of a tablespace and one of a physical backup.

Short recap: For physical backup to be consistent, we can generally take two routes. The first is to shut down the instance, or otherwise make data read-only in a guaranteed manner. The second is to make data files consistent to point in time, and then accumulate all changes between that point in time and end of the backup. The transportable tablespaces feature works in the first way, requiring the table to be made read-only for a short while.

Tablespace is a file that stores data for the tables and indexes. By default, InnoDB uses an innodb_file_per_table option, which forces the creation of a dedicated tablespace file for each table. But it’s possible to create a tablespace that will contain data for multiple tables. Additionally, you can use the “old” behavior of having all tables reside in a single ibdata tablespace. Export is only supported for the default configuration when a dedicated tablespace is used by each table. Tablespaces exist separately for each partition in a partitioned table, which allows for an interesting ability to transfer partitions between separate tables or create a table from a partition.

Export Tablespace

Now that both concepts have been covered, you know what needs to be done for the export. However, one thing that’s still missing is the table definition. Even though most of the InnoDB tablespace files actually contain a redundant copy of the data dictionary records for their tables, current implementation of transportable tablespaces requires for a table to be present on the target before import.

Steps for exporting of a tablespace are:

1. Get the table definition

2. Stop all the writes to the table (or tables) and make it consistent

3. Prepare the extra files necessary for import of the tablespace later

- .cfg file stores metadata used for schema verification

- .cfp file is generated only when encryption is used, will have a transition key necessary for decrypting the tablespace by the target server

To get the table definition, you can use a SHOW CREATE TABLE command that we showed quite a few times throughout this book. All other steps are done automatically by MySQL with a single command: FLUSH TABLE ... FOR EXPORT. That command locks the table and generates the additional file (or files, if encryption is used) near the regular .ibd file of the target table. Let’s export the actor table from sakila database:

mysql> USE sakila

mysql> FLUSH TABLE actor FOR EXPORT;

Query OK, 0 rows affected (0.00 sec)

The session where FLUSH TABLE was executed should remain open, because actor table will be released as soon as the session is terminated. A new file actor.cfg should appear near the regular actor.ibd under the MySQL data directory. Let’s verify:

# ls -1 /var/lib/mysql/sakila/actor.*

/var/lib/mysql/sakila/actor.cfg

/var/lib/mysql/sakila/actor.ibd

This pair of .ibd and .cfg files can now be copied somewhere and used later. Once you’ve copied the files, it’s generally advisable to release the locks on the table by running the UNLOCK TABLES statements, or closing the session where FLUSH TABLE was called. Once all that is done, you have a tablespace ready for import.

NOTE

Partitioned tables have multiple .ibd files, and each of them gets a dedicated .cfg.

learning_mysql_partitioned#p#p0.cfg

learning_mysql_partitioned#p#p0.ibd

learning_mysql_partitioned#p#p1.cfg

learning_mysql_partitioned#p#p1.ibd

Import Tablespace

Importing a tablespace is quite straightforward. It consists of the following steps:

1. Create a table using the preserved definition. It is not possible to change the table’s definition in any way.

2. Discard the table’s tablespace.

3. Copy .ibd and .cfg files over.

4. Alter the table to import the tablespace.

If the table exists on the target server and has the same definition, then there’s no need to perform step number 1.

Let’s restore the actor table in another database on the same server. The table needs to exist:

mysql> USE nasa

mysql> CREATE TABLE `actor` (

->  `actor_id` smallint unsigned NOT NULL AUTO_INCREMENT,

->  `first_name` varchar(45) NOT NULL,

->  `last_name` varchar(45) NOT NULL,

->  `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP

->    ON UPDATE CURRENT_TIMESTAMP,

->  PRIMARY KEY (`actor_id`),

->  KEY `idx_actor_last_name` (`last_name`)

-> ) ENGINE=InnoDB AUTO_INCREMENT=201 DEFAULT

->    CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

Query OK, 0 rows affected (0.04 sec)

As soon as actor table is created, MySQL creates an .ibd file for it:

# ls /var/lib/mysql/nasa/

actor.ibd  facilities.ibd

This brings us to the next step: discarding this new table’s tablespace. That’s done by running a special ALTER TABLE:

mysql> ALTER TABLE actor DISCARD TABLESPACE;

Query OK, 0 rows affected (0.02 sec)

And the .ibd file will be gone:

# ls /var/lib/mysql/nasa/

facilities.ibd

WARNING

Discarding the tablespace leads to total deletion of the associated tablespace files, and is not a recoverable operation. You will need to recover from a backup if you run ALTER TABLE ... DISCARD TABLESPACE accidentally.

We can now copy the exported tablespace of the original actor table along with a .cfg file:

# cp -vip /opt/mysql/actor. /var/lib/mysql/nasa/*

'/opt/mysql/actor.cfg' -> '/var/lib/mysql/nasa/actor.cfg'

'/opt/mysql/actor.ibd' -> '/var/lib/mysql/nasa/actor.ibd'

With all the steps done, it’s now possible to import the tablespace and verify the data:

mysql> ALTER TABLE actor IMPORT TABLESPACE;

Query OK, 0 rows affected (0.02 sec)

mysql> SELECT * FROM nasa.actor LIMIT 5;

+----------+------------+--------------+---------------------+

| actor_id | first_name | last_name    | last_update         |

+----------+------------+--------------+---------------------+

|        1 | PENELOPE   | GUINESS      | 2006-02-15 04:34:33 |

|        2 | NICK       | WAHLBERG     | 2006-02-15 04:34:33 |

|        3 | ED         | CHASE        | 2006-02-15 04:34:33 |

|        4 | JENNIFER   | DAVIS        | 2006-02-15 04:34:33 |

|        5 | JOHNNY     | LOLLOBRIGIDA | 2006-02-15 04:34:33 |

+----------+------------+--------------+---------------------+

5 rows in set (0.00 sec)

You can see that we have the data from sakila.actor in nasa.actor.

The best thing about transportable tablespaces is probably their efficiency. You can move very large tables between databases easily using this feature.

XtraBackup Single-Table Restore

Perhaps surprisingly, we’re going to mention XtraBackup once again in the context of transportable tablespaces. That’s because XtraBackup allows for the export of the tables from any existing backup. In fact, that’s the most convenient way to restore an individual table, and it’s also a first building block for a single-table or partial database PITR.

This functionality is completely based on the transportable tablespaces, and it carries over all of the limitations. For example, it won’t work on a non-file-per-table tablespaces.

To perform the single-table restore, you should first run xtrabackup with the --export command-line argument to prepare the table for export. You may notice that table’s name isn’t specified, and in reality each table will be exported. Let’s run that command on one of the backups we’ve taken earlier:

# xtrabackup --prepare --export --target-dir=/tmp/base_backup

# ls -1 /tmp/base_backup/sakila/

actor.cfg

actor.ibd

address.cfg

address.ibd

category.cfg

category.ibd

...

You can see that we have a .cfg file for each table: every tablespace is now ready to be transported and imported in another database. From here, you can repeat the steps from “Import Tablespace” to restore the data from one of the tables.

One of the most advanced backup and recovery techniques is based on top of the transportable tablespaces. That is the single-table or partial database point-in-time recovery. We’re not giving exact steps, and only cover this technique so that you know it’s possible.

Single-table or partial PITR is tricky, and that’s true for most of the database management systems out there. As you saw before in “Point-in-Time Recovery”, PITR in MySQL is based on binlog. What that means for partial recovery is that transactions concerning all tables in all databases are recorded, but binlogs can be filtered when applied through a replication. Very briefly, therefore, the partial recovery procedure is: you export the required tables, build a completely separate instance, and feed it with binlogs through a replication channel.

You can find more information in community blogs and articles like "MySQL Single Table PITR“, "Filtering Binary Logs with MySQL“, and "How to Make MySQL PITR Faster“.

Using the Export and Import Tablespaces feature is a powerful technique when used correctly and under certain circumstances.

Testing and Verifying Your Backups

Backups are only good when you you’re sure you can trust them. There are numerous examples of people having backup systems that failed when most needed. It’s entirely possible to be taking backups frequently and still lose the data.

There are multiple ways in which backups can be wrong or can fail:

- Inconsistent backups: The simplest example of this is a snapshot backup incorrectly taken from multiple volumes when the database is running. Unfortunately, some of the backups can even be consistent or at least not broken enough for you to notice until it’s too late. Backups can be broken or missing data.

- Corruption of the source database: Physical backups, as we covered extensively, will have copies of all of the database pages, corrupted or not. Some tools try to verify the data as they go, but this is not completely error-free. Your successful backups may contain bad data, that cannot be read later.

- Corruption of the backups: Backups are just data on their own, and as such are susceptible to the same issues as the original data. Your successful backup might end up being completely useless if its data corrupts while being stored.

- Bugs: Things happen. A backup tool you’ve been using for a dozen years might have a bug that you of all people will hit. In the best case, your backup will fail; in the worst case, it might fail to restore.

- Operational errors: We’re all human, and we make mistakes. If you automate everything, then this changes from human errors to bugs.

That’s not a comprehensive list of issues that you might face, but it gives you some insight into the problems you might face even when your backup strategy is sound. Let’s review some steps you can take to make you sleep better:

- Whenever implementing a backup system, test it thoroughly, and test it in various modes. Make sure you can back up your system, and use the backup for recovery. Test with and without load. Backup can be consistent when no connection is modifying the data, and fail when that’s not true.

- Use both physical and logical backups. They have different properties and failure modes, especially around source data corruption.

- Back up your backups, or just make sure that they are at least as durable as the database.

- Finally, periodically perform backup restoration tests.

The last point is especially interesting. No backup is safe until restored and tested. That means, that in the perfect world, your automation will actually try to use the backup to build a database server and only report success back when that went fine. Additionally, that new database can be attached to the source as a replica, and a data verification tool like pt-table-checksum from Percona Toolkit can be used to check the data consistency.

Here are some possible steps for backup data verification for physical backups:

1. Prepare the backup.

2. Restore the backup.

3. Run innochecksum on all of the *.ibd files.

The following command will run four innochecksum processes in parallel on Linux:

$ find . -type f -name "*.ibd" -print0 |\

xargs -t -0r -n1 --max-procs=4 innochecksum

4. Start a new MySQL instance using restored backup. Use a spare server, or just a dedicated .cnf file, and don’t forget to use non-default ports and paths.

5. Use mysqldump or any alternative to dump all of the data, making sure it’s readable, and providing another copy of the backup.

6. Attach the new MySQL instance as a replica to the original source database, and use pt-table-checksum or any alternative to verify that data matches. The procedure is nicely explained in the xtrabackup documentation, among other sources.

These steps are complex and might take a long time, so you should decide whether it’s applicable to your business and environment to utilize all of them.

Database Backup Strategy Primer

Now that we have a lot of the bits and pieces covered related to backups and recovery, we can piece together a very robust backup strategy. Let’s build it from the ground up.

Point-in-Time Recovery

We need to decide whether we’ll need PITR capabilities, as that’ll drive our decisions regarding the backup strategy. You have to decide for your specific case, but our suggestion is to default to having PITR available. It can be a life saver. If we decided that we’re going to need this capability, we need to set up binary logging and binlog copying.

Logical backups

We will likely need logical backups, either for their portability or for the corruption safeguard. Since logical backups load the source database significantly, schedule them for a time when there’s the least load. There might come a time when it won’t be possible to do logical backups from your production database either due to time or load constraints, or both. Since we still want to have the ability to run logical backups, we can use following techniques:

- Run logical backups on a replicated database. It can be problematic to track binlog position in this case, so it’s recommended to use GTID-based replication in this case.

- Incorporate creation of logical backups into your physical backup’s verification process. Prepared backup is a data directory that can by used by a MySQL server right away. If you run a server targeting the backup, you will spoil that backup, so you need to copy that prepared backup somewhere first.

Physical backups

Based on the OS, MySQL flavor, system properties, and careful review of documentation, we need to choose the tool we’ll be using for physical backups. For the sake of simplicity, we’re chosing XtraBackup here.

The first decision to make is how important the MTTR (Mean Time to Recovery) target is for us. For example, if you only do weekly base backups, you might end up needing to apply almost a week’s worth of transactions to recover that backup. To decrease the MTTR, implement incremental backups on a daily or perhaps even hourly basis.

Taking a step back, your system might be so large that even the hot backup with one of the physical backup tools is not viable for you. In that case, you need to go for snapshots of the volumes, if that’s possible.

Backup storage

We need to make sure our backups are safely, and ideally redundantly, stored. This may be a hardware storage setup utilizing a less-performant but redundant RAID array of level 5 or 6. This may be a less-reliable storage, but we’ll be adding continuous streaming of our backups to a cloud storage, like the Amazon’s object storage—S3—or other alternatives. In the end, we may just default to using S3 if that’s possible for us with the backup tools of choice.

Backup testing and verification

Finally, once we have backups in place, we would perfectly set up a backup testing process. Depending on the budget available for implementation and maintaining of this exercise, we should decide how many steps will be run each time, or once in a while.

With these steps implemented, we can say that we have our bases covered, and our database safely backed up. Sometimes it may feel like we’re never using backups, but you have to remember that you will eventually face a disaster — it’s just a question of time.

Chapter 7. Doing More with MySQL

MySQL is feature-rich. Over the past three chapters, you’ve seen the wide variety of techniques that can be used to query, modify, and manage data. However, there’s still much more that MySQL can do, and some of those additional features are the subject of this chapter.

In this chapter, you’ll learn how to:

- Insert data into a database from other sources, including with queries and from text files

- Perform updates and deletes using multiple tables in a single statement

- Replace data

- Use MySQL functions in queries to meet more complex information needs

- Analyze queries using the EXPLAIN statement and then improve their performance with simple optimization techniques

- Use alternative storage engines to change table properties

Inserting Data Using Queries

Much of the time, you’ll create tables using data from another source. The examples you’ve seen so far in Chapter 3 therefore illustrate only part of the problem: they show you how to insert data that’s already in the form you want—that is, formatted as an SQL INSERT statement. The other ways to insert data include using SQL SELECT statements on other tables or databases, and reading in files from other sources. This section shows you how to tackle the former method of inserting data; you’ll learn how to insert data from a file of comma-separated values in the next section, “Loading Data from Comma-Delimited Files”.

Suppose you’ve decided to create a new table in the sakila database. It’s going to store a random list of movies that we want to advertise more heavily. In the real world, you’d probably want to use some data science to find out what movies to highlight, but we’re going to stick to the basics. This list of films will be a way of tasting part of the catalog, rediscovering some old favorites, and learning about hidden treasures among movies customers haven’t explored. We’ve decided to structure the table as follows:

mysql> CREATE TABLE recommend (

->    film_id SMALLINT UNSIGNED,

->    language_id TINYINT UNSIGNED,

->    release_year YEAR,

->    title VARCHAR(128),

->    length SMALLINT UNSIGNED,

->    sequence_id SMALLINT AUTO_INCREMENT,

->    PRIMARY KEY (sequence_id)

-> );

Query OK, 0 rows affected (0.05 sec)

You can find this code in the file ch07/recommend.sql at the book’s github.

This table stores the details of the film, allowing you to find the actor, category, and other details using simple queries on the other tables. It also stores a sequence_id, which is a unique number that enumerates where the film is in our short list. When you first start using the recommendation feature, you’ll first see the movie with a sequence_id of 1, then 2, and so on. You can see that we’re using the MySQL auto_increment feature to allocate the sequence_id values.

Now we need to fill up our new recommend table with a random selection of films. Importantly, we’re going to do the SELECT and INSERT together in one statement. Here we go:

mysql> INSERT INTO recommend (film_id, language_id, release_year, title, length)

-> SELECT film_id, language_id, release_year, title, length

-> FROM film ORDER BY RAND() LIMIT 10;

Query OK, 10 rows affected (0.02 sec)

Records: 10  Duplicates: 0  Warnings: 0

Now, let’s investigate what happened before we explain how this command works:

mysql> SELECT * FROM recommend;

+---------+-------------+--------------+--------------------+--------+-------------+

| film_id | language_id | release_year | title              | length | sequence_id |

+---------+-------------+--------------+--------------------+--------+-------------+

|     542 |           1 |         2006 | LUST LOCK          |     52 |           1 |

|     661 |           1 |         2006 | PAST SUICIDES      |    157 |           2 |

|     613 |           1 |         2006 | MYSTIC TRUMAN      |     92 |           3 |

|     757 |           1 |         2006 | SAGEBRUSH CLUELESS |    106 |           4 |

|     940 |           1 |         2006 | VICTORY ACADEMY    |     64 |           5 |

|     917 |           1 |         2006 | TUXEDO MILE        |    152 |           6 |

|     709 |           1 |         2006 | RACER EGG          |    147 |           7 |

|     524 |           1 |         2006 | LION UNCUT         |     50 |           8 |

|      30 |           1 |         2006 | ANYTHING SAVANNAH  |     82 |           9 |

|     602 |           1 |         2006 | MOURNING PURPLE    |    146 |          10 |

+---------+-------------+--------------+--------------------+--------+-------------+

10 rows in set (0.00 sec)

You can see that we got 10 films into our recommendation list, numbered with sequence_id values from 1 to 10. We’re ready to start recommending the random movie selection. Don’t worry if your results differ; it’s a consequence of how the RAND() function works.

Let’s discuss how the command works. There are two parts to the SQL statement: an INSERT INTO and a SELECT. The INSERT INTO statement lists the destination table into which the data will be stored, followed by an optional list of column names in parentheses; if you omit the column names, all columns in the destination table are assumed in the order they appear in a DESCRIBE TABLE or SHOW CREATE TABLE statement. The SELECT statement outputs columns that must match the type and order of the list provided for the INSERT INTO statement (or the implicit, complete list if one isn’t provided). The overall effect is that the rows output from the SELECT statement are inserted into the destination table by the INSERT INTO statement. In our example, film_id, language_id, release_year, title, and length values from the film table are inserted into the five columns with the same names and types in the recommend table; the sequence_id is automatically created using MySQL’s AUTO_INCREMENT feature, and so isn’t specified in the statements.

Our example includes the clause ORDER BY RAND(); this orders the results according to the MySQL function RAND(). The RAND() function returns a pseudorandom number in the range 0 to 1:

mysql> SELECT RAND();

+--------------------+

| RAND()             |

+--------------------+

| 0.4593397513584604 |

+--------------------+

1 row in set (0.00 sec)

A pseudorandom number generator doesn’t generate truly random numbers, but rather generates numbers based on some property of the system, such as the time of day; this is sufficiently random for most applications. A notable exception is cryptography applications that depend on the true randomness of numbers for security.

If you ask for the RAND() value in a SELECT operation, you’ll get a random value for each returned row:

mysql> SELECT title, RAND() FROM film LIMIT 5;

+------------------+---------------------+

| title            | RAND()              |

+------------------+---------------------+

| ACADEMY DINOSAUR |  0.5514843506286706 |

| ACE GOLDFINGER   | 0.37940252980161693 |

| ADAPTATION HOLES |  0.2425596278557178 |

| AFFAIR PREJUDICE | 0.07459058060738312 |

| AFRICAN EGG      |  0.6452740502034072 |

+------------------+---------------------+

5 rows in set (0.00 sec)

Since the values are effectively random, you’ll almost certainly see different results than we’ve shown here. Moreover, if you repeat the statement, you’ll also see different values returned. It is possible to pass RAND() an integer argument—a seed. That will result in the RAND() function generating same values for same inputs with that seed. Not really useful for what we’re trying to achieve here, but a possibility nonetheless. You can try running the following statement on your side as many times as you want, and the results won’t change:

mysql> SELECT title, RAND(1) FROM film LIMIT 5;

Let’s return to the INSERT operation. When we ask that the results be ordered by RAND(), the results of the SELECT statement are sorted in a pseudorandom order. The LIMIT 10 is there to limit the number of rows returned by the SELECT; we’ve limited in this example simply for readability.

The SELECT statement in an INSERT INTO statement can use all of the features of SELECT statements. You can use joins, aggregation, functions, and any other features you choose. You can also query data from one database into another, by prefacing the table names with the database name followed by a period (.) character. For example, if you wanted to insert the actor table from the film database into a new art database, you could do the following:

mysql> CREATE DATABASE art;

Query OK, 1 row affected (0.01 sec)

mysql> USE art;

Database changed

mysql> CREATE TABLE people (

->   person_id SMALLINT UNSIGNED,

->   first_name VARCHAR(45),

->   last_name VARCHAR(45),

->   PRIMARY KEY (person_id)

-> );

Query OK, 0 rows affected (0.03 sec)

mysql> INSERT INTO art.people (person_id, first_name, last_name)

-> SELECT actor_id, first_name, last_name FROM sakila.actor;

Query OK, 200 rows affected (0.01 sec)

Records: 200  Duplicates: 0  Warnings: 0

You can find this code in the file ch07/art.sql at book’s github.

You can see that the new people table is referred to as art.people (though it doesn’t need to be, since art is the database that’s currently in use), and the actor table is referred to as sakila.actor (which it needs to be, since it isn’t the database being used). Note also that the column names don’t need to be the same for the SELECT and the INSERT.

Sometimes, you’ll encounter duplication issues when inserting with a SELECT statement. This occurs if you try to insert the same primary key value twice; it won’t happen in the recommend table, as long as you automatically allocate a new sequence_id using the auto_increment feature. However, when you try to insert duplicate key values, MySQL will abort. Let’s force a duplicate into the recommend table to show the behavior:

mysql> USE sakila;

Database changed

mysql> INSERT INTO recommend (film_id, language_id, release_year,

-> title, length, sequence_id )

-> SELECT film_id, language_id, release_year, title, length, 1

-> FROM film LIMIT 1;

ERROR 1062 (23000): Duplicate entry '1' for key 'recommend.PRIMARY'

If you want MySQL to ignore this and keep going, add an IGNORE keyword after the INSERT:

mysql> INSERT IGNORE INTO recommend (film_id, language_id, release_year,

-> title, length, sequence_id )

-> SELECT film_id, language_id, release_year, title, length, 1

-> FROM film LIMIT 1;

Query OK, 0 rows affected, 1 warning (0.00 sec)

Records: 1  Duplicates: 1  Warnings: 1

MySQL doesn’t complain, but it does report that it encountered a duplicate. Note that the data is not changed, all we did was ignore the error. This is useful in bulk load operation where you don’t want to fail halfway into running a script that inserts a million rows. We can inspect the warning to see the Duplicate entry error as a warning now:

mysql> SHOW WARNINGS;

+---------+------+-------------------------------------------------+

| Level   | Code | Message                                         |

+---------+------+-------------------------------------------------+

| Warning | 1062 | Duplicate entry '1' for key 'recommend.PRIMARY' |

+---------+------+-------------------------------------------------+

1 row in set (0.00 sec)

Finally, note that it’s possible to insert into a table that’s listed in the SELECT statement. However, you still need to avoid duplicate primary keys:

mysql> INSERT INTO actor SELECT

-> actor_id, first_name, last_name, NOW() FROM actor;

ERROR 1062 (23000): Duplicate entry '1' for key 'actor.PRIMARY'

There are two ways to avoid getting the error. The artist table has an auto increment for actor_id, thus if you don’t explicitly declare its values in the insert statement, you won’t get an error. The second way is to modify actor_id in the select query in a way that prevents collision. Let’s try the second way:

mysql> INSERT INTO actor SELECT

-> actor_id+200, first_name, last_name, NOW() FROM actor;

Query OK, 200 rows affected (0.01 sec)

Records: 200  Duplicates: 0  Warnings: 0

Here, we’re copying the rows but increasing their actor_id values by 200 before we insert them, because we remember that there are 200 rows initially. This is the result:

mysql> SELECT * FROM actor;

+----------+-------------+--------------+---------------------+

| actor_id | first_name  | last_name    | last_update         |

+----------+-------------+--------------+---------------------+

|        1 | PENELOPE    | GUINESS      | 2006-02-15 04:34:33 |

|        2 | NICK        | WAHLBERG     | 2006-02-15 04:34:33 |

...

|      198 | MARY        | KEITEL       | 2006-02-15 04:34:33 |

|      199 | JULIA       | FAWCETT      | 2006-02-15 04:34:33 |

|      200 | THORA       | TEMPLE       | 2006-02-15 04:34:33 |

|      201 | PENELOPE    | GUINESS      | 2021-02-28 10:24:49 |

|      202 | NICK        | WAHLBERG     | 2021-02-28 10:24:49 |

...

|      398 | MARY        | KEITEL       | 2021-02-28 10:24:49 |

|      399 | JULIA       | FAWCETT      | 2021-02-28 10:24:49 |

|      400 | THORA       | TEMPLE       | 2021-02-28 10:24:49 |

+----------+-------------+--------------+---------------------+

400 rows in set (0.00 sec)

You can see how fist names, last names, and last_update start repeating from the actor_id 201.

It’s also possible to use subqueries in the INSERT SELECT statements. For example, the next statement is valid:

INSERT INTO actor SELECT * FROM

(SELECT actor_id+400, first_name, last_name, NOW() FROM actor) foo;

Loading Data from Comma-Delimited Files

In the modern days, databases are usually not an afterthought. They are ubiquitous, easier than ever to use, and most IT professionals know about them. Nevertheless, end users find them difficult, and unless specialized UIs are created, a lot of data entry and analysis is done in various spreadhseet programs. Most of these programs have unique file formats, open or closed, but most of them will allow you to export data as rows of comma-separated values (CSV), also called a comma-delimited format. You can then import the data with a little effort into MySQL.

Another common task that can be accomplished by working with CSVs is transferring data in a heterogenous environment. If you have various database software running in your setup, and especially if you’re using a Database as a Service in a cloud, moving data between these systems can be daunting. However, the basic CSV can be a lowest common denominator for them. Note that in case of any data transfer you should always remember that CSV does not have a notion of schema, data types, or any constraints. But as a flat data file format, it works well.

If you’re not using a spreadsheet program, you can still often use command-line tools such as sed and awk--very old and powerful UNIX utilities—to convert text data into a CSV format suitable for import by MySQL. Some cloud databases allow export of their data directly into CSV. In some other cases, small programs have to be written that read data and produce a CSV file. This section shows you the basics of how to import CSV data into MySQL.

Figure 7-1. List of NASA facilities stored in a spreadsheet file

Let’s work through an example. We have a list of NASA facilities with their addresses and contact information that we want to store in a database. At present, it’s stored in a CSV file named NASA_Facilities.csv and has the format shown in Figure 7-1. You can see that each facility is associated with a center, and may have a date it was occupied and optionally its status. The full column list is as follows:

- Center

- Center Search Status

- Facility

- FacilityURL

- Occupied

- Status

- URL Link

- Record Date

- Last Update

- Country

- Contact

- Phone

- Location

- City

- State

- Zipcode

This example comes directly from NASA’s open data site publicly available at NASA’s Open Data Portal. We have also put this file on our book’s github as ch07/NASA_Facilities.csv. Since this is a CSV file from the beginning, we don’t need to convert another file format (like XLS) into CSV. However, if we were to do that, it’s usually as easy as doing a Save As command in the spreadsheet program, just don’t forget to pick CSV as an output format.

If you open the file using a text editor, you’ll see the result: the file has one line per spreadsheet row, with the value for each column separated by a comma. If you’re on a non-Windows platform, you may find that in some CSV files each line is terminated with a ^M, but don’t worry about this; it’s an artifact of the origins of Windows. Data in this format is often referred to as DOS format, and most software applications can handle it without problem. In our case, data is in UNIX format and thus on Windows you may see that all lines are concatenated. You can try to use another text editor if that’s the case. Here are a few width-truncated lines selected from NASA_Facilities.csv:

Center,Center Search Status,Facility,FacilityURL,Occupied,Status,...

Kennedy Space Center,Public,Control Room 2/1726/HGR-S ,,...

Langley Research Center,Public,Micometeroid/LDEF Analysis Laboratory,,...

Kennedy Space Center,Public,SRM Rotation and Processing Facility/K6-0494 ,...

Marshall Space Flight Center,..."35812(34.729538, -86.585283)",Huntsville,...

If there are commas or other special symbols within values, the whole value is enclosed in quotes, as in the last line shown here.

Let’s import this data into MySQL. First, create the new nasa database:

mysql> CREATE DATABASE nasa;

Query OK, 1 row affected (0.01 sec)

Choose this as the active database:

mysql> USE nasa;

Database changed

Now, create the facilities table to store the data. This needs to handle all of the fields that we see in the CSV file, which conveniently has a header:

mysql> CREATE TABLE facilities (

->   center TEXT,

->   center_search_status TEXT,

->   facility TEXT,

->   facility_url TEXT,

->   occupied TEXT,

->   status TEXT,

->   url_link TEXT,

->   record_date DATETIME,

->   last_update TIMESTAMP NULL,

->   country TEXT,

->   contact TEXT,

->   phone TEXT,

->   location TEXT,

->   city TEXT,

->   state TEXT,

->   zipcode TEXT

-> );

Query OK, 0 rows affected (0.03 sec)

You can find this code in the file ch07/nasa.sql at book’s github.

We’re cheating here somewhat with the data types. NASA provides the schema of the dataset, but for most of the fields type is given as Plain Text, and we can’t really store Website URL as anything but text, too. We don’t, however, know how much data each column will hold. Thus, we default to using the TEXT type, which is similar to defining a column as VARCHAR(65535). There are some more differences between the two types as you can probably remember from “String types”, but they are not important in this example. We don’t define any indexes and don’t put any constraints on our table. If you’re loading a completely new dataset which is quite small, it can be beneficial to load it, and only then analyze. For larger datasets, make sure that table is structured as well as possible, or you’ll spend considerable amount of time changing it later.

Now that we’ve set up the database table, we can import the data from the file using the LOAD DATA INFILE command:

mysql> LOAD DATA INFILE 'NASA_Facilities.csv' INTO TABLE facilities

-> FIELDS TERMINATED BY ',';

ERROR 1290 (HY000): The MySQL server is running with

the --secure-file-priv option so it cannot execute this statement

Oh, no! We got an error. By default, MySQL doesn’t let you load any data using the LOAD DATA INFILE command. The behavior is controlled by the secure_file_priv system variable. If the variable is set to a path, the file to be loaded should reside over in that particular path, and be readable by the MySQL server. If the variable isn’t set, which is considered insecure, then the file to be loaded should only be readable by the MySQL server. By default, MySQL 8.0 on Linux sets this variable as follows:

mysql> SELECT @@secure_file_priv;

+-----------------------+

| @@secure_file_priv    |

+-----------------------+

| /var/lib/mysql-files/ |

+-----------------------+

1 row in set (0.00 sec)

And on Windows:

mysql> SELECT @@secure_file_priv;

+------------------------------------------------+

| @@secure_file_priv                             |

+------------------------------------------------+

| C:\ProgramData\MySQL\MySQL Server 8.0\Uploads\ |

+------------------------------------------------+

1 row in set (0.00 sec)

NOTE

Your installation of MySQL may differ, and the value of secure_file_priv system variable can be different, or even empty. NULL value for secure_file_prive means that MySQL will allow loading a file in any location as long as that file is accessible to the MySQL server. On Linux, that means that the file has to be readable by the mysqld process, which usually runs under mysql user. You can change secure_file_priv variable’s value by updating MySQL configuration and restarting the server. You can find information on how to configure MySQL in Chapter 9.

On Linux or other UNIX-like systems, we need to copy the file our file into that directory, possibly using sudo to allow the operation, and then change its permissions so that mysqld program can access the file. On Windows, we only need to copy the file to a correct destination.

Let’s do this. On Linux or similar systems, you can run commands like this:

$ ls -lh $HOME/Downloads/NASA_Facilities.csv

-rw-r--r--. 1 skuzmichev skuzmichev 114K

Feb 28 14:19 /home/skuzmichev/Downloads/NASA_Facilities.csv

$ sudo cp -vip ~/Downloads/NASA_Facilities.csv /var/lib/mysql-files

[sudo] password for skuzmichev:

'/home/skuzmichev/Downloads/NASA_Facilities.csv'

-> '/var/lib/mysql-files/NASA_Facilities.csv'

$ sudo chown mysql:mysql /var/lib/mysql-files/NASA_Facilities.csv

$ sudo ls -lh /var/lib/mysql-files/NASA_Facilities.csv

-rw-r--r--. 1 mysql mysql 114K

Feb 28 14:19 /var/lib/mysql-files/NASA_Facilities.csv

On Windows, you can use regular file manager to copy or move the file.

Now we’re ready to try the loading again. When our target file is not in the current directory, the full path must be passed to the command:

mysql> LOAD DATA INFILE '/var/lib/mysql-files/NASA_Facilities.csv'

-> INTO TABLE facilities FIELDS TERMINATED BY ',';

ERROR 1292 (22007): Incorrect datetime value:

'Record Date' for column 'record_date' at row 1

Well, that doesn’t look correct: Record Date is indeed not a date, but a column name. A silly but common mistake: loading the CSV file with header. We need to tell MySQL to omit it:

mysql> LOAD DATA INFILE '/var/lib/mysql-files/NASA_Facilities.csv'

-> INTO TABLE facilities FIELDS TERMINATED BY ','

-> IGNORE 1 LINES;

ERROR 1292 (22007): Incorrect datetime value:

'03/01/1996 12:00:00 AM' for column 'record_date' at row 1

Turns out, that date format we have is not something MySQL expects. That’s an extremely common issue. There are a couple of ways out. First, we can just change our record_date column to TEXT type. We lose the niceties of a proper date-time datatype, but we can get the data into our database. Second, we can convert the data ingested from the file on the fly. To demonstrate the difference in the results, we specified the occupied column (which is a date field) to be TEXT. Before we jump into the conversion complexities, though, let’s try running the same command on Windows:

mysql> LOAD DATA INFILE

-> 'C:\ProgramData\MySQL\MySQL Server 8.0\Uploads\NASA_Facilities.csv'

-> INTO TABLE facilities FIELDS TERMINATED BY ',';

ERROR 1290 (HY000): The MySQL server is running with

the --secure-file-priv option so it cannot execute this statement

Even though the file is present in that directory, LOAD DATA INFILE errors out. The reason for that is how MySQL works with paths on Windows. You can’t just use regular Windows-style paths with this (or other) MySQL command. We need to escape each backslash (\) with another backslash, or change our path to forward slashes (/). Both will work. Well, in this case, both will error out due to the expected record_date conversion issue:

mysql> LOAD DATA INFILE

-> 'C:\\ProgramData\\MySQL\\MySQL Server 8.0\\Uploads\\NASA_Facilities.csv'

-> INTO TABLE facilities FIELDS TERMINATED BY ',';

ERROR 1292 (22007): Incorrect datetime value:

'Record Date' for column 'record_date' at row 1

mysql> LOAD DATA INFILE

-> 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/NASA_Facilities.csv'

-> INTO TABLE facilities FIELDS TERMINATED BY ',';

ERROR 1292 (22007): Incorrect datetime value:

'Record Date' for column 'record_date' at row 1

With that covered, let’s get back to our date conversion issue. As we have mentioned, that is an extremely common issue. You will face type conversion problems, because CSV is typeless, and different databases have different expecations about various types. In this case, the open dataset that we obtained has dates in the following format: 03/01/1996 12:00:00 AM. While this will make our operation more complex, we believe converting the date values from our CSV is a good exercise. To convert an arbitrary string to date, or at least to attempt such conversion, we can use STR_TO_DATE() function. After reviewing the documenation, we can come up with the following cast:

mysql> SELECT STR_TO_DATE('03/01/1996 12:00:00 AM',

-> '%m/%d/%Y %h:%i:%s %p') converted;

+---------------------+

| converted           |

+---------------------+

| 1996-03-01 00:00:00 |

+---------------------+

1 row in set (0.01 sec)

Since the function returns NULL when cast is unsuccessful, we know we have managed to find a correct invocation. Now we’ll need to find out how to use the function in the LOAD DATA INFILE command. The much longer version using the function is like this:

mysql> LOAD DATA INFILE '/var/lib/mysql-files/NASA_Facilities.csv'

-> INTO TABLE facilities FIELDS TERMINATED BY ','

-> OPTIONALLY ENCLOSED BY '"'

-> IGNORE 1 LINES

-> (center, center_search_status, facility, facility_url,

-> occupied, status, url_link, @var_record_date, @var_last_update,

-> country, contact, phone, location, city, state, zipcode)

-> SET record_date = IF(

->   CHAR_LENGTH(@var_record_date)=0, NULL,

->     STR_TO_DATE(@var_record_date, '%m/%d/%Y %h:%i:%s %p')

-> ),

-> last_update = IF(

->   CHAR_LENGTH(@var_last_update)=0, NULL,

->     STR_TO_DATE(@var_last_update, '%m/%d/%Y %h:%i:%s %p')

-> );

Query OK, 485 rows affected (0.05 sec)

Records: 485  Deleted: 0  Skipped: 0  Warnings: 0

That’s a lot of command! Let’s break it down. The first line specifies our command LOAD DATA INFILE and the path to a file. The second line specifies the target table, and starts the FIELDS specification, starting with TERMINATED BY ',', which means our fields are delimited by commas, as expected for CSV. The third line adds another parameter to fields specification and tells MySQL that some fields (but not all) are enclosed by the " symbol. That’s important, because our dataset has some entries with commas within "..." fields. On line four we specify that we skip the first line of the file, where we know the header resides.

Lines five through seven have the column list specification. We need to convert two date-time columns, and for that, we need to read their values into variables, which are then set to the nasa.facilities table’s columns values. However, we can’t tell that to MySQL without also specifying all other columns. If we were to omit some columns from the list, or specify them in wrong order, MySQL would not assign values correctly. CSV is inherently a position-based format. By default, when fields specification is not given, MySQL will read each CSV line, and will expect each field in all lines to map to a column in the target table (in the order of columns that DESCRIBE or SHOW CREATE TABLE commands give). By changing the order of columns in this specification, we can populate a table from CSV that has fields misplaced. By specifying fewer columns, we can populate a table from CSV that misses some of the fields.

Lines eight through fifteen are our function calls to convert the date time values. In the preceding column spec, we defined that field number eight is read into @var_record_date variable, and field number nine into @var_last_update. We know that fields eight and nine are our problematic date time fields. With variables populated, we can define the SET parameter, which allows modification of the target table column values based on fields read from the CSV. In the very basic example, you could multiply a specific value by two. In our case, we cast two functions: first, we check that a variable is not empty (,, in CSV) by assessing number of characters read from the file, and second we call the actual conversion if the previous check doesn’t return zero. If we found the length to be zero, we set values to NULL.

Finally, when the command had been executed, it’s possible to check the results:

mysql> SELECT facility, occupied, last_update

-> FROM facilities

-> ORDER BY last_update DESC LIMIT 5;

+---------------------...+------------------------+---------------------+

| facility            ...| occupied               | last_update         |

+---------------------...+------------------------+---------------------+

| Turn Basin/K7-1005  ...| 01/01/1963 12:00:00 AM | 2015-06-22 00:00:00 |

| RPSF Surge Building ...| 01/01/1984 12:00:00 AM | 2015-06-22 00:00:00 |

| Thermal Protection S...| 01/01/1988 12:00:00 AM | 2015-06-22 00:00:00 |

| Intermediate Bay/M7-...| 01/01/1995 12:00:00 AM | 2015-06-22 00:00:00 |

| Orbiter Processing F...| 01/01/1987 12:00:00 AM | 2015-06-22 00:00:00 |

+---------------------...+------------------------+---------------------+

5 rows in set (0.00 sec)

Remember we mentioned that occupied will remain TEXT. You can see that here. While it can be used for sorting, no date functions will work on values in this column unless they are explicitly cast to datetime.

This was a complex example, but it shows the unexpected complexity of loading data, and power of the LOAD DATA INFILE command.

Writing Data into Comma-Delimited Files

You can use the SELECT INTO OUTFILE statement to write out the result of a query into a comma-separated values (CSV) file that can be opened by a spreadsheet or other program.

Let’s export the list of current managers from our employees database into a CSV file. The query used to list all the current managers is shown here:

mysql> USE employees;

Database changed

mysql> SELECT emp_no, first_name, last_name, title, from_date

-> FROM employees JOIN titles USING (emp_no)

-> WHERE title = 'Manager' AND to_date = '9999-01-01';

+--------+------------+------------+---------+------------+

| emp_no | first_name | last_name  | title   | from_date  |

+--------+------------+------------+---------+------------+

| 110039 | Vishwani   | Minakawa   | Manager | 1991-10-01 |

| 110114 | Isamu      | Legleitner | Manager | 1989-12-17 |

| 110228 | Karsten    | Sigstam    | Manager | 1992-03-21 |

| 110420 | Oscar      | Ghazalie   | Manager | 1996-08-30 |

| 110567 | Leon       | DasSarma   | Manager | 1992-04-25 |

| 110854 | Dung       | Pesch      | Manager | 1994-06-28 |

| 111133 | Hauke      | Zhang      | Manager | 1991-03-07 |

| 111534 | Hilary     | Kambil     | Manager | 1991-04-08 |

| 111939 | Yuchang    | Weedman    | Manager | 1996-01-03 |

+--------+------------+------------+---------+------------+

9 rows in set (0.13 sec)

We can change this SELECT query slightly to write this data into an output file as comma-separated values. The INTO OUTFILE is subject to the same --secure-file-priv option rules as the LOAD DATA INFILE. The file path by default is limited, and we listed default options in “Loading Data from Comma-Delimited Files”.

mysql> SELECT emp_no, first_name, last_name, title, from_date

-> FROM employees JOIN titles USING (emp_no)

-> WHERE title = 'Manager' AND to_date = '9999-01-01'

-> INTO OUTFILE '/var/lib/mysql-files/managers.csv'

-> FIELDS TERMINATED BY ',';

Query OK, 9 rows affected (0.14 sec)

Here, we’ve saved the results into the file managers.csv in the /var/lib/mysql-files directory; the MySQL server must be able to write to the directory that you specify, and it should be one listed in secure_file_priv system variable (if set). On a Windows system, specify a path such as C:\ProgramData\MySQL\MySQL Server 8.0\Uploads\managers.csv instead. If you omit the FIELDS TERMINATED BY clause, the server will use tabs as the default separator between the data values.

You can view the contents of the file managers.csv in a text editor, or import it into a spreadsheet program:

110039,Vishwani,Minakawa,Manager,1991-10-01

110114,Isamu,Legleitner,Manager,1989-12-17

110228,Karsten,Sigstam,Manager,1992-03-21

110420,Oscar,Ghazalie,Manager,1996-08-30

110567,Leon,DasSarma,Manager,1992-04-25

110854,Dung,Pesch,Manager,1994-06-28

111133,Hauke,Zhang,Manager,1991-03-07

111534,Hilary,Kambil,Manager,1991-04-08

111939,Yuchang,Weedman,Manager,1996-01-03

When our data fields contain commas or another delimiter of our choice, MySQL by default will escape the delimiters within fields. Let’s switch to sakila database and test this:

mysql> USE sakila;

Database changed

mysql> SELECT title, special_features FROM film LIMIT 10

-> INTO OUTFILE '/var/lib/mysql-files/film.csv'

-> FIELDS TERMINATED BY ',';

Query OK, 10 rows affected (0.00 sec)

If you take a look at the data in the film.csv file now (again, feel free to use a text editor, a spreadsheet program, or a command-line utility like head on Linux):

ACADEMY DINOSAUR,Deleted Scenes\,Behind the Scenes

ACE GOLDFINGER,Trailers\,Deleted Scenes

ADAPTATION HOLES,Trailers\,Deleted Scenes

AFFAIR PREJUDICE,Commentaries\,Behind the Scenes

AFRICAN EGG,Deleted Scenes

AGENT TRUMAN,Deleted Scenes

AIRPLANE SIERRA,Trailers\,Deleted Scenes

AIRPORT POLLOCK,Trailers

ALABAMA DEVIL,Trailers\,Deleted Scenes

ALADDIN CALENDAR,Trailers\,Deleted Scenes

Notice how the comma in rows within second field has been automatically escaped with a backslash to distinguish it from the separator. Some spreadsheet programs may understand this and remove the backslash when importing the file, and some may not. MySQL will respect the escaping and not treat such commas as separators. If we specified FIELDS TERMINATED BY '^', all ^ symbols within fields would get escaped, this is not specific to commas.

Since not all programs may deal with escapes gracefully, we can ask MySQL to explicitly define fields by using the ENCLOSED option:

mysql> SELECT title, special_features FROM film LIMIT 10

-> INTO OUTFILE '/var/lib/mysql-files/film_quoted.csv'

-> FIELDS TERMINATED BY ',' ENCLOSED BY '"';

Query OK, 10 rows affected (0.00 sec)

We saw this option used before when loading data. Take a look at the results in the film_quoted.csv:

"ACADEMY DINOSAUR","Deleted Scenes,Behind the Scenes"

"ACE GOLDFINGER","Trailers,Deleted Scenes"

"ADAPTATION HOLES","Trailers,Deleted Scenes"

"AFFAIR PREJUDICE","Commentaries,Behind the Scenes"

"AFRICAN EGG","Deleted Scenes"

"AGENT TRUMAN","Deleted Scenes"

"AIRPLANE SIERRA","Trailers,Deleted Scenes"

"AIRPORT POLLOCK","Trailers"

"ALABAMA DEVIL","Trailers,Deleted Scenes"

"ALADDIN CALENDAR","Trailers,Deleted Scenes"

Our delimiters—commas—are now not escaped, which may work better with modern spreadsheet programs. You may ask what will happen if there are double quotes within exported fields? MySQL will escape those instead of commas, which again may cause problems. When doing data export, do not forget to make sure that resulting output will work for your consumers.

Creating Tables with Queries

You can create a table or easily create a copy of a table using a query. This is useful when you want to build a new database using existing data—for example, you might want to copy across a list of countries—or when you want to reorganize data for some reason. Data reorganization is common for producing reports, merging data from two or more tables, and redesigning on the fly. This short section shows you how it’s done.

TIP

We base all examples here on a clean state of the sakila database. You should repeat the steps given in “Entity Relationship Modeling Examples” to get the database back to its clean state.

In MySQL, you can easily duplicate the structure of a table using a variant of the CREATE TABLE syntax:

mysql> USE sakila;

Database changed

mysql> CREATE TABLE actor_2 LIKE actor;

Query OK, 0 rows affected (0.24 sec)

mysql> DESCRIBE actor_2;

+-------------+-------------------+------+-----+...

| Field       | Type              | Null | Key |...

+-------------+-------------------+------+-----+...

| actor_id    | smallint unsigned | NO   | PRI |...

| first_name  | varchar(45)       | NO   |     |...

| last_name   | varchar(45)       | NO   | MUL |...

| last_update | timestamp         | NO   |     |...

+-------------+-------------------+------+-----+...

...+-------------------+-----------------------------------------------+

...| Default           | Extra                                         |

...+-------------------+-----------------------------------------------+

...| NULL              | auto_increment                                |

...| NULL              |                                               |

...| NULL              |                                               |

...| CURRENT_TIMESTAMP | DEFAULT_GENERATED on update CURRENT_TIMESTAMP |

...+-------------------+-----------------------------------------------+

4 rows in set (0.01 sec)

mysql> SELECT * FROM actor_2;

Empty set (0.00 sec)

The LIKE syntax allows you to create a new table with exactly the same structure as another, including keys. You can see that it doesn’t copy the data across. You can also use the IF NOT EXISTS and TEMPORARY features with this syntax.

If you want to create a table and copy some data, you can do that with a combination of the CREATE TABLE and SELECT statements. Let’s remove the actor_2 table and re-create it using this new approach:

mysql> DROP TABLE actor_2;

Query OK, 0 rows affected (0.08 sec)

mysql> CREATE TABLE actor_2 AS SELECT * from actor;

Query OK, 200 rows affected (0.03 sec)

Records: 200  Duplicates: 0  Warnings: 0

mysql> SELECT * FROM actor_2 LIMIT 5;

+----------+------------+--------------+---------------------+

| actor_id | first_name | last_name    | last_update         |

+----------+------------+--------------+---------------------+

|        1 | PENELOPE   | GUINESS      | 2006-02-15 04:34:33 |

|        2 | NICK       | WAHLBERG     | 2006-02-15 04:34:33 |

|        3 | ED         | CHASE        | 2006-02-15 04:34:33 |

|        4 | JENNIFER   | DAVIS        | 2006-02-15 04:34:33 |

|        5 | JOHNNY     | LOLLOBRIGIDA | 2006-02-15 04:34:33 |

+----------+------------+--------------+---------------------+

5 rows in set (0.01 sec)

An identical table actor_2 is created, and all the data is copied across by the SELECT statement. The CREATE TABLE AS SELECT or CTAS is a common name for this action, but it’s actually not mandatory to specify the AS part, and we’ll omit that later.

This technique is powerful. You can create new tables with new structures and use powerful queries to populate them with data. For example, here’s a report table that’s created to contain the names of films and their categories in our database:

mysql> CREATE TABLE report (title VARCHAR(128), category VARCHAR(25))

-> SELECT title, name AS category FROM

-> film JOIN film_category USING (film_id)

-> JOIN category USING (category_id);

Query OK, 1000 rows affected (0.06 sec)

Records: 1000  Duplicates: 0  Warnings: 0

You can find this code in the file ch07/report.sql at book’s github.

You can see that the syntax is a little different from the previous example. In this example, the new table name, report, is followed by a list of column names and types in parentheses; this is necessary because we’re not duplicating the structure of an existing table. Moreover, we actually change name into category. Then, the SELECT statement follows, with its output matching the new columns in the new table. You can check the contents of the new table:

mysql> SELECT * FROM report LIMIT 5;

+---------------------+----------+

| title               | category |

+---------------------+----------+

| AMADEUS HOLY        | Action   |

| AMERICAN CIRCUS     | Action   |

| ANTITRUST TOMATOES  | Action   |

| ARK RIDGEMONT       | Action   |

| BAREFOOT MANCHURIAN | Action   |

+---------------------+----------+

5 rows in set (0.00 sec)

So, in this example, the title and name values from the SELECT statement are used to populate the new title and category columns in the report table.

Creating tables with a query has a major caveat that you need to be careful about. It doesn’t copy the indexes (or foreign keys, if you use them); this is a feature, since it gives you a lot of flexibility, but it can be a catch if you forget. Have a look at our actor_2 example:

mysql> DESCRIBE actor_2;

+-------------+-------------------+------+-----+...

| Field       | Type              | Null | Key |...

+-------------+-------------------+------+-----+...

| actor_id    | smallint unsigned | NO   |     |...

| first_name  | varchar(45)       | NO   |     |...

| last_name   | varchar(45)       | NO   |     |...

| last_update | timestamp         | NO   |     |...

+-------------+-------------------+------+-----+...

...+-------------------+-----------------------------------------------+

...| Default           | Extra                                         |

...+-------------------+-----------------------------------------------+

...| 0                 |                                               |

...| NULL              |                                               |

...| NULL              |                                               |

...| CURRENT_TIMESTAMP | DEFAULT_GENERATED on update CURRENT_TIMESTAMP |

...+-------------------+-----------------------------------------------+

4 rows in set (0.00 sec)

mysql> SHOW CREATE TABLE actor_2;

+---------+----------------------------------------------------------+

| Table   | Create Table                                             |

+---------+----------------------------------------------------------+

| actor_2 | CREATE TABLE `actor_2` (

`actor_id` smallint unsigned NOT NULL DEFAULT '0',

`first_name` varchar(45) NOT NULL,

`last_name` varchar(45) NOT NULL,

`last_update` timestamp NOT NULL

DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP

) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci   |

+---------+----------------------------------------------------------+

1 row in set (0.00 sec)

You can see that there’s no primary key; the idx_actor_last_name key is missing as well. Not only that, but AUTO_INCREMENT property of actor_id column is missing, too.

To copy indexes across to the new table, there are at least three things you can do. The first is to use the LIKE statement to create the empty table with the indexes, as described earlier and then copy the data across using an INSERT with a SELECT statement as described earlier in this chapter in “Inserting Data Using Queries”.

The second thing you can do is to use CREATE TABLE with a SELECT statement, and then add indexes using ALTER TABLE as described in Chapter 4.

The third way is to use the UNIQUE (or PRIMARY KEY or KEY) keyword in combination with the CREATE TABLE and SELECT to add a primary-key index. Here’s an example of this approach:

mysql> DROP TABLE actor_2;

Query OK, 0 rows affected (0.04 sec)

mysql> CREATE TABLE actor_2 (UNIQUE(actor_id))

-> AS SELECT * from actor;

Query OK, 200 rows affected (0.05 sec)

Records: 200  Duplicates: 0  Warnings: 0

mysql> DESCRIBE actor_2;

+-------------+-------------------+------+-----+...

| Field       | Type              | Null | Key |...

+-------------+-------------------+------+-----+...

| actor_id    | smallint unsigned | NO   | PRI |...

| first_name  | varchar(45)       | NO   |     |...

| last_name   | varchar(45)       | NO   |     |...

| last_update | timestamp         | NO   |     |...

+-------------+-------------------+------+-----+...

...+-------------------+-----------------------------------------------+

...| Default           | Extra                                         |

...+-------------------+-----------------------------------------------+

...| 0                 |                                               |

...| NULL              |                                               |

...| NULL              |                                               |

...| CURRENT_TIMESTAMP | DEFAULT_GENERATED on update CURRENT_TIMESTAMP |

...+-------------------+-----------------------------------------------+

4 rows in set (0.01 sec)

The UNIQUE keyword is applied to the actor_id column, making it the primary key in the newly created table. The keywords UNIQUE and PRIMARY KEY can be interchanged.

You can use different modifiers when you’re creating tables using these techniques. For example, here’s a table created with defaults and other settings:

mysql> CREATE TABLE actor_3 (

->   actor_id SMALLINT UNSIGNED NOT NULL AUTO_INCREMENT,

->   first_name VARCHAR(45) NOT NULL,

->   last_name VARCHAR(45) NOT NULL,

->   last_update TIMESTAMP NOT NULL

->     DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

->   PRIMARY KEY (actor_id),

->   KEY idx_actor_last_name (last_name)

-> ) SELECT * FROM actor;

Query OK, 200 rows affected (0.05 sec)

Records: 200  Duplicates: 0  Warnings: 0

You can find this code in the file ch07/actor.sql at book’s github.

Here, we’ve set NOT NULL for the new columns, used the AUTO_INCREMENT feature on actor_id, and created two keys. Anything you can do in a regular CREATE TABLE statement can be done in this variant; just remember to add those indexes explicitly!

Updates and Deletes with Multiple Tables

In Chapter 3, we showed you how to update and delete data. In the examples there, each update and delete affected one table and used properties of that table to decide what to modify. This section shows you more complex updates and deletes, with which you can delete or update rows from more than one table in one statement and can use those or other tables to decide what rows to change.

Deletion

Imagine you’re doing a cleanup of the database. Perhaps you’re running out of space, or there’s just too much unnecessary data within the database. One way to solve this problem is to remove some data, and it’d make sense to remove films that are present in our inventory, but were never rented. Unfortunately, this means you need to remove data from the inventory table using information from the rentals table.

With the techniques we’ve described so far in the book, there’s no way of doing this without creating a table that combines the two tables (perhaps using INSERT with SELECT), removing unwanted rows, and copying the data back to its source. This section shows you how you can perform this procedure and other more advanced types of deletion.

Consider the query you need to write to find films in the inventory that were not rented. One way to do it is to use a nested query—following the techniques we showed you in Chapter 5—with the NOT EXISTS clause. Here’s the query:

mysql> SELECT * FROM inventory WHERE NOT EXISTS

-> (SELECT 1 FROM rental WHERE

-> rental.inventory_id = inventory.inventory_id);

+--------------+---------+----------+---------------------+

| inventory_id | film_id | store_id | last_update         |

+--------------+---------+----------+---------------------+

|            5 |       1 |        2 | 2006-02-15 05:09:17 |

+--------------+---------+----------+---------------------+

1 row in set (0.01 sec)

This and other snippets from this section are available in ch07/multitable_deletes.sql

You can probably see how the query works, but let’s briefly discuss it anyway before we move on. You can see it uses a correlated subquery, where the current row being processed in the outer query is referenced by the subquery; you can tell this because the inventory_id column from inventory is referenced, but the inventory table isn’t listed in the FROM clause of the subquery. The subquery produces output when there’s a row in the rental table that matches the current row in the outer query (and so an inventory entry was rented). However, since the query uses NOT EXISTS, the outer query doesn’t produce output when this is the case, and so the overall result is that rows are output for inventory records of movies that haven’t been rented.

Now let’s take our query and turn it into a DELETE statement. Here it is:

mysql> DELETE FROM inventory WHERE NOT EXISTS

-> (SELECT 1 FROM rental WHERE

-> rental.inventory_id = inventory.inventory_id);

Query OK, 1 row affected (0.04 sec)

You can see that the subquery remains the same, but the outer SELECT query is replaced by a DELETE statement. Here, we’re following the standard DELETE syntax: the keyword DELETE is followed by FROM and a specification of table or tables from which rows should be removed; and, last, a WHERE clause (and any other query clauses, such as GROUP BY or HAVING) follow. In this query, rows are deleted from the inventory table, but in the WHERE clause a subquery is specified within a NOT EXISTS statement.

While this statement does indeed delete rows from one table based on data from another table, it’s basically a variation of a regular DELETE. To convert this particular statement into a multi-table DELETE, we should switch from a nested subquery to a LEFT JOIN, like so:

DELETE inventory FROM inventory LEFT JOIN rental

USING (inventory_id) WHERE rental.inventory_id IS NULL;

Note how the syntax changes to include the specific table (or tables) where we want to delete the rows we find. These tables are specified after DELETE but before the FROM and query specification. There’s another way to write down this query, however, and it’s the one we prefer:

DELETE FROM inventory USING inventory

LEFT JOIN rental USING (inventory_id)

WHERE rental.inventory_id IS NULL;

This query is a mix between the previous two. We do not specify the deletion targets between DELETE and FROM, and write them down as this was a regular deletion. Instead, we use a special USING clause, which indicates that a filter query (a join or otherwise) is going to follow. This is slightly clearer in our opinion than the previous example of DELETE <tbl> FROM <tbl>.... One dowside of using the USING keyword is it can be mixed up with the USING keyword of a JOIN statement. With some practice, you’ll never make that mistake, though.

Now that we know both multitable syntax variants, we can construct a query that actually requires a multitable delete. One example of a situation which could require such statement is deleting referenced records. In the sakila database, there are records for films in the film table, which have no associated records in inventory table. That is, there are films that there’s info on, but that cannot be rented. As a part of the database cleanup operation, you’re tasked with removing such dangling data. Initially this seems easy enough:

mysql> DELETE FROM film WHERE NOT EXISTS

-> (SELECT 1 FROM inventory WHERE

-> film.film_id = inventory.film_id);

ERROR 1451 (23000): Cannot delete or update a parent row:

a foreign key constraint fails (

`sakila`.`film_actor`, CONSTRAINT `fk_film_actor_film`

FOREIGN KEY (`film_id`) REFERENCES `film` (`film_id`)

ON DELETE RESTRICT ON UPDATE CASCADE)

Alas, the integrity constraint prevents this deletion. We will have to remove not only the films, but also relationships between those films and actors. That may generate orphan actors, which can be deleted next. We could delete films and reference to actors in one go like this:

DELETE FROM film_actor, film USING

film JOIN film_actor USING (film_id)

LEFT JOIN inventory USING (film_id)

WHERE inventory.film_id IS NULL;

Unfortunately, this statement fails all the same. Note how the film_actor table is before the film table, but deletion from film still fails. It’s not possible to control the order of rows deleted or the order of deletion itself. Strictly speaking, this statement shows how MySQL is different from the SQL standard: by standard, the foreign keys are checked at transaction commit, whereas MySQL checks them immediately, preventing this statement from succeeding. Not only that, but a film is also related to categories, so that will have to be taken care of separately.

MySQL allows few ways out of this situation. First one is to execute few DELETE statements within one transaction:

mysql> BEGIN;

Query OK, 0 rows affected (0.00 sec)

mysql> DELETE FROM film_actor USING

-> film JOIN film_actor USING (film_id)

-> LEFT JOIN inventory USING (film_id)

-> WHERE inventory.film_id IS NULL;

Query OK, 216 rows affected (0.01 sec)

mysql> DELETE FROM film_category USING

-> film JOIN film_category USING (film_id)

-> LEFT JOIN inventory USING (film_id)

-> WHERE inventory.film_id IS NULL;

Query OK, 42 rows affected (0.00 sec)

mysql> DELETE FROM film USING

-> film LEFT JOIN inventory USING (film_id)

-> WHERE inventory.film_id IS NULL;

Query OK, 42 rows affected (0.00 sec)

mysql> ROLLBACK;

Query OK, 0 rows affected (0.02 sec)

You can see that we executed ROLLBACK instead of COMMIT to preserve the rows. In reality, you would of course execute COMMIT to “save” the results of your operation. We talk more about transactions in Chapter 6.

The second way is dangerous. It is possible to suspend foreign key constraints by setting foreign_key_checks system variable to 0 on the session level temporarily. We recommend against that practice. However, that’s the only way to delete from all three tables at the same time:

mysql> SET foreign_key_checks=0;

Query OK, 0 rows affected (0.00 sec)

mysql> BEGIN;

Query OK, 0 rows affected (0.00 sec)

mysql> DELETE FROM film, film_actor, film_category

-> USING film JOIN film_actor USING (film_id)

-> JOIN film_category USING (film_id)

-> LEFT JOIN inventory USING (film_id)

-> WHERE inventory.film_id IS NULL;

Query OK, 300 rows affected (0.03 sec)

mysql> ROLLBACK;

Query OK, 0 rows affected (0.00 sec)

mysql> SET foreign_key_checks=1;

Query OK, 0 rows affected (0.00 sec)

While we don’t recommend disabling foreign key checks, doing so allows to show the power of multitable deletes. Here, with one query it was possible to achieve what took three queries in the previous example.

Let’s break down this query. The tables where rows will be deleted (if matched) are film, film_actor, and film_category. We specified them between DELETE FROM and USING terms for clarity. USING starts our query, the filtering part of the delete statement. In this example, we have constructed a four-table join. We have joined film, film_actor, and film_category using INNER JOIN, as we need only matching rows. To the result of those joins we LEFT JOIN the inventory table. In this context, left join is extremely important, because we are actually only interested in rows where inventory will have no records. We express that with WHERE inventory.film_id IS NULL. The result of this query is that we get all films not in inventory, then all film-actor relantionships for those films, along with all category relations for the films.

Is it possible to make this query safe to use with foreign keys? Not without breaking it down, unfortunately, but we can do better than having to run three queries:

mysql> BEGIN;

Query OK, 0 rows affected (0.00 sec)

mysql> BEGIN;

mysql> DELETE FROM film_actor, film_category USING

-> film JOIN film_actor USING (film_id)

-> JOIN film_category USING (film_id)

-> LEFT JOIN inventory USING (film_id)

-> WHERE inventory.film_id IS NULL;

Query OK, 258 rows affected (0.02 sec)

mysql> DELETE FROM film USING

-> film LEFT JOIN inventory USING (film_id)

-> WHERE inventory.film_id IS NULL;

Query OK, 42 rows affected (0.01 sec)

mysql> ROLLBACK;

Query OK, 0 rows affected (0.01 sec)

What we’ve done here is combined deletion from film_actor and film_category tables into a single DELETE statement, thus allowing deletion from film without any error. The difference with previous example is that we DELETE FROM two tables instead of three.

Let’s talk about the number of rows affected. In the first example, we deleted 42 rows from film, 42 rows from film_category, and 216 rows from the film_actor table. In the second example, our single DELETE query removed 300 rows. In the final example, we removed 258 rows combined from film_category and film_actor tables, and 42 rows from film table. You can probably guess by now that for a multi-table delete MySQL will output the total number of rows deleted, without a breakdown into individual tables. This makes it harder to keep track of exactly how many rows were touched in each table.

In multiple table deletes, you can’t use ORDER BY or LIMIT clauses.

Updates

Now we’ll contrive an example using the sakila database to illustrate multiple-table updates. We’ve decided to change ratings of all horror films to R regardless of the original. To begin, let’s display horror films and their ratings:

mysql> SELECT name category, title, rating

-> FROM film JOIN film_category USING (film_id)

-> JOIN category USING (category_id)

-> WHERE name = 'Horror';

+----------+-----------------------------+--------+

| category | title                       | rating |

+----------+-----------------------------+--------+

| Horror   | ACE GOLDFINGER              | G      |

| Horror   | AFFAIR PREJUDICE            | G      |

| Horror   | AIRPORT POLLOCK             | R      |

| Horror   | ALABAMA DEVIL               | PG-13  |

...

| Horror   | ZHIVAGO CORE                | NC-17  |

+----------+-----------------------------+--------+

56 rows in set (0.00 sec)

mysql> SELECT COUNT(title)

-> FROM film JOIN film_category USING (film_id)

-> JOIN category USING (category_id)

-> WHERE name = 'Horror' AND rating <> 'R';

+--------------+

| COUNT(title) |

+--------------+

|           42 |

+--------------+

1 row in set (0.00 sec)

This and other snippets from this section are available in ch07/multitable_updates.sql

We don’t know about you, but we’d love to see a G-rated horror. Now, let’s put that query into an UPDATE statement:

mysql> UPDATE film JOIN film_category USING (film_id)

-> JOIN category USING (category_id)

-> SET rating = 'R' WHERE category.name = 'Horror';

Query OK, 42 rows affected (0.01 sec)

Rows matched: 56  Changed: 42  Warnings: 0

Let’s look at the syntax: a multiple-table update looks similar to a SELECT query. The UPDATE statement is followed by a list of tables that incorporates whatever join clauses you need or prefer; in this example, we’ve used JOIN (remember, that’s INNER JOIN) to bring together the film and film_category tables. This is followed by the keyword SET, with assignments to individual columns; in this example, you can see that only one column is modified (to change rating to R), so columns in all other tables besides film aren’t modified. The following WHERE is optional, but is necessary in this example to only touch rows with category name Horror.

Note how MySQL reports that 56 rows were matched, but only 42 updated. To better illustrate that, previous select queries included both total number of fitting films, and also a count of films that do not already have the R rating. 42 rows that were updated did not have R rating out of 56 horror films.

As with multiple-table deletes, there are some limitations on updates:

- You can’t use ORDER BY.

- You can’t use LIMIT.

- You can’t update a table that’s read from in a nested subquery.

Other than that, multiple-table updates are much the same as single-table ones.

Replacing Data

You’ll sometimes want to overwrite data. You can do this in two ways using the techniques we’ve shown previously:

- Delete an existing row using its primary key and then insert a new replacement with the same primary key.

- Update a row using its primary key, replacing some or all of the values (except the primary key).

The REPLACE statement gives you a third, convenient way to change data. This section explains how it works.

The REPLACE statement is just like INSERT, but with one difference. You can’t INSERT a new row if there is an existing row in the table with the same primary key. You can get around this problem with a REPLACE query, which first removes any existing row with the same primary key and then inserts the new one.

Let’s try an example, where we’ll replace the row for actor PENELOPE GUINESS in the sakila database:

mysql> REPLACE INTO actor VALUES (1, 'Penelope', 'Guiness', NOW());

ERROR 1451 (23000): Cannot delete or update a parent row:

a foreign key constraint fails (`sakila`.`film_actor`,

CONSTRAINT `fk_film_actor_actor` FOREIGN KEY (`actor_id`)

REFERENCES `actor` (`actor_id`) ON DELETE RESTRICT ON UPDATE CASCADE)

This and other snippets from this section are available in ch07/replacing_data.sql

Unfortunately, as you could guess after reading the previous paragraph, REPLACE has to actually perform a DELETE. If your database is highly constrained referentially, like the sakila is, the REPLACE will not work a lot of times. Let’s not fight against the database here, and instead use the actor_2 table that was created in “Creating Tables with Queries”.

mysql> REPLACE actor_2 VALUES (1, 'Penelope', 'Guiness', NOW());

Query OK, 2 rows affected (0.00 sec)

You can see that MySQL reports that two rows were affected: first, the old row was deleted, and, second, the new row was inserted. You can see that the change we made was minor—we just changed the fully upper-case name to a proper spelling—and therefore, it could easily have been accomplished with an UPDATE. Because the tables in the sakila database is relatively small, it’s difficult to illustrate an example in which REPLACE looks simpler than UPDATE.

You can use the different INSERT syntaxes with REPLACE, including using SELECT queries. Here are some examples:

mysql> REPLACE INTO actor_2 VALUES (1, 'Penelope', 'Guiness', NOW());

Query OK, 2 rows affected (0.00 sec)

mysql> REPLACE INTO actor_2 (actor_id, first_name, last_name)

-> VALUES (1, 'Penelope', 'Guiness');

Query OK, 2 rows affected (0.00 sec)

mysql> REPLACE actor_2 (actor_id, first_name, last_name)

-> VALUES (1, 'Penelope', 'Guiness');

Query OK, 2 rows affected (0.00 sec)

mysql> REPLACE actor_2 SET actor_id = 1,

-> first_name = 'Penelope', last_name = 'Guiness';

Query OK, 2 rows affected (0.00 sec)

The first variant is almost identical to our previous example, except it includes the optional INTO keyword (which, arguably, improves the readability of the statement). The second variant explicitly lists the column names that the matching values should be inserted into. The third variant is the same as the second, without the optional INTO keyword. The final variant uses the SET syntax; you can add the optional keyword INTO to this variant if you want. Note that if you don’t specify a value for a column, it’s set to its default value, just like for INSERT.

You can also bulk-replace into a table, removing and inserting more than one row. Here’s an example:

mysql> REPLACE actor_2 (actor_id, first_name, last_name)

-> VALUES (2, 'Nick', 'Wahlberg'),

-> (3, 'Ed', 'Chase');

Query OK, 4 rows affected (0.00 sec)

Records: 2  Duplicates: 2  Warnings: 0

Note that four rows are affected: two deletions and two insertions. You can also see that two duplicates were found, meaning the replacement of existing rows succeeded. In contrast, if there isn’t a matching row in a REPLACE statement, it acts just like an INSERT:

mysql> REPLACE actor_2 (actor_id, first_name, last_name)

-> VALUES (1000, 'William', 'Dyer');

Query OK, 1 row affected (0.00 sec)

You can tell that only the insert occurred, since only one row was affected.

Replacing also works with a SELECT statement. Recall the recommend table from “Inserting Data Using Queries”, at the beginning of this chapter. Suppose you’ve added 10 films to it, but you don’t like the choice of the seventh film in the list. Here’s how you can replace it with a random choice of another film:

mysql> REPLACE INTO recommend SELECT film_id, language_id,

-> release_year, title, length, 7 FROM film

-> ORDER BY RAND() LIMIT 1;

Query OK, 2 rows affected (0.00 sec)

Records: 1  Duplicates: 1  Warnings: 0

Again, the syntax is the same as with INSERT, but a deletion is attempted (and succeeds!) before the insertion. Note that we keep the value of the sequence_id as 7.

If a table doesn’t have a primary key or another unique key, replacing doesn’t make sense. This is because there’s no way of uniquely identifying a matching row in order to delete it. When you use REPLACE on such a table, its behavior is identical to INSERT. Also, as with INSERT, you can’t replace rows in a table that’s used in a subquery. Finally, note the difference between INSERT IGNORE and REPLACE: the first keeps the existing data with the duplicate key and does not insert the new row, while the second deletes the existing row and replaces it with the new one.

When specifying a list of columns for REPLACE, you have to list every column that does not have a default value. In our examples above, we had to specify actor_id, first_name, and last_name, but omitted the last_update column which has a default value of CURRENT_TIMESTAMP.

WARNING

REPLACE is a powerful statement, but be careful when using it, as the results can be unexpected. Pay special attention when you have auto increment columns and multiple unique keys defined.

MySQL provides another non-standard extension of SQL, which is INSERT ... ON DUPLICATE KEY UPDATE. It similar to REPLACE, but instead of DELETE followed by INSERT, it executes an UPDATE whenever a duplicate key is faced. In the beginning of this section we had issue replacing a row in the actor table. MySQL refused running a REPLACE, because deleting a row from actor table would violate a foreign key constraint. It is, however, easily possible to achieve the desired result with the following statement:

mysql> INSERT INTO actor_3 (actor_id, first_name, last_name)

-> VALUES (1, 'Penelope', 'Guiness')

-> ON DUPLICATE KEY UPDATE first_name = 'Penelope', last_name = 'Guiness';

Query OK, 2 rows affected (0.00 sec)

Note that we’re using actor_3 table created in “Creating Tables with Queries”, as it has all the same constraints as the original actor table. The statement that we’ve just shown is very similar to REPLACE semantically, but has a few key differences. When you do not specify a value for a field in REPLACE command, that field must have a DEFAULT value, and that default value will be set. That naturally follows from the fact that a completely new row is inserted. In case of INSERT ... ON DUPLICATE KEY UPDATE, we are updating an existing row, thus it’s not necessary to list every column. We can do that if we want:

mysql> INSERT INTO actor_3 VALUES (1, 'Penelope', 'Guiness', NOW())

-> ON DUPLICATE KEY UPDATE

-> actor_id = 1, first_name = 'Penelope',

-> last_name = 'Guiness', last_update = NOW();

Query OK, 2 rows affected (0.01 sec)

To minimize the amount of typing necessary for this command, and to allow inserting multiple rows, we can refer to the new field values in our UPDATE part. Let’s see an example with multiple rows, one of which is new:

mysql> INSERT INTO actor_3 (actor_id, first_name, last_name) VALUES

-> (1, 'Penelope', 'Guiness'), (2, 'Nick', 'Wahlberg'),

-> (3, 'Ed', 'Chase'), (1001, 'William', 'Dyer')

-> ON DUPLICATE KEY UPDATE first_name = VALUES(first_name),

-> last_name = VALUES(last_name);

Query OK, 5 rows affected (0.01 sec)

Records: 4  Duplicates: 2

Let’s review this query in more detail. We’re inserting four rows into the actor_3 table, and by using ON DUPLICATE KEY UPDATE we tell MySQL to run an update on any duplicate rows it finds. Unlike our previous example, however, this time we don’t set updated column values explicitly. Instead, we use special VALUES() function to obtain the value of each column in rows we passed to the INSERT. For example, for row 2, Nick, Walhberg, VALUES(first_name) will return Nick. Notice that MySQL reports we’ve update an odd number of rows: five. Whenever a new row is inserted, affected rows is incremented by one. Whenever an old row is updated, affected rows is incremented by two. Since we’ve already update the record for Penelope by running the previous query, our new insert doesn’t add anything new, and MySQL will skip update as well. We are left with two updates for duplicate rows, and insertion of a completely new row, or 5 rows affected in total.

TIP

In most of the situations, we recommend that you default to using INSERT ... ON DUPLICATE KEY UPDATE instead of REPLACE.

The EXPLAIN Statement

You’ll sometimes find that MySQL doesn’t run queries as quickly as you expect. For example, you’ll often find that a nested query runs slowly. You might also find—or, at least, suspect—that MySQL isn’t doing what you hoped, because you know an index exists but the query still seems slow. You can diagnose and solve query optimization problems using the EXPLAIN statement.

Analyzing query plans, understanding optimizer decisions, and tuning query performance are advanced topics. Not only that, but it’s also a form of art, very similar to solving integrals: there’s no one way to do it. We are adding this section so that you know this capability exists, but we won’t get too deep into this topic.

The EXPLAIN statement helps you learn about a SELECT or any other query. Specifically, it tells you how MySQL is going to do the job in terms of the indexes, keys, and steps it’ll take if you ask it to resolve a query. EXPLAIN does not actually execute a query (unless you ask it to), and in general doesn’t take a lot of time to run.

Let’s try a simple example that illustrates the idea:

mysql> EXPLAIN SELECT * FROM actor\G

*************************** 1. row ***************************

id: 1

select_type: SIMPLE

table: actor

partitions: NULL

type: ALL

possible_keys: NULL

key: NULL

key_len: NULL

ref: NULL

rows: 200

filtered: 100.00

Extra: NULL

1 row in set, 1 warning (0.00 sec)

This and other snippets from this section are available in ch07/explain.sql

The statement gives you lots of information. It tells you in this example that:

- The id is 1, meaning the row in the output refers to the first (and only!) SELECT statement in this query. If we utilize a subquery, each SELECT statement will have a different id in the EXPLAIN output. Note that some subqueries will not result in multiple id reported, as MySQL might re-write the query.

mysql> EXPLAIN SELECT * FROM actor WHERE actor_id IN

-> (SELECT actor_id FROM film_actor JOIN

-> film USING (film_id)

-> WHERE title = 'ZHIVAGO CORE');

+----+--------------+-------------+------------+------+...

| id | select_type  | table       | partitions | type |...

+----+--------------+-------------+------------+------+...

|  1 | SIMPLE       | <subquery2> | NULL       | ALL  |...

|  1 | SIMPLE       | actor       | NULL       | ALL  |...

|  2 | MATERIALIZED | film        | NULL       | ref  |...

|  2 | MATERIALIZED | film_actor  | NULL       | ref  |...

+----+--------------+-------------+------------+------+...

...+------------------------+----------------+---------+---------------------+...

...| possible_keys          | key            | key_len | ref                 |...

...+------------------------+----------------+---------+---------------------+...

...| NULL                   | NULL           | NULL    | NULL                |...

...| PRIMARY                | NULL           | NULL    | NULL                |...

...| PRIMARY,idx_title      | idx_title      | 514     | const               |...

...| PRIMARY,idx_fk_film_id | idx_fk_film_id | 2       | sakila.film.film_id |...

...+------------------------+----------------+---------+---------------------+...

...+------+----------+--------------------------------------------+

...| rows | filtered | Extra                                      |

...+------+----------+--------------------------------------------+

...| NULL |   100.00 | NULL                                       |

...|  200 |     0.50 | Using where; Using join buffer (hash join) |

...|    1 |   100.00 | Using index                                |

...|    5 |   100.00 | Using index                                |

...+------+----------+--------------------------------------------+

4 rows in set, 1 warning (0.01 sec)

- The select_type is SIMPLE, meaning it doesn’t use a UNION or subqueries. In the example with a subquery we’ve just given, you can see that a query with id of 2 has a select_type of MATERIALIZED. It means optimizer materialized the results of the subquery, or in other words stored them into a temporary table in memory. The outside query (id 1) will be looking up the results of the inner query (id 2) from this temporary table. This is just one of many optimizations that MySQL can perform while executing complex queries.

- The table that this row is referring to is actor.

- The partitions column is empty, because no tables are partitioned.

- The type of join is ALL, meaning all rows in the table are processed by this SELECT statement. This is often bad—but not in this case—and we’ll explain why later.

- The possible_keys that could be used are listed. In this case, no index will help find all rows in a table, so NULL is reported.

- The key that is actually used is listed, taken from the list of possible_keys. In this case, since no key is available, none is used.

- The key_len (key length) of the key MySQL plans to use is listed. Again, no key means a NULL key_len is reported.

- The ref (reference) columns or constants that are used with the key is listed. Again, none in this example.

- The rows that MySQL thinks it needs to process to get an answer are listed.

- The filtered tells us a percentage of rows from the table that this stage will return. 100 means all rows will be returned. This is expected as we’re asking for all rows.

- Any Extra information about the query resolution is listed. Here, there’s none.

- Every EXPLAIN statement reports a warning. The warning message will have the re-written query. Each query we send to MySQL gets re-written first before the execution. Sometimes, it may be * expanded to an explicit list of columns. Sometimes, it might be subquery optimized implicitly into a JOIN. See below:

mysql> EXPLAIN SELECT * FROM actor WHERE actor_id IN

-> (SELECT actor_id FROM film_actor

-> WHERE film_id = 11);

+----+-------------+------------+------------+--------+...

| id | select_type | table      | partitions | type   |...

+----+-------------+------------+------------+--------+...

|  1 | SIMPLE      | film_actor | NULL       | ref    |...

|  1 | SIMPLE      | actor      | NULL       | eq_ref |...

+----+-------------+------------+------------+--------+...

...+------------------------+----------------+---------+...

...| possible_keys          | key            | key_len |...

...+------------------------+----------------+---------+...

...| PRIMARY,idx_fk_film_id | idx_fk_film_id | 2       |...

...| PRIMARY                | PRIMARY        | 2       |...

...+------------------------+----------------+---------+...

...+----------------------------+------+----------+-------------+

...| ref                        | rows | filtered | Extra       |

...+----------------------------+------+----------+-------------+

...| const                      |    4 |   100.00 | Using index |

...| sakila.film_actor.actor_id |    1 |   100.00 | NULL        |

...+----------------------------+------+----------+-------------+

2 rows in set, 1 warning (0.00 sec)

mysql> SHOW WARNINGS\G

*************************** 1. row ***************************

Level: Note

Code: 1003

Message: /* select#1 */ select

`sakila`.`actor`.`actor_id` AS `actor_id`,

`sakila`.`actor`.`first_name` AS `first_name`,

`sakila`.`actor`.`last_name` AS `last_name`,

`sakila`.`actor`.`last_update` AS `last_update`

from `sakila`.`film_actor` join `sakila`.`actor` where

((`sakila`.`actor`.`actor_id` = `sakila`.`film_actor`.`actor_id`)

and (`sakila`.`film_actor`.`film_id` = 11))

1 row in set (0.00 sec)

In summary, the output of EXPLAIN SELECT * FROM actor\G tells you that all rows from the actor table will be processed (there are 200 of them), and no indexes will be used to resolve the query. This makes sense and is probably exactly what you expected would happen.

We’ll now give the EXPLAIN statement some work to do. Let’s ask it to explain an INNER JOIN between actor, film_actor, film, film_category, and category:

mysql> EXPLAIN SELECT first_name, last_name FROM actor

-> JOIN film_actor USING (actor_id)

-> JOIN film USING (film_id)

-> JOIN film_category USING (film_id)

-> JOIN category USING (category_id)

-> WHERE category.name = 'Horror';

+----+-------------+---------------+------------+--------+...

| id | select_type | table         | partitions | type   |...

+----+-------------+---------------+------------+--------+...

|  1 | SIMPLE      | category      | NULL       | ALL    |...

|  1 | SIMPLE      | film_category | NULL       | ref    |...

|  1 | SIMPLE      | film          | NULL       | eq_ref |...

|  1 | SIMPLE      | film_actor    | NULL       | ref    |...

|  1 | SIMPLE      | actor         | NULL       | eq_ref |...

+----+-------------+---------------+------------+--------+...

...+-----------------------------------+---------------------------+---------+...

...| possible_keys                     | key                       | key_len |...

...+-----------------------------------+---------------------------+---------+...

...| PRIMARY                           | NULL                      | NULL    |...

...| PRIMARY,fk_film_category_category | fk_film_category_category | 1       |...

...| PRIMARY                           | PRIMARY                   | 2       |...

...| PRIMARY,idx_fk_film_id            | idx_fk_film_id            | 2       |...

...| PRIMARY                           | PRIMARY                   | 2       |...

...+-----------------------------------+---------------------------+---------+...

...+------------------------------+------+----------+-------------+

...| ref                          | rows | filtered | Extra       |

...+------------------------------+------+----------+-------------+

...| NULL                         |   16 |    10.00 | Using where |

...| sakila.category.category_id  |   62 |   100.00 | Using index |

...| sakila.film_category.film_id |    1 |   100.00 | Using index |

...| sakila.film_category.film_id |    5 |   100.00 | Using index |

...| sakila.film_actor.actor_id   |    1 |   100.00 | NULL        |

...+------------------------------+------+----------+-------------+

5 rows in set, 1 warning (0.00 sec)

Before we discuss the output, think about how the query could be evaluated. MySQL could go through each row in the actor table, then match that with film_actor, then with film, film_category, and finally category. We have a filter on the category table, so in this imaginery case MySQL would only be able to match fewer rows once it gets to that table. That is a poor execution strategy. Try to think what would be a better strategy.

Let’s see now what MySQL has actually decided to do. This time, there are five rows because there are five tables in the join. Let’s run through this, focusing on those things that are different from the previous examples:

- The first row is similar to what we saw before. MySQL will read all sixteen rows from the category table. This time, the Extra column reads Using where. That means a filter based on WHERE clause is going to be applied. In this example, filtered column shows 10, meaning that roughly 10% of table rows will be produced by this stage for further opreations. MySQL optimizer expects 16 rows in the table, and expects that 1-2 rows will be returned here.

- The join type for the film_category table is ref, meaning that all rows in the film_category table that match rows in the category table will be read. In practice, this means one or more rows from the film_category table will be read for each category_id from the category table.

- The possible_keys for film_category shows both PRIMARY and fk_film_category_category, and the latter is being chosen. PRIMARY key of the film_category table has two columns, and first one of them is film_id, making that index less optimal for filtering on category_id.

- The key used to search film_category has a key_len of 1 and is searched using the sakila.category.category_id value from the category table.

- Moving to the next row, join type for film table is eq_ref. This means that for each row we got from the previous stage (scanning film_category), we’ll read exactly one row on this stage. MySQL can guarantee that, because the index used to access film table is PRIMARY. In general, if a unique not null index is used, eq_ref is possible. This is one of the best join strategies.

Further three rows in the output do not show us anything new. In the end, we see that MySQL selected an optimal execution plan. Usually, the fewer rows are read on the first step of the execution, the faster the query will be.

MySQL 8.0 introduced a new format of explain plan output, which is available through EXPLAIN ANALYZE statement. While it may be somewhat easier to read, the caveat here is that the statement will actually have to be executed, unlike with the regular EXPLAIN. We’ll not gow into details of this new format, but we’ll show an example here:

mysql> EXPLAIN ANALYZE SELECT first_name, last_name

-> FROM actor JOIN film_actor USING (actor_id)

-> JOIN film USING (film_id)

-> WHERE title = ZHIVAGO CORE\G

*************************** 1. row ***************************

EXPLAIN:

-> Nested loop inner join

(cost=3.07 rows=5)

(actual time=0.036..0.055 rows=6 loops=1)

-> Nested loop inner join

(cost=1.15 rows=5)

(actual time=0.028..0.034 rows=6 loops=1)

-> Index lookup on film

using idx_title (title='ZHIVAGO CORE')

(cost=0.35 rows=1)

(actual time=0.017..0.018 rows=1 loops=1)

-> Index lookup on film_actor

using idx_fk_film_id (film_id=film.film_id)

(cost=0.80 rows=5)

(actual time=0.010..0.015 rows=6 loops=1)

-> Single-row index lookup on actor

using PRIMARY (actor_id=film_actor.actor_id)

(cost=0.27 rows=1)

(actual time=0.003..0.003 rows=1 loops=6)

1 row in set (0.00 sec)

This output is even more advanced than the regular explain output, as it gives more data. We leave analyzing this output as an exercise. You may cross-check it with our explanation for regular EXPLAIN ’s output to do so.

Alternative Storage Engines

One of the features of MySQL, distinguishing it from many other RDBMSes is its support for different storage engines. The mechanism of MySQL’s support of multiple engines is complicated, and to explain it properly we’d need to go into depth of MySQL architecture and implementation. We can, however, try to give you a bird’s eye overview of what engines are availalble, what could be a reason to use non-default engine, and why even having this choice is important.

Instead of saying storage engine, which sounds complicated, we could say table type. In very simplified terms, MySQL allows you to create tables of different types, with each type giving those tables distinct properties. There’s no universally good table type, as each storage engine has pros and cons.

In the book so far, we’ve used only the default InnoDB table type. The reason is simple: almost everything you would want from a modern database can be achieved using InnoDB. It’s generally fast, reliable, and a proven and well-supported engine, which provides in our (and not only our) opinion the best sum of pros and cons. We’ve seen this engine used successfully by applications requiring very high throughput of short queries, and also by data warehous applications that run few but “large” queries.

While writing this book, we’ve seen eight additional storage engines documented in official MySQL documentation, and eighteen engines documented for MariaDB. In reality, there are even more of the storage engines available, but not each of them makes to major MySQL flavor’s documentation. Here we’ll only describe those engines we find useful and at least somewhat commonly used. It may well be that the storage engine perfect for your use-case is not one we described. Take no offense; there are just too many of them to cover fairly.

Before we dive into our overview of different engines, let’s briefly look at why this matters. The pluggable nature of storage engines in MySQL and the ability to create tables with different types is important, because it allows you to unify your database access layer. Instead of using multiple database products, each with its own driver, query language, configuration, management, backups, and so on, you may just use MySQL and achieve different behaviors by changing table types. Your apps may not even need to know what types tables have. That said, it’s not all that simple and rosy. You may not be able to use each backup solution we’ll explain in Chapter 10. You will also need to understand trade-offs each engine provides. However, we still think that it’s better to have this ability to change table types than to not.

We may start our review of various storage engines by defining broad categories based on important properties of the engines. One of the most important divisions and one of the most important properties is ability of the engine to support transactions. You can read more about transactions, locking, and why all of it is important in Chapter 6. We can group transactional and non-transactional storage engines together.

Transactional engines currently available include the default InnoDB, an actively developed MyRocks, and deprecated TokuDB. From the different engines available across major MySQL flavors, only these three support transactions. Every other engine is non-transactional.

The next broad division we can perform is based on crash safety, or an ability of the engine to guarantee the durability property of the ACID set of properties. If a table uses a crash-safe engine, then we can expect every bit of data a committed transaction wrote to be available after an unclean instance restart. Crash safe engines include the already mentioned InnoDB, MyRocks, and TokuDB. This would be a very redundant division then, but there’s actually one more engine fitting into this category: Aria. Every other engine does not guarantee crash safety.

We could probably imagine few more ways to group the table types, but we’ve been going on for quite some time already. Let’s get to actually describing few of the engines and their properties. First things first, let’s see how to actually view the list of engines available. To achieve that, a special SHOW ENGINES command is used. Here’s its output on a default MySQL 8.0.23 Linux installation:

mysql> SHOW ENGINES;

+--------------------+---------+...

| Engine             | Support |...

+--------------------+---------+...

| ARCHIVE            | YES     |...

| BLACKHOLE          | YES     |...

| MRG_MYISAM         | YES     |...

| FEDERATED          | NO      |...

| MyISAM             | YES     |...

| PERFORMANCE_SCHEMA | YES     |...

| InnoDB             | DEFAULT |...

| MEMORY             | YES     |...

| CSV                | YES     |...

+--------------------+---------+...

...+----------------------------------------------------------------+...

...| Comment                                                        |...

...+----------------------------------------------------------------+...

...| Archive storage engine                                         |...

...| /dev/null storage engine (anything you write to it disappears) |...

...| Collection of identical MyISAM tables                          |...

...| Federated MySQL storage engine                                 |...

...| MyISAM storage engine                                          |...

...| Performance Schema                                             |...

...| Supports transactions, row-level locking, and foreign keys     |...

...| Hash based, stored in memory, useful for temporary tables      |...

...| CSV storage engine                                             |...

...+----------------------------------------------------------------+...

...+--------------+------+------------+

...| Transactions | XA   | Savepoints |

...+--------------+------+------------+

...| NO           | NO   | NO         |

...| NO           | NO   | NO         |

...| NO           | NO   | NO         |

...| NULL         | NULL | NULL       |

...| NO           | NO   | NO         |

...| NO           | NO   | NO         |

...| YES          | YES  | YES        |

...| NO           | NO   | NO         |

...| NO           | NO   | NO         |

...+--------------+------+------------+

9 rows in set (0.00 sec)

You can see that MySQL conveniently tells us whether an engine supports transactions. The XA column is for distributed transactions—we won’t be covering these at all. Savepoints is basically an ability to create mini transactions within transactions, another advanced topic. As a bit of an exercise, consider executing SHOW ENGINES; in MariaDB and Percona Server installations.

InnoDB

Before we move on to “alternative” storage engines, let’s discuss the default one: the InnoDB. InnoDB is a storage engine that is reliable, high-performant, and full-featured. Pretty much everything you’d expect from a modern RDBMS is achieved in some way or form with InnoDB. In this book, we never change engine of a table, and so every example concerns InnoDB. While you are learning MySQL, we recommend that you stick with InnoDB. Frankly, until you understand its downsides, there’s almost no reason to not use it all the time.

The InnoDB table type includes the following features:

Support for transactions

This is discussed in detail in Chapter 6.

Advanced crash recovery features

The InnoDB table type uses logs, which are files that contain the actions that MySQL has taken to change the database. With the combination of a log and the database, MySQL can recover effectively from power losses, crashes, and other basic database failures. Of course, nothing can help you recover from loss of a machine, failure of a disk drive, or other catastrophic failures. For these, you need offsite backups and new hardware. Every backup tool we explore in Chapter 10 works with InnoDB.

Row-level locking

Unlike the previous default engine --Myisam-- that we’ll explore later, InnoDB provides fine-grained locking infrastructure. The lowest level of locking is row-level, meaning that an individual row can be locked by a running query or transaction. This is important for most write-heavy OLTP applications; otherwise you end up with too much concurrency issues if you’re locking at a higher level, like at a table level.

Foreign-key support

InnoDB is currently the only MySQL table type that supports foreign keys. If you are building a system which requires high level of data safety enforced by referential constraints, InnoDB is your only choice.

Encryption support

InnoDB tables can be encrypted transparently by MySQL.

Partitioning support

InnoDB supports partitioning, that is, spreading of data physically between multiple data files based on some rules. This allows InnoDB to work with tables of tremendous size efficiently

The InnoDB type has the following limitations:

Complexity

InnoDB is relatively complex. This means that there’s quite a lot to configure and understand. Out of almost a thousand of server options of MySQL, more than two hundred are specific to InnoDB. This downside is, however, much smaller than benefits this engine provides.

Data footprint

InnoDB is relatively disk-hungry storage engine, making it less appealing for storing extremely large data sets.

Scaling with database size

InnoDB shines when the so-called “hot” dataset, or frequently accessed data, is present in its buffer pool. This limits its scalability.

MyISAM and Aria

MyISAM long been the default storage engine in MySQL, and a staple of this database. It is simple in use and design, quite performant, and has low overhead. Why has it stopped being the default then? There are actually quite a lot of good reasons for that. We’ll try to cover them under its limitations list below.

Nowadays, we do not recommend using MyISAM unless for legacy reasons. You may read on the internet that its performance is higher than InnoDB’s. Unfortunately, most of that info is very old and didn’t age well. When you are reading this book, the truth is, that statement is simply not correct in the vast majority of the cases. For example, in January 2018, Spectre and Meltdown security vulnerabilities necessiated changes for Linux kernel, resulting in up to 90% performance decrease for MyISAM.

Until MySQL 8.0, MyISAM was used in MySQL for all data dictionary objects. Starting with major version 8.0, the data dictionary is now fully InnoDB, to support advanced features like atomic DDL.

Aria is a reworked MyISAM provided in MariaDB. Apart from promising higher performance and being improved and worked on continuously, the most important feature of Aria is its crash safety. MyISAM, unlike InnoDB, does not guarantee data safety when your write succeeds, which is a major drawback of this storage engine. Aria, on the other hands, allows the creation of durable tables, supported by a global transaction log. In the future, Aria may also support full-fledged transactions, but this is not the case in the time of writing.

The MyISAM table type includes the following features:

Table-level locking

Unlike InnoDB, MyISAM only supports locks on the high level of whole tables. This is much simpler and less nuanced than row-level locking, and has lower overhead and memory requirements. However, a major drawback comes out during highly concurrent write load: even if each session would update or insert a separate row, they will each execute in turn. Reads in MyISAM can coexist simultaneously, but they will block concurrent writes. Writes also block reads.

Partitioning support

Until MySQL 8.0, MyISAM supported partitioning. Later, the only similar thing can be achieved through the use of MERGE or MRG_MyISAM engine.

Compression

It’s possible to create read-only compressed tables with the myisampack utility, which are quite a lot smaller than InnoDB without compression. Since InnoDB supports compression, we recommend you first check whether this option will give you better results.

The MyISAM type has the following limitations:

Crash-safety and recovery

MyISAM tables are not crash safe. MySQL does not guarantee that when a write succeeds, the data actuall reaches files on the disk. If MySQL doesn’t exit cleanly, MyISAM tables may get corrupted, require repairs, and lose data.

Transactions

MyISAM does not support transactions. Thus, MyISAM only provides atomicity for each individual statement, which may not be enough in your case.

Encryption

MyISAM tables do not support encryption.

MyRocks and TokuDB

One of the most significant problems with InnoDB is its relative difficulty in dealing with large data sets. We’ve mentioned that it is desirable to have your frequently-accessed data in memory, but that is not always possible to achieve. Moreover, when data sizes go into multi-terabyte terrytory, InnoDB’s on-disk performance suffers, too. The objects in InnoDB also have quite a large overhead in terms of size. In recent years, a few different projects have appeared that attempt to fix issues inherent to InnoDB’s basic data structure (that of B-Tree) by basing the storage engine on a different data structure. Here we’ll take a look at MyRocks based on the LSM-tree, and mention TokuDB, based on proprietary fractal tree data structure.

Understanding how data structures at the cornerstone of storage engines and types affect their properties is a complex topic, arguably falling out of the scope of database administration and operation. We try to keep things reasonably simple in this book, and so we won’t go into that particular topic. You should also remember what we wrote above about InnoDB: that default is not unreasonable, and more often than not, just using InnoDB is going to give you the best set of trade-offs. It continues to be our recommendation that you use InnoDB while learning MySQL, and beyond that, but we also feel that we should cover the alternatives.

We said we’re only going to mention TokuDB. Why? Its developer—Percona—deprecated this storage engine, and its future is unclear. TokuDB has similar properties to MyRocks, and in fact MyRocks is a preferable migration path for those who continue using TokuDB. This engine, thus, is only mentioned here for completeness.

The MyRocks table type includes the following features:

Support for transactions

MyRocks is a transactional storage engine, supporting regular transactions and distributed transactions. Savepoints are not fully supported.

Advanced crash recovery features

MyRocks relies on internal log files called WAL (“write-ahead log”) to provide crash recovery guarantees. You can expect everything that was committed to be present once the database is restarted after a crash.

Encryption support

MyRocks tables can be encrypted.

Partitioning support

MyRocks tables can be partitioned.

Data compression and compactness

Storage footprint of MyRocks tables is usually lower than that of InnoDB. There are two properties leading to that: storage structure of MyRocks tables on disk is more compact than that of InnoDB, and data within that storage structure can be compressed. While compression is not unique to MyRocks, and InnoDB in fact provides compression options, MyRocks consistently shows better results.

Consistent write performance at scale

This one is difficult to properly explain without going deep into the weeds. However, the minimal version is: MyRocks write performance is almost unaffected by the volume of the data. In the real world, it means that MyRocks tables show worse performance than InnoDB tables until the size of the data becomes much larger than memory. What happens then is that InnoDB’s performance decreases faster than MyRock’s, eventually falling below MyRocks’s level.

The MyRocks type has the following limitations:

Transactions and locking

MyRocks doesn’t support the SERIALIZABLE isolation level and gap-locking that we described in Chapter 6.

Foreign keys

Only InnoDB supports foreign key constraints.

Performance tradeoffs

MyRocks does not cope well with read-heavy and analytical workloads. InnoDB provides better generalized performance.

Complexity

We mentioned that InnoDB is more complex than MyISAM. In some respects, MyRocks is more complex than InnoDB. It is not well-documented, sees active development, and can be difficult to operate.

General availability

MyRocks is not available in Community or Enterprise MySQL, and you need to use another version of MySQL like MariaDB’s or Percona’s to be able to use MyRocks. That may have operational difficulties. Packaged versions lag behind development, and to use all of the current features, a dedicated MySQL server has to be built with MyRocks sources.

Other Table Types

At this point we covered all major table types. There are few more that we will summarize briefly. Some of these storage engines are rarely used, with the exception of Memory, may have documentation issues and bugs.

Memory

Tables of this type are stored entirely in memory, and are never persisted on disk. The obvious advantage is performance: memory is many times faster than disk and will probably always be. The disadvantage is, well, the data is lost as soon as MySQL is restarted or crashes. Memory tables are usually used as temporary tables. Apart from that, Memory tables can be used to hold small-sized frequently-accessed hot data, such as a dictionary of sorts.

Archive

Provides a way to store data in a highly-compressed and append-only manner. You cannot modify or delete data in tables using Archive storage engine. As its name suggests, mostly useful for long-term storage of data. In reality, it’s rarely used, and has few issues with primary key and auto-increment handling. InnoDB with compressed tables and MyRocks may provide a better alternative.

CSV

This storage engine stores tables on disk in CSV format. That allows you to view and manipulate such tables with spreadsheet applications or just text editors. Rarely used, but can be an alternative approach to what we explained in “Loading Data from Comma-Delimited Files”, and also can be used for data export.

Federated

Provides a way to query data in remote MySQL systems. Tables of Federated storage type do not contain any data, only some metadata related to connection details. Interesting way of getting or modifying remote data without setting up replication. Compared to just connecting to remote MySQL, this has a benefit of simultaneously having access to local and remote tables.

Blackhole

This storage engine discards every bit of data that would be stored within its tables. In other words, whatever is written into a blackhole table is immediately lost. That doesn’t sound terribly useful, but there are use-cases for this engine. Usually it’s used to filter replication through an intermediate server where unneded tables are blackholed. One more potential use-case is to get rid of a table under a closed-source application: you can’t just drop the table, as that’ll break the app, but by making it blackhole you remove any processing and storage overhead.

These storage engines are pretty exotic, and are rarely seen in the wild. However, you should know they exist, as we may never know when something might become useful.

Chapter 8. Managing Users and Privileges

The simplest database system is just a bunch of files lying around, having some data in them, and no unified access process. With any RDBMS we come to expect a significantly higher level of sophistication and abstraction. For one, we want to be able to access the database from multiple clients simultaneously. However, not all of the clients are similar, and not every one of them necessarily needs access to all of the data in the database. It’s possible to imagine a database where every user is a superuser, but that would mean you have to install a dedicated database for every app and dataset: wasteful. Instead, databases evolved to support multiple users and roles and provide means to control privileges and access on a very fine-grained level to guarantee secure shared environments.

Understanding users and privileges is an important part of working efficiently with a database system. Well-planned and managed roles result in a secure system, which is easy to manage and work with. In this chapter, we will review most of the things one needs to know about user and privileges management, starting from the basics and moving towards new features like roles. After finishing this chapter, you should have all the basics required to manage access within a MySQL database.

Understanding Users and Privileges

The first building block in the foundation of a shared system is going to be the concept of user. Most modern operating systems have user-based access, so it’s highly likely that you already know what that means. Users in MySQL are special objects used for a few purposes:

- Users act as an identity during authentication.

- Users are actors interacting with schema objects.

And one point which makes MySQL distinct from other DBMSes:

- Users do not own schema objects.

Elaborating on these, every time you access the database, you must specify a user to be used during authentication. Once the authentication is passed, and identity is confirmed, you have access to the database. Usually, the user you will be acting as will be the same you used for authentication, but that’s not strictly necessary, and that’s why we separate the second point about actors. Proxy user is a user which is used for the purpose of checking privileges and actually acting within the database, when another user is used during authentication. That’s a rather complex topic, and requires non-default configuration, but it’s still possible.

This is an important distinction to remember between authentication and authorization. While you can authenticate with one user, you can be authorized as another, and have or not have various permissions.

With these two covered, let’s discuss the last point. In some database management systems, there’s a concept of object ownership. That is, when user creates a database object — a database or a schema, a table, or a stored procedure — it automatically becomes the new object’s owner. Such an owner usually has an ability to modify objects it owns and grant other users access to them. The important thing here is that MySQL does not in any way have a concept of object ownership.

This lack of ownership makes it even more important to have a flexible sets of rules so that users can create objects and then potentially share access to those objects with other users. That is achieved using privileges. Privileges can be thought of as sets of rules controlling what user is allowed to do what actions and access what data. It’s important to understand that by default in MySQL a database user has no privileges at all. Granting a privilege means allowing some action that by default is forbidden.

Users in MySQL are also a bit different than in other databases, because the user object includes a network ACL. Usually, MySQL user is represented not just by its name, like bob, but by an appended network-address, like bob@localhost. This particular example defines a user, which can only be accessed locally through loopback interface or a UNIX socket connection. We will touch this topic and elaborate on that later, once we start discussing the SQL syntax for creating and manipulating existing users.

MySQL stores all information related to users and privileges in special tables in the mysql system database, called grant tables. We’ll talk about this concept in a bit more depth in “Grant tables”.

This short theoretical foundation should be sufficient to form a basic understanding of MySQL’s system of users and privileges. Let’s get practical and review the commands and capabilities that the database provides for managing users and privileges.

root user

Every MySQL installation comes with a few users installed by default. Most of them you don’t need to ever touch, but there’s one that’s extremely frequently used. Some might even say that it’s overused, but that’s not the discussion we want to have here. The user we’re talking about is the ubiquitous and all-powerful root user. Called the same as the default UNIX and Linux superuser, this user is just that in MySQL. A user that can do anything by default.

To be more specific and exact, the user is root@localhost, sometimes called the initial user. The localhost part of the username, as we now know, limits its use to only local connections. When you install MySQL, depending on the specific MySQL flavor, as well as packaging preferences, you might be able to access root@localhost from the OS root account by just executing mysql command. In some other cases, a temporary password for the root will be generated. It is recommended that you set one for the root anyway.

The initially created root user is not the only superuser you can create. In fact, we’ll talk about that later in “SUPER privilege”. While you can create a root@<ip> user, or even a root@% user, we strongly discourage you from doing so, as it is a security hole waiting to be exploited. Not every MySQL needs to even listen on any interface apart from loopback (that is, on localhost), let alone having a superuser with a default name available for login. Of course, you can set secure passwords for all users, and should probably set one for root, but it is arguably that little bit safer to not allow remote super-user access, if that is possible.

For all intents and purposes, the root@localhost is just a regular user with all privileges granted. You can even drop it, which can happen by mistake. One of the pretty common problems faced when running MySQL is losing access to the root@localhost user. You could set a password and forgot it, or you just inherited a server, or something else happened. We cover the procedure to recover a forgotten password for the root@localhost initial user in “Changing root’s password and insecure startup”. If you dropped your last available superuser, you will have to follow the same procedure, but create a new user instead of changing an existing one.

Creating and Using New Users

The very first task we’ll cover is creating a new user. Let’s start with a rather simple example, and review each part:

CREATE USER

'bob'@'10.0.2.%'

IDENTIFIED BY 'password';

Specific SQL statement

User and host definition

Password specification

Or a more complex example:

CREATE USER

'bob'@'10.0.2.%'

IDENTIFIED WITH mysql_native_password

BY 'password'

DEFAULT ROLE 'user_role'

REQUIRE SSL

AND CIPHER 'EDH-RSA-DES-CBC3-SHA'

WITH MAX_USER_CONNECTIONS 10

PASSWORD EXPIRE NEVER;

Specific SQL statement

User and host definition

Authentication plugin specification

Authentication string / password

Role set once user is authenticated and connected

Require SSL for connections for this user

Require specific ciphers

Limit maximum number of connections from this user

Override global password expiration settings

This is just scratching the surface, but should give an idea of the parameters that can be changed for a user during its creation. There are quite a lot of them. Let’s review specific parts of that statement.

User and host

We mentioned before in “Understanding Users and Privileges” that users in MySQL are defined not only by their name, but also by the hostname. In the example given above, the user is 'bob'@'10.0.2.%', where 'bob’ is the username, and '10.0.2.%' is a hostname specification. Even more, it’s a hostname specification with a wildcard. Each time someone connects as a user 'bob’ using TCP, MySQL will do a few things:

1. Get IP address of the connecting client

2. Perform a reverse DNS lookup of the IP address to a hostname

3. Perform a DNS lookup for that hostname (to make sure reverse lookup wasn’t compromised)

4. Check hostname or IP address with the user’s hostname specification

Only if the hostnames match is the access granted. For our user 'bob’, connection from IP address 10.0.2.121 would be allowed, while connection from 10.0.3.22 would be denied. In fact, to allow connections from another hostname, a new user should be created. Internally, 'bob'@'10.0.2.%' is a completely different user from 'bob'@'10.0.3.%'. It’s of course possible to use FQDN in the hostname specification, creating something like 'bob'@'acme.com'. However, DNS lookups take time, and it’s a common optimization to disable them completely.

Specifying all possible hostnames for all users might be tedious, but it’s a useful security feature. However, sometimes a database is set up behind a firewall, or it’s simply impractical to specify hostnames. To completely subvert this system, a single wildcard can be used in host specification, for example 'bob'@'%'. The '%' wildcard is also used when you do not specify the host at all.

NOTE

When proxying connections to MySQL, pay attention to what IP address MySQL “sees” for incoming connections. For example, when HAProxy is used, by default all connections will come from the IP addresses of machines where HAProxy is running. This fact should be taken into consideration when configuring users.

You can notice that we’ve enclosed both the username and host specification in ' symbols. That is not mandatory, and username and host specification follow a similar set of rules to those that were outlined for table and column names and aliases in “Creating and Using Databases” and “Aliases”. For example, when creating or altering user bob@localhost, there’s no need to use any quoting, as is the case with bob@172.17.0.2. You can’t, though, create this user without using quotes: 'username with a space'@'172.%'. Double quotes, single quotes or backticks can be used to enclose usernames and hostnames with special symbols.

Authentication plugins

MySQL supports a wide variety of ways to authenticate users through its system of authentication plugins. These plugins also provide a way for developers to implement new means of authentication without changing MySQL itself. You can set a particular plugin for a user in the creation phase or later.

You might never need to change the plugin for a user, but it’s still worth knowing about this subsystem. In particular, an LDAP authentication with MySQL is achieved by using special authentication plugin. MySQL Enterprise Edition supports first-class LDAP plugin, and other MySQL versions and flavors can use PAM as a middleman.

PAM stands for Pluggable Authentication Modules, and it’s a standard interface on UNIX-like systems that, in very simple terms, allows MySQL to provide authentication by many various methods like OS passwords, or an already-mentioned LDAP. PAM hides complexity of those authentication methods, and programs like MySQL only need to interface with the PAM itself.

MySQL 8.0 uses the caching_sha2_password plugin by default, which provides superior security and performance compared to legacy mysql_native_password, but is not compatible with every client library. To change the default plugin, you can change a default_authentication_plugin variable, which will cause new users to be created with a set plugin.

Authentication string / Password

Some authentication plugins, including the default one, require you to set a password for the user. Other plugins, like the PAM one, require you to define a mapping from OS users to MySQL users. `auth_string` will be used in both cases. Let’s take a look at an example mapping with PAM:

mysql> CREATE USER ''@'' IDENTIFIED WITH auth_pam

-> AS 'mysqld, dba=dbausr, dev=devusr';

Query OK, 0 row affected (0.01 sec)

What’s defined here is a mapping that can be read as follows. The PAM configuration file mysqld will be used (usually, located as /etc/pam.d/mysqld). OS users with group dba will be mapped to MySQL user dbausr, and OS users with group dev — to devusr. The mapping alone is not enough, however, as the necessary permissions have to be assigned.

Note that either the Percona PAM plugin or MySQL Enterprise Edition is required for this to work. This example creates a proxy user, which we briefly discussed in “Understanding Users and Privileges”. The mapping states that OS users in group dba will be mapped to a dbausr MySQL user, and dev to devusr respectively.

Using non-default authentication plugins is a relatively advanced topic, so we’re only bringing up PAM here to show you that the authentication string is not always a password.

For details on installing the Percona plugin, please see “PAM Authentication Plugin”. For the MySQL Enterprise Edition plugin, please see MySQL Enterprise Authentication .

Default roles

Roles are a fairly recent addition to MySQL. You may think of a role as of a collection of privileges. We discuss them later in “Roles”.

SSL configuration

You can force connections to a particular users to use SSL by passing REQUIRE SSL to CREATE USER or ALTER USER commands. Unencrypted connections to the user will be forbidden. Additionally, you can, as shown in the sample above, specify a particular cipher suite or a number of suites that can be used for this user. Ideally, you should set up acceptable cipher suites on system level, but setting on user level is useful to allow some less safe suite for specific connections. You don’t need to specify REQUIRE SSL to specify REQUIRE CIPHER, and in that case connections can be unencrypted, but once encrypted will only be done using a particular set of ciphers.

mysql> CREATE USER 'john'@'192.168.%' IDENTIFIED BY 'P@ssw0rd#'

-> REQUIRE CIPHER 'EDH-RSA-DES-CBC3-SHA';

Query OK, 0 row affected (0.02 sec)

Additional configurable options available include the following:

- X509, forcing a client to present a valid certificate. This, as well as following options, implies SSL.

- ISSUER issuer, forcing a client to present a valid certificate issued by a particular CA, specified in issuer.

- SUBJECT subject, forcing a client to present a valid certificate with a particular subject.

These options can be combined together to specify a very particular certificate and encryption requirement.

mysql> CREATE USER 'john'@'192.168.%'

-> REQUIRE SUBJECT '/C=US/ST=NC/L=Durham/

-> O=BI Dept certificate/

-> CN=client/emailAddress=john@nonexistent.com'

-> AND ISSUER '/C=US/ST=NC/L=Durham/

-> O=MySQL/CN=CA/emailAddress=ca@nonexistent.com'

-> AND CIPHER 'EDH-RSA-DES-CBC3-SHA';

Resource consumption limits

You can define resource consumption limits. In the example listed, we’re limiting the maximum number of concurrent connections using this user to 10. This and other resource options default to 0, meaning unlimited. The other possible constraints are: MAX_QUERIES_PER_HOUR, MAX_UPDATES_PER_HOUR, and MAX_CONNECTIONS_PER_HOUR. All of these options are a part of WITH specification.

Let’s create a fairly restricted user, which can only run ten queries during each given hour, can only have a single concurrent connection, and might not connect more than twice per hour:

mysql> CREATE USER 'john'@'192.168.%'

-> WITH MAX_QUERIES_PER_HOUR 10

-> MAX_CONNECTIONS_PER_HOUR 2

-> MAX_USER_CONNECTIONS 1;

Note that number of MAX_QUERIES_PER_HOUR is inclusive of MAX_UPDATES_PER_HOUR, and will limit updates as well. The number of queries does also include everything that MySQL CLI executes, so setting really low values is not recommended.

Password management options overrides

For authentication plugins that deal with passwords, which are stored in grant tables (covered in “Grant tables”), you can specify a variety of options related to passwords. In our example, we’re setting up a user that has PASSWORD EXPIRE NEVER policy, meaning that its password will never expire based on time. You could also create a user that would have a password expiring every other day.

MySQL 8 extends control capabilities to include tracking of failed authentication attempts and ability to lock account temporarily. Let’s consider an important user with strict control:

mysql> CREATE USER 'app_admin'@'192.168.%'

-> IDENTIFY BY '...'

-> WITH PASSWORD EXPIRE INTERVAL 30 DAY

-> PASSWORD REUSE INTERVAL 180 DAY

-> PASSWORD REQUIRE CURRENT

-> FAILED_LOGIN_ATTEMPTS 3

-> PASSWORD_LOCK_TIME 1;

This user’s password will need to be changed every 30 days, and previous passwords will not be re-used for 180 days. When changing the password, the current password must be presented. For a good measure, we’ll also only allow 3 consecutive failed login attempts, and will block this user for 1 day if those are made.

Note that these are overrides on the default system options. It’s impractical to set up each individual user, so we’d rather recommend you set up the defaults and only use overrides for particular users. For example, you can have your DBA users’ passwords expire more often.

There are some more options available for user creation, which we won’t cover. As MySQL evolves, more options become available, but we believe the ones we’ve covered so far should be enough while learning your way around MySQL.

Since this section is called not only creating, but also using the new user, let’s talk about how any new user can be used. Usually, that falls under few categories.

Connecting and authenticating as a user

This is the default and most widespread use of any user entity. You specify the user and password, MySQL authenticates you with that user and your origin host. Then that pair forms a user as defined within grants tables, which will be used to check privileges and act upon objects. Unless proxy users are involved, this is going to be the default situation.

You can see confirm this easily by running the following query that shows the authenticated user as well as the user provided by the client:

mysql> SELECT CURRENT_USER(), USER();

+----------------+----------------+

| CURRENT_USER() | USER()         |

+----------------+----------------+

| root@localhost | root@localhost |

+----------------+----------------+

1 row in set (0.00 sec)

Quite unsurprisingly, the records match. Even though it’s the most common occurrence, it’s not the only possibility.

Stored objects security

Any user can be named within a stored object (like a stored procedure or a view) in a security context. That allows you to execute an object from the standpoint of another user: definer, instead of invoker. That can be a useful way to provide elevated privileges for some specific operation. That can also be a security hole in your system.

We can see the effect of the DEFINER clause here:

DELIMITER ;;

CREATE DEFINER = `root`@`localhost` PROCEDURE test_proc()

BEGIN

SELECT CURRENT_USER(), USER();

END;

;;

DELIMITER ;

This procedure, when called by bob, will return:

mysql> CALL test_proc();

+----------------+---------------+

| CURRENT_USER() | USER()        |

+----------------+---------------+

| root@localhost | bob@localhost |

+----------------+---------------+

1 row in set (0.00 sec)

The preceding output is important to remember. Sometimes users are not what they are, and to keep your database safe that has to be noted.

Proxy

Some authentication methods like pam or ldap do not operate with a one-to-one mapping from authenticating users to database ones. We showed before how to create a PAM-authenticated user, let’s see what such a user would see if it tried to see authenticating and provided users:

mysql> SELECT CURRENT_USER(), USER();

+------------------+------------------------+

| CURRENT_USER()   | USER()                 |

+------------------+------------------------+

| dbausr@localhost | localdbauser@localhost |

+------------------+------------------------+

1 row in set (0.00 sec)

It should be noted that it’s also possible to create multiple users at once. For example, when the user already exists, CREATE USER doesn’t fail, but will change the user in subtle ways. This can be dangerous. To avoid this, you can specify an IF NOT EXISTS option to the command. Doing so, you tell MySQL to only create the user, and just do nothing if such user already exists.

At this point, you should have a good understanding of what is a MySQL user and how it can be used. Let’s see how these users can be modified. Before that, though, we need to understand how the user-related information is stored internally.

Grant tables

MySQL stores both user information and privileges as records in grant tables. These are special internal tables in the mysql database, which should ideally never be modified manually, and are implicitly modified when statements like CREATE USER or GRANT are run. For example, here we’ll show a slightly truncated output of a select query on mysql.user table, which contains user records, including their passwords (in hashed form):

mysql> SELECT * FROM user WHERE user = 'root'\G

*************************** 1. row ***************************

Host: localhost

User: root

Select_priv: Y

Insert_priv: Y

Update_priv: Y

Delete_priv: Y

...

Create_routine_priv: Y

Alter_routine_priv: Y

Create_user_priv: Y

Event_priv: Y

Trigger_priv: Y

Create_tablespace_priv: Y

ssl_type:

ssl_cipher: 0x

x509_issuer: 0x

x509_subject: 0x

max_questions: 0

max_updates: 0

max_connections: 0

max_user_connections: 0

plugin: mysql_native_password

authentication_string: *E1206987C3E6057289D6C3208EACFC1FA0F2FA56

password_expired: N

password_last_changed: 2020-09-06 17:20:57

password_lifetime: NULL

account_locked: N

Create_role_priv: Y

Drop_role_priv: Y

Password_reuse_history: NULL

Password_reuse_time: NULL

Password_require_current: NULL

User_attributes: NULL

1 row in set (0.00 sec)

You can immediately see that a lot of the fields directly correspond to the specific invocations of the CREATE USER or ALTER USER statements. For example, you can see that this root user doesn’t have any specific rules set regarding its password’s lifecycle. You can also see quite a lot of privileges, though we have omitted even more for brevity. These are privileges that don’t require a target, like a table. Such privileges are called global. We’ll later show how to view targeted privileges.

The other grant tables, as of MySQL 8.0 are, as listed in the documentation:

- mysql.user: User accounts, static global privileges, and other nonprivilege columns.

- mysql.global_grants: Dynamic global privileges.

- mysql.db: Database-level privileges.

- mysql.tables_priv: Table-level privileges.

- mysql.columns_priv: Column-level privileges.

- mysql.procs_priv: Stored procedure and function privileges.

- mysql.proxies_priv: Proxy-user privileges.

- mysql.default_roles: Default user roles.

- mysql.role_edges: Edges for role subgraphs.

- mysql.password_history: Password change history.

You don’t need to remember each of these tables, let alone their structure and contents. But you should remember that these tables exist. When necessary, you can easily look up the necessary structure information in the docs or in the database itself

Internally, MySQL caches grant tables in memory and refreshes this cached representation every time an account management statement is run and thus modifies grant tables. Cache invalidation happens only for the specific user. Ideally, you should never modify these grant tables directly, and there’s rarely a use-case for that. However, in an unfortunate event that you do require to modify any of these tables, there’s a way to tell MySQL to re-read them by running FLUSH PRIVILEGES statement. Failure to do so will mean that the in-memory cache won’t get updated until either of the following happens: the database is restarted, FLUSH PRIVILEGES is executed, an account management statement is run against the same user that was updated directly in grant tables. Even though the command reads as though we’re only touching privileges, MySQL will refresh information from all of the tables.

User and Privileges Management Commands, Logging and Replication

There’s a direct consequence of the fact that all the commands we’re discussing in this chapter are, under the hood, modifying the grant tables. That is, they are close to DML operations in some regards. They are atomic: any CREATE USER, ALTER USER, GRANT, or other such operation, either succeeds or fails without actually changing its target. They are logged: all of the changes done to grant tables either manually or through the relevant commands are logged to the binary log. Thus, they are replicated (see Chapter 13), and will also be available for point-in-time recovery (see Chapter 10).

Application of these statements on the source can break the replication if targeted user doesn’t exist on the replica. We recommend that you keep your replicas consistent with their sources not only in data, but also in users and other “metadata”. Of course, it’s only “meta” in the sense that it exists outside of your real application data, but users are records in the mysql.user table, and that should be remembered when setting up replication.

Mostly, replicas are full copies of their sources. In more complex topologies, like fan-in, that doesn’t need to be true. However, even in such cases, we recommend keeping users consistent across the topology. In general, it is easier and safer than fixing broken replicas, or remembering whether you need to disable binary logging before altering a user.

While we do say that execution of CREATE USER is similar to insert to mysql.user table, the statement is not changed in any way before being logged. That is true for binary logs, slow log (with a caveat), general log, and audit logs. That is also true for every other operation discussed in this chapter. The caveat for slow log is that an extra server option log_slow_admin_statements has to be enabled.

TIP

You can find locations of the logs we’ve mentioned under the following system variable names: log_bin_basename, slow_query_log_file, general_log_file. Their values can include full path to the file, or the file name. In the latter case, that file will be in the MySQL server’s data directory. Binary logs always have a numeric suffix, for example, binlog.000271. We do not cover audit log configuration in this book.

Here’s an example of how the same CREATE USER command is reflected in the general, slow, and binary logs:

mysql> CREATE USER 'vinicius' IDENTIFIED BY '...';

Query OK, 0 rows affected (0.02 sec)

General log

2020-11-22T15:53:17.354270Z        29 Query

CREATE USER 'vinicius'@'%' IDENTIFIED BY <secret>

Slow log

# Time: 2020-11-22T15:53:17.368010Z

# User@Host: root[root] @ localhost []  Id:    29

# Query_time: 0.013772  Lock_time: 0.000161 Rows_sent: 0  Rows_examined: 0

SET timestamp=1606060397;

CREATE USER 'vinicius'@'%' IDENTIFIED BY <secret>;

Binary log

#201122 18:53:17 server id 1  end_log_pos 4113 CRC32 0xa12ac622

Query   thread_id=29    exec_time=0     error_code=0    Xid = 243

SET TIMESTAMP=1606060397.354238/*!*/;

CREATE USER 'vinicius'@'%' IDENTIFIED WITH 'caching_sha2_password'

AS '$A$005$|v>\ZKe^R...'

/*!*/;

Don’t worry if the binary log output is intimidating. It’s not intended for easy human consumption. However, you can see that the actual hash of the password, as it would appear in mysql.user gets written to the binary log. We’ll talk about that more in the relevant chapter. Note, still, that binary logs should be considered an integral part of the database as a whole, including its security.

Modifying and Dropping a User

Whenever the user is created, its lifecycle usually doesn’t end. You may later need to change its properties and require an encrypted connection, for example. It also happens that users need to be dropped. Neither of these operations is too different from the user creation, but you need to know how to perform them in order to fully grasp user management.

Modifying the User

Any parameter that is possible to set during user creation can also be modified at a later time. This is generally achieved using the ALTER USER command, and sometimes instead of modifying, the word altering will be used. MySQL 5.7 and before also have the RENAME USER and SET PASSWORD shortcuts, while version 8.0 expanded that list to include SET DEFAULT ROLE. We cover the role system in “Roles”. Note that ALTER USER can be used to change everything about the user, and other commands are just convienient ways to run common maintenance operations.

Even though we called RENAME USER a shorthand, it is special in that it doesn’t have a “full” ALTER USER alternative. The privileges requried to run the RENAME USER are also different, and are the same as for the user creation.

We will start with the regular ALTER USER. In the first example we’re going to modify the authentication plugin used. A lot of older programs do not support the new and standard in MySQL 8.0 caching_sha2_password plugin, and require you to either create the users, or alter them later, using the older mysql_native_password plugin. First, we’ll check the current plugin used by querying one of the grant tables. See “Grant tables” for more information.

mysql> SELECT plugin FROM mysql.user WHERE

-> user = 'bob' AND host = 'localhost';

+-----------------------+

| plugin                |

+-----------------------+

| caching_sha2_password |

+-----------------------+

1 row in set (0.00 sec)

And now we can change the user and make sure the change is reflected:

mysql> ALTER USER bob@localhost IDENTIFIED WITH mysql_native_password;

Query OK, 0 rows affected (0.01 sec)

mysql> SELECT plugin FROM mysql.user WHERE

-> user = 'bob' AND host = 'localhost';

+-----------------------+

| plugin                |

+-----------------------+

| mysql_native_password |

+-----------------------+

1 row in set (0.00 sec)

Since the change was done via an ALTER command, there’s no need to run FLUSH PRIVILEGES. Immediately after the ALTER succeeds, each new authentication attempt will be using the new plugin. You could modify the record directly, but again, we recommend against that.

The properties that ALTER USER can modify are pretty numerous, and were explained or at least outlined in the “Creating and Using New Users”. There are, however, some frequently required operations that you should now.

Changing user’s password

Probably the single most frequent operation that’s ever done on a user. Changing the given user’s password can be done by another user that has necessary privileges, or by the user itself. We’ll talk about privileges soon, for now let’s concentrate on users.

mysql> ALTER USER bob@localhost IDENTIFIED by 'new password';

Query OK, 0 rows affected (0.01 sec)

Again, the next authentication will require the updated password. There’s a SET PASSWORD shortcut for this command. It can be executed by the authenticated user without any target specification like this:

mysql> SET PASSWORD = 'new password';

Query OK, 0 rows affected (0.01 sec)

Or it can be executed by another user, but requires the target specification. Otherwise, another user would just change its own password:

mysql> SET PASSWORD FOR bob@localhost = 'new password';

Query OK, 0 rows affected (0.01 sec)

Locking and unlocking a user

If you need to temporarily, or forever, block access to a specific user, you can do that using the ACCOUNT LOCK option of ALTER USER. The user in this case is only blocked for authentication. While nobody will be able to connect to MySQL as the user, it can still be used both as a proxy and within a security definer. That makes such users slightly more secure and easier to manage. Apart from this purpose, the ACCOUNT LOCK can also be used to, for example, block traffic from an application that is generating excessive load.

Let’s see this in action:

mysql> ALTER USER bob@localhost ACCOUNT LOCK;

Query OK, 0 rows affected (0.00 sec)

Only new connections will be affected. The message that MySQL produces in this case is clear:

$ mysql -ubob -p

Enter password:

ERROR 3118 (HY000): Access denied for user 'bob'@'localhost'.

Account is locked.

The counterpart to ACCOUNT LOCK is ACCOUNT UNLOCK. This option to ALTER USER does exactly what it says. Let’s allow access to bob:

mysql> ALTER USER bob@localhost ACCOUNT UNLOCK;

Query OK, 0 rows affected (0.01 sec)

And confirming by creating a new connection. Now it will succeed:

$ mysql -ubob -p

Enter password:

mysql>

Expiring user’s password

Instead of blocking off the user’s access completely, or changing the password for them, you may instead want to force them to change their password. That is possible in MySQL with a EXPIRE PASSWORD option of the ALTER USER command. Once the call succeeds, new connections coming from the user will be successfully authenticated using the previous password. However, as soon as the first query is run from the new connection and, thus, as soon as privileges are checked, the user will be presented with an error and forced to change the password. Existing connections are not affected.

Let’s see what it looks like for the user. First, the actual alter:

mysql> ALTER USER bob@localhost PASSWORD EXPIRE;

Query OK, 0 rows affected (0.01 sec)

Now, what the user gets. Note the successful authentication:

$ mysql -ubob -p

Enter password:

mysql> SELECT id, data FROM bobs_db.bobs_private_table;

ERROR 1820 (HY000): You must reset your password using ALTER USER

statement before executing this statement.

Even though the error states you have to run ALTER USER, you now know that SET PASSWORD can be run instead. You could, of course, execute the full command, too. It also doesn’t matter who changes the password: the user itself or another user. The EXPIRE PASSWORD just forces the password change. You should also remember that without password reuse and history controls in place, the user could just set the original password back. If another user changes the password, then sessions authenticated with the old password after the password had been expired would need to be re-opened:

mysql> SET PASSWORD = 'new password';

Query OK, 0 rows affected (0.06 sec)

mysql> SELECT id, data FROM bobs_db.bobs_private_table;

Empty set (0.00 sec)

Now that the password was changed, the same rules apply as explained earlier in the “Changing user’s password” section. Existing connections will continue to work normally, and every new connection will be authenticated with a new password only.

Renaming a user

Somewhat rare, but still an occurrence, is the operation of changing the user name. This operation has a special command: RENAME USER. It requires the CREATE USER privilege, or an UPDATE privilege on the mysql database or just grant tables. There’s no ALTER USER alternative for the command.

You can change both the “name” part of the username and the “host” parts. Since, as you know by now, the “host” part acts as a firewall, be cautious when changing it, as you may cause outages. Although, the same is true of the “name” part as well. Let’s rename our bob user into something more formal:

mysql> RENAME USER 'bob'@'localhost' TO 'robert'@'172.%';

Query OK, 0 rows affected, 1 warning (0.01 sec)

When the user name changes, MySQL automatically scans through its internal tables to see whether that user was a part of a security definer of a view or a stored object. Whenever that is the case, a warning is produced. Since that did happen when we renamed bob, let’s check that warning:

mysql> SHOW WARNINGS;

*************************** 1. row ***************************

Level: Warning

Code: 4005

Message: User 'bob'@'localhost' is referenced as a definer

account in a stored routine.

1 row in set (0.00 sec)

Failure to address this issue can potentially result in orphaned objects that will error out when accessed or executed. We discuss this in detail in the next section.

Dropping the User

The final part of the lifecycle of a database user is its end of life. As with any database object, users can be deleted. In MySQL, the DROP USER command is used to achieve that. This is one of the simplest commands discussed in this chapter and potentially in the whole book. DROP USER takes a user, or, optionally, a list of users as an argument, and has a single modifier: IF NOT EXISTS. Succesful execution of the command irrevokably deletes the user-related information from grant tables (with a caveat we’ll discuss later) and thus prevents further logins.

When you drop a user that is currently used by connections, then even though the drop succeeds, records will be removed when the last connection by this user ends. The next attempt to connect with the given user will result in the ERROR 1045 (28000): Access denied.

The IF NOT EXISTS works similarly to CREATE USER: if the user you target is missing, no error will be returned. This is useful in unattended scripts. If the host part of the user name is not specified, the % is used by default.

In its most basic form, the DROP USER command looks like this:

mysql> DROP USER jeff@localhost;

Query OK, 0 rows affected (0.02 sec)

Executing the DROP again will result in an error.

mysql> DROP USER jeff@localhost;

ERROR 1396 (HY000): Operation DROP USER failed for 'jeff'@'localhost'

If you were to construct an idempotent command that wouldn’t fail, then the following construct should be used instead:

mysql> DROP USER IF EXISTS jeff@localhost;

Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> SHOW WARNINGS;

+-------+------+-----------------------------------------------------+

| Level | Code | Message                                             |

+-------+------+-----------------------------------------------------+

| Note  | 3162 | Authorization ID 'jeff'@'localhost' does not exist. |

+-------+------+-----------------------------------------------------+

1 row in set (0.01 sec)

If you do not specify the host part of the user, MySQL will assume the default %. It’s also possible to drop multiple users at once:

mysql> DROP USER jeff, bob@'192.168.%';

Query OK, 0 rows affected (0.01 sec)

In MySQL, as users own no objects, they can be dropped quite easily and withot much preparation. However, as we have already discussed, users can fulfill extra roles. If the dropped user is used as a proxy user, or is a part of a security definer of some object, then dropping it can create an orphaned record. Whenever the user you drop is part of such relationship, MySQL emits a warning. Note that the DROP USER will still suceed, so it is left for you to resolve the resulting inconsistency and fix orphaned records:

mysql> DROP USER bob@localhost;

Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> SHOW WARNINGS;

*************************** 1. row ***************************

Level: Warning

Code: 4005

Message: User 'bob'@'localhost' is referenced as a definer

account in a stored routine.

1 row in set (0.00 sec)

We recommend that you check this before actually dropping the user. If you fail to notice the warning and take action, the objects are left orphan. Orphaned objects will produce errors when used:

mysql> CALL test.test_proc();

ERROR 1449 (HY000): The user specified as a definer ('bob'@'localhost') does not exist

For users in the proxy relationship, no warning is produced. However, an attempt to use the proxied user will still result in an error. As proxy users are used in authentication, the end result will be inability to log into MySQL with users dependent on the dropped one. This arguably can be more impactful than temporarily losing the ability to call a procedure or query a view, but still, no warning will be emitted. If you are using pluggable authentication relying on proxy users, remember this.

If you find yourself in a situation where you dropped the user and unexpectedly got the warning, you can easily create it again to avoid the errors. Note how the CREATE USER statement results in the now familiar warning. However, the problem is that if you don’t know or remember the initial privileges of the account, the new one can become a security issue. It is best to avoid creating orphaned objects:

mysql> CREATE USER bob@localhost IDENTIFIED BY 'new password';

Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> SHOW WARNINGS;

*************************** 1. row ***************************

Level: Warning

Code: 4005

Message: User 'bob'@'localhost' is referenced as a definer

account in a stored routine.

1 row in set (0.00 sec)

To identify the orphaned records, you need to manually review MySQL’s catalog tables. Specifically, you’re going to need to look at the DEFINER column of the following tables:

mysql> SELECT table_schema, table_name FROM information_schema.columns

-> WHERE column_name = 'DEFINER';

+--------------------+------------+

| TABLE_SCHEMA       | TABLE_NAME |

+--------------------+------------+

| information_schema | EVENTS     |

| information_schema | ROUTINES   |

| information_schema | TRIGGERS   |

| information_schema | VIEWS      |

+--------------------+------------+

Now you can quite easily construct a query to check if a user we’re going to drop or had already dropped is specified within any definer:

SELECT EVENT_SCHEMA AS obj_schema

, EVENT_NAME obj_name

, 'EVENT' AS obj_type

FROM INFORMATION_SCHEMA.EVENTS

WHERE DEFINER = 'bob@localhost'

UNION

SELECT ROUTINE_SCHEMA AS obj_schema

, ROUTINE_NAME AS obj_name

, ROUTINE_TYPE AS obj_type

FROM INFORMATION_SCHEMA.ROUTINES

WHERE DEFINER = 'bob@localhost'

UNION

SELECT TRIGGER_SCHEMA AS obj_schema

, TRIGGER_NAME AS obj_name

, 'TRIGGER' AS obj_type

FROM INFORMATION_SCHEMA.TRIGGERS

WHERE DEFINER = 'bob@localhost'

UNION

SELECT TABLE_SCHEMA AS obj_scmea

, TABLE_NAME AS obj_name

, 'VIEW' AS obj_type

FROM INFORMATION_SCHEMA.VIEWS

WHERE DEFINER = 'bob@localhost';

That query might look intimidating, but by now you should’ve seen UNION used in “The Union”, and the query is just a union of four simple queries. Each individual query looks for an object with a DEFINER value of bob@localhost in one of the following tables: EVENTS, ROUTINES, TRIGGERS, and VIEWS.

In our example, the code above returns a single record for bob@localhost:

+------------+-----------+-----------+

| obj_schema | obj_name  | obj_type  |

+------------+-----------+-----------+

| test       | test_proc | PROCEDURE |

+------------+-----------+-----------+

It’s similarly easy to check if the proxy privilege was granted for this user:

mysql> SELECT user, host FROM mysql.proxies_priv

-> WHERE proxied_user = 'bob'

-> AND proxied_host = 'localhost';

+------+------+

| user | host |

+------+------+

| jeff | %    |

+------+------+

We recommend that you always check for possible orphaned objects and proxy privileges before you drop a particular user. Such gaps left in the database will not only cause obvious issues (errors), but are, in fact, a security risk.

Privileges

When a user connects to a MySQL server, only authentication is performed at first, with the username and host information, as explained before. However, permissions to perform any actions aren’t checked before any commands are executed. MySQL grants privileges according to the identity of the connected user and actions it performs. As discussed in the beginning of this chapter, privileges are sets of permissions to perform actions on various objects. By default, a user is not entitled to any permissions, and thus it has no privileges assigned after CREATE USER is executed.

There are a lot of privileges that you can grant to a user, and later revoke. For example, you can allow a user to read from a table or modify data in it. You can grant a privilege to create tables and databases, and another to create stored procedures. The list is quite vast. Curiously, you will not find a connection privilege anywhere: it’s impossible to disallow a user from connecting to MySQL, assuming the host part of username matches. That’s a direct consequence of what was outlined in the previous paragraph: privileges are only checked when an action is performed, so by nature they will only apply once a user is authenticated.

To get a full list of privileges supported and provided by your MySQL installation, we always recommend checking the manual. However, we’ll try to cover the few broad categories of privileges here. We’ll also talk about levels of privileges, as the very same privilege can be allowed on multiple levels. Actually, that’s what we’ll start with.

Global privileges

These privileges allow the grantee (the user who is granted the privilege — we cover the GRANT in “Privilege Management Commands”) to either act on every object in every database, or to act on the cluster as a whole. The latter applies to commands, which are usually considered administrative. For example, you can allow a user to shut down the cluster.

Privileges in this category are stored within mysql.user and mysql.global_grants. The first one stores conventional static privileges, and the second one stores dynamic privileges. The difference is explained in “Static Versus Dynamic Privileges”. MySQL versions prior to 8.0 have all global privileges in mysql.user.

Database privileges

Privileges granted on a database level will allow the user to act upon objects within that database. As you can imagine, the list of privileges is narrower at this level, since there’s little sense in breaking down the SHUTDOWN privilege below the global level, for example. Records for these privileges are stored within the mysql.db table, and include ability to run DDL and DML queries within the target database.

Object privileges

A logical continuation of the database-level privileges, these target a particular object. Tracked in mysql.tables_priv, mysql.procs_priv, and mysql.proxies_priv, they, respectively, cover tables and views, all types of stored routines, and finally the proxy user permissions. Proxy privileges are special, and other privileges in this category are again regular DDL and DML permissions.

Column privileges

Stored in mysql.columns_priv, these are an interesting set of privileges. You can separate permisions within a particular table column by column. For example, a reporting user may not have a need to read the password column of a particular table. This is a powerful tool, but can be difficult to manage and maintain.

The complete list of privileges, frankly, is very long. It is always advisable to consult with MySQL documentation for your particular version. You should remember that any action a user can perform will either have a dedicated privilege assigned or will be a part of a privilege controlling a wider range of behaviors. In general, database- and object-level privileges will be dedicated (UPDATE, SELECT, and so on), and global privileges will be groups (GROUP_REPLICATION_ADMIN).

You can always access the list of privileges available in your MySQL instance by running a SHOW PRIVILEGES command:

mysql> SHOW PRIVILEGES;

+----------------------------+----------------------+--------------------+

| Privilege                  | Context              | Comment            |

+----------------------------+----------------------+--------------------+

| Alter                      | Tables               | To alter the table |

| Alter routine              | Functions,Procedures | ...

...

| REPLICATION_SLAVE_ADMIN    | Server Admin         |                    |

| AUDIT_ADMIN                | Server Admin         |                    |

+----------------------------+----------------------+--------------------+

58 rows in set (0.00 sec)

Static Versus Dynamic Privileges

Before we go on to review the commands used to manage privileges in MySQL, we must pause and talk about an important distinction. There are two types of privileges in MySQL starting with 8.0: static and dynamic. The static privileges are built into the server, and every installation of MySQL (of the same version) will have them available and usable. The dynamic privileges are, on the other hand, “volatile”: they are not guaranteed to be present all the time.

What are the dynamic privileges then? They are privileges that are registered within the server at runtime. Only registered privileges can be granted, thus it is possible that some of the privileges will never be registered and will never be grantable. All of that is a fancy way of saying that it’s now possible to extend privileges via plugins and components. However, most of the dynamic privileges currently available are registered by default in a regular community server installation.

The important role of dynamic privileges provided with MySQL is that they are aimed at reducing the necessity of using SUPER privilege, that was previously abused. The next section talks about SUPER. The other distinction of dynamic privileges is that they usually control a set of activities users can perform. Unlike a direct SELECT privilege on a table, which just allows querying the data, the CONNECTION_ADMIN privilege allows a whole list of actions. In this particular example, that’s going to be: killing other accounts’ connections, updating data in a read only server, connecting through an extra connection when the limit is reached, and some more. You can easily spot the difference.

SUPER privilege

This section is not long, but it is important. We mentioned earlier in “root user” that there’s a superuser created by default with every MySQL installation: root@localhost. Sometimes you might want to provide the same capabilities to another user, for example one used by a DBA. The convenient built-in way of doing so is by using the special SUPER privilege.

SUPER privilege is basically a catch-all privilege that turns a user it is assigned to into a superuser. As with any privilege, it can be assigned via a GRANT statement, which we’ll review in a moment in “Privilege Management Commands”.

There are two huge problems with the SUPER privilege, however. First, starting with MySQL 8.0 it is deprecated, and is going to be removed in a future release of MySQL. Second, it is a security and operational nightmare. The first point is clear, so let’s talk about the second one, and about the alternatives we have.

Using SUPER privilege poses the same risks and results in the same issues as using the default root@localhost user. Instead of carefully inspecting the scope of privileges, we resort to using a hammer to solve a problem. The main problem with SUPER is its all-encompassing scope. When you create a superuser, you create a liability: the user must be restricted, ideally audited, and operators and programs authenticating as the user must be extremely precise and careful. With great power comes great responsibility and, among other things, the ability to just outright shut down the MySQL instance. Imagine executing that by mistake.

In MySQL versions before 8.0, it’s not feasible to avoid using the SUPER privilege, as there are no alternatives provided for some of the permissions. Starting with 8.0, which deprecates SUPER, MySQL provides a whole set of dynamic privileges that are aimed at removing the need for the single catch-all privilege. You should try to avoid using SUPER privilege, if possible.

Consider an example of a user that needs to control group replication. In MySQL 5.7, you would need to grant SUPER to that user. Starting with version 8.0, however, you can grant a special GROUP_REPLICATION_ADMIN privilege, which only allows to perform a very small subset of actions related to group replication.

Sometimes, you will still need a full-on DBA user that can do anything. Instead of granting SUPER, consider looking at root@localhost privileges and copying them instead. We show the ways to do that in “Checking Privileges”. Taking this further, you can skip granting some of the privileges, for example a INNODB_REDO_LOG_ENABLE privilege, which authorizes one to basically enable a crash-unsafe mode. It is much safer to not have the privilege granted at all, and be able to grant it to yourself, when absolutely required, then run a statement by mistake.

Privilege Management Commands

Now that we know quite a bit about privileges, we can proceed to the basic commands that allow you to control them. You can never ALTER the privilege itself, though, so by controlling privileges here we mean giving and removing them to users.

GRANT statement

This statement is used to grant users (or roles) permissions to perform activities, either in general or on specific objects. The same statement can also be used to assign roles to users, but you cannot at the same time alter permissions and assign roles. To be able to grant a permission (privilege), you need to have that privilege assigned yourself, and have the special GRANT OPTION privilege (we’ll review it later). Users with SUPER (or, newer, CONNECTION_ADMIN) privilege can grant anything, and there’s a special condition related to grant tables, which we’ll discuss later.

For now, let’s check the basic structure of a GRANT statement:

mysql> GRANT SELECT ON app_db.* TO 'john'@'192.168.%';

Query OK, 0 row affected (0.01 sec)

That statement, once executed, tells MySQL that user 'john'@'192.168.%' is allowed to perform read-only queries (SELECT) on any table in an app_db database. Note that we have used a wildcard in the object specification. We could allow a particular user access to every table of every database by specifying a wildcard for database as well:

mysql> GRANT SELECT ON *.* TO 'john';

Query OK, 0 row affected (0.01 sec)

The preceding invocation notably lacks the host specification for user 'john'. This shortcut translates to 'john'@'%', thus it will not be the same user as the 'john'@'192.168.%' we used before. Speaking of wildcards and users, it is not possible to specify a wildcard for the user. Instead, you can specify multiple users or roles in one go like that:

mysql> GRANT SELECT ON app_db.* TO 'john'@'192.168.%',

-> 'kate'@'192.168.%';

Query OK, 0 row affected (0.06 sec)

We talk about available privileges later, but it’s useful to remember that there’s an ALL shortcut that allows to grant every possible privilege on an object or set of objects. That can come handy when you define permissions for the “owner” user — for example, a read-write application user:

mysql> GRANT ALL ON app_db.* TO 'app_db_user';

Query OK, 0 row affected (0.06 sec)

You cannot chain an object specification, so you won’t be able to grant select on two tables, unless that statement can be expressed using wildcards. As we’ll see in the REVOKE section below, you can combine wildcard grants and specific revokes for extra flexibility.

An interesting property of the GRANT is that it doesn’t check for the presence of the objects that you allow. That is, a wildcard is not expanded, but stays a wildcard forever. No matter how many new tables are added to the app_db database, both John and Kate will be able to issue SELECT statements on them. Earlier versions of MySQL also would create a user if it wasn’t found in the mysql.user table, but that is deprecated starting with MySQL 8.

Internally, the GRANT statement updates grant tables in the mysql database. Per-database permissions are tracked in table mysql.db, and there are dedicated tables for other objects. You don’t ever need to modify these tables manually, and doing so may damage your database beyond repair. We’ll see their contents later when we talk about showing privileges.

mysql> SHOW TABLES FROM mysql LIKE '%priv';

+-------------------------+

| Tables_in_mysql (%priv) |

+-------------------------+

| columns_priv            |

| procs_priv              |

| proxies_priv            |

| tables_priv             |

+-------------------------+

4 rows in set (0.00 sec)

As we already discussed in “User and Privileges Management Commands, Logging and Replication”, GRANT, like other such commands, is close to an update on a table. It’s going to get logged in binary logs, slow log, and general log.

Another thing that follows from the fact that there’s an update on grant tables is that if a user has the UPDATE privilege on those tables, that user can grant any account any privilege. Be extremely careful with permissions on objects in mysql schema. There’s little benefit from granting anything there to any user. And yet more: when read_only system variable is enabled, any grant requires super privileges (SUPER or CONNECTION_ADMIN).

In “Privileges” we mentioned column privileges. This set of privileges controls whether a user can read and update data in the particular column of a table. As all other privileges, they can be permitted using the GRANT command:

mysql> GRANT SELECT(id), INSERT(id, data)

-> ON bobs_db.bobs_private_table TO kate@'192.168.%';

Query OK, 0 rows affected (0.01 sec)

User kate will now be able to issue SELECT id FROM bobs_db.bobs_private_table, but not SELECT * or SELECT data.

Finally, you can grant every static privilege on a particular object, or globally, by running GRANT ALL PRIVILEGES instead of specifying every privilege. ALL PRIVILEGES is just a shorthand, and is not itself a special privilege, unlike SUPER, for example.

REVOKE statement

The REVOKE statement is an opposite of the GRANT statement. Using this statement, you can revoke privileges and roles assigned using GRANT. Unless otherwise specified, every property of GRANT applies to REVOKE. For example, you will need to have the GRANT OPTION privilege and the particular privileges that you are revoking.

Partial revokes. Starting with MySQL version 8.0.16 it’s possible to revoke privileges on particular schemas from users that have privileges granted globally. That makes it possible to easily restrict access to a list of databases, while having access to all others, including ones that are newly created. For example, consider a database system where you have a single restricted schema. You need to create a user for your BI application. You start by running the usual:

mysql> GRANT SELECT ON *.* TO 'bi_app_user';

Query OK, 0 rows affected (0.03 sec)

But wait, now the requirements change. The user has to be forbidden from querying any data in the restricted database. It’s extremely easy to set up using partial revokes:

mysql> REVOKE SELECT ON restricted_database.* FROM 'bi_app_user';

Query OK, 0 rows affected (0.03 sec)

mysql> SHOW GRANTS FOR 'bi_app_user';

Before 8.0.16, you would need to fall back to explicitly running GRANT SELECT for each individual allowed schema.

A special invocation of REVOKE exists, allowing the removal of all privileges from a particular user. Remember that you need to have all the privileges you are revoking, thus this option is likely be used by an administrative user. The following statemet will strip a user of its privileges, including the ability to assign any privileges:

mysql> REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'john'@'192.168.%';

Query OK, 0 rows affected (0.03 sec)

The REVOKE statement doesn’t under any circumstances remove the user itself. You can use the DROP USER statement for that.

Checking Privileges

An important part of managing privileges is reviewing them as it’s impossible to remember every privilege granted. As you might imagine, you can query the grant tables mentioned in “Privilege Management Commands”, but that’s not always convenient. That is still an option, however, as it can be a good way to find, for example, every user that has write privileges on a certain table. The more straightforward way, though, is to use a built-in SHOW GRANTS command. Let’s take a look at it:

mysql> SHOW GRANTS FOR john@'192.168.%';

+--------------------------------------------------+

| Grants for john@192.168.%                        |

+--------------------------------------------------+

| GRANT UPDATE ON *.* TO `john`@`192.168.%`        |

| GRANT SELECT ON `sakila`.* TO `john`@`192.168.%` |

+--------------------------------------------------+

2 rows in set (0.00 sec)

In general, you can expect to see every privilege in this output, but there’s a special case. When a user has every static privilege granted for a particular object, instead of listing each and every one of them, MySQL will output ALL PRIVILEGES instead. This is not a special privilege itself, but rather a shorthand for every possible privilege. Internally, ALL PRIVILEGES just translates to Y set for every privilege in the respective grant table:

mysql> SHOW GRANTS FOR bob@localhost;

+----------------------------------------------------------+

| Grants for bob@localhost                                 |

+----------------------------------------------------------+

...

| GRANT ALL PRIVILEGES ON `bobs_db`.* TO `bob`@`localhost` |

...

You can view permissions as related to roles using the SHOW GRANTS, but we’ll talk about that in more detail in “Roles”. To review the permissions of the currently authenticated and authorized user, you can use any of the following statements, which are synonymous:

- SHOW GRANTS;

- SHOW GRANTS FOR CURRENT_USER;

- SHOW GRANTS FOR CURRENT_USER();

Whenever you do not remember what a specific privilege means, you can either consult with the documentation or run the SHOW PRIVILEGES command, which will list every privilege currently available. That covers both static object privileges and dynamic server privileges.

Sometimes you might need to review privileges related to all accounts, or transfer those privileges to another system. One option that you have is to use the mysqldump command provided with MySQL server for all supported platforms. We will be reviewing that command in a lot of detail in Chapter 10. In short, you’ll need to dump all of the grant tables, otherwise you might miss some of the permissions. The safest way to go is to dump all of the data in mysql database:

$ mysqldump -uroot -p mysql

Enter password:

The output will include all the table definitions, along with a lot of INSERT statements. This output can be redirected to a file, then used to seed a new database. We talk more about that in Chapter 10. If your server versions don’t match, or the target server already has some users and privileges stored, it might be best to avoid dropping the existing objects. Add the --no-create-info option to the mysqldump invocation to only receive the INSERT statements.

By using mysqldump you get a portable list of users and privileges, but it’s not exactly easily readable. Here is an example of some of the rows in the output:

--

-- Dumping data for table `tables_priv`

--

LOCK TABLES `tables_priv` WRITE;

/*!40000 ALTER TABLE `tables_priv` DISABLE KEYS */;

INSERT INTO `tables_priv` VALUES ('172.%','sakila','robert'...

'Select,Insert,Update,Delete,Create,Drop,Grant,References,...

('localhost','sys','mysql.sys','sys_config','root@localhost'

'2020-07-13 07:14:57','Select','');

/*!40000 ALTER TABLE `tables_priv` ENABLE KEYS */;

UNLOCK TABLES;

Another option to review the privileges would be to write custom queries over grant tables, as already mentioned. We won’t give any guidelines on that, as there’s no one-size-fits-all solution.

Yet another way is by running SHOW GRANTS for every user in the database. By combining that with SHOW CREATE USER statement, you can generate the list of privileges, which can also be used to re-create the users and their privileges in another database:

mysql> SELECT CONCAT("SHOW GRANTS FOR ", user, "@", host,

-> "; SHOW CREATE USER ", user, "@", host, ";") grants

-> FROM mysql.user WHERE user = 'bob';

+----------------------------------------------------------------+

| grants                                                         |

+----------------------------------------------------------------+

| SHOW GRANTS FOR bob@%; SHOW CREATE USER bob@%;                 |

| SHOW GRANTS FOR bob@localhost; SHOW CREATE USER bob@localhost; |

+----------------------------------------------------------------+

2 rows in set (0.00 sec)

As you can imagine, the idea of automating this procedure is not new. In fact, there’s a tool in Percona-Toolkit — pt-show-grants — that does exactly that, and more. Unforunately, the tool can only be used on Linux officially, and might not work at all on any other platform:

$ pt-show-grants

-- Grants dumped by pt-show-grants

-- Dumped from server Localhost via UNIX socket,

MySQL 8.0.22 at 2020-12-12 14:32:33

-- Roles

CREATE ROLE IF NOT EXISTS `application_ro`;

-- End of roles listing

...

-- Grants for 'robert'@'172.%'

CREATE USER IF NOT EXISTS 'robert'@'172.%';

ALTER USER 'robert'@'172.%' IDENTIFIED WITH 'mysql_native_password'

AS '*E1206987C3E6057289D6C3208EACFC1FA0F2FA56' REQUIRE NONE

PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK PASSWORD HISTORY DEFAULT

PASSWORD REUSE INTERVAL DEFAULT PASSWORD REQUIRE CURRENT DEFAULT;

GRANT ALL PRIVILEGES ON `bobs_db`.* TO `robert`@`172.%`;

GRANT ALL PRIVILEGES ON `sakila`.`actor` TO `robert`@`172.%` WITH GRANT OPTION;

GRANT SELECT ON `sakila`.* TO `robert`@`172.%` WITH GRANT OPTION;

GRANT SELECT ON `test`.* TO `robert`@`172.%` WITH GRANT OPTION;

GRANT USAGE ON *.* TO `robert`@`172.%`;

...

The GRANT OPTION Privilege

As discussed in the opening of this chapter, MySQL does not have a concept of object ownership. Thus, unlike in some other systems, the fact that some user created a table does not automatically mean that the same user can allow another user to do anything with that table. To make this slightly less convoluted, let’s review an example.

User bob has permissions to create tables in a database called bobs_db:

mysql> CREATE TABLE bobs_db.bobs_private_table

-> (id SERIAL PRIMARY KEY, data TEXT);

Query OK, 0 rows affected (0.04 sec)

Operator using the bob user wants to allow the john user to read data in the newly-created table, but, alas, that is not possible:

mysql> GRANT SELECT ON bobs_db.bobs_private_table TO john@'192.168.%';

ERROR 1142 (42000): SELECT, GRANT command denied

to user 'bob'@'localhost' for table 'bobs_private_table'

Let’s check what privileges bob actually has:

mysql> SHOW GRANTS FOR 'bob'@'localhost';

+----------------------------------------------------------+

| Grants for bob@localhost                                 |

+----------------------------------------------------------+

| GRANT USAGE ON *.* TO `bob`@`localhost`                  |

| GRANT SELECT ON `sakila`.* TO `bob`@`localhost`          |

| GRANT ALL PRIVILEGES ON `bobs_db`.* TO `bob`@`localhost` |

+----------------------------------------------------------+

3 rows in set (0.00 sec)

The missing piece here is a privilege that would allow a user to grant other users privileges it has been granted. If a DBA wants to allow bob to grant other users access to tables in bobs_db database, an extra privilege needs to be granted. bob can’t grant that to itself, so a user with administrative privileges is required:

mysql> GRANT SELECT ON bobs_db.*

-> TO 'bob'@'localhost'

-> WITH GRANT OPTION;

Query OK, 0 rows affected (0.01 sec)

Note the WITH GRANT OPTION addition. That’s exactly the privilege that we were looking for. This option will allow bob to pass its privileges to other users. Let’s confirm again by running the GRANT SELECT statement as bob:

mysql> GRANT SELECT ON bobs_db.bobs_private_table TO john@'192.168.%';

Query OK, 0 rows affected (0.02 sec)

As expected, the statement was accepted and executed. There are still few clarifications to make, however. First, we may want to know how granular the GRANT OPTION privilege is. That is, what exactly (apart from SELECT on bobs_private_table) can bob grant to other users? SHOW GRANTS can answer that question for us neatly:

mysql> SHOW GRANTS FOR 'bob'@'localhost';

+----------------------------------------------------------------------------+

| Grants for bob@localhost                                                   |

+----------------------------------------------------------------------------+

| GRANT USAGE ON *.* TO `bob`@`localhost`                                    |

| GRANT SELECT ON `sakila`.* TO `bob`@`localhost`                            |

| GRANT ALL PRIVILEGES ON `bobs_db`.* TO `bob`@`localhost` WITH GRANT OPTION |

+----------------------------------------------------------------------------+

3 rows in set (0.00 sec)

That’s much clearer. We can see that WITH GRANT OPTION is applied to privileges that bob has on a particular database. That’s important to remember. Even though we executed GRANT SELECT ... WITH GRANT OPTION, bob got the ability to grant every privilege it has in bobs_db database.

Second, we may want to know if it is possible to revoke just the GRANT OPTION:

mysql> REVOKE GRANT OPTION ON `bobs_db`.* FROM `bob`@`localhost`;

Query OK, 0 rows affected (0.01 sec)

mysql> SHOW GRANTS FOR 'bob'@'localhost';

+----------------------------------------------------------+

| Grants for bob@localhost                                 |

+----------------------------------------------------------+

| GRANT USAGE ON *.* TO `bob`@`localhost`                  |

| GRANT SELECT ON `sakila`.* TO `bob`@`localhost`          |

| GRANT ALL PRIVILEGES ON `bobs_db`.* TO `bob`@`localhost` |

+----------------------------------------------------------+

3 rows in set (0.00 sec)

Finally, looking at how GRANT OPTION can be revoked, we may want to know whether it can be granted alone. The answer is yes, with a caveat that we’ll show. Let’s grant the GRANT OPTION on sakila and test databases. As you can see above, bob has a SELECT privilege on sakila, but no privileges on the test database:

mysql> GRANT GRANT OPTION ON sakila.* TO bob@localhost;

Query OK, 0 rows affected (0.00 sec)

mysql> GRANT GRANT OPTION ON test.* TO bob@localhost;

Query OK, 0 rows affected (0.01 sec)

Both statements succeeded. It’s pretty clear what exactly bob can grant on sakila: the SELECT privilege. However, it’s less clear what happened with test. Clearly, the statement succeeded. Let’s check it out:

mysql> SHOW GRANTS FOR 'bob'@'localhost';

+-------------------------------------------------------------------+

| Grants for bob@localhost                                          |

+-------------------------------------------------------------------+

| GRANT USAGE ON *.* TO `bob`@`localhost`                           |

| GRANT SELECT ON `sakila`.* TO `bob`@`localhost` WITH GRANT OPTION |

| GRANT USAGE ON `test`.* TO `bob`@`localhost` WITH GRANT OPTION    |

| GRANT ALL PRIVILEGES ON `bobs_db`.* TO `bob`@`localhost`          |

+-------------------------------------------------------------------+

4 rows in set (0.00 sec)

Okay, so the GRANT OPTION alone only gives the user a USAGE privilege, which is the “no privileges.” However, the GRANT OPTION can be seen as a switch, and, when “turned on” it’ll apply for privileges bob has in test database:

mysql> GRANT SELECT ON test.* TO bob@localhost;

Query OK, 0 rows affected (0.00 sec)

mysql> SHOW GRANTS FOR 'bob'@'localhost';

+-------------------------------------------------------------------+

| Grants for bob@localhost                                          |

+-------------------------------------------------------------------+

...

| GRANT SELECT ON `test`.* TO `bob`@`localhost` WITH GRANT OPTION   |

...

+-------------------------------------------------------------------+

4 rows in set (0.00 sec)

All this time we were using wildcard privileges, but it is possible to enable GRANT OPTION for a specific table:

mysql> GRANT INSERT ON `sakila`.`actor`

-> TO `bob`@`localhost` WITH GRANT OPTION;

Query OK, 0 rows affected (0.01 sec)

mysql> SHOW GRANTS FOR 'bob'@'localhost';

+-------------------------------------------------------------------------+

| Grants for bob@localhost                                                |

+-------------------------------------------------------------------------+

...

| GRANT INSERT ON `sakila`.`actor` TO `bob`@`localhost` WITH GRANT OPTION |

+-------------------------------------------------------------------------+

5 rows in set (0.00 sec)

By now, it should be clear that the GRANT OPTION is a powerful addition to the privileges system. Given that MySQL lacks the ownership concept, it’s the only way to make sure users that aren’t superusers can grant each other permissions. However, it is also important, as always, to remember that GRANT OPTION applies to every permission the user has.

Roles

Roles in MySQL are collections of privileges. They simplify user and privilege management by grouping and “containerizing” necessary permissions. You may have few different DBA users that all have the same permissions. Instead of granting privileges individually to each of the users, you can create a role, grant permissions to that role, and assign users that role. In doing so, you also simplify management in that you won’t need to update each user individually. Should your DBAs need adjusted privileges, you can simply adjust the role. Roles were introduced in MySQL 8.0.

Roles are quite similar to users in how they are creted, stored, and managed. To create a role, you need to execute a CREATE ROLE [IF NOT EXISTS] role1[, role2[, role3 ...]] statement. To remove a role, you should execute DROP ROLE [IF EXISTS] role1[, role2[, role3 ...]] statement. When you drop a role, the assignments of the role to various users is removed. Privileges required to create a role are CREATE ROLE or CREATE USER. To drop a role, the DROP ROLE or DROP USER privilege is required. As with the user management commands, if read_only is set, an admin privilege is additionally required to create and drop roles. Direct modification privileges on grant tables allow to modify anything, as we’ve also discussed.

Just like users, role names consist of two parts: the name itself, and the host specification. When host is not specified, the % is assumed. The host specification for role is only rudimentary, and does not limit the role use in any way. In fact, the reason why the host part is even there is that the roles are stored just like users in the mysql.user grant table. As a consequence, you cannot have the same role name@host as an existing user. To have a role with the same name as an existing user, specify a different hostname for the role.

Unlike just plain privileges, roles are not active all the time. When a user is granted a role, the user is just authorized to assign the role, but is not obliged to do so. In fact, a user can have multiple roles, and can “enable” one or more of them within the same connection.

One or more roles can be assigned as a default role to a user during the user’s creation or at a later time through ALTER USER command. Such roles will be active as soon as a user is authenticated.

Let’s review commands, settings, and terminology related to role management.

GRANT PRIVILEGE and REVOKE PRIVILEGE commands

We covered these commands quite extensively in “Privilege Management Commands”. For all intents and purposes, roles can be used just the same as users with both GRANT and REVOKE privilege commands. That is, you can assign all the same privileges to a role as to a user, and revoke them, too.

GRANT role [, role ...] TO user command

The basic command related to role management. By running this command, you authorize a user to assign a particular role. As mentioned above, user is not really forced to use the role. Let’s create a couple of roles that will be able to operate on sakila database:

mysql> CREATE ROLE application_rw;

Query OK, 0 rows affected (0.01 sec)

mysql> CREATE ROLE application_ro;

Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL ON sakila.* TO application_rw;

Query OK, 0 rows affected (0.06 sec)

mysql> GRANT SELECT ON sakila.* TO application_ro;

Query OK, 0 rows affected (0.00 sec)

Now you can assign these roles to an arbitrary number of users and only change the roles when needed. We’ll allow our bob user a read-only access to the sakila database:

mysql> GRANT application_ro TO bob@localhost;

Query OK, 0 rows affected (0.00 sec)

You could always grant more than one role in a single statement.

WITH ADMIN OPTION modifier

When you grant a role to a user, that user is only allowed to activate the role, but not alter it in any way. That user may not grant the role to any other user. If you wish to allow both modification of the role, and granting it to other users, you can specify WITH ADMIN OPTION to the GRANT ROLE command. The result will be reflected in grant tables, and will be visible in the SHOW GRANTS command’s output:

mysql> SHOW GRANTS FOR bob@localhost;

+-------------------------------------------------------------------+

| Grants for bob@localhost                                          |

+-------------------------------------------------------------------+

| GRANT USAGE ON *.* TO `bob`@`localhost`                           |

| GRANT `application_ro`@`%` TO `bob`@`localhost` WITH ADMIN OPTION |

+-------------------------------------------------------------------+

2 rows in set (0.00 sec)

SHOW GRANTS and roles

The SHOW GRANTS command which we reviewed in “Checking Privileges” is capable of showing you both assigned roles, as well as the effective permissions with one or more roles activated. This is possible by adding an optional USING <role> modifier. You can see the default invocation without USING in the previous paragraph. However, we can also show effective privileges that bob will have as soon as application_ro is activated.

mysql> SHOW GRANTS FOR bob@localhost USING `application_ro`;

+-------------------------------------------------------------------+

| Grants for bob@localhost                                          |

+-------------------------------------------------------------------+

| GRANT USAGE ON *.* TO `bob`@`localhost`                           |

| GRANT SELECT ON `sakila`.* TO `bob`@`localhost`                   |

| GRANT `application_ro`@`%` TO `bob`@`localhost` WITH ADMIN OPTION |

+-------------------------------------------------------------------+

3 rows in set (0.00 sec)

SET ROLE DEFAULT | NONE | ALL | ALL EXCEPT role [, role1 ...] | role [, role1 ...] command

Another command of role management, SET ROLE is invoked by an authenticated user to assign the particular role to itself. Once the role is set, its permissions apply to the user. Let’s continue with our example for bob. It’s possible to SET more than one role at once.

$ mysql -ubob

mysql> SELECT staff_id, first_name FROM sakila.staff;

ERROR 1142 (42000): SELECT command denied to user 'bob'@'localhost' for table 'staff'

mysql> SET ROLE application_rw;

ERROR 3530 (HY000): `application_rw`@`%` is not granted to `bob`@`localhost`

mysql> SET ROLE application_ro;

Query OK, 0 rows affected (0.00 sec)

mysql> SELECT staff_id, first_name FROM sakila.staff;

+----------+------------+

| staff_id | first_name |

+----------+------------+

|        1 | Mike       |

|        2 | Jon        |

+----------+------------+

2 rows in set (0.00 sec)

Only when the role is assigned, can bob use its privileges. And you cannot SET ROLE to a role which you aren’t authorized (through GRANT ROLE) to use.

There’s no UNSET ROLE command, but there are few other extensions of the SET ROLE. To unset every role, run SET ROLE NONE. A user can set all roles it has access to by running SET ROLE ALL. User can also get back to the default set of roles by executing SET ROLE DEFAULT. If you need to set a subset of roles which is neither default nor all, you can construct a SET ROLE ALL EXCEPT role [, role]... statement, and explicitly avoid setting one or more roles.

DEFAULT ROLE user option

When you run CREATE USER, or later through ALTER USER, you can set a role or few roles to be the default for a particular user. What it means is that the roles will be implicitly set once the user is authenticated, saving you a SET ROLE statement. This is convenient for example for application users that use a single role or a known set of roles most of the time. Let’s set application_ro as a default role for bob.

$ mysql -uroot

mysql> ALTER USER bob@localhost DEFAULT ROLE application_ro;

Query OK, 0 rows affected (0.02 sec)

$ mysql -ubob

mysql> SELECT CURRENT_ROLE();

+----------------------+

| CURRENT_ROLE()       |

+----------------------+

| `application_ro`@`%` |

+----------------------+

1 row in set (0.00 sec)

As soon as bob@localhost is logged in, the CURRENT_ROLE() function returns the desired application_ro.

Mandatory roles

It is possible to grant one or more roles to every user in the database implicitly. This is achieved by setting a mandatory_roles system parameter (global in scope, and dynamic) to a list of roles. Roles granted this way are not activated automatically until SET ROLE is run. It’s impossible to revoke roles assigned this way, but you can grant them explicitly to a user. Roles listed in the mandatory_roles cannot be dropped until removed from the setting.

Automatically activating roles

By default, roles are not active until SET ROLE is executed. However, it is possible to override that behavior and automatically activate every role available to a user upon authentication. This is analogous to running SET ROLE ALL upon login. This behavior can be enabled or disabled (which is the default) by changing the activate_all_roles_on_login system parameter (global in scope, and dynamic). When activate_all_roles_on_login is set to ON, both explicitly and implicitly (through mandatory_roles) granted roles will be activated for every user.

Cascading role permissions

Roles can be granted to roles. What happens then, is that all permissions of the granted role are inherited by the grantee role. Once the grantee role is activated by a user, you can think of that user as having activated a granted role. Let’s make our example slightly more comples. We will have an application role, that is granted application_ro and application_rw roles. application role itself has no direct permissions assign. We will assign the application role to our bob user and examine the result.

mysql> CREATE ROLE application;

Query OK, 0 rows affected (0.01 sec)

mysql> GRANT application_rw, application_ro TO application;

Query OK, 0 rows affected (0.01 sec)

mysql> *REVOKE application_ro FROM bob;`

Query OK, 0 rows affected (0.02 sec)

mysql> GRANT application TO bob@localhost;

Query OK, 0 rows affected (0.00 sec)

What happens now is that when bob activates the application role, it will have permissions of both rw and ro roles. We can easily verify this. Note that bob cannot activate any of the roles it was granted indirectly.

$ mysql -ubob

mysql> SET ROLE application;

Query OK, 0 rows affected (0.00 sec)

mysql> SELECT staff_id, first_name FROM sakila.staff;

+----------+------------+

| staff_id | first_name |

+----------+------------+

|        1 | Mike       |

|        2 | Jon        |

+----------+------------+

2 rows in set (0.00 sec)

Roles graph

Since roles can be granted to roles, the resulting hierarchy can be pretty hard to follow. You can review it by examining the mysql.role_edges grant table. This can rather quickly get too complex, though, as you can see.

mysql> SELECT * FROM mysql.role_edges;

+-----------+----------------+-----------+-------------+-------------------+

| FROM_HOST | FROM_USER      | TO_HOST   | TO_USER     | WITH_ADMIN_OPTION |

+-----------+----------------+-----------+-------------+-------------------+

| %         | application    | localhost | bob         | N                 |

| %         | application_ro | %         | application | N                 |

| %         | application_rw | %         | application | N                 |

| %         | developer      | 192.168.% | john        | N                 |

| %         | developer      | localhost | bob         | N                 |

| 192.168.% | john           | %         | developer   | N                 |

+-----------+----------------+-----------+-------------+-------------------+

6 rows in set (0.00 sec)

MySQL conveniently includes a built-in function that allows to generate an XML document, which is in a valid graphml format. You can use any capable software to visualize the output. Here’s the function call and a heavily formatted output (XML doesn’t work well in books):

mysql> SELECT * FROM mysql.roles_graphml()\G

*************************** 1. row ***************************

roles_graphml(): <?xml version="1.0" encoding="UTF-8"?>

<graphml xmlns="...

...

<node id="n0">

<data key="key1">`application`@`%`</data>

</node>

<node id="n1">

<data key="key1">`application_ro`@`%`</data>

</node>

...

+ Ideally, a SELECT ... INTO OUTFILE should be used (see “Writing Data into Comma-Delimited Files”). Then we can use the yEd Graph Editor, which is a powerful, cross-platform, and free software, to visualize that output. You can see a zooomed-in section of the complete graph, concentrating on our bob user and surrounding roles.

+ .Section of a visualized MySQL roles graph image::figures/ch08/web/0801.png[]

+ The privilege required to run this function is ROLE_ADMIN.

Differences between roles and users

You could notice that CREATE USER and DROP USER privileges allow modifying the roles. Given that roles are also stored in mysql.user, we can also guess that the regular user management commands will work for roles. That’s quite easy to test and confirm: just run RENAME USER or a DROP USER on a role. Another thing to note is how GRANT and REVOKE privilege commands target roles as if they were users.

Roles are at their core are just regular users. In fact, it is possible to use GRANT ROLE to grant an unlocked user to another unlocked user, or to a role.

mysql> CREATE ROLE `developer`;

Query OK, 0 rows affected (0.02 sec)

mysql> GRANT `john`@`192.168.%` TO `developer`;

Query OK, 0 rows affected (0.01 sec)

mysql> SELECT from_user, to_user FROM mysql.role_edges;

+-----------+-----------+

| from_user | to_user   |

+-----------+-----------+

| john      | developer |

+-----------+-----------+

1 row in set (0.00 sec)

Roles are a powerful and flexible addition to the user and privilege system that MySQL had. As with almost every feature, they can be overused, resulting in an unnecessarily complex hierarchies, which will be hard to follow. However, if you work and try to keep the things simple, roles can save you a lot of work.

Changing root’s password and insecure startup

Sometimes, it can become necessary to gain access to MySQL instance without knowing any user’s password. Or you could accidentally drop every user in the database, effectively locking you out. MySQL provides a way out, but requires you to be able to change its configuration and restart that instance. You may think that is something unlawful, but actually it’s just a protection from one of the simplest incidents that happen: forgotten passwords. Just imagine having a production instance running, which has no superuser access available: that’s obviously not something desirable. Luckily, it’s possible to bypass authorization.

To perform the authentication and privileges bypass, you have to restart a MySQL instance with the --skip-grant-tables option specified. Since most installations use service scripts to start the instance, you can specify skip-grant-tables in the my.cnf configuration file under [mysqld] section. When mysqld is started in this mode, it, pretty obviously, skips reading grant tables, which has the following effects:

- No authentication is performed, thus there’s no need to know any username or password.

- No privileges are loaded, and no permissions are checked.

- MySQL will implicitly set --skip-networking to prevent any but local access while it’s running in the unsafe configuration.

When you connect to MySQL running with --skip-grant-tables, you’ll be authorized as a special user. This user has access to every table and can alter any user. Before altering, for example, the root’s user lost password, you need to run FLUSH PRIVILEGES; command first, otherwise the alter will fail. Once password is reset, it’s recommended to restart the MySQL instance in a normal mode.

mysql> SELECT current_user();

+-----------------------------------+

| current_user()                    |

+-----------------------------------+

| skip-grants user@skip-grants host |

+-----------------------------------+

1 row in set (0.00 sec)

mysql> ALTER USER root@localhost IDENTIFIED BY 'P@ssw0rd!';

ERROR 1290 (HY000): The MySQL server is running with the --skip-grant-tables

option so it cannot execute this statement

mysql> FLUSH PRIVILEGES;

Query OK, 0 rows affected (0.02 sec)

mysql> ALTER USER root@localhost IDENTIFIED BY 'P@ssw0rd!';

Query OK, 0 rows affected (0.01 sec)

There’s also another alternative way to recover root’s password, which is arguably more secure. One of the numerous command-line arguments that mysqld takes is --init-file (or init_file if used through my.cnf). This argument specifies a path to a file containing some SQL statments, which will be executed during MySQL startup. No privilege checks are done at that time, thus it’s possible to put an ALTER USER root statement there. It’s recommended to delete the file and unset the option. Same goes for creating the new root user.

WARNING

Both of the options presented can potentially lead to security issues. Please use them carefully!

Some ideas for secure setup

During the course of this chapter we outlined a few practices related to user and privilege management that can help make your server more secure and safe. Here we will do a short summary of those techniques.

From the administrative side, we have following recommendations.

- Avoid overusing the built-in superuser root@localhost. Imagine five people having access to this user. Even if you have auditing enabled inside MySQL, you won’t be able to efficiently discern which particular person accessed the user and when. This user will also be the first one that potential attackers will try to exploit.

- Starting with MySQL 8.0, avoid creating new superusers through the SUPER privilege. Instead, you can create a special DBA role, that has either all dynamic privileges assigned individually, or just some of them that are frequently required.

- Consider organizing privileges for DBA functions into separate roles. For example, privileges INNODB_REDO_LOG_ARCHIVE and INNODB_REDO_LOG_ENABLE could be a part of innodb_redo_admin role. Since roles are not by default automatically activated, one would first need to SET ROLE explicitly before running potentially dangerous administrative commands.

For regular users, the recommendations are pretty similar.

- Try to minimize the scope of permissions. Always ask if this user need access to every database in the cluster, or even every table in a particular database.

- With MySQL 8.0, using roles is a convenient and arguably safer way to group and manage privileges. If you have three users that need same or almost same privileges, they may share a single role.

- Never allow any non-superuser modification permissions on tables in mysql database. This is a simple mistake that follows from the first recommendation in this list. Granting UPDATE on *.* will allow the grantee to grant itself any permissions.

- To make things even more secure and visible, you can consider periodically saving all of the privileges currently assigned to users, and comparing that with the previously saved sample. You can easily diff the pt-show-grants output, or even the mysqldump output.

With this chapter done, you should be comfortable administering MySQL from the side of users and privileges.

Chapter 9. Using Option Files

Almost every piece of software is capable of being configured, or even must be configured. MySQL is not much different in this regard. While the default configuration will probably suit an astonishing number of installations, more likely than not, you will end up needing to configure the server, or a client. Though the configuration of the server itself and some tuning ideas are discussed in-depth in the Chapter 11 chapter, the configuration files, or options files, in use by MySQL server and clients is covered here.

MySQL provides two ways to configure itself: through command-line argument options, and through specifying configurations in the config file. Since this config file contains only the options that could be specified on the command line, it’s called the option file.

The option file is not exclusive to MySQL server. It’s also not strictly correct to talk about the option file, as pretty much every installation of MySQL will have multiple option files. Most of MySQL software supports inclusion in the config files, and we cover that, too.

Knowing your way around the option file, understanding its sections and options precedence, is an important part of efficiently working with MySQL server and related software. After going through this chapter, you should feel comfortable configuring MySQL server and other programs that use option files.

Structure of the Option File

Option files, or just simply configuration files, in MySQL follow the ubiquitous INI file scheme. In short, they are regular text files that are intended to be edited manually. Of course, you can automate the editing process, but the structure of these files is purposefuly very simple. Every MySQL configuration file can be created and modified with any text editor. There are just two exceptions to this rule, reviewed in “Special Option Files”.

It’s probably going to be easier to have the fille structure visualized somehow. Let’s take a look at a configuration file shipped with MySQL 8 on Fedora Linux. We redacted few lines for brevity:

$ cat /etc/my.cnf

...

[mysqld]

#

# Remove leading # and set to the amount of RAM for the most important data

# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.

# innodb_buffer_pool_size = 128M

...

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

log-error=/var/log/mysqld.log

pid-file=/run/mysqld/mysqld.pid

TIP

On some Linux distributions, for example Ubuntu, /etc/my.cnf configuration file doesn’t exist in a default MySQL installation. Look for /etc/mysql/my.cnf on those systems, or refer to “Search Order for Option Files” for a way to get a full list of option files mysqld reads.

NOTE

Contents of the option files in your system can be different. Treat this as a visual example.

There are few main “parts” to the file structure.

Section (group) headers

These are the values in square brackets preceding other configuration parameters. All programs using option files look for parameters in one or more named sections. For example, [mysqld] is a section used by the MySQL server, and [mysql] is used by the mysql CLI program. The name of the sections are, strictly speaking, arbitrary, and you can put anything there. However, if you change [mysqld] to [section], your MySQL server will ignore all the options following such a header.

MySQL documentation calls sections groups, but both terms can be used interchangeably.

Headers control how the files are parsed, and by which programs. Each option after a section header and before the next section header will be attributed to the first header. It’s probably easier to show this than describe it:

[mysqld]

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

[mysql]

default-character-set=latin1

Options datadir and socket will go under the [mysqld] section (and they are under, quite literally), whereas the default-character-set will be under [mysql]. Note that some MySQL programs read multiple sections, but we’ll talk about that later.

Section headers can be intertwined. The following example is completely valid:

[mysqld]

datadir=/var/lib/mysql

[mysql]

default-character-set=latin1

[mysqld]

socket=/var/lib/mysql/mysql.sock

[mysqld_safe]

core-file-size=unlimited

[mysqld]

core-file

Such configuration might be difficult to read for a person, but programs that parse the file will not care. Still, it’s probably best to keep the configuration files as readable by humans as possible.

Option value pairs

The main part of the config file: the configuration variables themselves, and their values. Each of these is defined on the new line, and follows two general patterns. There is the option=value pattern, as you can see above, but there’s also just the option pattern. For example, the same standard MySQL 8 configuration file has the following lines:

# Remove the leading "# " to disable binary logging

# Binary logging captures changes between backups and is enabled by

# default. It's default setting is log_bin=binlog

# disable_log_bin

disable_log_bin is an option without a value. If we uncomment it, MySQL server will apply the option. In all our earlier examples, we had whitespaces around the = sign in the options specification. That’s not necessary, though, as the whitespaces preceding and following the option name and value are truncated automatically.

Option values can be enclosed in single or double quote characters. That is useful if you’re not sure whether the value is going to be interpreted correctly. For example, on Windows, paths contain the \ symbol, which is treated like an escape symbol. Thus, you should put paths on Windows in double quotes, although you could also escape the \ by doubling it as \\. Quoting the option values is also required when the value includes the # symbol, which would otherwise be treated as a start of the comment.

The rule of thumb that we recommend is to use double quotes when you’re not sure. Here are some valid option value pairs that illustrate previous points:

slow_query_log_file="C:\mysqldata\query.log"

slow_query_log_file=C:\\mysqldata\\query.log

innodb_temp_tablespaces_dir="./#innodb_temp/"

When setting values for numerical options, like sizes of different buffers and files, working with bytes can get tedious. To make life easier, MySQL understands a variety of suffixes standing in for different units. Here’s an example:

innodb_buffer_pool_size = 268435456

innodb_buffer_pool_size = 256M

innodb_buffer_pool_size = 256MB

innodb_buffer_pool_size = 256MiB

All of these lines define the buffer pool of the same size of 268435456 bytes. You can also specify G, GB, GiB for gigabytes and T, TB, and TiB for terabytes if you have a server large enough. Of course, K and other forms are also accepted. MySQL always uses base 2 units: 1GB is 1024M, not 1000.

You cannot specify fractional values for options. So, for example, 0.25G is an incorrect value for the innodb_buffer_pool_size variable. Unlike when setting values from mysql cli or another client connection, you cannot use mathematical notation for option values. You can run SET GLOBAL max_heap_table_size=16*1024*1024; but cannot put the same value in the option file.

The example with innodb_buffer_pool_size is also valid even though the same option is configured multiple times. Last setting value take precedence over the previous ones, and the files are scanned top to bottom. Option files have a global order of precedence as well, but we’ll talk about that in “Search Order for Option Files”.

A very important thing to remember is that setting an incorrect option name will lead to programs not starting. Of course, if an incorrect option is under a section that particular program doesn’t read, it’s fine. But mysqld will fail if it finds an option it doesn’t know under [mysqld]. In MySQL 8 you can validate some of the changes you do to option files by using the --validate-config command-line argument with mysqld. But that validation will only cover core server functionality and won’t verify storage engine options.

Sometimes you still need to set an option that MySQL doesn’t know at startup time. For example, this can be useful when configuring plugins that may be loaded later than startup time. You can prepend options with loose- prefix (or --loose- on the command line), and MySQL will only output a warning but not fail to start:

# mysqld --validate-config

2021-02-11T08:02:58.741347Z 0 [ERROR] [MY-000067] [Server] ...

... unknown variable audit_log_format=JSON.

2021-02-11T08:02:58.741470Z 0 [ERROR] [MY-010119] [Server] Aborting

After the option is changed to loose-audit_log_format, we can see the following instead. No output means that options were successfully validated:

# mysqld --validate-config

#

Comments

An overlooked but important feature of MySQL option files is the ability to leave comments. Comments allow you to put arbitrary text, usually a description of why the setting is here, that will not be parsed by any of MySQL programs. As you can see in a couple of examples above, comments are lines started with #. However, it’s also possible to create coments that start with ;, either is accepted. You don’t necessarily need to have a line dedicated to a comment, as you can inline them, but only with a #, not ;. Once MySQL founds an # symbol on a line (unless escaped), everything past it is treated as a comment. The following line is a valid configuration:

innodb_buffer_pool_size = 268435456 # 256M

Inclusion directives

Configuration files (and whole directories) can be included within other config files. This can make it easier to manage complex configurations, but also makes reading the options more difficult, as you, unlike a program, can’t really merge the files together easily. Still, it’s fairly useful to separate configurations of different MySQL programs. The xtrabackup utility (see Chapter 10) for example doesn’t have any special config file, and reads standard system option files. With inclusion, you can have xtrabackup ’s configs neatly organized in a dedicated file, and declutter your main MySQL option file:

$ cat /etc/my/cnf

!include /etc/mysql.d/xtrabackup.cnf

...

You can see that /etc/my.cnf refers to the /etc/mysql.d/xtrabackup.cnf, which in turn has few configuration options listed in the [xtrabackup] section.

It is not necessary to have different sections in different files, though. For example, Percona XtraDB Cluster has wsrep library configuration options under the [mysqld] section. There are plenty of such configurations, and they aren’t necessarily useful to have in your my.cnf. You may create a separate file, for example, /etc/mysql.d/wsrep.conf, and list wsrep variables under the [mysqld] section there. Any program reading the main my.cnf file will also read all of the included files, and only then parse the variables under sections.

When a lot of such extra configuration files are created, you may want to just go ahead and include the whole directory (or a few) instead of an each individual option file. That’s done with another directive--includedir--that expects a directory path as an argument. MySQL programs will understand that path as a directory and try to include every option in that directory’s tree. On Unix-like systems, .cnf files are included, on Windows either .cnf or .ini files are included.

!includedir /etc/mysql.d

Usually, inclusion is defined in the beginning of a particular config file, but that isn’t mandatory. You can think of the inclusion as appending contents of the included file (or files) to the parent file. Wherever inclusion is defined in the file, the included file’s contents will be placed right under that inclusion. In reality, things are a bit more complicated, but this mental model works when, for example, thinking about options precedence, that we cover in “Search Order for Option Files”.

Each included file must have at least one configuration section defined. For example, it may have [mysqld] in the beginning.

Empty lines

There’s no meaning to empty lines in the option files. You can use them to add more visual space into the config file to make it easier to read for you.

Scope of Options

We can talk about scope of options in MySQL from two perspectives. First, each individual option can have a variety of scopes: global, session, or both, and can be set dynamically or statically. Second, we can talk how options set in option files are scoped, and what’s the scope for option files themselves.

We mentioned that section headers define which particular program (or programs, as nothing prevents one from reading multiple sections) is intended to be reading the options under a particular header. Some configuration options do not make sense outside of their sections, but some, usually affecting session level, can be defined under multiple sections and do not necessarily need to be set equally.

Let’s consider an example, where we have a MySQL server configured with latin1 character set for legacy reasons. However, there are now newer tables with the utf8mb4 charset. We want our mysqldump logical dumps to just be in UTF-8, so we want to override the charset for this program. Conveniently, mysqldump reads its own configuration section, so we can write down an option file like this:

[mysqld]

character_set_server=latin1

[mysqldump]

default_character_set=utf8mb4

This small example shows how options can be set on different levels. In this particular case we used different options, but it could be the same one on different scopes. For example, we want to limit the future size of BLOB and TEXT values (see “String types”) to 32 MiB, but we already have rows of up to 256 MiB in size. We can put an artificial barrier for local clients using a configuration as follows:

[mysqld]

max_allowed_packet=256M

[client]

max_allowed_packet=32M

Server’s max_allowed_packet value will be set on a global scope, and will act as a hard limit on the maximum query size (and also on BLOB or TEXT field size). Client’s setting value will be set on a session scope, and will act as a soft limit. If a particular client requires a larger value (to read old row, for example), it can use SET statement to go up to the server’s limit.

The other scope we can talk about is how the variables relate to each other, how they are scoped within the configuration files, and what is the different option files scope. Let’s start with the latter.

MySQL option files can be divided by scope in few groups: global, client, server, extra. Global option files are read by all or most MySQL programs, whereas client and server are only read by client programs and mysqld respectively. Since it’s possible to specify an extra configuration file to be read by a program, we’re also listing the extra category.

Let’s outline the option files installed and read on Linux and Windows by regular MySQL 8.0 installation. Let’s start with Windows, in Table 9-1.

Table 9-1. MySQL Option Files on Windows

File name

Scope and purpose

%WINDIR%\my.ini, %WINDIR%\my.cnf

Global options read by all programs

C:\my.ini, C:\my.cnf

Global options read by all programs

BASEDIR\my.ini, BASEDIR\my.cnf

Global options read by all programs

Extra config file

File optionally specified with  --defaults-extra-file

%APPDATA%\MySQL\.mylogin.cnf

Login path config file

DATADIR\mysqld-auto.cnf

Option file for persisted variables

Table 9-2 breaks down a typical Linux installation on Fedora Linux.

Table 9-2. MySQL Option Files on Fedora Linux

File name

Scope and purpose

/etc/my.cnf, /etc/mysql/my.cnf, /usr/etc/my.cnf

Global options read by all programs

$MYSQL_HOME/my.cnf

Server options, only read if the variable is set

~/.my.cnf

Global options read by all programs run by a particular OS user

Extra config file

File optionally specified with  --defaults-extra-file

~/.mylogin.cnf

Login path config file under a particular OS user

DATADIR/mysqld-auto.cnf

Option file for persisted variables

With Linux, it’s slightly difficult to identify a universal, complete list of configuration files, as MySQL packages for different Linux distributions may read slightly different files or locations. As a rule of thumb, /etc/my.cnf is a good starting point on Linux, and either %WINDIR%\my.cnf or BASEDIR\my.cnf on Windows.

A couple of the configuration files above in particular may differ in their paths between different systems. /usr/etc/my.cnf can be also written down as SYSCONFIGDIR/my.cnf, and the path is defined at compilation time. $MYSQL_HOME/my.cnf is only read if the variable is set. The default packaged mysqld_safe program (used to start the mysqld daemon) will set $MYSQL_HOME to BASEDIR before running mysqld. You won’t find $MYSQL_HOME set for any of the OS users, and setting that variable is only relevant if you’re starting mysqld manually, or in other words, not using service or systemctl commands.

There’s one peculiar difference between Windows and Linux here. On Linux, MySQL programs read some configuration files located under the given OS user’s HOME directory. In Table 9-2, HOME directory is represented by ~. MySQL on Windows lacks this ability. One frequent use-case for such config files is controlling options for clients based on their OS user. Usually, they will contain credentials. However, the Login Path facility makes this a bit redundant.

An extra config file, specified on the command line with --defaults-extra-file, will be read after every other global file is read, according to its position in the table. This is a useful option when you want to do a one-off run of a program — to test new variables, for example. Overusing this option, though, can lead to troubles in understanding the current set of options in effect (see “Determining the Options in Effect”). The --defaults-extra-file option is not the only one that alters option file handling. --no-defaults prevents the program from reading any configuration files at all. --defaults-file forces the program to read a single file, which can be useful if you have your custom configuration all in one place.

The next section talks more about how different programs read different files, in which order, and what specific group or groups from those files. But you should have a firm grasp on what the option files are that could be used by MySQL on most installations.

Search Order for Option Files

At this point you should know the structure of an option file and where to find them. Most MySQL programs read one or more option files, and it’s important to know in which specific order a program searches for these files and reads them. This section covers the questions of search order and discusses its importance and options precedence.

If a MySQL program reads any option files, you can find the specific files it reads, as well as the order in which it reads them. The general order of the configuration files read will be either exactly the same or very similar to the ones outlined in Tables 9-1 and 9-2. You can use the following command to see the exact order:

$ mysqld --verbose --help | grep "Default options" -A2

Default options are read from the following files in the given order:

/etc/my.cnf /etc/mysql/my.cnf /usr/etc/my.cnf ~/.my.cnf

The following groups are read: mysqld server mysqld-8.0

On Windows, you need to run mysqld.exe instead of mysqld but the output will stay the same. That output includes the list of configuration files read, and their order. You can also see the list of option groups read by mysqld: [mysqld], [server], and [mysqld-8.0]. Note that you can alter the list of options groups that any program reads by adding the --defaults-group-suffix option:

$ mysqld --defaults-group-suffix=-test --verbose --help | grep "groups are read"

The following groups are read: mysqld server mysqld-8.0 ...

... mysqld-test server-test mysqld-8.0-test

You know at this point what option files and option groups are read. However, it’s also importatnt to know the order of precedence for those option files. Nothing prevents you from setting one or more options in multiple, or all, configuration files, after all. In the case of MySQL programs, the order of precedence for config files is simple: options from files read later take precedence over previously read files. Options passed to commands directly as command-line arguments take precedence over any configuration options in any config files.

You can look at the Tables 9-1 and 9-2 from top to bottom. The lower the config file, the higher the “weight” of options there. For example, for any programs that are not mysqld, values in .mylogin.cnf take precedence over any other config files, and only lower than values set through command-line arguments. For mysqld, the same is true for persisted variables set in DATADIR/mysqld-auto.cnf.

The ability to include configuration files in other files through inclusion directives makes things slightly more complicated. But you always include extras within one or more of the option files listed in the Tables 9-1 and 9-2. You can think of the inclusion as if right before MySQL reads the parent config file, included files are just appended to the parent file as is. Thus, the precedence of options globally is that of the parent configuration file. Within the “resulting” file itself (again, imagine included files just all appended to their parent) options take precedence, as we now know, from top to bottom, with lower ones overriding higher ones.

Special Option Files

There are two special configuration files used by MySQL, which are exceptions to the structure outlined in “Structure of the Option File”.

Login Path Configuration File

First, there’s a .mylogin.cnf file, which is used as a part of the login path system. Even though you can think of its structure as of the regular option file, this particular file is not a regular text file. In fact, that’s an encrypted text file. This file is intended to be created and modified through use of the special mysql_config_editor program, which is supplied with MySQL, usually in the client package. It is encrypted, because the purpose of .mylogin.cnf and the whole login path system is to store MySQL connection options, including passwords, in a convenient and secure manner.

By default, mysql_config_editor and other MySQL programs will look for .mylogin.cnf in the $HOME of the current user on Linux and various UNIX flavors, and in %APPDATA%\MySQL on Windows. It is possible to change the location and name of the file by setting the MYSQL_TEST_LOGIN_FILE environment variable.

To start looking at the file’s contents, you first need to create it. Let’s store a password to the root user:

$ mysql_config_editor set --user=root --password

Enter password:

After entering the password, and confirming the input, let’s take a look at the .mylogin.cnf contents:

$ ls -la ~/.mylogin.cnf

-rw-------. 1 skuzmichev skuzmichev 100 Jan 18 18:03 .mylogin.cnf

$ cat ~/.mylogin.cnf

>pZ

prI

R86w"># &.h.m:4+|DDKnl_K3>73x$

$ file ~/.mylogin.cnf

.mylogin.cnf: data

$ file ~/.my.cnf

.my.cnf: ASCII text

We can see now that .mylogin.cnf is for sure not a regular configuration file, at least on the surface. As such, it requires special treatment. In addition to creating the file, you can view, and modify .mylogin.cnf with the mysql_config_editor we mentioned before. Let’s start with how to actually see what’s inside. The option for that is print:

$ mysql_config_editor print

[client]

user = "root"

password = *****

client is a default login path. All operations done with mysql_config_editor without an explicit login path specification affect the client login path. We didn’t specify any login path when running set earlier, and root ’s credentials were written under the client path. It’s possible to specify a specific login path for any operation. Let’s put root ’s credentials under a login path named root:

$ mysql_config_editor set --login-path=root --user=root --password

Enter password:

To specify the login path, use --login-path= or -G option, and to view all when using print add the --all option:

$ mysql_config_editor print --login-path=root

[root]

user = root

password = *****

$ mysql_config_editor print --all

[client]

user = root

password = *****

[root]

user = root

password = *****

You can see that the output is that of an option file, so you can think of .mylogin.cnf as of an option file with some special treatment. Just don’t edit it manually. Speaking of editing, let’s add few more options to the set command as mysql_config_editor calls it. We’ll create a new login path in the process.

mysql_config_editor supports the --help (or -?) argument, which can be combined with other options to get help specifically on print or set, for example. Let’s start by looking at a slightly truncated help output for set first:

$ mysql_config_editor set --help

...

MySQL Configuration Utility.

Description: Write a login path to the login file.

Usage: mysql_config_editor [program options] [set [command options]]

-?, --help          Display this help and exit.

-h, --host=name     Host name to be entered into the login file.

-G, --login-path=name

Name of the login path to use in the login file. (Default

: client)

-p, --password      Prompt for password to be entered into the login file.

-u, --user=name     User name to be entered into the login file.

-S, --socket=name   Socket path to be entered into login file.

-P, --port=name     Port number to be entered into login file.

-w, --warn          Warn and ask for confirmation if set command attempts to

overwrite an existing login path (enabled by default).

(Defaults to on; use --skip-warn to disable.)

...

You can see here another interesting property of .mylogin.cnf: you can’t put arbitrary parameters into it. Now we know that we can basically only set a few options related to logging in to a MySQL instance or instances. Which is of course to be expected of the “login path”. But let’s get back to editing the file:

$ mysql_config_editor set --login-path=scott --user=scott

$ mysql_config_editor set --login-path=scott --user=scott

WARNING : scott path already exists and will be overwritten.

Continue? (Press y|Y for Yes, any other key for No) : y

$ mysql_config_editor set --login-path=scott --user=scott --skip-warn

Here we’ve shown all behaviors that mysql_config_editor can exhibit when modifying or creating a login path. If the login path doesn’t exist, no warning is produced. If there’s already such a path, a warning and confirmation will be printed, but only if --skip-warn is not specified. Note that we’re talking here in terms of the whole login path! It is not possible to modify a single property in the path: the whole login path is written out every time. If you want to change a single property, you’ll need to specify all other properties that you need.

Let’s add some more details and view the result:

$ mysql_config_editor set --login-path=scott \

--user=scott --port=3306 --host=192.168.122.1 \

--password --skip-warn

Enter password:

$ mysql_config_editor print --login-path=scott

[scott]

user = scott

password = *****

host = 192.168.122.1

port = 3306

Persistent System Variables Configuration File

Second, there’s a mysqld-auto.cnf file, which has resided in the data directory since MySQL 8.0. It is a part of the new Persisted System Variables process, which allows you to update MySQL options on disk using the regular SET statements. Before, you could not change MySQL’s configuration from within a database connection. The usual flow was to change the option files on disk, then run a SET GLOBAL statement to change the configuration variable online. As you can imagine, this can lead to mistakes, and changes being only done online, for example. The new SET PERSIST statement takes care of both tasks: a variable is updated online, and a variable is updated on disk. It’s also possible to update a variable on disk only.

The file itself is, surprisingly, not like any other configuration file in MySQL at all. Where .mylogin.cnf was an encrypted but still a regular option file, mysqld-auto.cnf uses a common, but a completely different format: JSON.

Before you persist anything, mysqld-auto.cnf doesn’t exist. So we’ll start by changing a system variable:

mysql> SELECT @@GLOBAL.max_connections;

--------------------------

| @@GLOBAL.max_connections |

--------------------------

|                      100 |

--------------------------

1 row in set (0.00 sec)

mysql> SET PERSIST max_connections = 256;

Query OK, 0 rows affected (0.01 sec)

mysql> SELECT @@GLOBAL.max_connections;

--------------------------

| @@GLOBAL.max_connections |

--------------------------

|                      256 |

--------------------------

1 row in set (0.00 sec)

As expected, the variable was updated on a global scope online. Let’s now explore the resulting config file. Since we know that contents are in JSON format, we’ll use a jq utility to format it nicely. That’s not necessary, but makes the file easier to read:

$ cat /var/lib/mysql/mysqld-auto.cnf | jq .

{

"Version": 1,

"mysql_server": {

"max_connections": {

"Value": "256",

"Metadata": {

"Timestamp": 1611728445802834,

"User": "root",

"Host": "localhost"

}

}

}

}

Just by looking at this file containing a single variable value, you can see why plain ini is used for config files that are intended to be edited by humans. This is verbose! However, JSON is excellent for reading by computers, so it’s a good match for a configuration written and read by MySQL itself. As an added benefit, you get auditing of the changes. See how the max_connection property has Metadata containing a time when the change occurred, and the author of the change.

Since this is still a text file, unlike the login path config, which is binary, it’s possible to edit mysqld-auto.cnf manually. However, it’s unlikely that there are a lot of cases where that’s needed.

Determining the Options in Effect

The last routine task that pretty much anyone working with MySQL will face is finding out values for the variables, and in what option file they are set. Also why, but no amount of technology helps with human reasoning sometimes.

At this point, we know what files MySQL programs read, in what order, and their precedence. We also know that command-line arguments override any other settings. Still, understanding where exactly some variable is set can be a daunting task. Multiple files scanned, potentially with nested inclusions, can make for a long investigation.

Let’s start by looking at how to determine the options currently used by a program. For some, like MySQL server (mysqld), that is easy. You can get the list of current values used by mysqld by running SHOW GLOBAL VARIABLES. It’s impossible to change an option value that mysqld uses and not see the effect reflected in global variables state. For other programs, things get more complicated. To understand what options are used by mysql, you’d have to run it, then check outputs of SHOW VARIABLES and SHOW GLOBAL VARIABLES to see which options are overridden on the session level. But even before a successful connection to the server is established, mysql must read or receive connection information.

There are two easy ways to determine the list of options in effect when the program starts: by passing the --print-defaults argument to that program, or by using a special my_print_defaults program. Let’s take a look at the former option as executed on Linux. You can ignore the sed part, but it makes the output slightly nicer for human eyes:

$ mysql --print-defaults

mysql would have been started with the following arguments:

--user=root --password=*****

$ mysqld --print-defaults | sed 's/--/\n--/g'

/usr/sbin/mysqld would have been started with the following arguments:

--datadir=/var/lib/mysql

--socket=/var/lib/mysql/mysql.sock

--log-error=/var/log/mysqld.log

--pid-file=/run/mysqld/mysqld.pid

--max_connections=100000

--core-file

--innodb_buffer_pool_in_core_file=OFF

--innodb_buffer_pool_size=256MiB

The variables picked up here come from all the option files we discussed before. If a variable value was set multiple times, the last occurrence will take precedence. However, --print-defaults will actually output every option set. For example, the output could look like this — even though innodb_buffer_pool_size is set five times, the value in effect will be 384 M:

$ mysqld --print-defaults | sed 's/--/\n--/g'

/usr/sbin/mysqld would have been started with the following arguments:

--datadir=/var/lib/mysql

--socket=/var/lib/mysql/mysql.sock

--log-error=/var/log/mysqld.log

--pid-file=/run/mysqld/mysqld.pid

--max_connections=100000

--core-file

--innodb_buffer_pool_in_core_file=OFF

--innodb_buffer_pool_size=268435456

--innodb_buffer_pool_size=256M

--innodb_buffer_pool_size=256MB

--innodb_buffer_pool_size=256MiB

--large-pages

--innodb_buffer_pool_size=384M

You can also combine the --print-defaults with other command-line arguments, particularly with configuration options. If you intend to run a program with command-line arguments, you could see whether they override or repeat an already set value, for example:

$ mysql --print-defaults --host=192.168.4.23 --user=bob | sed 's/--/\n--/g'

mysql would have been started with the following arguments:

--user=root

--password=*****

--host=192.168.4.23

--user=bob

The other way to print variables, as mentioned, is the my_print_defaults program. It takes one or more section headers as arguments and will print all options it finds in scanned files that fall into the requested groups. That can be beneficial to --print-defaults when we need to just review one option group. In MySQL 8, the mysqld program reads following groups: mysqld, server, mysqld-8.0. The combined output of options may be lengthy, but what if we only need to view options specifically set for 8.0? For this example, we’ve added [mysqld-8.0] option group to the option file, and put a couple of configuration parameter values there:

$ my_print_defaults mysqld-8.0

--character_set_server=latin1

--collation_server=latin1_swedish_ci

That can also help with other software like PXC or with MariaDB flavor of MySQL, both of which include multiple configuration groups. In particular, you would likely want to review the [wsrep] section without any other options. my_print_defaults can, of course, be used to output a complete set of options, it just needs to be passed all the section headers a program reads. For example, mysql program reads mysql and client option groups:

$ my_print_defaults mysql client

--user=root

--password=*****

--default-character-set=latin1

User and password definition come from the client group in the Login path config we set before, and charset from the mysql option group in regular .my.cnf. Note that we added that group and charset config manually for better clarity, by default that option is not set.

You can see that while both ways to read options talk of defaults, they actually output the options that we have explicitly set, making them non-default. This is an interesting tidbit, but it doesn’t change anything in the grand scheme of things.

Unfortunately, both of these ways to review options are not perfect in determining a complete set of options in effect. The problem is they only read configuration files, as defined in Tables 9-1 and 9-2. But it’s possible for MySQL programs to read other config files, or to be started with command-line arguments. Additionally, the variables persisted in DATADIR/mysqld-auto.cnf through SET PERSIST are also not provided by defaults-printing routines.

We mentioned that MySQL programs do not read options from any other files than the ones that were listed in the Tables 9-1 and 9-2. However, there was the extra file there, which can be an arbitrary location. Unless you specify the same extra file when invoking my_print_defaults or another program with --print-defaults, options from that extra file won’t be read. The extra file is specified with a command-line argument --defaults-extra-file, and can be specified for most if not all MySQL programs. The two defaults printing routines only read pre-defined config files, and will miss that file. You can, however, specify --defaults-extra-file both for my_print_defaults and for the program invoked with --print-defaults, and both will read the extra file then. Same applies to the --defaults-file, which we also described and which basically forces MySQL program to only read the single file passed as a value for this option.

Both --defaults-extra-file and --defaults-file share a thing in common: they are command-line arguments. Command-line arguments passed to MySQL program override any options read from configuration files, but at the same time you can miss them when you do --print-defaults or my_print_defaults, as they are coming from outside of any config files. To put it more concisely: a particular MySQL program, for example, mysqld, can be started by someone with unknown and arbitrary command-line arguments. Thus, when we’re talking about options in effect we must also consider the presence of such arguments.

On Linux and UNIX-like systems, you can use a ps utility (or an equivalent) to view information the processes currently running in the system, including their full command line. Let’s see an example on Linux where mysqld was started with --no-defaults but rather with all config options passed as arguments:

$ ps auxf | grep mysqld | grep -v grep

root      397830  ... \_ sudo -u mysql bash -c mysqld ...

mysql     397832  ...   \_ mysqld --datadir=/var/lib/mysql ...

Or, if we print just the command line for the mysqld process, and make it cleaner with sed:

$ ps -p 397832 -ocommand ww | sed 's/--/\n--/g'

COMMAND

mysqld

--datadir=/var/lib/mysql

--socket=/var/lib/mysql/mysql.sock

--log-error=/var/log/mysqld.log

--pid-file=/run/mysqld/mysqld.pid

...

--character_set_server=latin1

--collation_server=latin1_swedish_ci

Note that for this example we started mysqld without using any provided scripts. You won’t often see this way of starting MySQL server, but it’s possible.

You can pass any configuration option as an argument, so the output can be quite large. However, when you’re not sure how exactly mysqld or another program was executed, that’s an important thing to check. On Windows, you can view the command-line arguments of a running program either by executing the Task Manager, and adding a “Command Line” column to the Processes tab (through the View menu), or by using the Process Explorer tool from the sysinternals package.

If your MySQL program is started from within a script, you should inspect that script to find all the arguments used. While this is probably going to be a rare occasion for mysqld, it’s a common practice to run mysql, mysqldump, and xtrabackup from custom scripts.

Understanding the current used options can be a daunting task, but it’s going to be extremely important at times. Hopefully, these guidelines and hints will help you.

Chapter 11. Configuring and Tuning the Server

The MySQL installation process (see Chapter 1) provides everything necessary to install the MySQL process and start using it. However, it is required for production systems to fine-tuning, adjusting MySQL parameters and the operating system to optimize the MySQL Server performance. This chapter will cover the recommended best practices for every installation and parameters that need to be adjusted based on the expected or current workload. As you’ll see, it is not necessary to memorize all the MySQL parameters. Based on the Pareto mathematical principle, which states that, for many events, roughly 80% of the effects come from 20% of the causes, we will work the main MySQL and operating system parameters that are responsible for most of the performance issues. As we understand, there are some advanced topics in this section related to computer architecture (such as NUMA); the intent here is to introduce you to a few components that can change MySQL performance that you will face sooner or later in your career.

The MySQL Server Daemon

Since 2015, the majority of Linux distributions have adopted systemd. Because of that, Linux operating systems do not use the mysqld_safe process to start MySQL anymore. The mysqld_safe is called an angel process. This name is because the mysqld_safe adds some safety features such as restarting the server when an error occurs and logging runtime information to the MySQL error log. For operating systems that use systemd(the systemctl command), these functionalities were incorporated by systemd and the mysqld process.

The mysqld is the core process of the MySQL Server. It is a single multithreaded program that does most of the work in a MySQL server. It does not spawn additional processes. Note that we are talking about a single process with multiple threads, making MySQL a multithreaded process.

A brief explanation about the terms mentioned previously. A program is the code that is designed to accomplish a specific objective. There are many types of programs, including ones to assist parts of the operating system and others that are designed for user experience, such as web browsing.

A process is what we call a program that has been loaded into memory along with all the resources it needs to operate. The operating system allocates memory and other resources for it.

A thread is the unit of execution within a process. A process can have from just one thread to many threads. In single-threaded processes, the process contains one thread. The process and the threads are the same, and there is only one thing happening.

Because modern CPUs have multiple cores, they can execute multiple threads at the same time, so multithreaded processes are widespread nowadays. The multithread concept will be fundamental to understand some proposed settings in the following sections.

To conclude, MySQL is a single process software that spawns multiple threads for various purposes, such as serving user activities and executing background tasks.

MySQL Server Variables

MySQL server has many variables that allow tuning its operation. For example, MySQL Server 8.0.19 version has impressive 555 server variables!

Each system variable has a default value. Also, we can adjust most system variables dynamically (or on-the-fly); however, a few of them are static, which means that we need to change the my.cnf file and restart the MySQL process so they can take effect.

The system variables can have two different scopes, which are SESSION and GLOBAL. A system variable can have a global value that affects server operation as a whole, like the innodb_log_file_size or a session value that affects only a specific session like the sql_mode.

Checking Server Settings

Databases are not static entities; on the opposite, their workload is dynamic and changes along its way, and tends to grow over time. This organic behavior requires constant monitoring, analysis, and adjustment. The command to show the MySQL settings is:

mysql> SHOW [GLOBAL|SESSION] VARIABLES;

When using the GLOBAL modifier, the statement displays global system variable values. When using SESSION, the statement indicates the system variable values that affect the current connection. Observe that different connections can have different values.

Finally, if no modifier is present, the default is SESSION.

Best Practices

There are many aspects to optimize on a database. If the database runs on bare metal(physical host), we can control hardware and operating system resources. When we move to virtualized machines, we have reduced control over these resources because we can’t control what happens with the underlying host. The last option is the managed databases in the cloud, like RDS(Relational Database Service) from AWS, where only a few database settings are available. It is a tradeoff between fine-grained tuning to extract the most performance and the comfort of having most of the tasks automated (at the cost of a few extra dollars).

Let’s start reviewing some settings at the operating system level, and after that, we will check the MySQL parameters.

Operating System Best Practices

The swappiness setting and the swap usage

+ The swappiness parameter controls the behavior of the Linux operating system in the swap area. Swap is the process of sending memory blocks in/out from the disk. And it is well known that disks are at least a magnitude higher slow than memory access. In the Numbers Everyone Should Know section, you will see the difference between memory and disk access time.

The recommended value is 1, which means: do not swap until it is absolutely necessary for the OS to be functional. To adjust this parameter:

# Non persistent - the value will change to the previous value if you reboot the OS

# echo 1 > /proc/sys/vm/swappiness

And to make this change persistent when we reboot the operating system:

# Change in sysctl.conf will make the change persistent across reboots

# sudo sysctl -w vm.swappiness=1

We can check if the operating system is consuming swap using the following command:

# free -m

Or if we want detailed information, it is possible to run the following snippet in the shell:

#!/bin/bash

SUM=0

OVERALL=0

for DIR in `find /proc/ -maxdepth 1 -type d | egrep "^/proc/[0-9]"` ; do

PID=`echo $DIR | cut -d / -f 3`

PROGNAME=`ps -p $PID -o comm --no-headers`

for SWAP in `grep Swap $DIR/smaps 2>/dev/null| awk '{ print $2 }'`

do

let SUM=$SUM+$SWAP

done

echo "PID=$PID - Swap used: $SUM - ($PROGNAME )"

let OVERALL=$OVERALL+$SUM

SUM=0

done

echo "Overall swap used: $OVERALL"

NOTE

The difference between setting vm.swappiness to 1 and 0 is negligible. We choose the value of 1 because in some kernels, there was a bug that could lead MySQL to be killed by the OOM.

IO Scheduler

The IO scheduler is an algorithm the kernel will use to commit reads and writes to disk. By default, most Linux installs use the CFQ (Completely-Fair Queue) scheduler. The CFQ works well for many general use cases but with little latency guarantees. Two other schedulers are deadline and noop. The deadline excels at latency-sensitive use cases (like databases), and noop is closer to no schedule at all. For bare metals, any algorithm between deadline or noop (the performance difference between them is imperceptible) will be better than CFQ.

If you are running MySQL in a VM (which has it’s own IO scheduler beneath it), it is best to use “noop” and let the virtualization layer take care of the IO scheduling itself.

First, to verify which is the current algorithm in use by Linux:

# Verifying

#  cat /sys/block/xvda/queue/scheduler

noop [deadline] cfq

To change it run these as root (accordingly to the disk):

# Adjusting the value dynamically

# echo "noop" > /sys/block/xvda/queue/scheduler

To make this change persistent, you must edit the GRUB configuration file (usually /etc/sysconfig/grub ) and add an elevator option to GRUB_CMDLINE_LINUX_DEFAULT. For example, you would change this line:

GRUB_CMDLINE_LINUX="console=tty0 crashkernel=auto console=ttyS0,115200

With this line:

GRUB_CMDLINE_LINUX="console=tty0 crashkernel=auto console=ttyS0,115200 elevator=noop"

It is essential to take extra care when editing grub. Inserting an inadequate or misplaced configuration can make the server unusable and required installing the operating system again.

NOTE

Note for AWS setups: There are cases where the I/O scheduler has a value of none, most notably in AWS VM instance types where EBS volumes are exposed as NVMe block devices. This is because the setting has no use in modern PCIe/NVMe devices. The reason is that they have a substantial internal queue, and they bypass the IO scheduler altogether. The setting, in this case, is none, and it is optimal in such disks.

Filesystems and mount options

Choosing the file system appropriate for your database is an important decision due to the many options available and the trade-offs involved. It is worth mentioning two important ones most used: XFS and EXT4.

- XFS is a high-performance journaling file system designed for high scalability. It provides near-native I/O performance even when the file system spans multiple storage devices. XFS has features that make it suitable for very large file systems, supporting files up to 8EiB in size. Fast recovery, fast transactions, delayed allocation for reduced fragmentation, and near raw I/O performance with DIRECT I/O.

The default options for mkfs.xfs are good for optimal speed, so the simple command will provide good performance while ensuring data safety:

# Use default mkfs options

mkfs.xfs /dev/target_volume

Regarding mount options, the defaults should fit most cases. You can see a performance increase on some filesystems by adding the noatime mount option to the /etc/fstab. For XFS filesystems, the default atime behavior is relatime, which has almost no overhead compared to noatime and still maintains sane atime` values. Create an XFS file system on a LUN that has a battery-backed, non-volatile cache. You can further increase the filesystem’s performance by disabling the write barrier with the mount option nobarrier. These settings help you to avoid flushing data more often than necessary. If a BBU (backup battery unit) is not present, however, or you are unsure about it, leave barriers on; otherwise, you may jeopardize data consistency. With this option on, an /etc/fstab file should look like the one below:

/dev/sda2              /datastore              xfs     noatime,nobarrier

/dev/sdb2              /binlog                 xfs     noatime,nobarrier

- EXT4 has been developed as the successor to ext3 with added performance improvements. It is a solid option that will fit most workloads. We should note here that it supports files up to 16TB in size, a smaller limit than XFS. This is something you should consider if excessive tablespace size/growth is a requirement. Regarding mount options, the same considerations apply. We recommend the defaults for a robust filesystem without risks to data consistency. However, if an enterprise storage controller with a BBU cache is present, the following mount options will provide the best performance:

/dev/sda2              /datastore              ext4

noatime,data=writeback,barrier=0,nobh,errors=remount-ro

/dev/sdb2              /binlog                 ext4

noatime,data=writeback,barrier=0,nobh,errors=remount-ro

Transparent Huge Pages

The operating system manages memory in blocks known as pages. A page has a size of 4096 bytes (or 4Kb). 1MB of memory is equal to 256 pages; 1GB of memory is equivalent to 256,000 pages, etc. CPUs have a built-in memory management unit that contains a list of these pages, with each page referenced through a page table entry. It is common to see servers nowadays with hundreds or terabytes of memory. There are two ways to enable the system to manage large amounts of memory:

- Increase the number of page table entries in the hardware memory management unit

- Increase the page size

The first method is expensive since the hardware memory management unit in a modern processor only supports hundreds or thousands of page table entries. Besides, hardware and memory management algorithms that work well with thousands of pages (megabytes of memory) may have problems performing well with millions (or even billions) of pages. To address the scalability issue, operating systems started using huge pages. Simply put, huge pages are blocks of memory that can come in 2MB, 4MB, 1GB size, etc. Using huge page memory increases the CPU cache hits against the Transaction Lookaside Buffer (TLB).

You can run cpuid to verify the processor cache and TLB:

# cpuid | grep "cache and TLB information" -A 5

cache and TLB information (2):

0x5a: data TLB: 2M/4M pages, 4-way, 32 entries

0x03: data TLB: 4K pages, 4-way, 64 entries

0x76: instruction TLB: 2M/4M pages, fully, 8 entries

0xff: cache data is in CPUID 4

0xb2: instruction TLB: 4K, 4-way, 64 entries

Transparent Hugepages (THP for short), as the name suggests, are intended to bring huge page support automatically to applications without requiring custom configuration.

Now, for MySQL in particular, using THP is not recommended for a couple of reasons. First, MySQL databases use small memory pages (16KB), and using THP can cause excessive I/O because MySQL believes it is accessing 16Kb while THP is scanning a page larger than that. Also, the Transparent Huge Pages tend to become fragmented and impact performance. Lastly, some cases were reported over the years, whereas THP can result in memory leaking (eventually crashing MySQL).

- Disable THP for RHEL/CentOS 6 and RHEL/Centos 7 To disable it on the runtime for RHEL/CentOS 6 and RHEL/Centos 7:

# echo "never" > /sys/kernel/mm/transparent_hugepage/enabled

# echo "never" > /sys/kernel/mm/transparent_hugepage/defrag

To make this change survive a server restart, you’ll have to add the flag transparent_hugepage=never to your kernel options (/etc/sysconfig/grub):

GRUB_CMDLINE_LINUX="console=tty0 crashkernel=auto console=ttyS0,115200

elevator=noop transparent_hugepage=never"

Rebuild the /boot/grub2/grub.cfg. Before rebuilding the GRUB2 configuration file, ensure the backup of the existing /boot/grub2/grub.cfg. To rebuild it on BIOS-Based Machines.

# grub2-mkconfig -o /boot/grub2/grub.cfg

If THP is still not disabled, it may be necessary to disable the tuned services.

# systemctl stop tuned

# systemctl disable tuned

- Disable THP for Ubuntu 20.04 (Focal Fossa) To disable THP for Ubuntu 20.04, we recommend you use the sysfsutils package. To install it, execute the following command:

# apt install sysfsutils

And append the following lines to the /etc/sysfs.conf file:

kernel/mm/transparent_hugepage/enabled = never

kernel/mm/transparent_hugepage/defrag = never

Reboot the server and check if the settings are in-place:

# cat  /sys/kernel/mm/transparent_hugepage/enabled

always madvise [never]

# cat /sys/kernel/mm/transparent_hugepage/defrag

always defer defer+madvise madvise [never]

Use of jemalloc memory allocator

MySQL Server uses dynamic memory allocation, so a good memory allocator is important for proper CPU and RAM resource utilization. An efficient memory allocator should improve scalability, increase throughput and keep the memory footprint under control.

It is important to mention a characteristic of InnoDB. InnoDB creates a read view for every transaction and allocates memory for this structure from the heap area. The problem is that MySQL deallocates the heap on each commit, and thus the read view memory is reallocated on the next transaction leading to memory fragmentation.

The jemalloc is a memory allocator that emphasizes fragmentation avoidance and scalable concurrency support.

Using jemalloc (along with Transparent Huge Pages disabled), you have less memory fragmentation and more efficient resource management of the available server memory. We can install the jemalloc package from the jemalloc repository or using the Percona repository. We prefer to use the Percona repository because we consider it simpler to install and manage. We describe the steps to install the yum repository in “Installing Percona Server 8” and for the apt in “Installing Percona Server 8”.

Once you have the repo, you run the install command (according to your operating system).

TIP

In CentOS, if the server has the Extra Packages for Enterprise Linux (EPEL) installed, it is possible to install jemalloc from this repository with yum. To install the EPEL package:

# yum install epel-release -y

If you are using Ubuntu 20.04. then you need to execute the following steps to enable jemalloc:

1. Install jemalloc

# apt-get install libjemalloc2

# dpkg -L libjemalloc2

2. The dpkg command will show the location of the jemalloc library:

#  dpkg -L libjemalloc2

/.

/usr

/usr/lib

/usr/lib/x86_64-linux-gnu

/usr/lib/x86_64-linux-gnu/libjemalloc.so.2

/usr/share

/usr/share/doc

/usr/share/doc/libjemalloc2

/usr/share/doc/libjemalloc2/README

/usr/share/doc/libjemalloc2/changelog.Debian.gz

/usr/share/doc/libjemalloc2/copyright

3. Override the default configuration of the service with the command:

# systemctl edit mysql

Which will create the /etc/systemd/system/mysql.service.d/override.conf file.

4. Add the following configuration to the file:

[Service]

Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2"

Last, restart the MySQL service to enable the jemalloc library:

# systemctl restart mysql

5. Validate if MySQL is using jemalloc

To verify if it worked, with MySQL running (mysqld process), you can execute the following command:

# lsof -Pn -p $(pidof mysqld) |  grep "jemalloc"

And you should see similar output:

mysqld  3844 mysql  mem       REG              253,0   744776  36550 /usr/lib/x86_64-linux-gnu/libjemalloc.so.2

If you are using Centos/RHEL, you need to execute the following steps:

1. Install jemalloc package

# yum install jemalloc

# rpm -ql jemalloc

The rpm -ql command will show the library location:

/usr/bin/jemalloc.sh

/usr/lib64/libjemalloc.so.1

/usr/share/doc/jemalloc-3.6.0

/usr/share/doc/jemalloc-3.6.0/COPYING

/usr/share/doc/jemalloc-3.6.0/README

/usr/share/doc/jemalloc-3.6.0/VERSION

/usr/share/doc/jemalloc-3.6.0/jemalloc.html

2. Override the default configurations of the service with the command:

# systemctl edit mysqld

Which will create the /etc/systemd/system/mysqld.service.d/override.conf_ file.

3. Add the following configuration to the file:

[Service]

Environment="LD_PRELOAD=/usr/lib64/libjemalloc.so.1"

Last, restart the MySQL service to enable the jemalloc library:

# systemctl restart mysqld

4. Validate if MySQL is using jemalloc

To verify if it worked, with MySQL running (mysqld process), you can execute the following command:

# lsof -Pn -p $(pidof mysqld) |  grep "jemalloc"

And you should see similar output:

# mysqld  4784 mysql  mem       REG              253,0    212096  33985101 /usr/lib64/libjemalloc.so.1

CPU Governor

One of the most effective ways to reduce power consumption and heat output on your system is to use CPUfreq. CPUfreq — also referred to as CPU speed scaling — allows the processor’s clock speed to be adjusted on the fly. This feature enables the system to run at a reduced clock speed to save power. The rules for shifting frequencies, whether to a faster or slower clock speed and when to shift frequencies, are defined by the CPUfreq governor . The governor defines the power characteristics of the system CPU, which in turn affects CPU performance. Each governor has its unique behavior, purpose, and suitability in terms of workload.

However, for MySQL databases, we recommend using maximum performance setting to achieve the best throughput.

For CentosOS, you can view which CPU governor is being currently used by executing:

# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

You can enable performance mode by running:

# cpupower frequency-set --governor performance

For Ubuntu, we recommend installing the linux-tools-common to have the cpupower utility:

# apt install linux-tools-common

Once you installed it, you can change the governor to performance mode with the following command:

cpupower frequency-set --governor performance

MySQL Best Practices

Adjusting the Buffer Pool size

The innodb_buffer_pool_size controls the size in bytes of the InnoDB buffer pool, the memory area where InnoDB caches table and index data. There’s no question that for tuning InnoDB, the innodb_buffer_pool_size is the most important variable. The starting point to set the innodb_buffer_pool_size is around 70% of the RAM for a MySQL dedicated server.

However, as larger the server, the more likely the rule will end up wasting RAM. For a server with 512 GB of RAM, this means that we will leave 153 GB of RAM for the operating system, which, in general, is a waste.

So what’s a better rule of thumb? The rule is that you tune the innodb_buffer_pool_size as large as possible without using swap when the system is running the production workload.

For MySQL 5.7 onwards, the buffer pool resize feature is online, and we can change according to its needs without the need to restart the database.

To adjust it online:

# Setting to 1 GB

mysql> set global innodb_buffer_pool_size = 1024*1024*1024;

Query OK, 0 rows affected (0.00 sec)

And on the my.cnf below the [mysqld] section:

[mysqld]

innodb_buffer_pool_size = 1G

The innodb_buffer_pool_instances parameter

One of the obscure MySQL Parameters is the innodb_buffer_pool_instances. The innodb_buffer_pool_instances parameter defines the number of regions that the InnoDB will split the buffer pool. For systems with buffer pools in the multi-gigabyte range, dividing the buffer pool into separate instances can improve concurrency by reducing contention as different threads read and write to cached pages.

However, in our experience, setting high values for this parameter may also introduce additional overhead. The reason is that each buffer pool instance manages its free lists, flush lists, LRUs, and all other data structures connected to a buffer pool and is protected by its buffer pool mutex.

Unless you run benchmarks that prove performance gains, we suggest using the default value (8).

NOTE

MariaDB disabled the innodb_buffer_pool_instances in MariaDB 10.5 and removed it in MariaDB 10.6. According to Marko Makela (MariaDB architect), the original reasons for splitting the buffer pool have mostly gone away nowadays. You can find more details in the MariaDB JIRA ticket.

Adjusting the REDO log size

The redo log is a structure used during crash recovery to correct data written by incomplete transactions. The main goal is to guarantee the durability (D) property of ACID transactions by providing redo recovery for committed transactions. Because the REDO file logs all data written to MySQL even before the commit, having the right REDO log size is fundamental for MySQL to run smoothly without struggling. An undersized REDO can even lead to errors in operations!

An example of an error:

[ERROR] InnoDB: The total blob data length (12299456) is greater than 10% of the total redo log size (100663296). Please increase total redo log size.

MySQL, in this case, was using the default value for the innodb_log_file_size, which is 48 Mb. To estimate the REDO log size, there is a formula that we can use in the majority of the cases. Let’s take a look at the following commands:

mysql> pager grep sequence

PAGER set to 'grep sequence'

mysql> show engine innodb status\G select sleep(60); show engine innodb status\G

Log sequence number 3836410803

1 row in set (0.06 sec)

1 row in set (1 min 0.00 sec)

Log sequence number 3838334638

1 row in set (0.05 sec)

The log sequence number is the total number of bytes written to the redo log. Using the sleep command, we can calculate the delta for that period. Using the following formula, we can reach an estimated value:

mysql> select (((3838334638 - 3836410803)/1024/1024)*60)/2 as Estimated_innodb_log_file_size;

+--------------------------------+

| Estimated_innodb_log_file_size |

+--------------------------------+

|                55.041360855088 |

+--------------------------------+

1 row in set (0.00 sec)

We usually round up to the higher value so that the final number will be 56 Mb. As a rough rule of thumb, you can make the log big enough that it can hold at most an hour or so of logs. So this is the final value that needs to be added to my.cnf under the [mysqld] section:

[mysqld]

innodb_log_file_size=56M

The sync_binlog parameter

The binary log is a set of log files that contain information about data modifications made to a MySQL server instance. They are different from the REDO files and have another purpose. For example, they are used to create the replication, to create the InnoDB Cluster, and it is also helpful to perform a point-in-time-recovery(PITR).

By default, the MySQL Server synchronizes its binary log to disk (using fdatasync()) before transactions are committed. Using sync_binlog=1 is the safest choice but brings a great drop in performance. In the event of a power failure or operating system crash, transactions missing from the binary log are only in a prepared state. As it is a dynamic option, you can change it while the server is running with the following command:

mysql> SET global sync_binlog = 0;

Add the parameter in the my.cnf file below the [mysqld] section:

[mysqld]

sync_binlog=0

The binlog_expire_logs_seconds and the expire_logs_days parameters

To avoid MySQL fill the entire disk with binary logs, it is possible to use the parameters ` binlog_expire_logs_seconds` and the expire_logs_days. The expire_logs_days specifies the number of days before automatic removal of binary log files. But, be aware that expire_logs_days is deprecated, and you should expect it to be removed in a future release.

Because of the expire_logs_days deprecation that we mentioned the binlog_expire_logs_seconds. The default value for binlog_expire_logs_seconds is 2592000 (30 days). The binlog_expire_logs_seconds sets the binary log expiration period in seconds. After their expiration period ends, MySQL can automatically remove the binary log files. Possible removals happen at startup and when the binary log is flushed.

NOTE

If you want to flush the binary log manually, you can execute the following command:

mysql> FLUSH BINARY LOGS;

The innodb_flush_log_at_trx_commit parameter

The innodb_flush_log_at_trx_commit controls the balance between strict ACID compliance for commit operations and higher performance possible when commit-related I/O operations are rearranged and done in batches. It is a delicate option and many prefer to use innodb_flush_log_at_trx_commit=1 in the source servers, while for replicas they use innodb_flush_log_at_trx_commit equal 0 or 2. The value of 2 instructs InnoDB to write into log files after transaction commit, but don’t fsync. Instead, it fsyncs log files every second. This way, you can lose 1 second of updates in case the OS crashes. With modern hardware that supports up to one million inserts per second, we cannot disregard this. The value of 0 is even worse: you may lose up to 1 second of transactions even if the mysqld process crashes.

WARNING

Many operating systems and some disk hardware fool the flush-to-disk operation. They may tell mysqld that the flush has taken place, even though it has not. In this case, the durability of transactions is not guaranteed even with the recommended settings, and in the worst case, a power outage can corrupt InnoDB data. Using a battery-backed disk cache in the SCSI disk controller or in the disk itself speeds up file flushes and makes the operation safer. You can also disable the caching of disk writes in hardware caches if the battery is not working correctly.

The innodb_thread_concurrency parameter

MySQL has the innodb_thread_concurrency set to 0 by default. The default value means that an infinite number (hardware limit) of threads can be opened and executed inside MySQL. The usual recommendation is to leave innodb_thread_concurrency with its default value and only change it to solve contention problems.

If your workload is consistently heavy or have occasional spikes, you can set the innodb_thread_concurrency using the following formula:

innodb_thread_concurrency = Number of Cores * 2

Because MySQL does not use multiple cores to execute a single query(it is a 1:1 relation), each core will run one query at a single unit of time. Based on our experience, because modern CPUs are fast in general, setting the maximum number of executing threads to double the CPU available is a good start.

Once the number of executing threads reaches this limit, additional threads sleep for a number of microseconds, set by the configuration parameter innodb_thread_sleep_delay, before being placed into the queue.

The innodb_thread_concurrency is a dynamic variable, and we can change it at runtime, but it needs to be changed in the my.cnf to make it persistent.

mysql> SET global innodb_thread_concurrency = 0;

Add in the my.cnf file below the [mysqld] section:

[mysqld]

innodb-thread-concurrency=0

And to validate if MySQL applied the setting:

mysql> show global variables like '%innodb_thread_concurrency%';

NOTE

MySQL 8.0.14 contains this in the release notes: “As of MySQL 8.0.14, InnoDB supports parallel clustered index reads, which can improve CHECK TABLE performance.” Parallel clustered index reads also work for a simple count(*) (without a WHERE condition). You can control the number of parallel threads with the innodb_parallel_read_threads parameter.

This feature is currently limited and available only for queries without the WHERE condition(full scans). However, it is a great start for MySQL and opens a road to real parallel query executions.

NUMA Architecture

NUMA, or Non-Uniform Memory Access, is a shared memory architecture that describes the placement of main memory modules to processors in a multiprocessor system. Like most other processor architectural features, ignoring NUMA can result in the sub-par database memory performance.

In the NUMA shared memory architecture, each processor has its local memory module that can directly access a distinctive performance advantage. At the same time, it can also access any memory module belonging to another processor using a shared bus (or some other type of interconnect) as seen in Figure 11-1:

Figure 11-1. NUMA Architecture overview

Here is an example of a system that has NUMA enabled:

# NUMA system

shell> numactl --hardware

available: 2 nodes (0-1)

node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 24 25 26 27 28 29 30 31 32 33 34 35

node 0 size: 130669 MB

node 0 free: 828 MB

node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23 36 37 38 39 40 41 42 43 44 45 46 47

node 1 size: 131072 MB

node 1 free: 60 MB

node distances:

node   0   1

0:  10  21

1:  21  10

As we can see the node 0 has more free memory than node 1. There is an issue with this, which causes the OS to swap even with memory available. The swap issue is explained in the excellent article by Jeremy Cole in the Swap Insanity and NUMA Architecture.

Since MySQL 5.7, the innodb_buffer_pool_populate and numa_interleave parameters were removed and its functions now are presented on the innodb_numa_interleave parameter. When we enable it, we balance memory allocation across nodes in a NUMA system avoiding the swap insanity problem.

This parameter is not dynamic, so we need to restart MySQL. Add the parameter in the my.cnf file below the [mysqld] section:

[mysqld]

innodb_numa_interleave = 1

Chapter 12. Monitoring MySQL Servers

Monitoring can be defined as observing or checking the quality or progress of something over a period of time. Applying that definition to MySQL, what we observe and check the server’s “health” and performance. Quality, then, would be maintaining uptime and having performance meet desired levels. So really, monitoring is a continuous effort to keep things under observation and control. Usually, it’s thought of as something optional, something that may not be needed unless there’s some high load or high stakes. However, just like backups, monitoring benefits almost every installation of any database.

We think that having monitoring in place and understanding the metrics you’re getting from it is one of the most important tasks for anyone operating database systems — probably just after setting up proper verified backups. As with operating a database without backups, having unmonitored databases is dangerous: what use is a system that provides unpredictable performance and may be “down” randomly? The data may be safe, but it might not be usable.

In this chapter, we’ll try to give you a foundation for understanding how to monitor MySQL efficiently. This book is not called High Performance MySQL, and we’ll not be going in-depth on specifics of exactly which metrics mean what, or how to make complex analysis of a system. But we will talk about few basic metrics which should be checked on every MySQL installation, and we’ll discuss important OS level metrics and tools. We’ll then briefly touch on a few widely used methodologies for assessing systems performance. After that, we will review a few popular open source monitoring solutions, and finally we’ll show how to gather data for investigation and monitoring purposes manually.

After completing this chapter, you should feel comfortable picking a monitoring tool and understanding some of the most important metrics it shows.

Operating System Metrics

An operating system is a complex computer program: an interface level between applications, mainly MySQL in our case, and hardware. In the early days, OSes were simple, now they are arguably quite complex, but the idea behind them never really changed. An OS tries to hide, or abstract away, the complexity of dealing with the underlying hardware. It’s possible to imagine some special-purpose RDBMS running directly on hardware, being its own operating system, but realistically you’ll likely never see that. Apart from providing a convenient and powerful interface, operating systems also expose a lot of performance metrics. You don’t need to know each and every one of them, but it’s important to have basic understanding of how to assess performance of the layer underlying your database.

Usually, when talking about operting systems performance and metrics, what’s really being discussed is hardware performance assessed at the operating system level. There’s nothing wrong in saying “OS metrics”, but remember that at the end of the day they are mostly showing hardware performance.

Let’s take a look at the most important OS metrics that you’ll want to monitor and in general have a feel for. We will be covering two major OSes in this section: Linux and Windows. UNIX-like systems, like macOS and others, will either have the same tools as Linux, or at least tools showing the same or similar outputs.

CPU

The central processing unit (CPU) is the heart of any computer. Nowadays, CPUs are so complex they can be considered separate computers within computers. Luckily, the basic metrics we think you should undersand are universal. In this section, we’ll take a look at CPU utilization as reported by Linux and Windows, and see what contributes to the overall load.

Before we get into measuring CPU utilization, let’s do a quick recap of what a CPU is and what characteristics of it are most important for database operators. We called it the “heart of a computer,” but that’s oversimplified. In fact, a CPU is a device that can do a few basic (and not so basic) operations, on top of which we build layers and layers of complexity from machine code up to the high-level programming languages, running operating systems and ultimately (for us) database systems.

Every operation a computer does is done by a CPU. As Kevin Closson has said, everything is a CPU problem. When a program is actively being executed—for example, MySQL parsing a query—the CPU is doing all the work. When a program is waiting for a resource—MySQL waiting for a data read from disk—the CPU is involved in “telling” the program when the data is available. Such a list could go on forever.

Here are a few of the most important metrics of a CPU for a server (or any computer in general):

CPU frequency

The number of times per second a CPU core can “wake up” to execute a piece of work. This is basically the “speed” of the CPU. The more the merrier, but surprisingly often frequency is not the most important metric.

Cache memory

Size of cache memory defines the amount of memory located directly within the CPU, making it extremely fast. Again, the more, the better, and there are no downsides to having more.

Number of cores

This is the number of execution units within a single CPU “package” (a physical item), and the sum of those units across all CPUs we can fit into a server. Nowadays, it’s increasingly difficult to find a CPU that has a single core: most CPUs are multi-core systems. Some even have “virtual” cores, making the difference between the “actual” number of CPUs and total number of cores even higher.

Usually, having more cores is a good thing, but there are caveats to that. In general, the more cores available, the more processes can be scheduled by OS to be executed simultaneously. For MySQL, that means more queries executed in parallel, and less impact from background operations.

But if half of the available cores are “virtual,” you don’t get the 2x performance increase you might expect. Rather, you can get 2x increase, or you can get anywhere between 1x and 2x increase: not every workload (even within MySQL) benefits from virtual cores.

Also, having multiple CPUs in different sockets makes interfacing with memory (RAM) and other onboard devices (like network cards) more complicated. Usually, regular servers are physically laid out in such a way that some CPUs (and their cores) will access parts of RAM quicker than other parts. That is called NUMA, for non-uniform memory architecture. For MySQL, this means that memory allocation and memory-related issues can become a pain point. We cover necessary configuration on NUMA systems over at “NUMA Architecture”.

The basic measurement of CPU is its load in percents. When someone tells you “CPU 20,” you can be pretty sure that they mean “the CPU is 20% busy at the moment.” You can never be totally sure, though, so you’d better double-check. For example, 20% of one core on a multi-core system may be just 1% of the overall load. Let’s try to see this load.

On Linux, the basic command to get the CPU load is vmstat. It can be run without arguments and it will output current average values, then exit. If we run it with a digit argument (we’ll call it X here) it’ll print values every X seconds. We recommend that you run vmstat with a digit argument — for example, vmstat 1 for a few seconds. If you run just vmstat, you get averages since boot, which are usually misleading. vmstat 1 will execute forever until interrupted (pressing Ctrl+C is the easiest way out).

The vmstat program prints info not only on CPU load, but also memory and disk-related metrics, as well as advanced system metrics. We will be exploring some of the sections of vmstat output soon, but here we’ll concentrate on CPU and process metrics.

To start, let’s see the vmstat output on an idle system:

$ vmstat 1

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----

r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

2  0      0 1229924 1856608 6968268    0    0    39   125   61  144 18  7 75  0  0

1  0      0 1228028 1856608 6969384    0    0     0    84 2489 3047  2  1 97  0  0

0  0      0 1220972 1856620 6977688    0    0     0    84 2828 4712  3  1 96  0  0

0  0      0 1217420 1856644 6976796    0    0     0   164 2405 3164  2  2 96  0  0

0  0      0 1223768 1856648 6968352    0    0     0    84 2109 2762  2  1 97  0  0

^C

The first line of the output after the header is an average since boot, and latter lines represent current values when printed. The output can be hard to read at first, but you get used to it rather quickly. For clarity, we will provide a truncated output with only the information we want. We will truncate vmstat output across this whole section.

procs ------cpu-----

r  b us sy id wa st

2  0 18  7 75  0  0

1  0  2  1 97  0  0

0  0  3  1 96  0  0

0  0  2  2 96  0  0

0  0  2  1 97  0  0

Brief explanation of the output is as follows. r and b are process metrics — the number of processes actively running and the number of processes blocked (usually waiting for IO). Other columns represent a breakdown of CPU utilization in percentage points (from 0% to 100% even on a multi-core system). All of these cpu columns’ values added will always be 100.

us (User)

Time spent running user programs (otherwise, load put on a system by these programs). MySQL server is a user program, as is every piece of code which exists outside of the kernel. More importantly, this metric only shows time spent purely inside the program itself. For example, when MySQL is doing some computation, or parsing a complex query, this value will go up. When MySQL wants to perform a disk or network operation, this value will also go up, but so will two other values, as you’ll soon see.

sy (System)

Time spent running kernel code. Due to the way Linux and other UNIX-like systems are organized, user programs increase this counter. For example, whenever MySQL needs to do a disk read, some work will have to be done by the OS kernel. Time spent doing that work will be included in the sy value.

id (Idle)

Time spent doing nothing; idle time. On a perfectly idle server, this metric will be 100.

wa (IO wait)

Time spent waiting for IO. This is an important metric for MySQL, as reading and writing to various files are a relatively huge part of MySQL operation. When MySQL does a disk read, some time will be spent in MySQL’s internal functions, and reflected in us. Then some time will be spent inside the kernel, and reflected in sy. Finally, once the kernel has sent a read request to the underlying storage device (which could be a local or network device), and waits for the response and data, all the time spent is accumulated in wa. If our program and kernel are perfectly quick and storage is perfectly slow, and all we do is IO, this metric can read close to 100. In reality, double-digit values are rare, and usually indicate some IO issues. We’ll talk about IO in-depth in “Disk”.

st (Steal)

This is a difficult metric to explain without getting deep into the weeds. It’s defined by manual as “time stolen from a virtual machine.” You can think of this as the time during which VM wanted to execute its instructions but had to wait for the host server to allocate CPU time. There are multiple reasons for this behavior, a couple of those reasons are notable. The first is host overprovisioning: running too many too large VMs, resulting in a situation where the sum of resources VMs require is more than host capacity. The second one is the “noisy neighbor” situation, where one or more VMs suffer from a particularly loaded VM.

Other commands like top (which we’ll show in a bit) will have finer CPU load breakdown. However, the ones just listed are a good starting point and cover most of what you need to know about a running system.

Now let’s get back to our vmstat 1 output on an idle system:

procs ------cpu-----

r  b us sy id wa st

2  0 18  7 75  0  0

1  0  2  1 97  0  0

0  0  3  1 96  0  0

0  0  2  2 96  0  0

0  0  2  1 97  0  0

What can we tell from this output? The first line is an average since boot. On average, there are two processes running (r) on this system, 0 blocked (b), user CPU utilization is 18% (us), system CPU utilization is 7% (sy), and overall CPU is 75% idle (id). IO wait (wa) and steal time (st) are 0.

After the first, each line of the output printed is an average over a sampling interval, which is 1 second in our example. This is pretty close to what we could call “current” values. As this is an idle machine, we see that overall values are below average. Only one or no processes are running or blocked, user CPU time is 2-3%, system CPU time is 1-2%, and the system is idle 96-97% of the time.

For a good measure, let’s look at the vmstat 1 output on the same system doing a CPU-intensive computation in a single process:

procs ------cpu-----

r  b us sy id wa st

2  0 18  7 75  0  0

1  0 13  0 87  0  0

1  0 13  0 86  0  0

1  0 14  0 86  0  0

1  0 15  0 84  0  0

The averages since boot are the same, but we have a single process running in every sample, and it drives user CPU time to 13-15%. The problem with vmstat is that we can’t learn from its output which process specifically is burning the CPU. Of course, if this is a dedicated database server, you can suppose that most if not all user CPU time is going to be MySQL and its threads, but things happen. The other problem is that on machines with high CPU core count, you can mistakenly take low readings in vmstat output for a fact. But vmstat gives a reading from 0% to 100% even on a 256-core machine. If 8 cores of such machine are 100% loaded, the user time shown by vmstat will be 3%, but in reality some workload may be throttled.

Before we talk about a solution to those problems, let’s talk Windows a little bit. A lot of what we’ve said in general about CPU utilization and especially about CPUs will translate to Windows, with some notable differences:

- There’s no IO wait accounting in Windows, as the IO subsystem is fundamentally different. Time spent by threads waiting for IO is going into the idle counter.

- The system CPU time counterpart is, roughly, the Privileged CPU time.

- Steal information is not available.

User and idle counters remain unchanged. With that, you can base your CPU monitoring on user, privileged, and idle CPU time as it is exposed by Windows. There are other counters and metrics available, but this should have you covered quite well. Getting the current CPU utilization on Windows can be done using many different tools. The simplest one, and probably the closest one to vmstat in spirit, is the good old Task Manager, a staple of looking at Windows performance metrics. It’s readily available, simple, and you’ve probably used it before. Task Manager can show you CPU utilization in percentage points broken down by CPU cores, and also split between user and kernel time.

Figure 12-1 shows Task Manager running on an idle system.

Figure 12-1. Task Manager showing idle CPU

Figure 12-2 shows Task Manager running on a busy system.

Figure 12-2. Task Manager showing busy CPU

As we said earlier, vmstat has a couple of problems: it doesn’t break down load per process or per a CPU core. Solving both problems of vmstat requires running other tools. Why not just run them right away? vmstat is universal, gives more than just CPU readings, and is very concise. It’s a good way to quickly see if there’s anything very wrong with a given system. The same goes for Task Manager, although as we’ll show it is actually more capable than vmstat.

On Linux, the next simplest tool to use after vmstat is top, another basic element in the toolbox of anyone dealing with a Linux server. It expands on the basic CPU metrics we have discussed, and adds both per-core load breakdown and per-process load accounting. When you execute top without any arguments, it starts in a terminal UI or TUI mode. Press ? to see the help menu. To display per-core load breakdown, press 1. Figure 12-3 shows what top looks like.

Figure 12-3. top in TUI mode

You can see here that each process gets its own overall CPU utilization shown under the %CPU column. For example, mysqld is using 104.7% of the overall CPU time. Now we can also see how that load is distributed among the many cores the server has. In this particular case, one core (Cpu0) is slightly more loaded than the other one. There are cases when MySQL hits a limit of a single CPU throughput, thus having per-core load breakdown is important. Having a view on how load is distributed between processes is important if you suspect some rogue process is eating into server’s capacity.

There are many more tools out there that can show you even more data. We can’t talk here in detail about each and every one of them, but we’ll name a few. mpstat can give a very deep level of CPU statistics. pidstat is a universal tool that provides stats on CPU, memory, disk, and network utilization for each individual process running. atop is an advanced version of top. And there are more, and everyone has their favorite set of tools. We firmly believe that what really matters is not tools, though they help, but an understanding of core metrics and stats that they provide.

On Windows, the Task Manager program is actually much closer to top than it is to vmstat, although we’ve done just that comparison. Task Manager’s ability to show per-core load and per-process load makes it quite a useful first step in any investigation. We recommend to dive into the Resource Monitor right away, as it provides more details. Easiest way to access Resource Monitor is to click on the “Open Resource Monitor” link in the Task Manager.

Figure 12-4 shows a Resource Monitor window with CPU load details.

Figure 12-4. Resource Monitor showing CPU load details

Task Manager and Resource Monitor are not the only tools on Windows capable of showing performance metrics. Here are a couple tools that you may want to get comfortable using. They are more advanced, so we’re leaving them out here.

Performance Monitor

Another built-in tool, but this time it’s an advanced one. Unlike the trusty Task Manager, Performance Monitor is a GUI for the performance counters subsystem in Windows. In short, you can view and plot any (or all) of the various performance metrics Windows measures, not only related to CPU.

Process Explorer

This tool is a part of a suite of advanced system utilities called Windows Sysinternals. It’s more powerful and more advanced than other tools listed, and can be useful to learn. Unlike the other tools, you’ll have to install Process Explorer separately from its home page on the Sysinternals site.

Disk

Disk or IO subsystem is crucial for database performance. Although CPU underpins every operation done on any given system, for databases in particular, disk is likely to be the most problematic bottleneck. That’s only logical — after all, databases store data on disk and then serve that data from disk. There are many layers of caches on top of slow and durable long storage, but they cannot always be utilized, and are not infinitely large. Thus, understanding basic disk performance is extremely important when dealing with database systems. The other important and frequently underestimated property of any storage is not related to performance at all — it’s the capacity of the storage, which is going to be the first one in this section.

Disk capacity and disk utilization refer to the total amount of data that can be stored on a given disk (or a lot of disks together in a storage system), and how much of that data is already stored. These are very boring, but very important metrics. While it’s not really necessary to monitor the disk capacity, as it’s unlikely to change without you noticing, you absolutely must keep an eye on the disk utilization and available space.

Most databases usually only grow in size over time. MySQL in particular requires a healthy amount of available disk space headroom to accomodate for the table alters, long-running transactions, and spikes in write load. When there’s no more disk space available for use for a MySQL database instance, it may crash or stop working, and will unlikely start working again until some space is freed up or more disk capacity is added. Depending on your circumstances, adding more capacity may take from minutes to days. That’s something you probably want to plan ahead of time.

Luckily, monitoring disk space usage is very easy. On Linux, it can be done using the simple df command. Without arguments, it will show capacity and usage in 1K-blocks for every filesystem. You can add -h argument to get human-readable measurements, and specify a mountpoint (or just a path) to limit the check. Here’s an example:

$ df -h /var/lib/mysql

Filesystem      Size  Used Avail Use% Mounted on

/dev/sda1        40G   18G   23G  45% /

df ’s output is self-explanatory, and it’s one of the easiest tools to work with. We recommend that you try to keep your database mountpoints at 90% capacity unless you run multi-terabyte systems. In that case, go higher. A trick that you can use is to put some large dummy files on the same filesystems as your database. Should you start running out of space, you can remove a dummy file or files and give yourself some more time to react. We recommend that you rather have some disk space monitoring in place.

On Windows, the trusty File Explorer can give the diskspace utilization and capacity information, as can be seen in Figure 12-5.

Figure 12-5. File Explorer showing available disk space

With the disk space covered, we will now highlight the following performance properties of any IO subsystem.

Bandwidth

How many bytes of data can be pushed to (or pulled from) storage per unit of time.

IOPS

Number of operations a disk (or a storage system) is capable of serving per unit of time.

Latency

How long it takes for a read or a write to be served by the storage.

These three properties are enough to describe any storage system and start forming understanding of whether it’s good, bad, or ugly. As we did in the CPU section, we’ll show a couple of tools to inspect disk performance, and use their output to explain the specific metrics. We’re again showing Linux and Windows, and other systems will have something similar, so the knowledge is portable.

The IO load analogue to vmstat on Linux is the iostat program. Ergonomics should be familiar: invoke without arguments, and you get average values since boot; pass a number as an argument and get averages per a sample period. We also prefer running the tool with a -x argument which adds a lot of useful details. Unlike vmstat, iostat gives metrics broken down by a block device, similar to the mpstat we mentioned.

TIP

iostat is usually a part of sysstat package. Use apt on Ubuntu/Debian or yum on RHEL-based operating systems to install the package. You should be comfortable using these tools after following instructions in Chapter 1.

Let’s start with looking at an output example. Command we’ll be using is iostat -dxyt 5, which translates to: average over every five seconds, print device report, omit first average since boot, print extended statistics, and add the timestamp. Here’s an iostat output on a loaded system:

05/09/2021 04:45:09 PM

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s     wkB/s avgrq-sz avgqu-sz...

sda               0.00     0.00    0.00 1599.00     0.00 204672.00   256.00   141.67...

...await r_await w_await  svctm  %util

...88.63    0.00   88.63   0.63 100.00

There’s quite a lot to unpack here. We won’t cover every column, but let’s highlight the most importat ones, especially those corresponding to properties we mentioned before.

Bandwidth

In iostat output, columns rkB/s and wkB/s correspond to bandwidth utilization, read and write respectively. If you know the characteristics of underlying storage (for example, you could know it promises 200 MiB/s of combined read and write bandwidth), you can tell if you’re pushing the limits. Here you can see a respectable figure of just over 200,000 kilobytes per second being written to the /dev/sda device, and no reads taking place.

IOPS

This metric is represented by r/s and w/s columns, giving number of read and write operations per second respectively. Our example shows 1599 write operation per seconds happening. As expected, no read operations are registered.

Latency

Shown in a slightly more complex manner, latency is broken down in four columns (or more, in newer iostat versions). They are await, r_await, w_await and svctm. For a basic analysis you should be looking at the await, which is an average latency serving any request. r_await and w_await break await down by reads and writes. svctime is a deprecated metric, which attempts to show the pure device latency without any queueing.

Having these basic metric readings, and knowing some basic facts about the storage used, it is possible to tell what’s going on. Our basic example is running on a modern consumer-grade NVMe in one of the author’s laptops. While the bandwidth is pretty good, each request averages 88ms, which is a lot. You can also do some simple math to get an IO load pattern from these metrics. For example, if we divide bandwidth by IOPS, we get a figure of 128 kilobyte per request. iostat does, actually, include that metric in avgrq-sz column, which shows average request’s size in a historical unit of sector (512 bytes). You can go forward and measure that 1599 writes per second can only be served at ~40ms/request, meaning that there’s parallel write load (and also that our device is capable of serving parallel requests).

IO patterns—size of requests, degree of parallelism, random vs sequential—can shift the upper limits of the underlying storage. Most devices will advertise maximum bandwidth, IOPS, and minimal latency at specific conditions. Moreover, these conditions may vary for maximum IOPS and maximum bandwidth measurements, as well as latency. It is rather difficult to definitely answer the question of whether metric readings are good or bad. Without knowing anything about the underlying storage, one way to look at utilization is to try and assess saturation. Saturation, which we’ll touch on in the “Investigation Methods” in the USE Method, is a measure of how overloaded a resource is. This becomes increasingly difficult with modern storage, capable of servicing long queues efficiently in parallel, but in general, queueing on a storage device is a sign of saturation. In the iostat output, that is the avgqu-sz column (or aqu-sz in newer versions of iostat), and values larger than 1 usually mean that a device is saturated. Example we had shows the queue of 146 requests, which is a lot, likely telling us that IO is highly utilized and may be a bottleneck.

Unfortunately, as you might’ve noticed, there’s no simple straight measure of the IO utilization: there seems to be a caveat on every metric. Measuring storage performance is a difficult task!

The same metrics define storage devices on Linux, Windows, and any other OS.

Let’s now take a look at basic Windows tools for assessing IO performance. Their readings should be familiar by now. We recommend looking at Resource Monitor, which we’ve shown in the CPU section, but this time navigate to the Disk tab. Figure 12-6 shows that view with MySQL under heavy write load.

Figure 12-6. Resource Monitor showing IO load details

Metrics presented by the Resource Monitor are similar to those of iostat. You can see bandwidth, latency, and length of the request queue. One metric missing is IOPS. To get those, you’ll need to dig into Performance Monitor (or perfmon), but we’ll leave that as an exercise.

Resource Monitor actually shows a slightly more detailed view than one we saw in the iostat example. First, there’s a breakdown of IO load per process, and second there’s a further breakdown of that load per file. We don’t know of a single tool on Linux which is capable of showing load broken down like that simultaneously. To get the load breakdown per program, a pidstat tool can be used, which we mentioned before. Here’s an example output:

# pidstat -d 5

...

10:50:01 AM   UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command

10:50:06 AM    27      4725      0.00  30235.06      0.00  mysqld

10:50:06 AM   UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command

10:50:11 AM    27      4725      0.00  23379.20      0.00  mysqld

...

Getting a breakdown per file on Linux is quite easily achieved using the BCC Toolkit, specifically the filetop tool. There are many more tools to explore there, but they are usually quite advanced. Tools we’ve shown should be enough to cover basic investigation and monitoring needs.

Memory

Memory, or RAM, is another important resource for any database. Memory offers vastly superior performance than disk for reading and writing data, thus databases strive to operate “in memory” as much as possible. Unfortunately, memory is not persistent, so eventually everything should get to the disk. (For more on disk performance, look at the “Disk” section just before this one.)

In contrast to what we did with CPU and disk, we won’t actually be talking about memory performance. Even though it’s important, it’s also a very advanced and deep topic. Instead, we will be focusing on memory utilization. That can also get quite complex quite quickly, so we’ll be trying to stay focused.

Let’s start with some basic premises. Every program needs some memory to operate. Database systems, including MySQL, usually need a lot of memory. When you run out of it, applications start having performance issues, or may fail, as you’ll see at the end of the section. Monitoring memory utilization therefore is crucial to any system’s stability, even more so if it’s a database system.

In this case, we’ll actually start with Windows, since on the surface level it has slightly less convoluted memory accounting compared to Linux. To get the overall OS memory utilization on Windows, all you need to do is start the Task Manager, as done in the “CPU” section, and then navigate to the Performance tab and pick Memory. You can see Task Manager’s memory utilization details in Figure 12-7.

Figure 12-7. Task Manager showing memory utilization details

You can see that this machine has 4 GB of memory in total, with 2.4 GB being currently used, and 1.6 GB available, making overall utilization 60%. This is a safe amount, and we may even want to allocate more memory to MySQL to minimize “wasted” free memory. Some ideas on the MySQL’s InnoDB buffer pool sizing can be found in “Adjusting the Buffer Pool size”.

On Linux, the simplest tool to get memory utilization is a free command. We recommend using it with the -h argument, which converts all fields to a human-readable format. Here’s an output on a machine running CentOS 7:

$ free -h

total        used        free      shared  buff/cache   available

Mem:           3.7G        2.7G        155M        8.5M        905M        787M

Swap:          2.0G         13M        2.0G

Now, that’s more data than we had previously seen on Windows. In reality, Windows has most of these counters, they are just not as visible.

Let’s go through the output. For now, we’ll be covering the Mem row, and we’ll talk about Swap later.

Two main metrics here are used and available, which translate to Windows’ in use and available. A frequent mistake, one your authors used to make, is to look at the free metric reading. That’s not correct! Linux (or, in fact, Windows) doesn’t like to keep memory free. After all, free memory is a wasted resource. When there’s memory available that’s not needed by applications directly, Linux will use that memory to keep a cache of data being read and written from and to the disk. We’ll show later that Windows does the same, but you cannot see that from the Task Manager. The mistake of looking at free even has its own site: Linux ate my ram!.

Let’s further break down the output.

total

Total amount of memory available on the machine.

used

Amount of memory currently used by the applications.

free

Actual free memory not used by the OS at all.

shared

Special type of memory that needs to be specifically requested and allocated, and that multiple processes can access together; because it’s not used by MySQL, we’re skipping the details here.

buff/cache

Amount of memory the OS currently uses as a cache to improve IO.

available

Amount of memory that applications could use if they needed it; usually a sum of free and buff/cache.

In general, for a basic but robust monitoring, you only need to look at the total, used, and available amounts. Linux should be capable of handling the cached memory on its own. We’re deliberately not covering the page cache here — that’s an advanced topic. By default, MySQL on Linux will utilize the page cache, so you should size your instance to accomodate for that. An often recommended change, however, is to tell MySQL to avoide the page cache (look for the documentation on innodb_flush_method), which will allow more memory to be used by MySQL itself.

We’ve mentioned that Windows has mostly the same metrics, they’re just hidden. To see that, open the Resource Monitor, and navigate to the Memory tab. Figure 12-8 shows the contents of the Memory tab.

Figure 12-8. Resource Monitor showing memory utilization details

You’ll immediately notice that the amount of Free memory is just 52 MB, and there’s a hefty chunk of some Standby memory, with a little bit of Modified. A sum of Standby and Modified is the Cached listed below the Available figure. When the screenshot was taken, 1593 MB of memory was used by cache, with 33 MB of that being dirty (or modified). Windows, like Linux, caches filesystem pages in an attempt to minimize and smoothe the IO and utilize the memory to its fullest capacity.

Another thing you can see is a breakdown of memory utilization per process with mysqld.exe holding just under 500 MB of memory. On Linux, a similar output can be found in the top command, which we first used in the “CPU” section. Once top is running, press Shift+M to sort the output by memory usage and get a human-readable figures. top showing memory details can be seen in Figure 12-9.

Figure 12-9. top showing memory utilization details

On this system, the output is not very interesting, but you can quickly see that it’s MySQL that consumes most memory with its mysqld process.

Before finishing this section, let’s talk about what happens when you run out of memory. Before that, though, let’s discuss swapping, or paging. We should mention here that most modern OSes implement memory management in such a way that individual applications each have their own view of the whole memory (hence you can see application memory being called virtual memory), and that the sum total of the virtual memory that applications can use exceeds the total actual memory capacity of the system. Discussion of the former point is better suited for a university course on operating system design, but the latter point is very important when running database systems.

The implications of this design are important, because an OS can’t just magically extend the capacity of the system’s memory. In fact, what happens is the OS uses disk storage to extend the amount of memory, and as mentioned, RAM is usually far more performant than even the fastest disk. Thus, as you can imagine, there’s a price to pay for this memory extention. Paging can occur in a few different ways and for different reasons. Most important for MySQL is the type of paging called swapping — writing out parts of memory into a dedicated place on disk. On Linux, that place can be a separate partition, or a file. On Windows, there’s a special file called pagefile.sys, which has mostly the same use.

Swapping is not bad per se, but it’s problematic for MySQL. The problem is that our database thinks it’s reading something from memory, whereas in reality the OS has paged out some of that data away into swap, and will actually read it from disk. MySQL cannot predict when this situation will happen, and can do nothing to prevent it or optimize the access somehow. For the end user, this can mean a sudden unexplained drop in query response times. Having some swap, though, is an important protective measure, as we’ll show.

Let’s move on to answer the question of what really happens when you run out of memory. In short: nothing good. There are only a couple of general outcomes for MySQL when the system is running out of memory, so let’s talk them through:

- MySQL requests more memory from the OS, but there’s none available: everything that could be paged out is not in memory, and the swap file is absent or already full. Usually, this situation results in a crash. That’s a really bad outcome.

- On Linux, a variation of the preceding point is that the OS detects situations when system is close to running out of memory and forcefully terminates—in other words, kills—processes. Usually, the processes terminated will the ones holding most memory, and usually on a database server, MySQL will be the top memory consumer. This situation usually happens before the situation explained by the previous point.

- MySQL, or some other program, fills up memory to a point where OS has to start swapping. This assumes the swap (or pagefile in Windows) is set up. As explained a few paragraphs back, MySQL’s performance will degrade unexpectedly and unpredictably when its memory is swapped out. This arguably is a better outcome than just a crash or MySQL being terminated, but nevertheless it’s something to avoid.

So, MySQL will either get slower, crash, or get killed, as simple as that. You now should see clearly why monitoring available and used memory is very important. We also recommend leaving some memory headroom on your servers, and having swap/pagefile set up. For some advice on Linux swap setup, see Chapter 11.

Network

Of all the OS resources, network is probably the one most frequently blamed for random unexplained issues. There’s a good reason for that: monitoring the network is difficult. Understanding issues with the network sometimes requires a detailed analysis of the whole network stream. Network as a resource is peculiar, because unlike CPU, disk, and memory, it is not contained within a single server. At the very least, you need two machines communicating with each other for “network” to even appear. Of course, there are local connections, but they are usually stable. And granted, disk storage may be shared, and CPU and memory in case of virtual machines can be shared, too, but networking is always about multiple machines.

Since this chapter is about monitoring, we’re not going to cover connectivity issues here. Yet a surprising number of issues with networking boil down to the simple problem of one computer not being able to talk to another. Do not take connectivity for granted. Network topologies are usually complex, with each packet following a complex route through multiple machines. In cloud environments, the routes can be even more complex and less obvious. If you think you have some network issues, it’s wise to check that connections can be established at all.

We’ll be touching the following properties of any network:

Bandwidth and its utilization—throughput

Similar to the same concept defined in “Disk”. Every network connection has a maximum bandwidth capacity, usually expressed as some unit of volume of data per second. Internet connections usually use Mbps or megabits per second, but MBps or megabytes per second can also be used. Network links and equipment put a hard cap on maximum bandwidth. For example, currently common household network equipment rarely exceeds 1 Gbps bandwidth. More advanced data center equipment regularly has 10 Gbps. Special equipment exists that can drive bandwidth to hundreds of Gbps, but such connections are usually unrouted direct connections between two servers.

Errors, their number and sources

Network errors are unavoidable. In fact, TCP—Transmission Control Protocol, a backbone of the internet (and a protocol used by MySQL) — is built around the premise that packets will be lost. Yet, even though you’ll see errors from time to time, having a high rate of errors will cause connections to be slow, as communicating parties will need to re-send packets over and over.

Following analogy with the disk, we could also include latency and number of packets sent and received (loosely resembling IOPS). However, packet transmission latency can only be measured by an application that’s doing the actual transmission. The OS can’t measure and show some average latency for a network. Number of packets is usually just redundant, as it follows the bandwidth and throughput figures.

One particular metric that is useful to add when looking at networks is the number of retransmitted packets. Retransmission happens when a packet is lost or damaged. It is not an error, but is usually a result of some issues with a connection. Just like running out of bandwidth, an increased number of retransmits will lead to a choppy network performance.

On Linux, we can start by looking at the network interface statistics. The easiest way is to run an ifconfig command. Its output by default will include every network interface on a particular host. Since we know in this case all load comes through eth1, we can only see stats on it:

$ ifconfig eth1

...

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

inet 192.168.10.11  netmask 255.255.255.0  broadcast 192.168.10.255

inet6 fe80::a00:27ff:fef6:b4f  prefixlen 64  scopeid 0x20<link>

ether 08:00:27:f6:0b:4f  txqueuelen 1000  (Ethernet)

RX packets 6217203  bytes 735108061 (701.0 MiB)

RX errors 0  dropped 0  overruns 0  frame 0

TX packets 11381894  bytes 18025086781 (16.7 GiB)

TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

...

We can immediately see that the network is pretty healthy just by the fact there are no errors receiving (RX) or sending (TX) packets. The RX and TX total data stats (701.0 MiB and 16.7 GiB respectively) will be growing each time you run ifconfig, so you can easily measure bandwidth utilization by running it over time. That’s not terribly convenient, and there are programs that show transmission rates in real time, but none of those are shipped by default in common Linux distributions. To see a history of the transmission rate and errors, you can use a sar -n DEV or sar -n EDEV commands respectively. sar is a part of a sysstat package we’ve mentioned when talking about iostat:

$ sar -n DEV

IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s...

06:30:01 PM      eth0      0.16      0.08      0.01      0.01...

06:30:01 PM      eth1   7269.55  13473.28    843.84  21618.70...

06:30:01 PM        lo      0.00      0.00      0.00      0.00...

06:40:01 PM      eth0      0.48      0.28      0.03      0.05...

06:40:01 PM      eth1   7844.90  13941.09    893.95  19204.10...

06:40:01 PM        lo      0.00      0.00      0.00      0.00...

...rxcmp/s   txcmp/s  rxmcst/s

...   0.00      0.00      0.00

...   0.00      0.00      0.00

...   0.00      0.00      0.00

...   0.00      0.00      0.00

...   0.00      0.00      0.00

...   0.00      0.00      0.00

$ sar -n EDEV

04:30:01 PM     IFACE   rxerr/s   txerr/s    coll/s  rxdrop/s...

06:40:01 PM      eth0      0.00      0.00      0.00      0.00...

06:40:01 PM      eth1      0.00      0.00      0.00      0.00...

06:40:01 PM        lo      0.00      0.00      0.00      0.00...

...txdrop/s  txcarr/s  rxfram/s  rxfifo/s  txfifo/s

...    0.00      0.00      0.00      0.00      0.00

...    0.00      0.00      0.00      0.00      0.00

...    0.00      0.00      0.00      0.00      0.00

Again, we see that in our example, interface eth1 is quite loaded, but there are no errors being reported. If we stay within bandwidth limits, network performance should be nominal. To get a full detailed view on the various errors and issues that happened within the network, a netstat command can be used. With a -s flag it will report a lot of counters. To keep things basic, we will show just the Tcp section of the output with a number of retransmits. For a more detailed overview, check the TcpExt section of the output:

$ netstat -s

...

Tcp:

55 active connections openings

39 passive connection openings

0 failed connection attempts

3 connection resets received

9 connections established

14449654 segments received

25994151 segments send out

54 segments retransmited

0 bad segments received.

19 resets sent

...

Considering a sheer number of segments sent out, the retransmission rate is excellent. This network seems to be fine.

On Windows, we again resort to checking Resource Monitor, which has most of the metrics we want, and more. Figure 12-10 shows network-related views Resource Monitor has to offer on a host running some synthetic load against MySQL.

Figure 12-10. Resource Monitor showing network utilization details

To get a reading on the number of errors, you can use the netstat command. Note that even though it’s the same name as a similar Linux tool, they are slightly different. In this case, we have no errors:

C:\Users\someuser> netstat -e

Interface Statistics

Received            Sent

Bytes                      58544920         7904968

Unicast packets               62504           32308

Non-unicast packets               0             364

Discards                          0               0

Errors                            0               0

Unknown protocols                 0

We’ve used netstat -s on Linux, and the same modifier exists on Windows, too. Again, we’re only looking at a part of the output:

C:\Users\someuser> netstat -s

...

TCP Statistics for IPv4

Active Opens                        = 457

Passive Opens                       = 30

Failed Connection Attempts          = 3

Reset Connections                   = 121

Current Connections                 = 11

Segments Received                   = 61237201

Segments Sent                       = 30866526

Segments Retransmitted              = 0

...

Judging by the metrics we highlighted for monitoring—bandwidth utilization and errors—this system’s network is operating perfectly fine. We understand this is barely scratching the surface when it comes to the complexity of networking. However, this minimal set of tools can help you out immensely to understand whether you should be blaming your network at all.

That finishes a pretty lengthy overview of OS monitoring basics. We could probably have kept it shorter, and you may ask why we put all this in a book about MySQL. The answer is pretty simple: because it is important. Any program interacts with an OS and requires some of the system’s resources. MySQL, by nature, is usually going to be a very demanding program, which you expect to be performing well. For that, however, we must make sure that we have the necessary resources and we’re not running out of a performance capacity for disk, CPU, or network, or just out of capacity for disk and memory. Sometimes, an issue with system’s resources caused by MySQL can also lead you to uncovering issues within MySQL itself. For example, a badly written query may put a lot of load on CPU and disk while causing a spike in memory usage. The next section shows some basic ways to monitor and diagnose a running MySQL.

MySQL Server Observability

Monitoring MySQL is simultaneously easy and difficult. Easy, because MySQL exposes almost 500 status variables, which allow you to see almost exactly what is going on inside your database. In addition to that, InnoDB has its own diagnostic output. Monitoring is hard, though, because it may be tricky to make sense of the data you have.

In this section, we’re going to explain the basics of MySQL monitoring, starting with going over what the status variables are and how to get them, and moving on to InnoDB’s diagnostics. Once that’s covered, we’ll show a few basic recipes we believe should be a part of every MySQL database monitoring suite. With these recipes and an idea about OS monitoring that you got in the previous chapter, you should be able to understand what’s going on with your system.

Status Variables

We’ll start with MySQL’s Server Status Variables. These variables, unlike configuration options, are read-only, and they show you information about the current state of the MySQL server. They vary in nature: most of them are either ever-increasing counters, or gauges with values moving up and down. Some, though, are static text fields, which are helpful to understand the current server configuration. All status variables can be accessed at the global server level, and at the current session level. But not every variable makes sense on a session level, and some will show same values on both levels.

SHOW STATUS is used to get the current status variable values. It has two optional modifiers: GLOBAL and SESSION, and defaults to SESSION. You can also specify a name of the variable, or a pattern, but that’s not mandatory. The command in the following example shows all status variable values for the current session:

mysql> SHOW STATUS;

+-----------------------------------------------+-----------------------+

| Variable_name                                 | Value                 |

+-----------------------------------------------+-----------------------+

| Aborted_clients                               | 0                     |

| Aborted_connects                              | 0                     |

| Acl_cache_items_count                         | 0                     |

...

| Threads_connected                             | 2                     |

| Threads_created                               | 2                     |

| Threads_running                               | 2                     |

| Uptime                                        | 305662                |

| Uptime_since_flush_status                     | 305662                |

| validate_password.dictionary_file_last_parsed | 2021-05-22 20:53:08   |

| validate_password.dictionary_file_words_count | 0                     |

+-----------------------------------------------+-----------------------+

482 rows in set (0.01 sec)

Scrolling through hundreds of rows of output is suboptimal, so let’s instead use a wildcard to limit the number of variables we request. LIKE in SHOW STATUS works the same as it does for regular SELECT statements, as explained in Chapter 3:

mysql> SHOW STATUS LIKE 'Created%';

+-------------------------+-------+

| Variable_name           | Value |

+-------------------------+-------+

| Created_tmp_disk_tables | 0     |

| Created_tmp_files       | 7     |

| Created_tmp_tables      | 0     |

+-------------------------+-------+

3 rows in set (0.01 sec)

Now the output is much easier to read. To read a value of a single variable, just specify its full name within the quotes without percentage symbols, like so:

mysql> SHOW STATUS LIKE 'Com_show_status';

+-----------------+-------+

| Variable_name   | Value |

+-----------------+-------+

| Com_show_status | 11    |

+-----------------+-------+

1 row in set (0.00 sec)

You might notice in the output for Created% status variables that MySQL showed a value of 7 for Created_tmp_files. Does that mean this session created 7 temporary files, while creating 0 temporary tables? No, and in fact the Created_tmp_files status variable has only a global scope. It is unfortunately a known issue with MySQL at the moment. You always see all status variables regardless of the requested scope, but their values will be properly scoped. MySQL documentation includes a helpful page: “Server Status Variable Reference”.

Unlike Created_tmp_files, the Com_show_status variable has scope “both”, meaning that you can get a global counter as well as a per-session value. Let’s see that in practice:

mysql> SHOW STATUS LIKE 'Com_show_status';

+-----------------+-------+

| Variable_name   | Value |

+-----------------+-------+

| Com_show_status | 13    |

+-----------------+-------+

1 row in set (0.00 sec)

mysql> SHOW GLOBAL STATUS LIKE 'Com_show_status';

+-----------------+-------+

| Variable_name   | Value |

+-----------------+-------+

| Com_show_status | 45    |

+-----------------+-------+

1 row in set (0.00 sec)

Another important thing to note when looking at the status variables is that it’s possible to reset most of them back to 0 on a session level. That is achieved by running a FLUSH STATUS command. The command flushes status variables within all connected sessions back to 0, but before that adds those values to the global counters. Thus, FLUSH STATUS operates on a session level, but for all sessions. It’s impossible to reset global counters without a restart. For example, we’ll reset status variable values in the session we used before:

mysql> FLUSH STATUS;

Query OK, 0 rows affected (0.00 sec)

mysql> SHOW STATUS LIKE 'Com_show_status';

+-----------------+-------+

| Variable_name   | Value |

+-----------------+-------+

| Com_show_status | 1     |

+-----------------+-------+

1 row in set (0.00 sec)

mysql> SHOW GLOBAL STATUS LIKE 'Com_show_status';

+-----------------+-------+

| Variable_name   | Value |

+-----------------+-------+

| Com_show_status | 49    |

+-----------------+-------+

1 row in set (0.00 sec)

Even though global counter keeps on increasing, session counter was reset to 0, and only incremented to 1 when we ran the SHOW STATUS command. This can be useful to see in isolation, how for example running a single query changes the status variable values, in particular, the Handler_* family of status variables.

Basic Monitoring Recipes

You could monitor numerous metrics and their combinations. We believe, however, that there are a few that must be in the toolbox of every database operator. While you are learning MySQL, these should be enough to give you a reasonable sense of how your database is doing. Most of the existing monitoring systems should have these covered, and usually include many more metrics. You may never set up collection yourself, but our explanation should also allow you to get a better understanding of just what your monitoring system tells you.

We’re going to give few broad categories of metrics, within which we’ll detail some of the more important counters in our opinion.

MySQL server availability

This is the most important thing you should monitor. If MySQL server is not accepting connections or is not running, all the other metrics don’t matter at all.

MySQL server is a robust piece of software that’s capable of running with uptime of months and years. Yet, there are situations which can lead to a premature unplanned shutdown (or, less fancy, a crash). For example, “Memory” we discussed that out of memory conditions may lead to MySQL crashing or being killed. Other incidents happen, too. There are crashing bugs in MySQL, however rare they are nowadays. There are also operational mistakes: who didn’t forget to bring up a database after a planned maintenance? Hardware gets bad, servers restart — many things may compromise availability of MySQL.

There are few approaches to monitoring MySQL availability, and there’s not a single best one of them: better to combine a few. A very simple basic approach is to check that the mysqld (or mysqld.exe) process is actually running and visible from the OS level. On Linux and Unix-like systems, a ps command can be used, and on Windows, you could check the Task Manager or run a Get-Service PowerShell command. This is not a useless check, but it has its issues. For one, the fact that MySQL is running does not guarantee that it’s actually doing what it should, that is, processing clients’ queries. MySQL could be swamped by load, or suffer from a disk failure and be unbearably slow. From the OS side, the process is running, but from the client perspective, it’s as good as shut down anyway.

The second approach thus is to check MySQL’s availability from the application point of view. Usually, that’s achieved by running MySQL monitor and executing some simple short query. Unlike the previous check, this one makes sure that a new connection to MySQL can be established, and that the database does process queries. You can locate these checks on the application side to make them even closer to how apps see the database. Instead of setting up such checks as independent entities, applications can be adjusted to probe MySQL and report clear errors either to operators or to the monitoring system.

The third approach is in between the previous two, and is concentrated on the DB side. While monitoring MySQL, you will at least need to execute simple queries against the database to check status variables. If those queries fail, your monitoring system should alert, as potentially MySQL started to have issues. A variation would be to check that no data from the target instance was received by a monitoring system in the last few minutes.

What ideally should come out of these checks is not only the alert “MySQL is down”, but also some clue as to why it is. For example, if second type check cannot initiate a new connection because MySQL has run out of connections, then that should be a part of the alert. If third type check is failing, but the first type is okay, then it’s a different situation than a crash.

Client connections

MySQL server is a multi-threaded program, as was laid out in depth in “The MySQL Server Daemon”. Every client connecting to a database causes a new thread to be spawned within the process of MySQL server (mysqld or mysqld.exe). That thread will be responsible for executing statements sent by a client, and in theory there can be as many concurrent queries executing as there are client threads.

Each connection and its thread put some low overhead on MySQL server even when they are idle. Apart from that, from the database point of view, each connection is a liability: the database cannot know when a connection will send a statement. Concurrency, or the number of simultaneously running transactions and queries, usually increases with an increase in number of established connections. Concurrency is not bad in itself, but each system will have a limit of scalability. As you’ll remember from reading “Operating System Metrics”, CPU and disk resources have performance limits, and it’s impossible to push past them. Even with an infinite amount of OS resources, MySQL itself has internal scalability limits.

To put it simply: the amount of connections, and especially active connections should ideally be kept minimal. From the MySQL side, no connections is the perfect situation, but from application side that’s unacceptable. Some applications, though, make no attempt at limiting the number of connections they make and queries they send, assuming that the database will take care of the load. This can create a dangerous situation known as a thundering herd: for some reason, queries run longer, and the app reacts by sending more and more queries, overloading the database.

Finally, MySQL has an upper limit on the number of client connections, controlled by the system variable max_connections. Once the number of existing connections hits the value of that variable, MySQL will refuse to create new connections. That’s a bad thing. max_connections should be used as a protection from complete server meltdown if clients establish thousands of connections. But ideally, you should monitor number of connections and work with apps team to keep that number low.

Let’s review the specific connection and thread counters that MySQL exposes:

Threads_connected

Number of currently connected client threads, or in other words number of established client connections. We’ve been explaining the importance of this one for the last few paragraphs, so you should know why you have to check it.

Threads_running

Number of client threads that are currently executing a statement, where Threads_connected indicates a potential for high concurrency, Threads_running actually shows the current measure of that concurrency. Spikes in this counter indicate either an increased load from the application or a slowness in the database leading to queries stacking up.

Max_used_connections

Maximum number of connections established that was recorded since the last MySQL server restart. If you suspect that a connection flood happened, but don’t have a recorded history of changes in Threads_connected, you can check this status variable to see the highest peak recorded.

Max_used_connections_time

Date and time when MySQL server saw maximum number of connections since last restart to date.

Another important item to monitor about connections is their rate of failure. MySQL distinguishes between connections that clients failed to establish and existing connections that failed due to a timeout, for example. An increased rate of errors may indicate that your applications are having trouble communicating with your database.

Aborted_clients

Number of already established connections that were aborted. MySQL documentation mentions “the client died without closing the connection properly,” but this can also happen if there’s a network issue between server and client. A frequent source of increase in this counter is max_allowed_packet violations (see “Scope of Options”), and session timeouts (see wait_timeout and interactive_timeout system variables). Some errors are to be expected, but sharp spikes should be checked.

Aborted_connects

Number of new connections that failed to be established. This includes incorrect passwords, connecting to database to which a user has no permission, protocol mismatches, connect_timeout violations, and reaching max_connections among other things. It also includes various network-related issues. There’s a family of status variables under the Connection_errors_% wildcard that look more in depth into some of the specific issues. An increase in Aborted_connects should be checked, and it can indicate an application configuration issue (wrong user/password), or a database issue (running out of connections).

NOTE

MySQL Enterprise Edition, Percona Server, and MariaDB offer a thread pool functionality. This changes the connection and thread accounting. With thread pool, the number of connections stays the same, but the number of threads running within MySQL is limited by the size of the thread pool. When a connection needs to execute a statement, it will have to get an available thread from the pool, and wait if such thread is not available. Using thread pool improves MySQL performance with hundreds and thousands of connections. Since this feature is not available in regular MySQL, and we believe it’s an advanced one, we’re not covering it in this book.

Query counts

The next broad category of metrics is query-related metrics. Where Threads_running shows how many sessions are active at once, metrics in this category will show the quality of the load those sessions produce. 30 running threads may be executing a single hour-long query each, or be executing a few dozen queries per second. The conclusions you make will be completely different, and the load profile will likely change, too. Here, we’ll start by looking at the overall amount of queries, then move on to breaking down queries by type, and last but not least we’ll look into how the queries execute.

Queries

This global status variable, simply put, gives a number of statements executed by the server (excluding COM_PING and COM_STATISTICS). If you run SHOW GLOBAL STATUS LIKE 'Queries'; on an idle server, you will see the counter value increasing.

Questions

Almost the same as Queries, but excludes statements executed within stored procedures, and also the following types of queries: COM_PING, COM_STATISTICS, COM_STMT_PREPARE, COM_STMT_CLOSE, or COM_STMT_RESET. Unless your databases clients use stored procedures extensively, the Questions metric is closer to the amount of actual queries executed by server, compared to the amount of statements in Queries. In addition to that, Questions is both a session-level as well as a global status variable.

Both Queries and Questions are incremented when the query starts executing, so it’s necessary to also look at the Threads_running value to see how many queries are actually being executed right now.

QPS

Queries Per Second. A synthetic metric that you can arrive at by looking at how the Queries variable changes over time. QPS based on Queries will include almost any statement that the server executes.

The QPS metric does not tell us about the quality of queries executed, but it’s a useful gauge nevertheless. Usually, the load on a database from applications is regular. It may be moving in waves (more during the day, lower at night), but over a week or a month a pattern of number of queries over will show. When you get a report about a database being slow, looking at QPS may give you a quick indication of whether there’s a sudden unexpected growth in application load. A drop in QPS, on the other hand, may indicate that issues are on the database side, as it cannot process as many queries as usually in the same time.

Query types and quality

The next logical step from knowing the QPS is understanding what types of queries are being executed by the clients, and the quality of those queries. By quality here we broadly mean the impact of those queries on the server. Queries are not equal, and some, you may say, are bad, or produce unnecessary load on the system. Looking for and catching such queries is an important part of monitoring. In this section we’re trying to answer the question “are there a lot of bad queries?” and in “The Slow Query Log” we’ll learn how to catch the specific offenders.

Types of queries

Each query MySQL executes has a type. What’s more, any command that you can execute has a type. MySQL accounts for the number of different types of commands and queries executed under a Com_% family of status variables. The number of these variables is high: there are 172 of them in MySQL 8.0.25, almost a third of all status variables. As you can guess from the number of such variables, MySQL counts a lot of commands that you perhaps wouldn’t even think of: Com_uninstall_plugin counting the number of times UNINSTALL PLUGIN was called, or Com_help for the HELP statment.

Every Com_% status variable is available on both global and session levels, as was shown with Com_show_status in “Status Variables”. However, MySQL doesn’t expose other threads’ counters for Com_% variables, and so for monitoring purposes, global status variables are assumed here. It’s possible to get other sessions’ statement counters, but that’s achieved through the performance schema family of events called statement/sql/%. That can be useful to attempt to find a thread that’s sending a disproportional amount of some type of statements, but it’s a bit advanced and falls under investigation rather than monitoring. You can find more details about that in the Performance Schema Status Variable Tables section of MySQL documentation.

Since there are so many of Com_% status variables, monitoring every type of command is going to prove too noisy and unnecessary. You should, however, try to store values of all of them. It’s looking (and reacting) at 172 variables that is probably inessential. You can go two ways about these counters.

First is to pick the command types that are relevant for your database load profile. For example, if your database clients do not use stored procedures, then looking at Com_call_procedure will waste time. A good starting selection is to cover SELECT and basic DML statements, which usually comprise the bulk of any database system’s load. This selection can be: Com_select, Com_insert, Com_update, and Com_delete. Status variables’ names are self-explanatory here. One interesting thing MySQL does is it accounts for multi-table updates and deletes (see “Updates and Deletes with Multiple Tables”) separately under Com_update_multi and Com_delete_multi, so these should also be monitoried, unless you’re sure such statements are never run in your system.

Starting from there, you can look at all of the Com_% status variables and see which ones are growing, and then make a decision to add those to your selection of monitored variables. Unfortunately, the flaw of this approach is that you can miss some unexpected spikes.

Another way of looking at these counters could be to look at the top five or ten of them over time. This way, a sudden change of load pattern can be more difficult to miss.

Knowing what types of queries are running is important in shaping the overall understanding of the load on a given database. Moreover, it changes how you approach tuning the database, because, for example, an insert-heavy workload may require a different setup compared to a read-only or mostly-read workload. Changes in query load profile, like a sudden appearance of thousands of UPDATE statements executed per second, can indicate changes on the application side.

Query quality

The next step from knowing what queries are running is to understand their quality, or their impact on the system. We mentioned this, but it’s worth reiterating: queries are not equal. Some will put more burden on the system than others. Looking at the overall query-related metrics may give you advance warning of problems growing in the database. You will learn that it’s possible to notice problematic behaviors from just a few counters.

Select_scan counts the number of queries that caused a full table scan, or in other words forced MySQL to read the whole table to form the result. Now, we should immediately warn that full table scans are not always a problem. After all, sometimes just reading all of the data in a table is a viable query execution strategy, especially when the number of rows there is low. You can also expect to always see some amount of full table scans happening, as a lot of MySQL catalog tables are read that way. For example, just running a SHOW GLOBAL STATUS query will cause Select_scan to increase by two. Often, however, full table scans imply that there are queries hitting the database that perform suboptimally: either they are improperly written and don’t filter out data efficiently, or there are simply no indexes that can be used by the query. We give more information about query execution details and plans in “The EXPLAIN Statement”.

Select_full_join is similar to Select_scan, but counts number of queries that caused a full table scan on a referenced table within a JOIN query. Referenced table is the rightmost query in the JOIN condition — see “Joining Two Tables” for more information. Again, as was with Select_scan, it’s hard to say that Select_full_join is always bad. It’s common in large data warehouse systems, for example, to have compact dictionary tables, and reading those fully may not be a problem. Yet usually the high value of this status variable indicates presence of badly behaving queries.

Select_range is the number of queries that scanned data with some range condition (covered in “Selecting Rows with the WHERE Clause”). Usually this is not a problem at all. If the range condition is not possible to satisfy using an index, then Select_scan or Select_full_join grow alongside this status variable. Probably the only time when this counter’s value may indicate an issue is when you know that most of the queries running in the database in fact do not utilize ranges, and you see a growth in this counter. Without an associated growth of table scan counters, the issue is still benign.

Select_full_range_join is the combination of Select_range and Select_full_join. This variable holds a counter for queries that caused a range scan on referenced tables in JOIN queries.

So far, we’ve been counting individual queries, but MySQL also does a similar accounting for every row it reads from the storage engines! The family of status variables showing those counters are Handler_% variables. Simply put, every row MySQL reads increments some Handler_% variable. Combining this information with query types and query quality counters you’ve seen so far can tell you, for example, if full table scans that run in your database are a problem at all.

The first handler we’ll look at is the handler counting the number of rows read when a full or partial table scan is performed. Its name is Handler_read_rnd_next. Unlike the Select_% status variables, the Handler_% variables do not have a nice and easy to remember naming, so some memorization is necessary. High values in this status variable in general indicate that either a lot of tables are not indexed properly, or many queries do not utilize the existing indexes. Remember we mentioned explaining Select_scan that some full table scans are not problematic. To see whether that’s true or not, look at the ratio of Handler_read_rnd_next to other handlers. You want to see the value of that counter low. If your database returns on average a million rows per minute, then you probably want the number of rows returned by full scans to be in the thousands, not tens or hundreds of thousands.

Handler_read_rnd counts the number of rows usually read when the sorting of a result set was performed. High values may indicate presence of many full table scans and joins not using indexes. However, unlike Handler_read_rnd_next, this is not a sure sign of problems.

Handler_read_first counts how many times the first index entry was read. High value in this counter indicates that a lot of full index scans are occurring. This is better than full table scans, but still a problematic behavior. Likely, some of the queries are missing filters in their WHERE clauses. The value of this status variable should again be viewed in relation to the other handlers, as some full index scans are unavoidable.

Handler_read_key counts the number of rows read by an index. This is a read-related handler that you want to have in the top between other read handlers. In general, high number here means your queries are using indexes properly.

Note that handlers still can hide some issues. If a query only reads rows using indexes, but does so inefficiently, then Select_scan will not be increased, and Handler_read_key--our good read handler—will grow, but the end result is a slow query. We explain how to find specific slow queries in “The Slow Query Log”, but there’s also a special counter for them: Slow_queries. This status variable counts the queries that took longer than a value of Long_query_time to execute, regardless of whether the slow query log is enabled. You can gradually drop Long_query_time and see when Slow_queries start to approach the total number of queries executed by your server. This is a good way to assess how many queries in your system take, for example, longer than a second without actually turning on the slow query log, which has an overhead.

Not every query executed is read-only, and MySQL also counts the number of rows inserted, updated, or deleted under, respectively, Handler_insert, Handler_update, and Handler_delete status variables. Unlike with SELECT queries, it’s hard to make conclusions about the quality of your write statements based on the status variables alone. However, you can monitor these to see if, for example, your database clients started updating more rows. Without a change in the number of UPDATE statements (Com_update and Com_update_multi status variables) that may indicate a change in parameters passed to the same queries: wider ranges, more items in the IN clauses, and so on. Not a problem on its own, but it may be used during investigation of slowness to see whether more strain was put on the database.

Apart from INSERT statements, UPDATE, DELETE, and even INSERT SELECT statements have to look for rows to change. Thus, for example, a DELETE statement will increase read-related counters, and may result in an unexpected situation: no Select_scan growth, but a lot of Handler_read_rnd_next. Do not forget about this peculiarity if you see a discrepancy between status variables. Slow query log will include SELECT as well as DML statements.

Temporary objects

Sometimes, when queries execute, MySQL needs to create and use temporary objects, which may reside in memory or on disk. Examples of reasons for temporary object creation include use of the UNION clause, derived tables, common table expressions, and some variations of ORDER BY and GROUP BY clauses, among other things. We’ve been saying this about almost everything in this chapter, but temporary objects are not a problem: some number of them is unavoidable and actually desired. Yet they eat into your server’s resources: if temporary tables are small enough, they’ll be kept in memory and use it up, and if they grow large, MySQL will start offloading them to disk, using up both the disk space, and disk performance capacity.

MySQL maintains three status variables related to temporary objects created during query execution. Note that this doesn’t include temporary tables created explicitly through the CREATE TEMPORARY TABLE statement. Look for those under Com_create_table counter.

Created_tmp_tables is the number of temporary tables created implicitly by MySQL server while executing various queries. You cannot know why, or for which queries these were created, but every table will be accounted for here. Under a stable workload, you should see a uniform number of temporary tables created, as roughly the same queries run the same amount of times. Growth of this counter is usually associated with changing queries or their plans, for example due to growth of the database, and may be problematic. Creating temporary tables, even in memory, although useful, takes resources. You cannot completely avoid temporary tables, but you should check why their number is growing by performing a query audit with Slow Query Log, for example.

Created_tmp_disk_tables is the number of temporary tables that “spilled”, or were written to disk after their size surpassed the configured upper limits for in-memory temporary tables. With older MEMORY engine, the limit was controlled by tmp_table_size or max_heap_table_size. MySQL 8 moved by default to a new TempTable engine for the temporary tables, which, by default, does not spill to disk in the same way MEMORY tables did. If temptable_use_mmap variable is set to its default of ON, then TempTable temporary tables do not increase this variable even if they are written to disk.

Created_tmp_files is the number of temporary files created by MySQL. This is different from MEMORY engine temporary tables spilling to disk, but will account for TempTable tables being written out to disk. We understand that this may seem complicated, and it truly is, but major changes don’t usually come without some downsides.

Whatever configuration you’re using, sizing the temporary tables is important, as is monitoring the rate of their creation and spillage. If a workload creates a lot of temporary tables of roughly 32 MB in size, but upper limit for in-memory tables is 16 MB, then the server will see an increased rate of IO due to those tables being written out and read back to and from the disk. That’s fine for a server strapped for memory, but it’s a waste if you have memory available. Conversely, setting upper limits too high may result in the server swapping or outright crashing, as explained in “Memory”.

We’ve seen servers brought down by memory spikes when lots of simultaneously open connections all ran queries requiring temporary tables. We’ve also seen servers where the bulk of IO bandwidth was produced by temporary tables spilling to disk. As with most things related to operating databases, decision on temp table sizing is a balancing act. The three counters we’ve shown can help you make an informed decision.

InnoDB IO And Transaction Metrics

So far, we’ve been mostly talking about overall MySQL metrics, and ignoring the fact that there are things like transactions and locking. In this subsection we’ll take a look at some of the useful metrics the InnoDB storage engine exposes. Some of those metrics relate to how much data InnoDB reads and writes, and why. Some, however, can show important information on locking, which can be combined with MySQL-wide counters to get a solid grasp on current locks situation in a database.

InnoDB storage engine provides 61 status variables showing various information about the internal state of the engine. By looking at their change over time you can see how loaded InnoDB is and how much load on the OS it produces. Given that InnoDB is the default storage engine, that will likely be most of the load MySQL produces.

We perhaps should’ve put these in the section about query quality, but InnoDB maintains its own counters for the number of rows it read, inserted, updated, or deleted. The variables are, respectfully, Inndb_rows_read, Inndb_rows_inserted, Inndb_rows_updated, and Inndb_rows_deleted. Usually their values corresponds pretty well to the values of related Handler_% variables. If you primarily use InnoDB tables, it may be simpler to use the Innodb_rows_% counters instead of Handler_% ones to monitor relative load from queries expressed in number of rows processed.

Other important and useful status variables InnoDB provides show the amount of data that the storage engine reads and writes. In “Disk” we saw how to check and monitor overall and per-process IO utilization. InnoDB allows you to see exactly why it’s reading and writing data, and how much of it:

Innodb_data_read

The amount of data expressed in bytes read from disk since server startup. If you take measurements of this variable’s value over time, you can get to read bandwidth utilization in bytes/second. This metric is tightly related to InnoDB Buffer Pool sizing and its effectiveness, and we’ll get to that in a bit. All of this data can be assumed to be read from the data files to satisfy queries.

Innodb_data_written

The amount of data expressed in bytes written to disk since server startup. Same as the Innodb_data_read, but in the other direction. Usually, this value will amount for a large portion of the overall amount of write bandwidth MySQL will generate. Unlike with reading data, InnoDB writes data out in a variety of situations, thus there are additional variables specifying parts of this IO, as well as other sources of IO.

Innodb_os_log_written

The amount of data expressed in bytes written by InnoDB into its redo logs. This amount is also included in the Innodb_data_written, but it’s worth monitoring individually to see if your redo logs may need a size change. See “Adjusting the REDO log size” for more details.

Innodb_pages_written

The amount of data expressed in pages (16 KiB by default) written by InnoDB during its operation. This is the second half of the Innodb_data_written status variable. It’s useful to see the amount of non-redo IO that InnoDB generates.

Innodb_buffer_pool_pages_flushed

The amount of data expressed in pages written by InnoDB due to flushing. Unlike the writes covered by two previous counters, writes caused by flushing do not happen immediately after an actual write is performed. Flushing is a complex background operation, the details of which are beyond the scope of our book. However, you should at least know that flushing exists, and that it generates IO independent of other counters.

By combining Innodb_data_written and Innodb_buffer_pool_pages_flushed, you should be able to come up with a pretty accurate figure of the disk bandwidth utilized by the InnoDB and MySQL server. Adding Innodb_data_read completes the IO profile of InnoDB. MySQL doesn’t only use InnoDB, and there can be IO from other parts of the system, like temporary tables spilling to disk that we discussed earlier. Yet often InnoDB IO matches that of MySQL server observed from the OS.

One use of this information is to see how close your MySQL server is to hitting the limits of your storage system’s performance capacity. This is especially important in the cloud where storage often has strict limits. During the incidents related to database performance, you can check the IO-related counters to see if MySQL is writing or reading more, perhaps indicating increased load, or instead doing fewer actual IO operations. The latter may mean that MySQL is currently limited by some other resource like CPU, or suffers from other issues like locking. Unfortunately, decreased IO may also mean that the storage is having issues.

There are some status variables in InnoDB that may help to find issues with storage or its performance. Those variables are: Innodb_data_pending_fsyncs, Innodb_data_pending_reads, Innodb_data_pending_writes, Innodb_os_log_pending_fsyncs, and Innodb_os_log_pending_writes. You can expect to see some amount of pending data reads and writes, though as always it’s helpful to look at the trends and previous data. The most important of all of these is Innodb_os_log_pending_fsyncs. Redo logs are synced often, and performance of the syncing operation is extremely important for the overall performance and transaction throughput of InnoDB. Unlike many other status variables, all of these are gauges, meaning that their value goes up from 0 and back. You should sample these variables and look at how often there are pending operations, in particular for the redo log sync. Even small increases in Innodb_os_log_pending_fsyncs may indicate serious issues with disk: either you’re running out of performance capacity, or there are hardware issues.

While writing about the Innodb_data_read we mentioned that the amount of data that InnoDB reads is related to its buffer pool size and usage. Let’s elaborate on that. InnoDB caches pages it read from disk inside its buffer pool. The larger the buffer pool, the more pages will be there, and the less frequently pages will have to be read from disk. We talk about that in “Adjusting the Buffer Pool size”. Here, while discussing monitoring, let’s see how to monitor the effectiveness of the buffer pool. That’s easily done with just two status variables:

Innodb_buffer_pool_read_requests

MySQL documentation defines this as “the number of logical read requests”. Put simply, this is the amount of pages that various operations within InnoDB wanted to read from the buffer pool. Usually most of the pages are read due to query activity.

Innodb_buffer_pool_reads

This is the number of pages that InnoDB had to read from disk to satisfy the read requests by queries or other operations. This counter is usually smaller or equal to Innodb_buffer_pool_read_requests even in the very worst case with a completely empty (or “cold”) buffer pool, because reads from disk are performed to satisfy the read requests.

Under normal conditions even with the small buffer pool, you won’t get a 1:1 ratio of these variables. It will be possible to satisfy at least some reads from the buffer pool. Ideally you should try keep the amount of disk reads to minimum. That may not always be possible, especially if database size is much larger than server memory. The following formula is frequently used to get a numeric representation of the currently-sized buffer pool’s efficiency: Innodb_buffer_pool_read_requests / (Innodb_buffer_pool_read_requests + Innodb_buffer_pool_reads) * 100.

For example, on a MySQL server which shows Innodb_buffer_pool_read_requests of 33809532 and Innodb_buffer_pool_reads of 20751, the calculation will show 99.94% buffer pool efficiency. Even if we’d like to get the amount of disk reads lower, we may hit diminishing returns by increasing the buffer pool in this situation.

Finally, to close the InnoDB part, we move on to transaction and locking information. A lot of information on both topics was given in Chapter 6, so here we’ll do a brief overview of the related status variables:

Transaction-related command counters

BEGIN, COMMIT, and ROLLBACK are all special MySQL commands. Thus, MySQL will count the number of times they were executed within the Com_% status variables: Com_begin, Com_commit, and Com_rollback. By looking at these counters you can see how many transactions are started explicitly, and either committed or rolled back.

Locking-related status variables

You know by now that InnoDB provides locking with row-level granularity. This is a huge improvement over MyISAM’s table-level locking, as the impact of each individual lock is minimized. Still, there can be impact if transactions are waiting for each other even for a short time.

InnoDB provides status variables that let you see just how many locks are being created and lock waits happening. Those can be found under Innodb_row_lock_% names. Let’s look at them:

Innodb_row_lock_current_waits is how many transactions operating on InnoDB tables are currently waiting on a lock to be released by some other transactions. The value of this variable will go up from 0 when there are blocked sessions, then get back to 0 as soon as locking is resolved.

Innodb_row_lock_waits is how many times since server startup transactions on InnoDB tables waited for row-level locks. Unlike the current waits, this variable is a counter and will continualy increase until MySQL server is restarted.

Innodb_row_lock_time shows total time in milliseconds spent by sessions trying to acquire locks on InnoDB tables.

Innodb_row_lock_time_avg shows an average time in milliseconds that it takes for a session to acquire a row-level lock on an InnoDB table. You can arrive at the same value by dividing Innodb_row_lock_time by Innodb_row_lock_waits. This value may go up and down depending on how many lock waits are encountered and how much accumulated lock time grows.

Innodb_row_lock_time_max shows the maximum time in milliseconds it took to obtain a lock on an InnoDB table. This value will only go up if the record is broken by some other unfortunate transaction.

Here’s an example from MySQL server running moderate read-write load:

+-------------------------------+--------+

| Variable_name                 | Value  |

+-------------------------------+--------+

| Innodb_row_lock_current_waits | 0      |

| Innodb_row_lock_time          | 367465 |

| Innodb_row_lock_time_avg      | 165    |

| Innodb_row_lock_time_max      | 51056  |

| Innodb_row_lock_waits         | 2226   |

+-------------------------------+--------+

There were 2,226 individual transactions waiting for locks, and it took 367,465 milliseconds to obtain all of those locks, with an average lock acquisition duration of 165 milliseconds and the maximum duration of just over 51 seconds. There are currently no sessions waiting for locks. On its own this information doesn’t tell us much: it’s neither a lot nor a little. However, we know that at the same time more than 100,000 transactions were executed by this MySQL server. Resulting locking metric values are more than reasonable for the level of concurrency.

Locking issues are a frequent source of headache for database administrators and application developers alike. While these metrics, as everything we’ve discussed so far, are aggregated across every session running, deviation from the normal values may help you in pinning down some of the issues. To find and investigate indivindual locking situations, you may use the InnoDB status report — see “InnoDB Engine Status Report” for more details.

The Slow Query Log

In “Query types and quality” we showed how to look for tell-tale signs of unoptimized queries in MySQL. However, that’s not enough to start optimizing those queries. We need specific examples. There are few ways to do that, but probably the most robust one is using the Slow Query Log facility. The Slow Query Log is exactly what it sounds like: a special text log where MySQL puts information about slow queries. Just how slow those queries should be is controllable, and you can go as far as logging every query.

To enable the Slow Query Log, a slow_query_log system variable must be set to ON from its default of OFF. By default, when the Slow Query Log is enabled MySQL will log queries taking longer than 10 seconds. That’s configurable by changing the long_query_time variable, which has a minimum of 0, meaning every query executed by server will be logged. Log location is controlled by the slow_query_log_file variable, which defaults to a value of _host_name_-slow.log. When the path to the slow log is relative, meaning it doesn’t start from / on Linux or, for example, C:\ on Windows, then this file will be located in the MySQL data directory.

You can also tell MySQL to log queries not using indexes regardless of the time they take to execute. To do so, the log_queries_not_using_indexes variable has to be set to ON. By default, DDL and administrative statements are not logged, but this behavior can be changed by setting log_slow_admin_statements to ON.

MariaDB and Percona Server expand functionality of the Slow Query Log by adding filtering capabilites, rate limiting, as well as enhanced verbosity. If you’re using those products, it’s worth reading their documentation on the subject to see if you can utilize the enhanced Slow Query Log.

Here’s an example from a record in the Slow Query Log showing a SELECT statement taking longer than a configured long_query_time value of 1:

# Time: 2021-05-29T17:21:12.433992Z

# User@Host: root[root] @ localhost []  Id:    11

# Query_time: 1.877495  Lock_time: 0.000823 Rows_sent: 9  Rows_examined: 3473725

use employees;

SET timestamp=1622308870;

SELECT

dpt.dept_name

, emp.emp_no

, emp.first_name

, emp.last_name

, sal.salary

FROM

departments dpt

JOIN dept_emp ON dpt.dept_no = dept_emp.dept_no

JOIN employees emp ON dept_emp.emp_no = emp.emp_no

JOIN salaries sal ON emp.emp_no = sal.emp_no

JOIN (SELECT dept_emp.dept_no, MAX(sal.salary) maxsal

FROM dept_emp JOIN salaries sal

ON dept_emp.emp_no = sal.emp_no

WHERE

sal.from_date < now()

AND sal.to_date > now()

GROUP BY dept_no

) largest_sal_by_dept ON dept_emp.dept_no = largest_sal_by_dept.dept_no

AND sal.salary = largest_sal_by_dept.maxsal;

By analyzing this output you can immediately start making conclusions about this query. This is much more descriptive than looking at server-wide metrics. For example, we can see that this query was executed at 17:21:12 UTC by user root@localhost in employees database, took 1.88 seconds to run, produced 9 rows, but had to scan 3473725 rows to produce that result. That information on its own can tell you a whole lot about the query, if not now while learning MySQL, but for sure after looking at it for quite some time. However, you now also have the complete query text, which you can turn into execution plan information to see exactly how MySQL is executing this query. You can find more details on that process in “The EXPLAIN Statement”.

If the long_query_time is set low, the Slow Query Log may grow large. Sometimes, it may be reasonable to do so, but reading through the resulting log may be nearly impossible if the number of queries is high. There’s a tool called mysqldumpslow that solves this problem. It takes path to the Slow Query Log file as an argument and will summarize queries from that file, and sort (by default by time) them. In the following example, the command is run such that it will show two top queries sorted by the number of rows returned:

$ mysqldumpslow -s r -t 2 /var/lib/mysql/mysqldb1-slow.log

Reading mysql slow query log from /var/lib/mysql/mysqldb1-slow.log

Count: 2805  Time=0.00s (0s)  Lock=0.00s (0s)  Rows=100.0 (280500), sbuser[sbuser]@localhost

SELECT c FROM sbtest1 WHERE id BETWEEN N AND N

Count: 2760  Time=0.00s (0s)  Lock=0.00s (0s)  Rows=100.0 (276000), sbuser[sbuser]@localhost

SELECT c FROM sbtest1 WHERE id BETWEEN N AND N ORDER BY c

You can see that just these two queries alone were recorded in the Slow Query Log 5565 times. Imagine reading that without help! Another tool that can help with summarizing the Slow Query Log information is pt-query-digest from Percona-Toolkit. The tool is a bit more advanced and difficult to use than mysqldumpslow, but gives a lot of information and has many features. A report it produces starts with a summary:

$ pt-query-digest /var/lib/mysql/mysqldb1-slow.log

# 7.4s user time, 60ms system time, 41.96M rss, 258.35M vsz

# Current date: Sat May 29 22:36:47 2021

# Hostname: mysqldb1

# Files: /var/lib/mysql/mysqldb1-slow.log

# Overall: 109.42k total, 15 unique, 7.29k QPS, 1.18x concurrency ________

# Time range: 2021-05-29T19:28:57 to 2021-05-29T19:29:12

# Attribute          total     min     max     avg     95%  stddev  median

# ============     ======= ======= ======= ======= ======= ======= =======

# Exec time            18s     1us    10ms   161us     1ms   462us    36us

# Lock time             2s       0     7ms    16us    14us   106us     5us

# Rows sent          1.62M       0     100   15.54   97.36   34.53    0.99

# Rows examine       3.20M       0     200   30.63  192.76   61.50    0.99

# Query size         5.84M       5     245   55.93  151.03   50.37   36.69

# Profile

# Rank Query ID                            Response time Calls R/Call V/M

# ==== =================================== ============= ===== ====== ====

#    1 0xFFFCA4D67EA0A788813031B8BBC3B329  11.1853 63.1%  5467 0.0020  0.00 COMMIT

#    2 0xB2249CB854EE3C2AD30AD7E3079ABCE7   1.5985  9.0%  5467 0.0003  0.00 UPDATE sbtest?

#    3 0xE81D0B3DB4FB31BC558CAEF5F387E929   1.5600  8.8% 54670 0.0000  0.00 SELECT sbtest?

#    4 0xF0C5AE75A52E847D737F39F04B198EF6   0.8853  5.0%  5467 0.0002  0.00 SELECT sbtest?

#    5 0x9934EF6887CC7A6384D1DEE77FA8D4C3   0.5959  3.4%  5467 0.0001  0.00 SELECT sbtest?

#    6 0xA729E7889F57828D3821AE1F716D5205   0.4748  2.7%  5467 0.0001  0.00 SELECT sbtest?

#    7 0xFF7C69F51BBD3A736EEB1BFDCCF4EBCD   0.4511  2.5%  5467 0.0001  0.00 SELECT sbtest?

#    8 0x6C545CFB55365122F1256A27240AEFC7   0.3092  1.7%  5467 0.0001  0.00 INSERT sbtest?

# MISC 0xMISC                               0.6629  3.7% 16482 0.0000   0.0 <7 ITEMS>

And each query is then summarized as follows:

# Query 2: 546.70 QPS, 0.16x concurrency, ID 0xB2249CB854EE3C2AD30AD7E3079ABCE7 at byte 1436377

# Scores: V/M = 0.00

# Time range: 2021-05-29T19:29:02 to 2021-05-29T19:29:12

# Attribute    pct   total     min     max     avg     95%  stddev  median

# ============ === ======= ======= ======= ======= ======= ======= =======

# Count          4    5467

# Exec time      9      2s    54us     7ms   292us     1ms   446us    93us

# Lock time     61      1s     7us     7ms   203us     1ms   437us     9us

# Rows sent      0       0       0       0       0       0       0       0

# Rows examine   0   5.34k       1       1       1       1       0       1

# Query size     3 213.55k      40      40      40      40       0      40

# String:

# Databases    sysbench

# Hosts        localhost

# Users        sbuser

# Query_time distribution

#   1us

#  10us  ################################################################

# 100us  ###############################################

#   1ms  ############

#  10ms

# 100ms

#    1s

#  10s+

# Tables

#    SHOW TABLE STATUS FROM `sysbench` LIKE 'sbtest2'\G

#    SHOW CREATE TABLE `sysbench`.`sbtest2`\G

UPDATE sbtest2 SET k=k+1 WHERE id=497658\G

# Converted for EXPLAIN

# EXPLAIN /*!50100 PARTITIONS*/

select  k=k+1 from sbtest2 where  id=497658\G

That’s a lot of valuable information in a dense format. One of the distinguishing features of this output is the query duration distribution visualization, which allows you to quickly see whether a query has parameter-dependent performnace issues. Explaining every feature of pt-query-digest would take another chapter, and it’s an advanced tool, so we leave this for you to try once you’re done learning MySQL.

The Slow Query Log is a powerful tool that allows you to get a very detailed view of queries executed by MySQL server. We recommend using the Slow Query Log like this:

- Set long_query_time to a value large enough that covers most of the queries normally running in your system, but small enough that you catch outliers. For example, in an OLTP system, where most of the queries are expected to complete in milliseconds, a value of 0.5 may be reasonable, only catching relatively slow queries. On the other hand, if your system has queries running in minutes, then long_query_time should be set accordingly.

- Logging to slow log has some performance cost, and you should avoid logging more queries than you need. If you have the slow log enabled, make sure you adjust the long_query_time if you find the log too noisy.

- Sometimes, you may want to perform a “query audit”, whereas you temporarily (for a few minutes) set long_query_time to 0 to catch every query. This is a good way to get a snapshot of your database load. Such snapshots may be saved and compared later. However, we recommend strongly against setting long_query_time too low.

- If you have the Slow Query Log set up, we recommend running mysqldumpslow or pt-query-digest, or a similar tool on your slow logs periodically to see if there are new queries appearing or if existing ones start behaving worse than usually.

InnoDB Engine Status Report

InnoDB storage engine has a built-in report that exposes deep technical details on the current state of the engine. A lot can be said about InnoDB load and performance from reading just this one report, ideally sampled over time. Reading InnoDB status report is an advanced topic that requires more learning than we can convey in our book, and also a lot of practice. Still, we believe you should know that this report exists, and we’ll give you some hints as to what to look for there.

To view the report, you only need to run a single command. We recommend using the vertical result display:

mysql> SHOW ENGINE INNODB STATUS\G

*************************** 1. row ***************************

Type: InnoDB

Name:

Status:

=====================================

2021-05-31 12:21:05 139908633830976 INNODB MONITOR OUTPUT

=====================================

Per second averages calculated from the last 35 seconds

-----------------

BACKGROUND THREAD

-----------------

srv_master_thread loops: 121 srv_active, 0 srv_shutdown, 69961 srv_idle

...

--------------

ROW OPERATIONS

--------------

0 queries inside InnoDB, 0 queries in queue

2 read views open inside InnoDB

Process ID=55171, Main thread ID=139908126139968 , state=sleeping

Number of rows inserted 2946375, updated 87845, deleted 46063, read 88688110

572.50 inserts/s, 1145.00 updates/s, 572.50 deletes/s, 236429.64 reads/s

Number of system rows inserted 109, updated 367, deleted 60, read 13218

0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s

----------------------------

END OF INNODB MONITOR OUTPUT

============================

The output, which we truncated above, is presented broken down in sections. At first, this output may seem intimidating, but over time you will come to appreciate the details. For now, we’ll show few sections that we believe provide information that will benefit operators of any experience level.

Transactions

This section provides information about the transactions of every session, including duration, current query, number of locks held, information on lock waits. You can also find some data there on transaction visibility, but that’s rarely required. Usually, you want to look at the transactions sections if you want to see the current state of transactions active within InnoDB. A sample record from this section looks like this:

---TRANSACTION 252288, ACTIVE (PREPARED) 0 sec

5 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 4

MySQL thread id 82, OS thread handle 139908634125888,...

...query id 925076 localhost sbuser waiting for handler commit

COMMIT

Trx read view will not see trx with id >= 252287, sees < 252285

This tells us that a transaction is currently waiting for COMMIT to finish, and it holds three row locks, and was pretty fast, likely to finish in under a second. Sometimes, you will see long transactions here: you should avoid that. InnoDB does not handle long transaction well, and you will have a performance impact even from an idle transaction staying open for too long.

This section will also show you information on the current locking behavior if there are transactions waiting to obtain locks. See this example:

---TRANSACTION 414311, ACTIVE 4 sec starting index read

mysql tables in use 1, locked 1

LOCK WAIT 2 lock struct(s), heap size 1136, 1 row lock(s)

MySQL thread id 84, OS thread handle 139908634125888,...

...query id 2545483 localhost sbuser updating

UPDATE sbtest1 SET k=k+1 WHERE id=347110

Trx read view will not see trx with id >= 414310, sees < 413897

------- TRX HAS BEEN WAITING 4 SEC FOR THIS LOCK TO BE GRANTED:

RECORD LOCKS space id 333 page no 4787 n bits 144 index PRIMARY of...

...table `sysbench`.`sbtest1` trx id 414311 lock_mode X locks...

...rec but not gap waiting

Record lock, heap no 33

------------------

Unfortunately, InnoDB status doesn’t point at the lock holder directly, so you’ll need to look for blocking transactions separately. Some information on that is available in Chapter 6. Usually, if you see one transaction active, while others are waiting, it’s a good indication that it’s that active transaction that’s holding the locks.

File I/O

Information on the current IO operations, as well as aggregated summaries over time. We discussed this in more detail in the recipes section, but this is an additional way of checking whether InnoDB has pending data and log operations.

Buffer pool and memory

Under this section InnoDB prints information about its buffer pool and memory usage. If you have multiple buffer pool instances configured, then this section will show both totals and per-instance breakdowns. There’s a lot of info, but a highlight that we can make is that there’s an info on the buffer pool size and internal state of the buffer pool:

Total large memory allocated 274071552

Dictionary memory allocated 1377188

Buffer pool size   16384

Free buffers       1027

Database pages     14657

Old database pages 5390

Modified db pages  4168

These are also exposed as Innodb_buffer_pool_% variables.

Semaphores

This section includes information on internal InnoDB semaphores: synchronization primitives. In some basic terms, semaphores are special internal in-memory structures that allow multiple threads to operate without interfering with each other. You will rarely see anything of value in this section unless there’s a semaphore contention on your system. Usually that happens when InnoDB is put under extreme load, thus every operation takes longer and there are more chances to see active semaphore waits in the status output.

Investigation Methods

Having so many available metrics that need to be monitored, checked, and understood may cause your head to spin. You may notice that we haven’t defined a single metric that has a definite range from good to bad. Is the locking bad? Could be, but it’s also expected and normal. The same can be said about almost every aspect of MySQL, with the exception of the server availability.

This problem is not unique to MySQL server monitoring, and in fact it’s common for any complex system. There are a lot of metrics, complicated dependencies, and almost no strictly definable rules for whether something is good or bad. To solve this problem, we need some approach, some methodology that we can apply to abundant data to quickly and easily come to conclusions about current system performance.

Luckily, such methodologies already exist. In this section, we will briefly describe two of them and give ideas on how to apply them to monitoring MySQL and OS metrics.

USE Method

The USE method — which stands for Utilization, Saturation, and Errors, popularized by Brendan Gregg — is a general-purpose methodology that can be applied to any system. Though better suited for resources with well-defined performance characteristics like CPU or disk, it can also be applied to some parts of MySQL.

USE is best used by creating a checklist for an individual part of the system. First, we need to define what is it about our resource that is a Utilization, a Saturation, and finally Errors. Some example checklists for Linux can be found on the USE homepage: “USE Method: Linux Performance Checklist”.

We will take the disk-related checklist as an example.

Utilization

For a disk subsystem, as we’ve laid out in “Disk”, utilization can be anything (or all) of the following:

- Overall storage bandwidth utilization

- Per-process storage bandwidth utilization

- Current IOPS (where upper IOPS limit for the device is known)

Saturation

While talking about the iostat output and disk metrics, we have already mentioned saturation. It’s a measure of work the resource cannot process, and that is usually queued. For disk, this is expressed as the IO request queue size (avgqu-sz in iostat output). On most systems, values >1 mean there’s a saturation of the storage, and thus some requests will be queued and thus waste time doing nothing.

Errors

Self-explanatory, amount of errors that a resource returns to its users. For disk devices, this may mean IO errors due to hardware degradation.

The problem with the USE method is that it’s difficult to apply to a complex system as a whole. Take MySQL for example: what should be the utilization, what’s the measure of saturation, and what are errors? Some attempts can be made to apply USE to MySQL as a whole, but it’s much better suited to be applied to some isolated parts of the system.

USE method for client connections

A good fit for USE method application is making sense of the client connection metrics we discussed in “Basic Monitoring Recipes”.

Utilization is both the Threads_connected and Threads_running.

Saturation can be defined arbitrarily based on Threads_running. For example, in an OLTP system that is read-heavy and doesn’t generate a lot of IO, a saturation point for Threads_running will likely be around the number of available CPU cores. In a system which is mostly IO-bound, a starting point could be twice the available CPU cores, but it’s much better to find what number of concurrently running threads start to saturate the storage subsystem.

Errors is the Aborted_clients and Aborted_connects. Once the value of Threads_connected status variable becomes equal to max_connections system variable, creation of new connections will be declined, and clients will get errors. Existing clients may be terminating their connections uncleanly without waiting for MySQL to respond.

USE method for transactions and locking

Another example of how USE can be applied is to look at the InnoDB locking-related metrics relative to the number of queries processed by the system.

Utilization is the synthetic QPS metric we defined earlier.

Saturation can be the fact that there’s some amount of lock waits occurring. Remember, by USE definition, saturation is a measure of work that cannot be performed and has to wait in the queue. There’s no queue, but transactions are waiting for locks to be acquired. Alternatively, the regular amount of lock waits can be multiplied twofold to construct an arbitrary threshold, or better yet, some experiments can be performed to find a number of lock waits relative to QPS that result in lock timeouts occurring.

Errors is the number of times sessions timed out waiting for locks or were terminated to resolve a deadlock situation. Unlike previously mentioned metrics, these two are not exposed as global status variables, but can instead be found in the information_schema.innodb_metrics table under lock_deadlocks (number of deadlocks registered) and lock_timeouts (number of times lock wait timed out).

Looking at the rate of errors from monitoring metrics alone can be difficult, so frequently just the US part of USE is used. As you can see, this method allows us to look at the system from a pre-defined, checklist, point of view. Instead of analyzing every possible metric when there’s an incident, an existing checklist can be reviewed saving time and effort.

RED Method

RED (Rate, Errors, and Duration) is a methodology similar to USE, but which was created to address shortcomings of USE. Specifically, RED is easier to apply to complex systems and services.

The logical application of RED to MySQL is by looking at query performance:

- Rate: QPS of the database

- Errors: Number or rate of errors and failing queries

- Duration: Average query latency

One problem with RED in the context of this book and learning MySQL in general is that applying this method requires monitoring data that is more difficult to obtain than reading status variables. Not every existing monitoring solution for MySQL can provide the necessary data, either, though you can look at the example shown by Peter Zaitsev in his blog post RED Method for MySQL Performance Analyses. One way to apply RED to MySQL or any other database system is by looking at the metrics from the application side instead of from the database side. Multiple monitoring solutions allow you to instrument (manually or automatically) your applications to capture data, such as number of queries, rate of failures, and query latency. Just what’s needed for RED!

You can, and should, use RED and USE together. In the article explaining RED — The RED Method: How to Instrument Your Services — its author Tom Wilkie mentions that “It’s really just two different views on the same system.”

One perhaps unexpected benefit of applying RED, USE, or any other method is that you do that before an incident happens. Thus you are forced to understand what it is you monitor and measure, and how that relates to what actually matters for your system and its users.

MySQL Monitoring Tools

Throughout this chapter we’ve kept on talking about metrics and monitoring methodologies, but we haven’t mentioned a single actual tool that can turn those metrics into dashboards or alerts. In other words, we haven’t talked about the actual monitoring tools and systems. The reason for that is simple enough: we believe that in the beginning it’s much more important for you to know what to monitor and why, rather than how. If we were to concentrate on the how we could spend this whole chapter talking about peculiarities and differences of monitoring tools. The second reason is that MySQL and OS metrics don’t change often, but if you’re reading this book in 2025, our choice of monitoring tools may already seem antiquated.

Nevertheless, we put together a list of notable popular open source monitoring tools that can be used to monitor MySQL availability and performance. We’re not going to go too deep into the specifics of their setup or configuration or into comparing them all together. Look at this as a list of suggestions. We cannot also list every possible monitoring solution available: brief research shows us that almost anything that is a “monitoring tool” or a “monitoring system” can monitor MySQL in some way. We’re also not covering non-open-source and non-free monitoring solutions, with the one exception of the Oracle Enterprise Monitor. We hold nothing against such systems in general, and a lot of them are great. Most of them, though, have excellent documentation and support available, so you should be able to get familiar with them quickly.

Following monitoring systems will be mentioned here:

- Prometheus

- InfluxDB and TICK stack

- Zabbix

- Nagios Core

- Percona Monitoring and Management

- Oracle Enterprise Monitor

Prometheus

Born out of Google’s internal monitoring system Borgmon, Prometheus is an extremely popular general-purpose monitoring and alerting system. At the core of it is a time-series database and a data-gathering engine based around a pull model. What that means is that it’s Prometheus server that actively gathers data from its monitoring targets.

Actual data gathering from Prometheus targets is performed by special programs called exporters. Exporters are purpose-built: there’s a dedicated MySQL exporter, a PostgreSQL exporter, a basic OS metrics or node exporter. What these programs do is collect metrics from the system they are written to monitor and present those metrics in a format suitable for Prometheus server to consume.

MySQL monitoring with Prometheus is done by running a mysqld_exporter program. Like most parts of Prometheus ecosystem, it’s written in Go and is available for Linux, Windows, and many other operating systems, making it a good choice for a heterogenous environment.

MySQL exporter gathers all the metrics we’ve covered in this chapter (and many more!), and since it actively tries to get info from MySQL, it can also report on MySQL server’s availability. In addition to standard metrics, it is possible to supply custom queries that exporter will execute and turn their results into additional metrics.

Prometheus offers only very basic visualization, so Grafana analytics and data visualization web application is usually added to the setup.

InfluxDB and TICK stack

Built around the InfluxDB time-series database, TICK, standing for Telegraf, InfluxDB, Chronograf, and Kapacitor, is a complete time-series and monitoring platform. Comparing to Prometheus, Telegraf takes the place of exporters, being a unified program that is capable of monitoring multitude of targets. Unlike exporters, Telegraf actively pushes data to InfluxDB instead of being pulled by the server. Chronograf is an administrative and data interface. Kapacitor is a data processing and alerting engine.

Where you had to install a dedicated exporter for MySQL, Telegraf is extended using plugins. MySQL plugin is a part of standard bundle, and provides a detailed overview of MySQL database metrics. Unfortunately, it is not capable of running arbitrary queries, so extensibility is limited. As a workaround, the exec plugin can be used. Telegraf is also a multi-platform program, and does support Windows among other OSes.

TICK stack is frequently used partially, with Grafana added to InfluxDB and Telegraf.

A common thread between Prometheus and TICK is that they are a collection of building blocks allowing you to build your own monitoring. Neither of them offers any out-of-the-box recipes for dashboards, alerting, and so forth. They are powerful, but they may require some getting used to. Apart from that, they are very automation- and infrastructure-as-code-oriented. Prometheus especially, but TICK as well, provides minimal GUI, and was not initially conceived for data exploration and visualization. It’s monitoring as in reacting to metric value changes by alerting, not in visually inspecting various metrics. Adding Grafana to the equation, especially with either home-brewed or community dashboards for MySQL, makes visual inspection possible. Still, most of the configuration and setup will not be done in a GUI.

Both of these systems saw an influx in popularity towards mid and late 2010s with the shift to running a multitude of small servers compared to running a few large servers. That shift required some change in monitoring approaches, and these systems became almost de facto standard monitoring for a lot of companies.

Next in our review are a couple of more “old school” monitoring solutions: Zabbix and Nagios (Core).

Zabbix

A completely free and open source monitoring system first released in 2001, Zabbix is proven and powerful. It supports a wide range of monitored targets and advanced auto-discovery and alerting capabilities.

MySQL monitoring with Zabbix can be done by using plugins or with the official MySQL templates. Metrics coverage is quite good, with every recipe we defined available. However, both mysqld_exporter and Telegraf offer more data. The standard MySQL metrics Zabbix collects are sufficient to set up a basic MySQL monitoring, but for deeper insight you will need to go custom or use some of the community templates and plugins.

Zabbix agent is cross-platform, so you can monitor MySQL running on almost any OS.

While Zabbix offers quite powerful alerting, its visualization capabilities may feel slightly dated. It is possible to set up custom dashboards based on MySQL data, and it’s also possible to use Zabbix as a data source for Grafana.

Zabbix is fully configurable through its GUI. The commercial offering includes various levels of support and consulting.

Nagios Core

Like Zabbix, Nagios is a veteran monitoring system, with its first release seeing light in 2002. Unlike other systems we’ve seen so far, Nagios is an “open core” system. The Nagios Core distribution is free and open source, but there’s also a commercial Nagios system.

Monitoring of MySQL is set up by use of plugins. They should provide enough data to set up basic monitoring similar to that of the official Zabbix templates. It is possible to extend the collected metrics if needed.

Alerting, visualizations, and configuration are similar to Zabbix. One notable feature of Nagios is that at its peak of popularity it was forked multiple times. Some of the most popular Nagios forks are Icinga and Shinken. Check_MK was also initially a Nagios extension that eventually moved on to become its own commercial product.

Both Nagios and its forks and Zabbix can and are successfully used by multiple companies to monitor MySQL. Even though they may feel outdated in their architecture and data representation, they can get the job done. Their biggest problem is that the standard data they collect may feel limited compared with alternatives, and you’ll need to use community plugins and extensions. Percona used to maintain Monitoring Plugins for Nagios, as well as for Zabbix, but has deprecated them concentrating on its own monitoring offering: Percona Monitoring and Management.

All systems we’ve covered so far have one thing in common: they are general-purpose monitoring solutions, not tailor-made for database monitoring. It’s their power, and also their weakness. When it comes to monitoring and investigating deep database internals, you’ll often be forced to manually extend those systems’ capabilities. One feature, for example, that none of them have, for example, is storing individual query execution statistics. Technically, it’s possible to add that feature, but it may be cumbersome and problematic.

We will finish this section by looking at two database-oriented monitoring solutions. MySQL Enterprise Monitor from Oracle and Percona Monitoring and Management. They are similar in functionality provided, and both are big improvements over nonspecialized monitoring systems.

MySQL Enterprise Monitor

Part of the MySQL Enterprise Edition, Enterprise Monitor is a complete monitoring and management platform for MySQL databases.

In terms of monitoring, MySQL Enterprise Monitor extends usual metrics gathered by monitoring systems by adding details on MySQL memory utilization, per-file IO details, and a wide variety of dashboards based on InnoDB’s status variables. The data is taken from MySQL itself without any agents involved. Theoretically, all the same data can be gathered and visualized by any other monitoring system, but here it’s tightly packed with thought-out dashboards and categories.

Enterprise Monitor includes the Events subsystem, which is a set of predefined alerts. Adding to database-specific features, the Enterprise Monitor adds replication topology overview for regular and multi-source replication, Group Replication, and NDB cluster. Another feature is monitoring of the backup execution statuses (for backups done with the MySQL Enterprise Backup).

We mentioned that individual query execution statistics and query history are something that’s usually missing in the general-purpose monitoring systems. MySQL Enterprise Monitor includes its Query Analyzer system that provides an insight into history of queries executed over time, as well as statistics gathered about the query. It’s possible to view information like an average count of queries read and returned, duration distribution, and even see the execution plan of the query.

Enterprise Monitor is a good database monitoring system. Its biggest downside, really, is that it’s only available in the Enterprise Edition of MySQL. Unfortunately, most of MySQL installations cannot benefit from the Enterprise Monitor and the level of insight it provides into the database and OS metrics. It’s also not suitable for monitoring anything apart from MySQL, queries it’s executing and the OS it’s running on, and MySQL monitoring is limited in scope to Oracle’s products.

There’s a 30-day trial of MySQL Enterprise Edition available, which includes the Enterprise Monitor, and Oracle also maitains a list of visual demos of the system on its site: MySQL Enterprise Edition: Demos.

Percona Monitoring and Management

Percona’s monitoring solution, PMM is similar in functionality to Oracle’s Enterprise Monitor, but is fully free and open source. Intended to be a “single pane of glass,” it tries to provide deep insight into MySQL and OS performance, and can also be used to monitor MongoDB and PostgreSQL databases.

PMM is built on top of existing open source components like the already-reviewed Prometheus and its exporters, and Grafana. Percona maintains forks of the database exporters it uses, including MySQL’s one, and adds functionality and metrics that were lacking in the original versions. In addition to that, PMM hides complexity usually associated with deploying and configuring those tools, and instead provides its own bundled package and configuration interface.

Like Enterprise Monitor, PMM offers a selection of dashboards, visualizing pretty much every aspect of MySQL’s and InnoDB’s operation, as well as giving a lot of details on the underlying OS state. This is extended to include technologies like PXC/Galera, discussed in Chapter 13, ProxySQL, discussed in Chapter 15. As PMM uses Prometheus and exporters, it’s possible to extend the range of monitored databases by adding external exporters. In addition to that, PMM supports DBaaS systems like RDS and CloudSQL.

PMM ships with a custom application called Query Analytics (QAN), which is a query monitoring and metrics system. Like Query Analyzer of the Enterprise Monitor, QAN shows the overall history of queries executed in a given system, as well as information about the individual query. That includes a history of the number of executions of that query over time, rows read and sent, locking, and temporary tables created, among few other things. QAN allows you to view the query plan, and structures of the involved tables.

The Management part of PMM for now exists only in its name, as at this moment it is purely a monitoring system. PMM supports alerting through Prometheus’ standard AlertManager, or through the use of internal templates.

One significant problem of PMM is that out of the box it only supports targets running on Linux. Since Prometheus exporters are cross-platform, you can add Windows (or other OS) targets to PMM, but you won’t be able to utilize some of the benefits of PMM like simplified exporter configuration and bundled installation of the client software.

Both authors of this book are currently employed by Percona, so you can dismiss our description of PMM as advertisement. However, we’ve tried to give a fair overview of a few monitoring systems, and we don’t try to say that PMM is perfect. If your company is already using an Enterprise version of MySQL, then you should absolutely first see what MySQL Enterprise Monitor has to offer.

Before we close this section, we want to note that in many cases the actual monitoring system doesn’t matter a lot. Every system we’ve mentioned provides MySQL availability monitoring, as well as some level of insight into the internal metrics, enough for the selection of recipes we gave earlier. Especially while you’re learning your way around MySQL, and perhaps starting to operate its installations in production, you should try to leverage existing monitoring infrastructure. Rushing to change everything to the best often leads to unsatisfactory results. As you get more experienced, you will see more and more data missing in the tools you have, and will be able to make an informed decision on which new tool to choose.

Incident/diagnostic and Manual Data Collection

Sometimes, you may not have a monitoring system set up for a database, or you may not trust it to contain all the information you may need to investigate some issue, or you may have a database-as-a-service instance running and want to get more data than your cloud provider gives you. In such situations, manual data collection can be a viable option in the short term to quickly get some data out of the system. We will show you few tools you can use to do just that: quickly gather a lot of diagnostic information from any MySQL instance.

The following sections are short, useful recipes that you can take away and use in your day-to-day work with MySQL databases.

Gathering System Status Variable Values Periodically

In “Status Variables” and “Basic Monitoring Recipes” we talked a lot about looking at different status variable values changing over time. Every monitoring tool mentioned in “MySQL Monitoring Tools” does that to accumulate data, which is then used for plots and alerting. The same sampling of status variables can be performed manually if you want to look at raw data, or simply sample at an interval lower than your monitoring system uses.

You could write a simple script that runs MySQL Monitor in a loop, but the better approach is to use the built-in mysqladmin utility. This program can be used to perform a wide range of administrative operations on a running MySQL server, though we should note that every one of those can also be done through the regular mysql. mysqladmin can, however, be used to sample global status variables easily, which is exactly how we’re going to use it here.

mysqladmin includes two status outputs: regular and extended. The regular one is less informative:

$ mysqladmin status

Uptime: 176190  Threads: 5  Questions: 5287160 ...

... Slow queries: 5114814  Opens: 761  Flush tables: 3 ...

... Open tables: 671  Queries per second avg: 30.008

The extended output will be familiar to you at this point. It’s just a SHOW GLOBAL STATUS output:

$ mysqladmin extended-status

+-----------------------------------------------+---------------------+

| Variable_name                                 | Value               |

+-----------------------------------------------+---------------------+

| Aborted_clients                               | 2                   |

| Aborted_connects                              | 30                  |

...

| Uptime                                        | 176307              |

| Uptime_since_flush_status                     | 32141               |

| validate_password.dictionary_file_last_parsed | 2021-05-29 21:50:38 |

| validate_password.dictionary_file_words_count | 0                   |

+-----------------------------------------------+---------------------+

Conveniently, mysqladmin is capable of repeating the commands it runs at a given interval a given amount of times. For example, the following command will cause mysqladmin to print status variable values every second for a minute (ext is a shorthand for extended-status):

$ mysqladmin -i1 -c60 ext

By redirecting its output to a file, you can have a minute-long sample of database metric changes. Text files are not as nice as proper monitoring systems, but again, this is usually done under special circumstances. For a long time, gathering information about MySQL like this with mysqladmin was a standard practice, and so there’s a tool called pt-mext that can turn plain SHOW GLOBAL STATUS outputs into a format better suited for consumption by humans. Unfortunately, the tool is only available on Linux.

$ pt-mext -r -- cat mysqladmin.output | grep Bytes_sent

Bytes_sent 10836285314 15120 15120 31080 15120 15120 31080 15120 15120

The initial large number is the status variable value at the first sample, and values after that represent the change to the initial number. If the value would decrease, a negative number would be shown.

Using pt-stalk to Collect MySQL and OS Metrics

pt-stalk is a part of Percona Toolkit, and is normally used to run alongside MySQL and continuously check for some specified condition. Once that condition is met, for example, the value of Threads_running is larger than 15, pt-stalk triggers a data collection routine gathering extensive information on MySQL and the operating system. However, it is possible to utilize the data collection part without actually stalking the MySQL server. Although it’s not a correct way to use pt-stalk, it’s a useful method to quickly glance at an unknown server, or try to gather as much information as possible on a misbehaving server.

pt-stalk, like other tools in the Percona Toolkit, is only available for Linux, even though a target MySQL server can run on any OS.

The basic invocation of pt-stalk to achive that is simple:

$ sudo pt-stalk --no-stalk --iterations=2 --sleep=30 \

--dest="/tmp/collected_data" \

-- --user=root --password=<root password>;

The utility will run two data collections, each of which will span a minute, and will sleep 30 seconds between each of the collection rounds. In case you don’t need OS information, or can’t get it because your target is a DBaaS instance, then you can utilize a --mysql-only flag:

$ sudo pt-stalk --no-stalk --iterations=2 --sleep=30 \

--mysql-only --dest="/tmp/collected_data" \

-- --user=root --password=<root password> \

--host=<mysql host> --port=<mysql port>;*

Below you can see the list of files a single collection. We omit OS-related files deliberately, but there are quite a lot of them.

2021_04_15_04_33_44-innodbstatus1

2021_04_15_04_33_44-innodbstatus2

2021_04_15_04_33_44-log_error

2021_04_15_04_33_44-mutex-status1

2021_04_15_04_33_44-mutex-status2

2021_04_15_04_33_44-mysqladmin

2021_04_15_04_33_44-opentables1

2021_04_15_04_33_44-opentables2

2021_04_15_04_33_44-processlist

2021_04_15_04_33_44-slave-status

2021_04_15_04_33_44-transactions

2021_04_15_04_33_44-trigger

2021_04_15_04_33_44-variables

Extended Manual Data Collection

pt-stalk is not always available, and it doesn’t run on all platforms. Sometimes, you may also want to add or remove some of the data it gathers. We can use the mysqladmin command we learned initially, gather a bit more data, and wrap it all up in a simple script. A version of this script is frequently used by authors of this book in their daily job.

This script, which should run on any Linux or UNIX-like system, will execute continuously either until terminated or until the /tmp/exit-flag file is found to be present. You can run touch /tmp/exit-flag to gracefully finish the execution of this script. We recommend putting it into a file and running through nohup ... &, or executing within a screen or tmux sessions. If you’re unfamiliar with the terms we’ve just mentioned, they are all ways to make sure a script continues to execute when your session disconnects.

DATADEST="/tmp/collected_data";

MYSQL="mysql --host=127.0.0.1  -uroot -proot";

MYSQLADMIN="mysqladmin  --host=127.0.0.1 -uroot -proot";

[ -d "$DATADEST" ] || mkdir $DATADEST;

while true; do {

[ -f /tmp/exit-flag ] \

&& echo "exiting loop (/tmp/exit-flag found)" \

&& break;

d=$(date +%F_%T |tr ":" "-");

$MYSQL -e "SHOW ENGINE INNODB STATUS\G" > $DATADEST/$d-innodbstatus &

$MYSQL -e "SHOW ENGINE INNODB MUTEX;" > $DATADEST/$d-innodbmutex &

$MYSQL -e "SHOW FULL PROCESSLIST\G" > $DATADEST/$d-processlist &

$MYSQLADMIN -i1 -c15 ext > $DATADEST/$d-mysqladmin ;

} done;

$MYSQL -e "SHOW GLOBAL VARIABLES;" > $DATADEST/$d-variables;

We’ve also created a Windows version of the same script written in PowerShell. It behaves exactly in the same way as does the previous script, and will terminate on its own as soon as the C:\tmp\exit-flag is found:

$mysqlbin='C:\Program Files\MySQL\MySQL Server 8.0\bin\mysql.exe'

$mysqladminbin='C:\Program Files\MySQL\MySQL Server 8.0\bin\mysqladmin.exe'

$user="root"

$password="root"

$mysqlhost="127.0.0.1"

$destination='C:\tmp\collected_data'

$stopfile='C:\tmp\exit-flag'

if (-Not (Test-Path -Path "$destination")) {

mkdir -p "$destination"

}

Start-Process -NoNewWindow $mysqlbin -ArgumentList `

"-h$mysqlhost","-u$user","-p$password",'-e "SHOW GLOBAL VARIABLES;"' `

-RedirectStandardOutput "$destination\variables"

while(1) {

if (Test-Path -Path "$stopfile") {

echo "Found exit monitor file, aborting"

break;

}

$d=(Get-Date -Format "yyyy-MM-d_HHmmss")

Start-Process -NoNewWindow $mysqlbin -ArgumentList `

"-h$mysqlhost","-u$user","-p$password",'-e "SHOW ENGINE INNODB STATUS\G"' `

-RedirectStandardOutput "$destination\$d-innodbstatus"

Start-Process -NoNewWindow $mysqlbin -ArgumentList `

"-h$mysqlhost","-u$user","-p$password",'-e "SHOW ENGINE INNODB MUTEX;"' `

-RedirectStandardOutput "$destination\$d-innodbmutex"

Start-Process -NoNewWindow $mysqlbin -ArgumentList `

"-h$mysqlhost","-u$user","-p$password",'-e "SHOW FULL PROCESSLIST\G"' `

-RedirectStandardOutput "$destination\$d-processlist"

& $mysqladminbin "-h$mysqlhost" -u"$user" -p"$password" `

-i1 -c15 ext > "$destination\$d-mysqladmin";

}

You should remember that script-based data collection is not a substitute for proper monitoring. It has its uses, which we described when opening this section, but it should always be an addition to what you already have, not the only way to look at MySQL metrics. By now, after reading through this chapter, you should have a pretty good idea of how to approach MySQL monitoring. Remember that issues, incidents, and outages will happen — they are unavoidable. With proper monitoring, however, you can make sure that the same issue doesn’t happen twice, as you’ll be able to find the root cause after the first occurrence. Of course, some issues won’t happen at all, as you’ll fix the problems brought out by exploring MySQL monitoring and changing the setup.

One closing thought is that perfect monitoring is unattainable, but even a pretty basic monitoring is better than no monitoring.

Chapter 13. High Availability

In the IT context, the term high availability defines a state of continuous operation for a specified length of time. We often measure availability against a 100% operational or never-fails standard. A common standard of availability is known as "five 9s,” or 99.999% availability. Two 9s would be a system that guarantees 99% availability in one year, allowing up to 1% downtime, or 3.65 days of unavailability.

Reliability engineering uses three principles of systems design to help achieve high availability: elimination of single-points-of-failure (SPOF); reliable crossover or failover points; and failure detection capabilities (monitoring is one of the components, see Chapter 12).

We need to consider many components need to achieve high availability. We can enumerate network redundancy, disk redundancy, different power supplies, multiple application and database servers, and many others, to mention only a few possible redundancies.

A simple example is an airplane with two engines. If one engine fails while flying, the aircraft can still land at an airport. A more complex example is a nuclear power plant. There are redundant protocols and components to avoid catastrophic failures.

If, nowadays, having multiple applications running into different data centers (like Facebook or Google do) is a must, who knows whether in a hundred years we will need data centers into other planets?

Exaggerations aside, when we talk about high availability, we are not eliminating the risk of failure (that’s impossible). We are trying to guarantee that under a failure situation, the operation can continue.

This chapter will focus on the options to achieve the high availability that MySQL databases offer.

TIP

In this chapter we are using the new name convention. We introduced it in Naming convention.

Asynchronous Replication

Replication enables data from one MySQL database server (known as a source) to be copied to one or more MySQL database servers (known as replicas). MySQL Replication by default is asynchronous. With asynchronous replication, the source writes events to its binary log, and replicas request them when ready. There is no guarantee that any event will ever reach any replica. It’s a loosely coupled source-replica relationship, where the following is true:

- The source does not wait for the replica.

- The replica determines how much to read and from which point in the binary log.

- The replica can be arbitrarily behind the source in reading or applying changes. This issue is known as replication lag, and we will see ways of minimizing this.

Asynchronous replication provides lower write latency since a write is acknowledged locally by a source before being written to the replicas.

MySQL implements its replication capabilities using three main threads, one on the source server and two on the replica:

- Binary log dump thread: The source creates a thread to send the binary log contents to a replica when the replica connects. We can identify this thread in the output of SHOW PROCESSLIST on the source as the Binlog Dump thread.

The binary log dump thread acquires a lock on the source’s binary log for reading each event sent to the replica. When the source reads the event, the lock is released, even before the source sends the event to the replica.

- Replication I/O thread: When we execute the START SLAVE statement on a replica server, the replica creates an I/O thread connected to the source and asks it to send the updates recorded in its binary logs.

The replication I/O thread reads the updates that the source’s Binlog Dump thread sends (see the previous item) and copies them to local files that comprise the replica’s relay log.

MySQL shows the state of this thread as Slave_IO_running in the output of SHOW SLAVE STATUS.

- Replication SQL thread: The replica creates an SQL thread to read the relay log written by the replication I/O thread and execute the transactions contained in it.

There are ways to improve replication parallelization, and we will discuss this in the following sections in this chapter.

The image Figure 13-1 shows how MySQL replication architecture is:

Figure 13-1. Asynchronous replication architecture flow

Replication works because events written to the binary log are read from the source and then processed on the replica, as shown in Figure 13-1. The events are recorded within the binary log in different formats according to the type of event. MySQL replication has three kinds of binary logging formats:

- Row-based replication (RBR): The source writes events to the binary log that indicate how individual table rows are changed. Replication of the source to the replica works by copying the events representing the replica’s table rows’ changes. For MySQL 5.7 and 8.0, this is the default replication format.

- Statement-based replication (SBR): The source writes SQL statements to the binary log. Replication of the source to the replica works by executing the SQL statements on the replica.

- Mixed replication: You can also configure MySQL to use a mix of both statement-based and row-based logging, depending on which one is most appropriate to log the changes. We call this mixed-format logging, in which MySQL uses a statement-based log by default. However, MySQL converts to row-based replication unsafe statements that have a non-deterministic behavior. Suppose we have the following statement:

mysql> UPDATE customer SET last_update=NOW() WHERE customer_id=1;

We know that the function NOW() returns the current date and time. Imagine that the source replicates the statement with 1 second of delay(the reason for the delay can be various, but let’s imagine my replica is in a different continent than my source). When the replica receives the statement and executes the statement, the date and time created will have a 1-second difference, leading to data inconsistency between source and replica. Every time MySQL parses a NOW() function and uses a mixed replication format, MySQL will convert the statement to row-based replication. There are many other functions that MySQL considers unsafe, and we can find them described in the documentation.

Basic Parameters to Set in the Source and the Replica

There are some basic settings that we need to set in both source server and the replica server in order to make the replication work. They are required for all methods explained in this section.

On the source server, you must enable binary logging and define a unique server ID. If you have not performed this, it is necessary to restart the MySQL server since these parameters are not dynamic.

TIP

The server ID does not need to be incremental or be in any order, like having the source server ID smaller than the replica server ID. The only requirement is to be unique in each server that is part of the replication topology.

Let’s see how the my.cnf file will look:

[mysqld]

log-bin=mysql-bin

server-id=1

binlog_format = row

As we saw, MySQL has a default value for all its parameters. However, we want to make explicit the binlog_format option by specifying the row-based format. The other options for binlog_format are MIXED and STATEMENT modes. The reason for that is from MySQL 5.7, MySQL 8.0 series and also Group Replication use the binlog_format set to ROW.

TIP

If you are using an automation tool like Ansible to deploy MySQL, you can use the following bash command to create server-ids:

# date '+%s'

1617565330

The command converts the current date and time to an integer value, so it increases monotonically. Note that the command does not guarantee the values’ uniqueness, but you may find it convenient to use the proposed command since it provides a relatively good uniqueness level.

On a replica, you must establish a unique server ID. Like in the source, if you haven’t done this yet, you need to restart the replica server. It is not mandatory to enable the binary log in the replica server, altought it is a recommended practice:

[mysqld]

log-bin=mysql-replica-bin

server-id=1617565330

binlog_format = row

log_slave_updates

Using the log_slave_updates options tells to the replica server that commands from a source server should be logged to the replica’s own binary log. Again the log_slave_updates is not mandatory, but it is recommended as a good practice.

Next, each replica connects to the source using a MySQL user name and password, so a user account on the source server that the replica can use to connect. Any account can be used for this operation, providing it has been granted the REPLICATION SLAVE privilege (in case you need to remember how to create a user, check “Creating and Using New Users”). Below is an example of how to create the user in the source server:

mysql> CREATE USER 'repl'@'%' IDENTIFIED BY 'learning_mysql';

mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%';

Create a Replica Using the Percona Xtrabackup Tool

The Percona Xtrabackup tools provide a method of performing a hot backup of your MySQL data while the system is running. It also offers advanced capabilities like parallelization, compression, and encryption. The Xtrabackup tool performs physical backups (“Physical and Logical Backups”).

The first step is taking a copy of the current source so we can start our replica. Using what we learned in Chapter 10, we will perform a physical backup of the source. We will use the commands provided in “Percona XtraBackup”:

# xtrabackup --defaults-file=my.sandbox.cnf -uroot -plearning_mysql \

-H 192.168.1.2 -P 3306 --backup --parallel=4 \

--datadir=./data/ --target-dir=./backup/

Alternatively, you can use rsync, NFS or any other method that you feel confortable with.

Once Xtrabackup finishes the backup, we will send the files to the replica server in a backup directory. In this example, we will send the files using the scp command:

# scp -r ./backup/* vinicius.grippa@192.168.1.3:/backup

At this point we’re finished with the source. The following steps will run only in the replica server. The next step is to prepare our backup:

# xtrabackup --prepare --apply-log --target-dir=./

With everything set, we are going to move the backup to the data directory:

# xtrabackup --defaults-file=my.sandbox.cnf --copy-back --target-dir=./backup

Next, we will ensure that your replica server does not have the same server-id as our source. Assuming both servers have an identical operating system and resources, you can copy the my.cnf file from the source to the replica and change my.cnf:

[mysqld]

server-id=2

On the replica, the content of the file xtrabackup_binlog_info will be something like this:

$ cat /backup/xtrabackup_binlog_info

mysql-bin.000003 156

This information is essential because it tells us where to start replicating. Remember that the source was still receiving operations, and we needed to know in which position of the binary log file MySQL was when the backup finished.

With that, let’s run the command to establish the replication:

mysql> CHANGE MASTER TO MASTER_HOST='192.168.1.2', MASTER_USER='repl',

-> MASTER_PASSWORD='learning_mysql',

-> MASTER_LOG_FILE='mysql-bin.000003', MASTER_LOG_POS=156;

mysql> START SLAVE;

Once you started, you can run the SHOW SLAVE STATUS\G command to check if the replication is working. For example:

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

Last_Errno: 0

Last_Error:

Skip_Counter: 0

Exec_Master_Log_Pos: 8332

Relay_Log_Space: 8752

Until_Condition: None

Seconds_Behind_Master: 0

Master_SSL_Verify_Server_Cert: No

Last_IO_Errno: 0

Last_IO_Error:

Last_SQL_Errno: 0

Last_SQL_Error:

It is important to check if both threads are running (Slave_IO_Running and Slave_SQL_Running), whether there is any error (Last_Error), and how many seconds the replica is behind the source. For large databases with an intensive write workload, the replication may take a while to catch up.

Create a Replica Using the Clone Plugin

MySQL 8.0.17 introduces the CLONE SQL statement to make the MySQL server a clone of another MySQL server running at a different server. We refer to the server instance, where the clone statement is executed, as the recipient. And we refer to the source server instance from which the recipient will clone the data as the donor. The donor clones all its data and metadata stored in the InnoDB storage engine at a consistent snapshot to replace the data in the recipient.

Let’s go through a real example and steps to perform the clone. We will also look at some interesting details, such as observability – how to monitor the progress of a long-running clone command, the privileges required to clone, and more. The following example uses the classic shell.

Choose the MySQL server to clone from and connect to it as the system user. Then install the clone plugin and provide backup privilege to the user that would transfer data from donor server:

mysql> INSTALL PLUGIN CLONE SONAME "mysql_clone.so";

mysql> CREATE USER clone_user@'%' IDENTIFIED BY "clone_password";

mysql> GRANT BACKUP_ADMIN ON *.* to clone_user;

Next, to observe the clone we need to grant privileges to view performance schema and execute functions:

mysql> GRANT SELECT ON performance_schema.* TO clone_user;

mysql> GRANT EXECUTE ON *.* to clone_user;

Now, we will move to the recipient server where we want to build using the clone process (the recipient). If you are provisioning a new node, first initialize a data directory and start the server. The clone process needs to have MySQL running (different from the Xtrabackup method mentioned previously).

Connect to the server as the root user. Then install the clone plugin and provide CLONE_ADMIN privilege to the user that would replace the current instance data with cloned data. We will also set a configuration that lists a set of valid donors that the recipient can clone:

mysql> INSTALL PLUGIN CLONE SONAME "mysql_clone.so";

mysql> SET GLOBAL clone_valid_donor_list = "127.0.0.1:21122";

mysql> CREATE USER clone_user IDENTIFIED BY "clone_password";

mysql> GRANT CLONE_ADMIN ON *.* to clone_user;

And we are going to provige grants to the PERFORMANCE_SCHEMA to observe the progress in the recipient side:

mysql> GRANT SELECT ON performance_schema.* TO clone_user;

mysql> GRANT EXECUTE ON *.* to clone_user;

We have everything we need in place, so it is the time to start the cloning process. Note that the donor server must be reachable from the recipient. The recipient will connect to the donor with the address and credential provided and start cloning:

mysql> CLONE INSTANCE FROM clone_user@192.168.1.2:3306 IDENTIFIED BY "clone_password";

The recipient will shut down and restart itself for the clone operation to succeed. We can monitor the progress with the following query:

select STAGE, STATE, CAST(BEGIN_TIME AS TIME) as "START TIME",

CASE WHEN END_TIME IS NULL THEN

LPAD(sys.format_time(POWER(10,12) * (UNIX_TIMESTAMP(now()) - UNIX_TIMESTAMP(BEGIN_TIME))), 10, ' ')

ELSE

LPAD(sys.format_time(POWER(10,12) * (UNIX_TIMESTAMP(END_TIME) - UNIX_TIMESTAMP(BEGIN_TIME))), 10, ' ')

END as DURATION,

LPAD(CONCAT(FORMAT(ROUND(ESTIMATE/1024/1024,0), 0), " MB"), 16, ' ') as "Estimate",

CASE WHEN BEGIN_TIME IS NULL THEN LPAD('0%', 7, ' ')

WHEN ESTIMATE > 0 THEN

LPAD(CONCAT(CAST(ROUND(DATA*100/ESTIMATE, 0) AS BINARY), "%"), 7, ' ')

WHEN END_TIME IS NULL THEN LPAD('0%', 7, ' ')

ELSE LPAD('100%', 7, ' ') END as "Done(%)"

from performance_schema.clone_progress;

With this result, we can observe each state of the clone process. The output will be similar to this:

+-----------+-----------+------------+------------+------------------+

---------+

| STAGE     | STATE     | START TIME | DURATION   | Estimate         | Done(%) |

+-----------+-----------+------------+------------+------------------+

---------+

| DROP DATA | Completed | 14:44:46   |     1.33 s |             0 MB |

100% |

| FILE COPY | Completed | 14:44:48   |     5.62 s |         1,511 MB |

100% |

| PAGE COPY | Completed | 14:44:53   |   95.06 ms |             0 MB |

100% |

| REDO COPY | Completed | 14:44:54   |   99.71 ms |             0 MB |

100% |

| FILE SYNC | Completed | 14:44:54   |     6.33 s |             0 MB |

100% |

| RESTART   | Completed | 14:45:00   |     4.08 s |             0 MB |

100% |

| RECOVERY  | Completed | 14:45:04   |  516.86 ms |             0 MB |

100% |

+-----------+-----------+------------+------------+------------------+

---------+

7 rows in set (0.08 sec)

As mentioned, there is a restart in the end. Note that we did not establish the replication yet. Both binary log position (filename, offset) and GTID coordinates are extracted and transferred from the donor MySQL server instance.

We can execute the following queries on the cloned MySQL server instance to view the binary log position or the GTID of the last transaction that was applied:

mysql> SELECT BINLOG_FILE, BINLOG_POSITION FROM performance_schema.clone_status;

+------------------+-----------------+

| BINLOG_FILE      | BINLOG_POSITION |

+------------------+-----------------+

| mysql-bin.000002 |       816804753 |

+------------------+-----------------+

1 row in set (0.01 sec)

mysql> SELECT @@GLOBAL.GTID_EXECUTED;

+------------------------+

| @@GLOBAL.GTID_EXECUTED |

+------------------------+

|                        |

+------------------------+

1 row in set (0.00 sec)

In this example, we are not using GTID, so the query does not return anything. Next, we will run the command to establish the replication:

mysql> CHANGE MASTER TO MASTER_HOST = '192.168.1.2', MASTER_PORT = 3306,

-> MASTER_USER = repl, MASTER_PASSWORD = 'learning_mysql',

-> MASTER_LOG_FILE = 'mysql-bin.000002',

-> MASTER_LOG_POSITION = 816804753;

mysql> START SLAVE;

And we can validate if everything is okay by running the SHOW SLAVE STATUS\G command.

Now, the advantage of the clone process is that the clone plugin automates the whole process, and only in the last part is it necessary to execute the CHANGE MASTER command. The disadvantage is that it is only available for MySQL 8.0.17 and higher, so it is relatively new. We believe that in years to come, this process may become the default.

Create a Replica Using mysqldump

This is what we can call a classic. It is the front door for those starting with MySQL who are still learning about the ecosystem.

Let’s see an example of using mysqldump to create a new replica. We will execute the backup from the replica server:

# mysqldump -uroot -plearning_mysql --single-transaction \

--all-databases --routines --triggers --events \

--master-data=2 > backup.sql

The dump succeeded if the message Dump completed appears in the end.

# tail -1f backup.sql

-- Dump completed on 2021-04-26 20:16:33

With the backup taken, we need to import it in the replica server. For example, you can use this command:

$ mysql < backup.sql

Once done, it is necessary to execute the CHANGE MASTER command with the coordinates extracted from the dump (for more details about mysqldump, revisit the “The mysqldump Program”). Because we used the --master-data=2, the inforomation will be written in the beginning of the dump. For example:

$ head -n 35 out

-- MySQL dump 10.13  Distrib 5.7.31-34, for Linux (x86_64)

--

-- Host: 127.0.0.1    Database:

-- ------------------------------------------------------

-- Server version 5.7.33-log

...

--

-- Position to start replication or point-in-time recovery from

--

-- *CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=4089; *

And in GTID’s case:

--

-- GTID state at the beginning of the backup

-- (origin: @@global.gtid_executed)

--

SET @@GLOBAL.GTID_PURGED=00048008-1111-1111-1111-111111111111:1-16;

Next, we are going to execute the command to establish the replication. For the GTID scenario:

mysql> CHANGE MASTER TO MASTER_HOST='192.168.1.2', MASTER_USER='repl',

-> MASTER_PASSWORD = 'learning_mysql', MASTER_AUTO_POSITION=1;

mysql> START SLAVE;

For traditional replication, you can configure the replication and start replication threads using the previously extracted position:

mysql> CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=4089,

-> MASTER_HOST='192.168.1.2', MASTER_USER='repl',

-> MASTER_PASSWORD='learning_mysql';

mysql> START SLAVE;

And check the status of the replica with SHOW SLAVE STATUS command.

Create a Replica Using mydumper and myloader

We know that mysqldump is the most essential tool for performing backups and building replicas. But there is a more efficient method: mydumper. This tool, just like mysqldump, generates a logical backup and, also just like mysqldump, can be used to create a consistent backup of the database. The main difference between mydumper and mysqldump is that mydumper, when paired with myloader, can dump and restore data in parallel, improving the dump and, especially, restoring time. Imagine a scenario where your database has a dump of 500 GB. Using mysqldump we will have a single huge file. With mydumper, we will have one file per each table, making it more efficient to restore later.

Setting up the mydumper and myloader utilities

We can run mydumper directly in the source server or from another server, which in general is better since it will avoid the overhead in the storage system of writing the backup files in the same server as the database server.

To install mydumper, download the package specific to the operating system version you are using. We can locate the releases in the mydumper GitHub repository. Let’s see an example for CentOS:

# yum install https://github.com/maxbube/mydumper/releases/download/v0

.10.3/mydumper-0.10.3-1.el7.x86_64.rpm -y

Now we should have both mydumper and myloader commands installed on the server. We can validate this:

$ mydumper --version

mydumper 0.10.3, built against MySQL 5.7.33-36

$ myloader --version

myloader 0.10.3, built against MySQL 5.7.33-36

Extracting data from the source

The mydumper tool extracts the DB data in parallel and creates separate files from schemas and tables data, so it is easy to modify them before restoring them.

TIP

You will need to give at least SELECT and RELOAD permissions to the mydumper user.

The following command will execute a dump of all databases (except mysql, test, and sys schemas), with 15 simultaneously threads, and will also include triggers, views and functions:

# mydumper --regex ^(?!(mysql\.|test\.|sys\.)) --threads=15

--user=learning_user --password=learning_mysql --host=192.168.1.2

--port=3306

--trx-consistency-only --events --routines --triggers --compress

--outputdir /home/vinicius.grippa/sandboxes/backup --logfile

/tmp/log.out --verbose=2

And if we check the output directory (outputdir), we will see the compressed files:

# ls -l backup/

total 5008

-rw-r--r--. 1 vinicius.grippa percona   182 May  1 19:30 metadata

-rw-r--r--. 1 vinicius.grippa percona   258 May  1 19:30

sysbench.sbtest10-schema.sql.gz

-rw-r--r--. 1 vinicius.grippa percona 96568 May  1 19:30

sysbench.sbtest10.sql.gz

-rw-r--r--. 1 vinicius.grippa percona   258 May  1 19:30

sysbench.sbtest11-schema.sql.gz

-rw-r--r--. 1 vinicius.grippa percona 96588 May  1 19:30

sysbench.sbtest11.sql.gz

-rw-r--r--. 1 vinicius.grippa percona   258 May  1 19:30

sysbench.sbtest12-schema.sql.gz

...

TIP

Decide the number of threads based on the CPU cores of the database server and server load. Using parallel dump can consume a lot of resources from the server.

Restoring data in a replica server

Like mysqldump, we need to have the MySQL instance already up and running. Once the data is ready to be imported, we will execute the following command:

# myloader --user=learning_user --password=learning_mysql

--threads=25 --host=192.168.1.3 --port=3306

--directory=/home/vinicius.grippa/sandboxes/backup --overwrite-tables

--verbose 3

Establishing the replication

Now that we’ve restored the data, we will set up replication. We need to find the correct binary log position at the start of the backup. This information is stored in the mydumper metadata file:

$ cat backup/metadata

Started dump at: 2021-05-01 19:30:00

SHOW MASTER STATUS:

Log: mysql-bin.000002

Pos: 9530779

GTID:00049010-1111-1111-1111-111111111111:1-319

Finished dump at: 2021-05-01 19:30:01

Now, we need to execute the CHANGE MASTER command like we did previously for mysqlump:

mysql> CHANGE MASTER TO MASTER_HOST=192.168.1.2, MASTER_USER=repl,

MASTER_PASSWORD=learning_mysql,  MASTER_LOG_FILE=mysql-bin.000002,

MASTER_LOG_POS=9530779, MASTER_PORT=49010;

mysql> START SLAVE;

Group Replication

It might be a bit controversial to include Group Replication under the asynchronous replication group. The short explanation is that Group Replication is asynchronous. The confusion here can be explained by the comparison with Galera (explained in the next section in “Galera Cluster (Percona XtraDB Cluster)”) that claims to be synchronous or virtually synchronous.

The more detailed answer is that it depends on how we define replication. In the MySQL world, we define replication as the process of writing (or changing or deleting) data to a source and the appearance of that data on the replica. The entire process is what we call replication. The fact of writing data on a source, adding that change in the binary log, sending it on to the relay log of a replica, and the replica applying that change gives us five different steps:

1. Locally applying

2. Generating a binlog event

3. Sending the binlog event to the replicas

4. Adding the binlog event on the relay log

5. Applying the binlog event from the relay log

And indeed, in MySQL Group Replication and in Galera (even if the Galera cache primarily replaces binlog and relay log files), only step #3 is synchronous, and this step is the streaming of the binary log event (or write set in Galera) to the replica (s).

So yes, the process of sending (replicating, streaming) the data to the other nodes is synchronous. But the applying of these changes is still wholly asynchronous.

TIP

Group Replication is available since MySQL 5.7. However, when the product was released, it was not mature enough, leading to constant performance issues and crashes. We highly recommend using MySQL 8 series if you want to test Group Replication.

Installing MySQL Group Replication

The first advantage of Group Replication compared to Galera is that you don’t have to install different binaries. MySQL Server provides Group Replication as a plugin. Group Replication is available for Oracle MySQL and Percona versions.

Since the installation process is the same as a standard installation, please refer to Chapter 1.

To confirm that the Group Replication plugin is enabled, run the following query:

SELECT

PLUGIN_NAME, PLUGIN_STATUS, PLUGIN_TYPE,

PLUGIN_LIBRARY, PLUGIN_LICENSE

FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_NAME like

group_replication;

The output should show ACTIVE like this example:

+-------------------+---------------+-------------------+-------------

--

-------+----------------+

| PLUGIN_NAME       | PLUGIN_STATUS | PLUGIN_TYPE       |

PLUGIN_LIBRARY

| PLUGIN_LICENSE |

+-------------------+---------------+-------------------+-------------

--

-------+----------------+

| group_replication | ACTIVE        | GROUP REPLICATION |

group_replication.so | GPL            |

+-------------------+---------------+-------------------+-------------

--

-------+----------------+

1 row in set (0.00 sec)

If the plugin is not installed, run the following command:

mysql> INSTALL PLUGIN group_replication SONAME group_replication.so;

With the plugin active, we will set the minimum parameters in the nodes to start Group Replication. Open my.cnf and add the following:

[mysqld]

server_id=175907211

log-bin=mysqld-bin

enforce_gtid_consistency=ON

gtid_mode=ON

log-slave-updates

transaction_write_set_extraction=XXHASH64

master_info_repository=TABLE

relay_log_info_repository=TABLE

binlog_checksum=NONE

Let’s go over each of those parameters:

- server_id: Like classic replication, this parameter helps to identify each member in the group using a unique id. You must use a different value for each server participating in group replication.

- log_bin: The value determines if binary logging should take place. We will set this value to binlog for group replication to take place. Binary logging is responsible for recording all changes taking place in the database.

- enforce_gtid_consistency: This value must be set to ON to instruct MySQL to execute transaction-safe statements to ensure consistency when replicating data.

- gtid_mode: This directive enables global transaction identifier when set to ON.

- log_slave_updates: This value is set to ON to allow members to log updates from each other. In other words, the directive chains the replication servers together.

- transaction_write_set_extraction: This instructs the MySQL server to collect write sets and encode them using a hashing algorithm. In this case, we are using the XXHASH64 algorithm. Write sets are defined by primary keys on each record.

- master_info_repository: When set to TABLE, this directive allows MySQL to store details about source binary log files and positions into a table rather than a file to enable faster replication and guarantee consistency using InnoDB ACID properties.

- relay_log_info_repository: When set to TABLE this configures MySQL to store replication information as an InnoDB table.

- binlog_checksum: We are setting this value to NONE. This disables MySQL from writing a checksum for each event taking place in the binary log. By setting the value to NONE, we instruct the server to verify events when they are written by checking their length rather than generating a checksum.

Next, we are going to add Group Replication settings:

[mysqld]

# Group configuration

loose-group_replication_group_name="8dc32851-d7f2-4b63-8989-5d4b467d8251

"

loose-group_replication_start_on_boot=off

loose-group_replication_local_address= "10.124.33.139:33061"

loose-group_replication_group_seeds= "10.124.33.139:33061,

10.124.33.90:33061, 10.124.33.224:33061"

loose-group_replication_bootstrap_group=off

bind-address = "0.0.0.0"

report_host = "10.124.33.139"

Let’s see what each parameter does:

- loose-group_replication_group_name: This is the name of the group that we are creating. We are going to use the built in Linux uuidgen command to generate Universally Unique IDentifier (UUID). To do this, run the command below:

$ uuidgen

Sample output:

8dc32851-d7f2-4b63-8989-5d4b467d8251

- loose-group_replication_start_on_boot: When set to off, the value instructs the plugin not to start working when the server starts. You may set this value to on once you are through with configuring all the group members.

- loose-group_replication_local_address: This is the internal IP and port address combination used for communicating with other MySQL server members in the group. The recommended port for Group Replication is 33061.

- loose-group_replication_group_seeds: This configures the IP addresses or hostnames of members participating in the group replication together with their communication port. New members use the value to establish themselves in the group.

- loose-group_replication_bootstrap_group: This option instructs the server whether to initiate a group or not. We will only run this option on-demand on server 1 to avoid creating multiple groups. So, it will remain off for now.

- bind_address: The value of 0.0.0.0 tells MySQL to listen to all networks.

- report_host`: This is the IP address or hostname the group members reports to each other when they are registered on the group.

We are using the loose- prefix to instruct the server to start even when the MySQL Group Replication plugin is not installed and configured. This avoids encountering server errors before you finish configuring all the settings.

Setting MySQL Group Replication

First, we will set the group_replication_recovery channel. MySQL Group Replication works on a group_replication_recovery channel to transfer transactions between members. Because of this, we must set up a replication user with REPLICATION SLAVE permission on each server.

So, on node 1, log in to the MySQL console and execute the following command:

mysql> SET SQL_LOG_BIN=0;

mysql> CREATE USER replication_user@% IDENTIFIED BY

learning_mysql;

mysql> GRANT REPLICATION SLAVE ON . TO replication_user@%;

mysql> FLUSH PRIVILEGES;

mysql> SET SQL_LOG_BIN=1;

We are using SQL_LOG_BIN equals 0 to avoid the new user’s details from being logged to the binary log, and in the end, we are re-enabling it.

To instruct the MySQL server to use the replication user we have created for the group_replication_recovery channel, run this command:

mysql> CHANGE MASTER TO MASTER_USER=replication_user,

MASTER_PASSWORD=learning_mysql FOR CHANNEL

group_replication_recovery;

The above settings will allow members joining the group to run the distributed recovery process to get to the same state with other members (donors).

Now we will start the Group Replication service on server 1. We will bootstrap the group using these commands:

mysql>SET GLOBAL group_replication_bootstrap_group=ON;

mysql>START GROUP_REPLICATION;

mysql>SET GLOBAL group_replication_bootstrap_group=OFF;

To avoid starting up different groups, we have set the value of group_replication_bootstrap_group back to OFF after successfully starting the group.

To check the status of the new member:

mysql>  SELECT * FROM performance_schema.replication_group_members;

-------------------------------------------------------------------

------------------------------------------------------------------

---

| CHANNEL_NAME              | MEMBER_ID                            |

MEMBER_HOST   | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE |

MEMBER_VERSION |

-------------------------------------------------------------------

------------------------------------------------------------------

---

| group_replication_applier | d58b2766-ab90-11eb-ba00-00163ed02a2e |

10.124.33.139 |        3306 | ONLINE       | PRIMARY     | 8.0.22

|

-------------------------------------------------------------------

------------------------------------------------------------------

---

1 row in set (0.00 sec)

Great. So far we’ve bootstrapped and initiated one node. Let’s proceed to the second node. Make sure you have installed the same MySQL version as in node 1 and add the following settings to the my.cnf file:

[mysqld]

loose-group_replication_group_name="8dc32851-d7f2-4b63-8989-5d4b467d8

51"

loose-group_replication_start_on_boot=off

loose-group_replication_local_address= "10.124.33.90:33061"

loose-group_replication_group_seeds= "10.124.33.139:33061,

10.124.33.90:33061, 10.124.33.224:33061"

loose-group_replication_bootstrap_group=off

bind-address = "0.0.0.0"

Similar to node 1, we are changing only the group_replication_local_address while the other settings remain the same. Also, note that the other MySQL configurations are necessary to node 2, and we strongly recommend having the same across all nodes.

With the configurations in place, restart the MySQL service:

# systemctl restart mysqld

Issue the following commands to configure the credentials for the recovery user on server 2:

mysql>SET SQL_LOG_BIN=0;

mysql>CREATE USER replication_user@% IDENTIFIED BY learning_mysql;

mysql>GRANT REPLICATION SLAVE ON . TO replication_user@%;

mysql>SET SQL_LOG_BIN=1;

mysql>CHANGE MASTER TO MASTER_USER=replication_user, MASTER_PASSWORD=PASSWORD FOR CHANNEL group_replication_recovery;

Next, add server 2 to the group that we bootstrapped earlier:

mysql> START GROUP_REPLICATION;

Run the query to check the member’s state:

mysql>  SELECT * FROM performance_schema.replication_group_members;

-------------------------------------------------------------------

------------------------------------------------------------------

---

| CHANNEL_NAME              | MEMBER_ID                            |

MEMBER_HOST   | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE |

MEMBER_VERSION |

-------------------------------------------------------------------

------------------------------------------------------------------

---

| group_replication_applier | 9e971ba0-ab9d-11eb-afc6-00163ec43109 |

10.124.33.90  |        3306 | ONLINE       | SECONDARY   | 8.0.22

|

| group_replication_applier | d58b2766-ab90-11eb-ba00-00163ed02a2e |

10.124.33.139 |        3306 | ONLINE       | PRIMARY     | 8.0.22

|

-------------------------------------------------------------------

------------------------------------------------------------------

---

2 rows in set (0.00 sec)

Now you can follow the same steps for node 3 as we used in node 2. In the end, we can validate whether all nodes are responsive by inserting some dummy data:

mysql> CREATE DATABASE learning_mysql;

Query OK, 1 row affected (0.00 sec)

mysql> use learning_mysql

Database changed

mysql> CREATE TABLE test (i int primary key);

Query OK, 0 rows affected (0.01 sec)

mysql> INSERT INTO test VALUES (1);

Query OK, 1 row affected (0.00 sec)

And connect to the other nodes to observe if you can visualize the data.

Synchronous replication

Synchronous replication is used by Galera clusters where we have more than one MySQl server, but they act as single entity for the application. The Figure 13-2 illustrates a Galera cluster with three nodes:

Figure 13-2. Galera cluster, all nodes communicate with each other

The primary difference between synchronous and asynchronous replication is that synchronous replication guarantees that if a change happened on one node in the cluster, then the change will occur on other nodes in the cluster synchronously or at the same time. Asynchronous replication gives no guarantees about the delay between applying changes on the source node and propagating changes to replica nodes. The delay with asynchronous replication can be short or long. This also implies that if the source node crashes in an asynchronous replication topology, some of the latest changes may be lost. This concept of source and replica does not exist for the Galera cluster. All nodes can receive reads and writes.

Theoretically, synchronous replication has several advantages over asynchronous replication:

- Clusters utilizing synchronous replication are always highly available. If one of the nodes crashed, then there would be no data loss. Additionally, all cluster nodes are always consistent.

- Clusters utilizing synchronous replication allow transactions to be executed on all nodes in parallel.

- Clusters utilizing synchronous replication can guarantee causality across the whole cluster. This means that if a SELECT is executed on one cluster node after a transaction is executed on a cluster node, it should see the effects of that transaction.

However, there are disadvantages to synchronous replication as well. Traditionally, eager replication protocols coordinate nodes one operation at a time. They use a two-phase commit or distributed locking. Increasing the number of nodes in the cluster leads to growth in the transaction response times and the probability of conflicts and deadlock rates among the nodes. This is because all nodes need to certify the transaction and reply with OK message back.

For this reason, asynchronous replication remains the dominant replication protocol for database performance, scalability, and availability. The lack of understanding or subestimating the impact of synchronous replication is one reason companies give up using Galera clusters and go back to using asynchronous replication.

Nowadays, two companies offer the Galera cluster: Percona and MariaDB. The following example shows how to install the Percona XtraDB cluster, but the settings are relatively similar for both platforms since they share the same Galera source code.

Galera Cluster (Percona XtraDB Cluster)

Installing the Percona XtraDB cluster is similar to Percona Server (the difference is the packages), so we won’t dive into details for all platforms. You may want to revisit Chapter 1 to review the installation process. Now the configuration process assumes there are three Percona XtraDB cluster nodes:

Node

Host

IP

Node 1

pxc1

172.16.2.56

Node 2

pxc2

172.16.2.198

Node 3

pxc3

172.16.3.177

Connect to one of the nodes and install the repository:

# yum install https://repo.percona.com/yum/percona-release-latest.noar

ch.rpm -y

With the repository installed, we will install the binaries:

# yum install Percona-XtraDB-Cluster-57 -y

Next you can apply the typical configurations that you would use for a regular MySQL process. With the changes made, start the mysqld process and get the temporary password:

# systemctl start mysqld

# grep temporary password/var/log/mysqld.log

Use the previous password to log in as root and change the password:

$ mysql -u root -p

mysql> ALTER USER root@localhost IDENTIFIED BY learning_mysql;

Stop the mysqld process:

# systemctl stop mysql

Repeat the previous steps for all three nodes.

With the binaries and basic configuration in place, we will start working on the cluster parameters.

We need to add the following configuration variables to /etc/my.cnf on the first node:

[mysqld]

wsrep_provider=/usr/lib64/galera3/libgalera_smm.so

wsrep_cluster_name=pxc-cluster

wsrep_cluster_address=gcomm://172.16.2.56,172.16.2.198,172.16.3.177

wsrep_node_name=pxc1

wsrep_node_address=172.16.2.56

wsrep_sst_method=xtrabackup-v2

wsrep_sst_auth=sstuser:learning_mysql

pxc_strict_mode=ENFORCING

binlog_format=ROW

default_storage_engine=InnoDB

innodb_autoinc_lock_mode=2

Use the same configuration for the second and third nodes, except the wsrep_node_name and wsrep_node_address variables:

For the second node:

wsrep_node_name=pxc2

wsrep_node_address=172.16.2.198

For the third node:

wsrep_node_name=pxc3

wsrep_node_address=172.16.3.177

Like regular MySQL, the XtraDB cluster has many configurable parameters, and the ones required above are the minimal settings to start the cluster. We are configuring the node name, its IP address, the cluster address, and the user that will be used by internal communication among the nodes. You can find more detailed information in the documentation.

We have all nodes configured at this point, but the mysqld process is not running in any node. To start the XtraDB cluster, we need to start one of the nodes in bootstrap mode. XtraDB cluster requires you to start a node in a cluster as a reference point for the others before the other nodes can join and form the cluster. Bootstrapping is an initial step to introduce a database node as a primary component before others see it as a reference point to sync up data.

Start the first node with the following command:

# systemctl start mysql@bootstrap

Before adding other nodes to your new cluster, connect to the node that you just started and create a user for SST and provide the necessary privileges for it. The credentials must match those specified in the wsrep_sst_auth that you set previously:

mysql> CREATE USER sstuser@localhost IDENTIFIED BY

learning_mysql;

mysql> GRANT RELOAD, LOCK TABLES, PROCESS, REPLICATION CLIENT ON .

TO sstuser@localhost;

mysql> FLUSH PRIVILEGES;

NOTE

The State Snapshot Transfer (SST) process is used by the cluster to provision nodes by transferring a full data copy from one node to another. When a new node joins the cluster, the new node initiates an SST to synchronize its data with a node that is already part of the cluster.

After this, you can initialize the other nodes regularly:

# systemctl start mysql

To verify that the cluster is up and running fine, we can check a few things, like creating a database in the first node, a table in the second node, and inserting data in the third node. First, let’s create the database in the first node (pxc1):

mysq> CREATE DATABASE learning_mysql;

Query ok, 1 row affected (0.01 sec)

Create a table on the third node:

In the second node (pxc2), create the table and insert data:

mysql> USE learning_mysql;

Database changed

mysql> CREATE TABLE example (node_id INT PRIMARY KEY, node_name VARCHAR(30));

Query ok, 0 rows affected (0.05 sec)

mysql> INSERT INTO learning_mysql.example VALUES (1, Vinicius1);

Query OK, 1 row affected (0.02 sec)

Retrieve rows from that table on the third node:

mysql>  SELECT * FROM learning_mysql.example;

--------------------+

| node_id | node_name |

--------------------+

|       1 | Vinicius1 |

--------------------+

1 row in set (0.00 sec)

Another more elegant solution is checking the wsrep_% global status variables, in particular the wsrep_cluster_size and wsrep_cluster_status:

mysql> show global status like wsrep_cluster_size;

---------------------------+

| Variable_name      | Value |

---------------------------+

| wsrep_cluster_size | 3     |

---------------------------+

1 row in set (0.00 sec)

mysql> show global status like wsrep_cluster_status;

-------------------------------+

| Variable_name        | Value   |

-------------------------------+

| wsrep_cluster_status | Primary |

-------------------------------+

1 row in set (0.00 sec)

The output of these statuses tells us that the cluster has three nodes and is in the primary state (it can receive reads and writes).

You might consider using ProxySQL in addition to the Galera cluster to ensure transparency for the application (Chapter 15).

Note that we are not going to approach cluster maintenance and how to optimize its performance. The idea here is just to start getting familiar with the different topologies and know they exist.

Chapter 14. MySQL in the Cloud

"You don’t need to worry, it’s in the Cloud" is a famous phrase we often hear. It reminds us of a curious history where a woman was worried that after her iPhone drowned in the toilet, she lost all her years of family and travel photos. To her surprise, when she bought the new iPhone, the device “recovered” all the photos. She was using the iCloud backup solution from Apple to back up her device content to the cloud. (The other surprise may have been the service subscription bill she hadn’t realized she was paying.)

As computer engineers, we don’t have the luxury of not knowing if our data will be recovered or not. And there are many stories on the web about companies that poorly designed an auto-scale solution for which Amazon AWS or Google Cloud charged a few thousand dollars for a couple of hours usage.

In this chapter, we will see a few options that companies can choose to embrace MySQL in the cloud for their environment, differentiate them, and the advantages and disadvantages of each.

Database as a service (DBaaS)

Database as a service (DBaaS) is an option where companies outsource their database, paying a cloud provider to launch and maintain a cloud database. Payment is usually per-usage, and the data owners can access their application data as they please. These databases-as-a-service provide the same functionalities as a standard relational or non-relational database. DBaaS is often beneficial for companies trying to avoid configuring, maintaining, and upgrading their databases and servers (although this is not always true). DBaaS lives in the realm of software-as-a-service (SaaS), like the platform-as-a-service (PaaS) and infrastructure-as-a-service (IaaS), where products like databases become a service.

Amazon RDS for MySQL/MariaDB

The most popular DBaaS is the Amazon RDS for MySQL. Choosing your RDS is almost like configuring a new car on the website. You choose the main product and add the options you want until it looks the way you like and launches it.

First, you can pick the product. In this case, we will go for MySQL (the MariaDB version has similar settings for deployment). The Figure 14-1 shows the products available:

Figure 14-1. Choosing the product

We can also choose the version, which in this case, we will go for 8.0.21. Next, we need to set the master user (similar to the root) and its password. Make sure to pick a strong password, especially if you will expose your database to the world. The Figure 14-2 shows how to define the username and the password for the root user:

Figure 14-2. Configuring the master username and password

Next is the instance size, which will impact directly on the final price. We will pick a top-level configuration so we can have an estimated cost in the end. The Figure 14-3 shows the instance classes available:

Figure 14-3. There are several options from the cheapest to costly ones!

Another option that can directly affect the billing is the storage options. Naturally, higher performance (more IOPs) and more storage lead to a higher cost. The Figure 14-4 illustrates the IOPs option:

Figure 14-4. Configuring storage size and its IOPs performance

The next option is an important choice: do we want to use multi-AZ deployment or not? The multi-AZ is all about high availability. When we provision a multi-AZ DB instance, Amazon RDS automatically creates a primary DB instance and synchronously replicates the data to a standby instance in a different Availability Zone (AZ). Availability Zones are physically distinct and have an independent infrastructure, which increases overall high availability.

If we don’t want to use multi-AZ deployment, AWS will install a single instance. AWS will spin up a new one in case of failure, and AWS will remount its data volume. This process takes some time, during which your database will not be available. Even big cloud providers are not bulletproof, and disasters can happen, so having a standby server is always recommended. The Figure 14-5 shows how to configure the replica:

Figure 14-5. Configuring a standby replica

The next part is networking. In this part, we set a general networking configuration. We recommend configuring the RDS to a private network, where only the application servers and developers’ IPs can access it. The Figure 14-6 demonstrates the network options:

Figure 14-6. Configuring the network settings

Finally, perhaps the saddest part comes the estimated costs. The Figure 14-7 shows how much it will cost in a month:

Figure 14-7. The bill can reach the stars under specific configurations!

Google Cloud for MySQL

As we will see now, Google Cloud SQL offers managed database services comparable to Amazon RDS (and as Azure) with slight differences. For example, the Google Cloud options for MySQL are more straightforward because there are fewer options to choose from. For example, you cannot choose between MySQL and MariaDB or the MySQL minor version (only the major version). The Figure 14-8 shows the options available:

Figure 14-8. Google Cloud SQL

To start the instance, you have to fill in a few options. As usual, it begins with the product. The Figure 14-9 show the options available for MySQL:

Figure 14-9. Choosing the product

After picking MySQL, it is necessary to set a few options like instance name, root password, version, and location. The Figure 14-10 shows how to configure these settings:

Figure 14-10. Setting basic configuration

Next are the settings that can impact the performance, and of course, the cost. It is naturally crucial to find the right balance between performance and costs. The Figure 14-11 shows the storage, memory, and CPU options available:

Figure 14-11. Setting server capacity

Now the instance is ready to launch in the Google Cloud.

Azure SQL

Finally, the last one from the top three cloud service providers is Azure SQL. The Figure 14-12 shows the database products available in Azure:

Figure 14-12. Choosing MySQL in Azure

Azure offers two options, to go with a simple server or a more robust solution with high availability in the setup. The Figure 14-13 shows the difference between the two options:

Figure 14-13. Choosing a single server vs. flexible server

Those choices are followed by similar configurations regarding the service performance and costs. The Figure 14-14 shows the MySQL managed services options:

Figure 14-14. Configuring our MySQL managed service instance

Amazon Aurora RDS

Amazon Aurora is an RDS solution provided by Amazon based on MSQL or Postgres under a commercial license.

Since it is built-in top of MySQL, it offers similar features plus a few extra features developed by Amazon AWS.

Two of the extra features are worth mentioning. First is the Aurora PQ (Parallel Query). The parallel query is a feature that parallelizes some of the I/O and computation involved in processing data-intensive queries.

Aurora PQ works by doing a full table scan (the storage level performs the parallel reads). When we use a parallel query, the query does not use the InnoDB buffer pool. It pushes down and parallelizes the query processing using the storage fleet.

The advantage is that moving the processing close to the data reduces network traffic and latency. However, the feature is not a silver bullet and does not work well for all cases—the feature focuses on analytical queries that need to run over large portions of data.

The parallel query feature is not available for all AWS instances. For instances that support the parallel feature, their instance class determines the number of parallel queries that can be active at a given time. The following instances are the ones that support the parallel query feature:

- db.r*.large: 1 concurrent parallel query session

- db.r*.xlarge: 2 concurrent parallel query sessions

- db.r*.2xlarge: 4 concurrent parallel query sessions

- db.r*.4xlarge: 8 concurrent parallel query sessions

- db.r*.8xlarge: 16 concurrent parallel query sessions

- db.r4.16xlarge: 16 concurrent parallel query sessions

The other feature is the Amazon Aurora Global Database, designed for applications with a global footprint. It allows a single Aurora database to span multiple AWS regions, with fast replication to enable low-latency global reads and disaster recovery from region-wide outages. The Aurora Global Database uses storage-based replication using the dedicated Amazon infrastructure across its data centers worldwide.

MySQL in the Cloud Instances

A cloud instance is nothing but a virtual server that each cloud provider names according to their business and marketing needs. Amazon EC2 stands for Elastic Compute Cloud, whereas Google uses the name Compute Engine, and Azure refers to them merely as Virtual Machines.

All of them have different instance types according to the user’s business needs and vary from shallow, basic configuration to outstanding limits. For example, the m2-megamem-416 uses Google compute engine. The m2-megamem-416 instance is a monster that has 416 CPUs and 5,888 GB of RAM.

The MySQL installation process for these instances is the standard one mentioned in Chapter 1. In this case, the most significant advantage of using the cloud instances is the freedom of customizing MySQL and the operating system according to its needs without the limitation that managed databases have.

MySQL in Kubernetes

The most recent option available to deploy MySQL instances is in Kubernetes. Kubernetes and the OpenShift platform have added a way to manage containerized systems, including database clusters. The management is achieved by controllers declared in configuration files. These controllers provide automation to create objects, such as a container or a group of containers called pods, to listen for a specific event and perform a task.

This automation adds complexity to the container-based architecture and stateful applications, such as a database. A Kubernetes operator is a particular type of controller introduced to simplify complex deployments. The operator extends the Kubernetes API with custom resources.

There are many good books written on how Kubernetes works. To keep this section as concise as possible, we will discuss the significant components relevant to the Percona Kubernetes Operator. For a quick introduction to Kubernetes, you can check the documentation from the Linux Foundation. The Figure 14-15 shows the Percona XtraDB cluster components in Kubernetes:

Figure 14-15. The Percona XtraDB cluster components in Kubernetes

To register it here, Oracle also has a MySQL operator. However, its operator has not been under development since 2018. We believe this is because Oracle has been developing its MySQL managed services in its Cloud (Oracle MySQL Database Service).

MariaDB also has its operator. At the time of this writing, the operator is in the alpha stage, so please check its maturity before using it in production.

Presslabs has released its operator. It deploys MySQL instances along with Orchestrator and backup functionalities. This operator is production-ready.

The following section describes how to deploy the Percona operator for Percona XtraDB Cluster, which is considered production-ready.

Deploying Percona XtraDB Cluster in Kubernetes

Let’s look at deploying a Kubernetes cluster in the Google Cloud using Google Cloud SDK and the Percona XtraDB operator.

1. Install Google Cloud SDK

The Google Cloud SDK provides tools and libraries for interacting with Google Cloud products and services. To start the installation, the first step is to download the binary according to your platform. Let’s see an example for macOS:

$ wget https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/goog

le-cloud-sdk-341.0.0-darwin-x86_64.tar.gz

$ tar -xvf google-cloud-sdk-341.0.0-darwin-x86_64.tar.gz

$ *cd google-cloud-sdk/ *

$ *./install.sh *

1. Install kubectl with gcloud

With gcloud installed, we are going to install the kubectl component using the following command:

$ gcloud components install kubectl

1. Create the Kubernetes vluster

To create the Kubernetes cluster, first, we need to authenticate in the Google Cloud service:

$ gcloud auth login

Once authenticated, we need to create the cluster. The command accepts a lot of parameters, but in this case, we will go with the basics to create a Kubernetes cluster:

$ gcloud container clusters create --machine-type n1-standard-4 \

--num-nodes 3 --zone us-central1-b --project support-211414\

*--cluster-version latest vinnie-k8 *

NOTE

The account needs to have the necessary privileges to create the cluster. Also, you need to replace the project with your project name. You may also be required to edit the zone location set to us-central1-b in the above example.

The parameters used are just a small subset of everything available, and we can see the options by running gcloud container clusters --help, but for this case, we just requested a cluster with three nodes of n1-standard-4 type instances.

This process may take a while, especially if there are a lot of nodes. The output will look like this:

Creating cluster vinnie-k8 in us-central1-b... Cluster is being

health-checked (master is healthy)...done.

Created [https://container.googleapis.com/v1/projects/support-211414/zo

nes/us-central1-b/clusters/vinnie-k8].

To inspect the contents of your cluster, go to:

https://console.cloud.google.com/kubernetes/workload_/gcloud/us-central

1-b/vinnie-k8?project=support-211414

kubeconfig entry generated for vinnie-k8.

NAME       LOCATION       MASTER_VERSION    MASTER_IP      MACHINE_TYPE

NODE_VERSION      NUM_NODES  STATUS

vinnie-k8  us-central1-b  1.19.10-gke.1000  34.134.67.128

n1-standard-4  1.19.10-gke.1000  3          RUNNING

And we can check the pods of our Kubernetes cluster in the Google Cloud:

$ kubectl get nodes

NAME                                       STATUS   ROLES    AGE

VERSION

gke-vinnie-k8-default-pool-376c2051-5xgz   Ready    <none>   62s

v1.19.10-gke.1000

gke-vinnie-k8-default-pool-376c2051-w2tk   Ready    <none>   61s

v1.19.10-gke.1000

gke-vinnie-k8-default-pool-376c2051-wxd7   Ready    <none>   62s

v1.19.10-gke.1000

It is also possible to use the Google Cloud interface to deploy the cluster as shown in Figure 14-16:

Figure 14-16. On the main menu, go to Kubernetes Engine and then cluster option

And to create a new cluster, press the CREATE option as shown in Figure 14-17:

Figure 14-17. Create the Kubernetes cluster by pressing CREATE

1. Installing the Percona XtraDB Cluster operator

The documentation for deploying the operator has very detailed steps. We follow the recommended steps.

First, we will configure the Cloud Identity and Access Management (Cloud IAM) to control access to the cluster. The following command will give you the ability to create Roles and RoleBindings:

$ kubectl create clusterrolebinding cluster-admin-binding --clusterrole

cluster-admin --user $(gcloud config get-value core/account)

The return statement confirms the creation:

clusterrolebinding.rbac.authorization.k8s.io/cluster-admin-binding created

Next, we will create a namespace and set the context for the namespace:

$ kubectl create namespace learning-mysql

$ kubectl config set-context $(kubectl config current-context) --namespace=learning-mysql

Now, we will clone the repository and get into the dir:

$ git clone -b v1.8.0 https://github.com/percona/percona-xtradb-cluster-operator

$ cd percona-xtradb-cluster-operator

Deploy the operator with the following command:

$ kubectl apply -f deploy/bundle.yaml

The following confirmation is returned:

customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc

.percona.com created

customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbacku

ps.pxc.percona.com created

customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterresto

res.pxc.percona.com created

customresourcedefinition.apiextensions.k8s.io/perconaxtradbbackups.pxc.

percona.com created

role.rbac.authorization.k8s.io/percona-xtradb-cluster-operator created

serviceaccount/percona-xtradb-cluster-operator created

rolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cl

uster-operator created

deployment.apps/percona-xtradb-cluster-operator created

The operator has been started and we can confirm by running:

$ kubectl get pods

Now we will create the Percona XtraDB cluster:

$  kubectl apply -f deploy/cr.yaml

This step can take some time. After that, you will see all the pods running:

$ kubectl get pods

NAME                                               READY   STATUS    RESTARTS   AGE

cluster1-haproxy-0                                 2/2     Running   0          4m54s

cluster1-haproxy-1                                 2/2     Running   0          3m15s

cluster1-haproxy-2                                 2/2     Running   0          2m52s

cluster1-pxc-0                                     3/3     Running   0          4m54s

cluster1-pxc-1                                     3/3     Running   0          3m16s

cluster1-pxc-2                                     3/3     Running   0          105s

percona-xtradb-cluster-operator-77bfd8cdc5-d7zll   1/1     Running   0          7m18s

During previous steps, the operator has generated several secrets, including the password for the root user, which you will need to access the cluster. To get the generated secrets run the following:

$ kubectl get secret my-cluster-secrets -o yaml

And you will see the following output:

apiVersion: v1

kind: Secret

metadata:

name: my-cluster-secrets

type: Opaque

stringData:

root: root_password

xtrabackup: backup_password

monitor: monitory

clustercheck: clustercheckpassword

proxyadmin: admin_password

pmmserver: admin

operator: operatoradmin

vinnie-macpro:percona-xtradb-cluster-operator vgrippa$ kubectl get secret my-cluster-secrets -o yaml

apiVersion: v1

data:

clustercheck: UFZjdjk0SU4xWGtBSTR2VlVJ

monitor: ZWZja01mOWhBTXZ4bTB0bUZ4eQ==

operator: Vm10R0IxbHA4cVVZTkxqVVI4Mg==

proxyadmin: VXVFbkx1S3RmUTEzVlNOd1c=

root: eU53aWlKT3ZXaXJaeG16OXJK

xtrabackup: V3VNNWRnWUdIblVWaU1OWGY=

kind: Secret

metadata:

creationTimestamp: "2021-05-23T00:49:07Z"

managedFields:

- apiVersion: v1

fieldsType: FieldsV1

fieldsV1:

f:data:

.: {}

f:clustercheck: {}

f:monitor: {}

f:operator: {}

f:proxyadmin: {}

f:root: {}

f:xtrabackup: {}

f:type: {}

manager: percona-xtradb-cluster-operator

operation: Update

time: "2021-05-23T00:49:07Z"

name: my-cluster-secrets

namespace: learning-mysql

resourceVersion: "16598"

selfLink: /api/v1/namespaces/learning-mysql/secrets/my-cluster-secrets

uid: 9d78c4a8-1926-4b7a-84a0-43087a601066

type: Opaque

The actual password is base64-encoded, so we need to run the following to get the root password:

$ echo eU53aWlKT3ZXaXJaeG16OXJK | base64 --decode

yNwiiJOvWirZxmz9rJ

Now that we have the password, to check connectivity with the cluster we will create a client pod:

$ kubectl run -i --rm --tty percona-client --image=percona:8.0 --restart=Never -- bash -il

Then we will connect to MySQL:

$ mysql -h cluster1-haproxy -uroot -pyNwiiJOvWirZxmz9rJ

Note that the operator comes with HAProxy, which is a load balancer (we will discuss in Chapter 15).

Chapter 15. Load Balancing MySQL

There are different ways to connect to MySQL. For example, to perform a write test, a connection is created, the statement is executed, and then the connection is closed. To avoid the cost of opening a connection every time it is needed, the concept of the connection pool was developed. Connection pooling is a technique of creating and managing a pool of connections that are ready for use by any thread of the application.

Extending the concept of high availability Chapter 13 to connections in order to improve the production systems resilience, it is possible to use load balancers to connect to a database cluster. With load balancing and MySQL high availability, it is possible to keep the application running without interruption (or minor downtime). Basically, if the source server or one of the nodes of the database cluster fails, the client would need to reconnect to another database node before it can continue to serve requests.

Load balancers were built to provide transparency for clients when connecting to MySQL infrastructure. In this way, the application does not need to be aware of the MySQL topology. Whether it is a classic replication, group replication, or galera cluster does not matter. The load balancer will provide a online node where it will be possible to read and write queries. Having a robust MySQL architecture and a proper load balancer in-place can help avoid sleepless nights for DBAs.

Load Balancing with Aplication Drivers

To connect an application to MySQL requires a driver. A driver is an adaptor to connect the application to a different system type. It is similar to connecting a video card to your computer; once you start it, a driver will be installed.

Modern MySQL drivers from the most used programming languages supports connection pooling, load balancing, and failover. For example, the JDBC driver for MySQL (MySQL Connector/J).

Another example is the PHP MySQL native driver for master slave (mysqlnd-ms).

The database drivers we’ve mentioned are built to provide transparency for clients when connecting to standalone MySQL server or MySQL replication setups. Adding its library improves code development, since the driver abstracts a substantial work for the developer. This book can’t show how to use the drivers in the code because that would be out of scope.

But for other topologies, like a clustering setup like Galera Cluster for MySQL or MariaDB, the JDBC and PHP drivers are not aware of internal Galera state information. For instance, a Galera donor node might be in read-only while it is helping another node resynchronize (if the SST method is mysqldump or rsync) or it could be up in non-primary state if split-brain happens. Another solution is to use a load balancer between the clients and the database cluster.

ProxySQL Load Balancer

ProxySQL is a SQL proxy. ProxySQL implements the MySQL protocol, and because of this, it allows it to do things that other proxies cannot do. Here are some of its advantages:

- It provides “intelligent” load balancing of application requests onto multiple databases.

- It understands the MySQL traffic that passes through it and can split reads from writes. Understanding the MySQL protocol is especially useful in a source-replica replication setup when writes should only go to the source and reads to the replicas. Or, in the case of Galera Cluster, distribute the read queries evenly (linear read scaling).

- It understands the underlying database topology, whether the instances are up or down, and therefore can route requests to healthy databases.

- It provides query workload analytics and cache, which is useful when analyzing and improving performance.

- It provides administrators with robust, rich query rule definitions to efficiently distribute queries and cache data to maximize the database service efficiency.

ProxySQL runs as a daemon watched by a monitoring process. The process monitors the daemon and restarts it in case of a crash to minimize downtime. The daemon accepts incoming traffic from MySQL clients and forwards it to backend MySQL servers.

The proxy is designed to run continuously without needing to be restarted. Most configurations can be done at runtime using queries similar to SQL statements in the ProxySQL admin interface. These include runtime parameters, server grouping, and traffic-related settings.

While it is common to install ProxySQL on a standalone node between the application and database, this can affect query performance due to the additional latency from network hops. The Figure 15-1 shows the ProxySQL as a middle layer:

Figure 15-1. ProxySQL between the Application and MySQL

To reduce the impact on performance (and avoid the additional network hop), another architecture option is installing ProxySQL on the application servers. The application then connects to ProxySQL (acting as a MySQL server) on localhost, using Unix Domain Socket, and avoiding extra latency. It would then use its routing rules to reach out and talk to the actual MySQL servers with their connection pooling. The application doesn’t have any idea what happens beyond its connection to ProxySQL. The Figure 15-2 shows ProxySQL in the same server as the application:

Figure 15-2. ProxySQL in the same server as the Application

Installing and Configuring ProxySQL

Let’s see how to deploy ProxySQL for a Source and Replica configuration.

The developers of ProxySQL provide official packages for a variety of Linux distributions for all ProxySQL releases on their GitHub releases page, so we’ll download the latest package version from there and install it.

Before installing, the following instances are the ones we will use in this process:

-------------------------------------------------------------+ | vinicius-grippa-default(mysql) | 10.124.33.5 (eth0) | -------------------------------------------------------------+ | vinicius-grippa-node1(mysql) | 10.124.33.169 (eth0) | -------------------------------------------------------------+ | vinicius-grippa-node2(mysql) | 10.124.33.130 (eth0) | -------------------------------------------------------------+ | vinicius-grippa-node3(proxysql) | 10.124.33.170 (eth0) | -------------------------------------------------------------+

Now, find the proper distribution for your operating system. In this example, we will install for Centos 7. First, we will become root, install the MySQL client to connect to ProxySQL, and install ProxySQL itself. We get the URL from the downloads page and refer it to yum:

$ sudo su - root

# yum -y install https://repo.percona.com/yum/percona-release-latest.noarch.rpm

# yum -y install Percona-Server-client-57

# yum install -y https://github.com/sysown/proxysql/releases/download/v2.0.15/proxysql-2.0.15-1-centos7.x86_64.rpm

We have all the requirements to run ProxySQL, but the service doesn’t automatically start after installation, so start it manually:

# sudo systemctl start proxysql

ProxySQL should now be running with its default configuration in place. We can check it by running the following:

# systemctl status proxysql

The output of the ProxySQL process in the active state should be similar to the following:

proxysql.service - High Performance Advanced Proxy for MySQL

Loaded: loaded (/etc/systemd/system/proxysql.service; enabled; vendor preset: disabled)

Active: active (running) since Sun 2021-05-23 18:50:28 UTC; 15s ago

Process: 1422 ExecStart=/usr/bin/proxysql --idle-threads -c /etc/proxysql.cnf $PROXYSQL_OPTS (code=exited, status=0/SUCCESS)

Main PID: 1425 (proxysql)

CGroup: /system.slice/proxysql.service

├─1425 /usr/bin/proxysql --idle-threads -c /etc/proxysql.cnf

└─1426 /usr/bin/proxysql --idle-threads -c /etc/proxysql.cnf

May 23 18:50:27 vinicius-grippa-node3 systemd[1]: Starting High Performance Advanced Proxy for MySQL...

May 23 18:50:27 vinicius-grippa-node3 proxysql[1422]: 2021-05-23 18:50:27 [INFO] Using config file /etc/proxysql.cnf

May 23 18:50:27 vinicius-grippa-node3 proxysql[1422]: 2021-05-23 18:50:27 [INFO] Using OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

May 23 18:50:27 vinicius-grippa-node3 proxysql[1422]: 2021-05-23 18:50:27 [INFO] No SSL keys/certificates found in datadir (/var/lib/proxysql). Generating new keys/certificates.

May 23 18:50:28 vinicius-grippa-node3 systemd[1]: Started High Performance Advanced Proxy for MySQL.

ProxySQL splits the application interface from the admin interface. This means that ProxySQL will listen on two network ports. The Admin interface will be on 6032, and the application will listen on 6033 (to make it easier to remember, that’s the reverse of MySQL default port 3306).

Next, ProxySQL needs to communicate with the MySQL nodes to be able to check their condition. To achieve this task, ProxySQL needs to connect to each server with a dedicated user.

First, we are going to create the user in the source server. Connect to the MySQL source instance and run this:

mysql> CREATE USER 'proxysql'@'%' IDENTIFIED by '$3Kr$t';

mysql> GRANT USAGE ON *.* TO 'proxysql'@'%';

Next, we will configure ProxySQL parameters to recognize the user. Let’s connect to ProxySQL:

# mysql -uadmin -padmin -h 127.0.0.1 -P 6032

And now, set the parameters:

proxysql> UPDATE global_variables SET variable_value='proxysql'

-> WHERE variable_name='mysql-monitor_username';

proxysql> UPDATE global_variables SET variable_value='$3Kr$t'

-> WHERE variable_name='mysql-monitor_password';

proxysql> LOAD MYSQL VARIABLES TO RUNTIME;

proxysql> SAVE MYSQL VARIABLES TO DISK;

Now that we’ve set the user in the database and ProxySQL, it is time to tell ProxySQL which MySQL servers are present in the topology:

proxysql> INSERT INTO mysql_servers(hostgroup_id, hostname, port)

-> VALUES (10,'10.124.33.5',3306);

proxysql> INSERT INTO mysql_servers(hostgroup_id, hostname, port)

-> VALUES (11,'10.124.33.169',3306);

proxysql> INSERT INTO mysql_servers(hostgroup_id, hostname, port)

-> VALUES (11,'10.124.33.130',3306);

proxysql> LOAD MYSQL SERVERS TO RUNTIME;

proxysql> SAVE MYSQL SERVERS TO DISK;

The next step is to define who will be our writer and reader groups. The servers present in the writer group will be able to receive DML operations, while SELECT queries will use the servers in the reader group. In this example, the host group 10 will be the writer and host group 11 will be the reader:

proxysql> INSERT INTO mysql_replication_hostgroups

-> (writer_hostgroup, reader_hostgroup) VALUES (10, 11);

proxysql> LOAD MYSQL SERVERS TO RUNTIME;

proxysql> SAVE MYSQL SERVERS TO DISK;

Next, ProxySQL must have users that can access backend nodes to manage connections. Let’s create the user in the backend source server:

mysql> CREATE USER 'app'@'%' IDENTIFIED by '$3Kr$t';

mysql> GRANT ALL PRIVILEGES ON *.* TO 'app'@'%';

And now we will configure ProxySQL with the user:

proxysql> INSERT INTO mysql_users (username,password,default_hostgroup)

-> VALUES ('app','$3Kr$t',10);

proxysql> LOAD MYSQL USERS TO RUNTIME;

proxysql> SAVE MYSQL USERS TO DISK;

The next step is the most exciting because it is here that we define the rules. The rules will tell ProxySQL where to send write and read queries, load balancing them in the servers:

proxysql> INSERT INTO mysql_query_rules

-> (rule_id,username,destination_hostgroup,active,match_digest,apply)

-> VALUES(1,'app',10,1,'^SELECT.*FOR UPDATE',1);

proxysql> INSERT INTO mysql_query_rules

-> (rule_id,username,destination_hostgroup,active,match_digest,apply)

-> VALUES(2,'app',11,1,'^SELECT ',1);

proxysql> LOAD MYSQL QUERY RULES TO RUNTIME;

proxysql> SAVE MYSQL QUERY RULES TO DISK;

ProxySQL has a thread responsible for connecting on each server listed in the mysql_servers table and checking the value of the read_only variable. So, in case the replica is showing up in the writer group like this:

proxysql> SELECT * FROM mysql_servers;

+--------------+---------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+

| hostgroup_id | hostname      | port | gtid_port | status | weight |

compression | max_connections | max_replication_lag | use_ssl |

max_latency_ms | comment |

+--------------+---------------+------+-----------+--------+--------+--

-----------+-----------------+---------------------+---------+---------

-------+---------+

| 10           | 10.124.33.5   | 3306 | 0         | ONLINE | 1      | 0

| 1000            | 0                   | 0       | 0              |         |

| 11           | 10.124.33.169 | 3306 | 0         | ONLINE | 1      | 0

| 1000            | 0                   | 0       | 0              |

|

| 11           | 10.124.33.130 | 3306 | 0         | ONLINE | 1      | 0

| 1000            | 0                   | 0       | 0              |

|

+--------------+---------------+------+-----------+--------+--------+--

-----------+-----------------+---------------------+---------+---------

-------+---------+

3 rows in set (0.00 sec)

We need to set the read_only option in the replica servers, so these servers become able to serve read queries:

mysql> SET GLOBAL read_only=1;

And now, we are good to use our application. Running the following command should return the hostname that ProxySQL connected:

$ mysql -uapp -p'$3Kr$t' -h 127.0.0.1 -P 6033 -e "select @@hostname;"

+-----------------------+

| @@hostname            |

+-----------------------+

| vinicius-grippa-node1 |

+-----------------------+

ProxySQL has a lot more features and flexibility than what we show here. The idea is to present the tool and make the DBA aware of such an option when deciding the architecture.

NOTE

As we saw when configuring replication earlier, we want to reinforce the idea that ProxySQL needs to reach the MySQL servers; otherwise, it won’t work.

HAProxy Load Balancer

HAProxy stands for High Availability Proxy, and it is a TCP/HTTP load balancer. It distributes a workload across a set of servers to maximize performance and optimize resource usage.

With the intent to expand reader knowledge regarding MySQL architectures and different topologies, we will configure a Percona XtraDB Cluster (Galera Cluster) with HAProxy in this section instead of a classic replication.

The architecture options are similar to ProxySQL. HAProxy can be placed together with the application or in a middle layer. The image Figure 15-3 is an example when HAProxy is placed in the same server as the application:

Figure 15-3. HAProxy together with the Application

And the figure Figure 15-4 shows a topology with HAProxy in a middle layer:

Figure 15-4. HAProxy in the middle layer with a dedicated server

Again, these are archictectures with different pros and cons. While in the first one we do not have an extra hop (reduces latency), we add extra load to the application server. Also, you have to configure each HAProxy on each applications server.

On the other hand, having HAProxy in the middle layer faciitates managing it. And you have expanded high availability because the application can connect to any HAProxy server.

Installing and Configuring HAProxy

Common operating systems such as Red Hat/Centos and Debian/Ubuntu provide the HAProxy package, and you can install it using the package manager. The installation process is relatively easy.

For Debian or Ubuntu:

# apt update

# apt install haproxy

For Red Hat or CentOS:

# sudo yum update

# sudo yum install haproxy

When installed, HAProxy will set the default path for the configuration file in /etc/haproxy/haproxy.cfg.

Now, before starting HAProxy, we need to configure it. For this demonstration, in our first scenario HAProxy will be located in the same server as the application. Also, the following will be the IPs of our three-node Galera Cluster:

172.16.3.45/Port:3306

172.16.1.72/Port:3306

172.16.0.161/Port:3306

Let’s open our /etc/haproxy/haproxy.cfg file and look at it. There are many parameters to customize and split in sections. The Table 15-1 shows the basic HAProxy parameters:

Table 15-1. HAProxy options (with links to HAProxy documentation)

Parameter

Description

global

A section in the configuration file for process-wide parameters.

defaults

A section in the configuration file for default parameters for all other following sections.

listen

A section in the configuration file that defines a complete proxy with its frontend and backend parts combined in one section.

balance

Load balancing algorithm to be used in a backend.

clitimeout

Set the maximum inactivity time on the client side.

contimeout

Set the maximum time to wait for a connection attempt to a server to succeed.

daemon

Makes the process fork into background (recommended mode of operation).

gid

Changes the process’ group ID to <number>.

log

Adds a global syslog server.

maxconn

Sets the maximum per-process number of concurrent connections to  <number>.

mode

Set the running mode or protocol of the instance.

option dontlognull

Disable logging of null connections.

optiontcplog

Enable advanced logging of TCP connections with session state and timers.

To make our HAProxy work, we will use the following configuration file based on our settings:

global

log /dev/log   local0

log /dev/log   local1 notice

maxconn 4096

#debug

#quiet

chroot      /var/lib/haproxy

pidfile     /var/run/haproxy.pid

user        haproxy

group       haproxy

daemon

# turn on stats unix socket

stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------

# common defaults that all the 'listen' and 'backend' sections will

# use if not designated in their block

#---------------------------------------------------------------------

defaults

log     global

mode    http

option  tcplog

option  dontlognull

retries 3

redispatch

maxconn 2000

contimeout      5000

clitimeout      50000

srvtimeout      50000

#---------------------------------------------------------------------

# round robin balancing between the various backends

#---------------------------------------------------------------------

listen mysql-pxc-cluster 0.0.0.0:3307

mode tcp

bind *:3307

timeout client  10800s

timeout server  10800s

balance roundrobin

option httpchk

server vinicius-grippa-node2 172.16.0.161:3306 check port 9200

inter 12000 rise 3 fall 3

server vinicius-grippa-node1 172.16.1.72:3306 check port 9200 inter 12000 rise 3 fall 3

server vinicius-grippa-default 172.16.3.45:3306 check port 9200

inter 12000 rise 3 fall 3

To start HAProxy, we need to use the haproxy command. We can pass any number of configuration parameters on the command line. To use a configuration file, use the -f option. For example:

Passing one configuration file:

# sudo haproxy -f /etc/haproxy/haproxy.cfg

Passing multiple configuration files:

# sudo haproxy -f /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy-2.cfg

Passing a directory:

# sudo haproxy -f conf-dir

With this configuration, HAProxy will balance the load between three nodes. In this case, it only checks if the mysqld process listens on port 3306, but it doesn’t take into an account the state of the node. So it could be sending queries to the node that has mysqld running even if it’s in JOINING or DISCONNECTED state.

To check the current status of a node we need a more complex check. This idea was taken from codership-team google groups.

To implement this setup, we will need two scripts:

- clustercheck: Located in /usr/local/bin and a config for xinetd

- mysqlchk: Located in /etc/xinetd.d on each node

Both scripts are available in binaries and source distributions for the Galera Cluster.

Change the /etc/services file by adding the following line on each node:

mysqlchk        9200/tcp                # mysqlchk

If /etc/services file does not exist, probably xinetd is not installed.

To install for Centos/Red Hat:

# yum install -y xinetd

To install for Debian/Ubuntu:

# sudo apt-get install -y xinetd

Next, we need to create a MySQL user so the script can check if the node is healthy. Ideally, for security reasons, we are going to create the user with minimum privileges required:

mysql> GRANT PROCESS ON *.* TO 'clustercheckuser'@'localhost' IDENTIFIED BY 'clustercheckpassword!'

And to validate how our node is performing the health check, we can run the following command and observe the output:

# /usr/bin/clustercheck

HTTP/1.1 200 OK

Content-Type: text/plain

Connection: close

Content-Length: 40

Percona XtraDB Cluster Node is synced.

If we do this for all nodes we will be ready to test whether our HAProxy is working. The easiest way is connecting to it and running some MySQL commands. Let’s run a MySQL command that retrieves the hostname in which we are connected:

# mysql -uroot -psecret -h 127.0.0.1 -P 3307 -e "select @@hostname"

mysql: [Warning] Using a password on the command line interface can be

insecure.

+-----------------------+

| @@hostname            |

+-----------------------+

| vinicius-grippa-node1 |

+-----------------------+

Running a second time:

$ mysql -uroot -psecret -h 127.0.0.1 -P 3307 -e "select @@hostname"

mysql: [Warning] Using a password on the command line interface can be

insecure.

+-----------------------+

| @@hostname            |

+-----------------------+

| vinicius-grippa-node2 |

+-----------------------+

Running a third time:

$ mysql -uroot -psecret -h 127.0.0.1 -P 3307 -e "select @@hostname"

mysql: [Warning] Using a password on the command line interface can be

insecure.

+-------------------------+

| @@hostname              |

+-------------------------+

| vinicius-grippa-default |

+-------------------------+

As you can see, our HAProxy is connecting in a round-robin fashion. If we shut down one of the nodes, HAProxy will route only to the remaining ones.

MySQL Router

The MySQL Router is responsible for distributing the traffic between members of a InnoDB cluster. It is a proxy-like solution to hide cluster topology from applications, so applications don’t need to know which member of a cluster is the primary node and which are secondaries. Note that MySQL router will not work with Galera clusters. The MySQL router was developed for InnoDB Cluster only.

The MySQL Router tool is capable of performing read/write splitting by exposing different interfaces. A common setup is to have one read-write interface and one read-only interface. This is the default behavior that also exposes two similar interfaces to use x-protocol (used for CRUD operations and async calls).

The read and write split is done using the concept of roles: Primary for writes and Secondary for read-only. This is analogous to how members of cluster are named. Additionally, each interface is exposed via a TCP port so applications only need to know the IP:port combination used for writes and the one used for reads. Then MySQL Router will take care of connections to cluster members depending on the type of traffic to the server.

When working in a production environment, the MySQL server instances that make up an InnoDB Cluster run on multiple host machines as part of a network rather than on single machine. So, as it happens, for ProxySQL and HAProxy, the MySQL router can be a middle layer in the architecture.

The image Figure 15-5 illustrates how the production scenario works.

Figure 15-5. MySQL InnoDB Cluster Production Deployment

Now, to start our example, let’s take a look in our MySQL members that are part of the InnoDB Cluster:

mysql> SELECT member_host, member_port, member_state, member_role

-> FROM performance_schema.replication_group_members;

+--------------+-------------+--------------+-------------+

| member_host  | member_port | member_state | member_role |

+--------------+-------------+--------------+-------------+

| 172.16.3.9   |        3306 | ONLINE       | SECONDARY   |

| 172.16.3.127 |        3306 | ONLINE       | SECONDARY   |

| 172.16.3.120 |        3306 | ONLINE       | PRIMARY     |

+--------------+-------------+--------------+-------------+

3 rows in set (0.00 sec)

mysql> SELECT cluster_name FROM mysql_innodb_cluster_metadata.clusters;

+--------------+

| cluster_name |

+--------------+

| cluster1     |

+--------------+

1 row in set (0.00 sec)

Now that we have the configuration of the MySQL nodes and the cluster name, we can start configuring our MySQL Router. For performance purposes it’s recommended to set up MySQL Router in the same place as the application, considering an instance per application server, so we will place our router in the application server. First, we are going to identify the compatible MySQL router with our OS:

# cat /etc/*release

CentOS Linux release 7.9.2009 (Core)

Now, we will check the download page and install using yum:

# yum install -y https://dev.mysql.com/get/Downloads/MySQL-Router/mysql-

router-community-8.0.23-1.el7.x86_64.rpm

Loaded plugins: fastestmirror

mysql-router-community-8.0.23-1.el7.x86_64.rpm                                                                                                                                           |  34 MB  00:00:01

Examining /var/tmp/yum-root-_ljdTQ/mysql-router-community-8.0.23-1.el7.x

86_64.rpm: mysql-router-community-8.0.23-1.el7.x86_64

Marking /var/tmp/yum-root-_ljdTQ/mysql-router-community-8.0.23-1.el7.x86

_64.rpm to be installed

Resolving Dependencies

--> Running transaction check

---> Package mysql-router-community.x86_64 0:8.0.23-1.el7 will be

installed

--> Finished Dependency Resolution

Dependencies Resolved

========================================================================

========================================================================

================================================================

Package                                            Arch

Version                                    Repository

Size

========================================================================

========================================================================

================================================================

Installing:

mysql-router-community                             x86_64

8.0.23-1.el7

/mysql-router-community-8.0.23-1.el7.x86_64

176 M

Transaction Summary

========================================================================

========================================================================

================================================================

Install  1 Package

Total size: 176 M

Installed size: 176 M

Downloading packages:

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

Installing : mysql-router-community-8.0.23-1.el7.x86_64

1/1

Verifying  : mysql-router-community-8.0.23-1.el7.x86_64

1/1

Installed:

mysql-router-community.x86_64 0:8.0.23-1.el7

Complete!

Now that our MySQL router is installed, we need to create the dedicated directory for the MySQL router operation:

# mkdir /var/lib/mysqlrouter

Next, we are going to bootstrap the MySQL router. The bootstrap will configure the router for operation with a MySQL InnoDB Cluster.

# mysqlrouter --bootstrap root@172.16.3.120:3306 \

--directory /var/lib/mysqlrouter --conf-use-sockets \

--account app_router --account-create always \

--user=mysql

Please enter MySQL password for root:

# Bootstrapping MySQL Router instance at '/var/lib/mysqlrouter'...

Please enter MySQL password for app_router:

- Creating account(s)

- Verifying account (using it to run SQL queries that would be run by

Router)

- Storing account in keyring

- Adjusting permissions of generated files

- Creating configuration /var/lib/mysqlrouter/mysqlrouter.conf

# MySQL Router configured for the InnoDB Cluster 'cluster1'

After this MySQL Router has been started with the generated

configuration

$ mysqlrouter -c /var/lib/mysqlrouter/mysqlrouter.conf

the cluster 'cluster1' can be reached by connecting to:

## MySQL Classic protocol

- Read/Write Connections: localhost:6446, /var/lib/mysqlrouter/mysql.sock

- Read/Only Connections:  localhost:6447,

/var/lib/mysqlrouter/mysqlro.sock

## MySQL X protocol

- Read/Write Connections: localhost:64460,

/var/lib/mysqlrouter/mysqlx.sock

- Read/Only Connections:  localhost:64470,

/var/lib/mysqlrouter/mysqlxro.sock

In the command line, we are telling to the router to connect with the user root, in our Primary server (172.16.3.120), at port 3306. We are also telling the router to create a socket file so we can connect using it.finally, we are creating a new user (app_router) to use in our application.

Let’s have a look at the contents that the bootstrap process created in our configuration directory(/var/lib/mysqlrouter):

# ls -l | awk '{print $9}'

data

log

mysqlrouter.conf

mysqlrouter.key

run

start.sh

stop.sh

A generated MySQL Router configuration file (mysqlrouter.conf) looks similar to this:

# cat mysqlrouter.conf

# File automatically generated during MySQL Router bootstrap

[DEFAULT]

user=mysql

logging_folder=/var/lib/mysqlrouter/log

runtime_folder=/var/lib/mysqlrouter/run

data_folder=/var/lib/mysqlrouter/data

keyring_path=/var/lib/mysqlrouter/data/keyring

master_key_path=/var/lib/mysqlrouter/mysqlrouter.key

connect_timeout=15

read_timeout=30

dynamic_state=/var/lib/mysqlrouter/data/state.json

client_ssl_cert=/var/lib/mysqlrouter/data/router-cert.pem

client_ssl_key=/var/lib/mysqlrouter/data/router-key.pem

client_ssl_mode=PREFERRED

server_ssl_mode=AS_CLIENT

server_ssl_verify=DISABLED

[logger]

level = INFO

[metadata_cache:cluster1]

cluster_type=gr

router_id=1

user=app_router

metadata_cluster=cluster1

ttl=0.5

auth_cache_ttl=-1

auth_cache_refresh_interval=2

use_gr_notifications=0

[routing:cluster1_rw]

bind_address=0.0.0.0

bind_port=6446

socket=/var/lib/mysqlrouter/mysql.sock

destinations=metadata-cache://cluster1/?role=PRIMARY

routing_strategy=first-available

protocol=classic

[routing:cluster1_ro]

bind_address=0.0.0.0

bind_port=6447

socket=/var/lib/mysqlrouter/mysqlro.sock

destinations=metadata-cache://cluster1/?role=SECONDARY

routing_strategy=round-robin-with-fallback

protocol=classic

[routing:cluster1_x_rw]

bind_address=0.0.0.0

bind_port=64460

socket=/var/lib/mysqlrouter/mysqlx.sock

destinations=metadata-cache://cluster1/?role=PRIMARY

routing_strategy=first-available

protocol=x

[routing:cluster1_x_ro]

bind_address=0.0.0.0

bind_port=64470

socket=/var/lib/mysqlrouter/mysqlxro.sock

destinations=metadata-cache://cluster1/?role=SECONDARY

routing_strategy=round-robin-with-fallback

protocol=x

[http_server]

port=8443

ssl=1

ssl_cert=/var/lib/mysqlrouter/data/router-cert.pem

ssl_key=/var/lib/mysqlrouter/data/router-key.pem

[http_auth_realm:default_auth_realm]

backend=default_auth_backend

method=basic

name=default_realm

[rest_router]

require_realm=default_auth_realm

[rest_api]

[http_auth_backend:default_auth_backend]

backend=metadata_cache

[rest_routing]

require_realm=default_auth_realm

[rest_metadata_cache]

require_realm=default_auth_realm

In this example, MySQL Router configured four ports (two ports to read/write using regular protocol, two read/write for the X protocol) and four sockets. Ports are added by default, and sockets were added by passing in --conf-use-sockets. The InnoDB Cluster named cluster1 is the source of the metadata, and the destinations are using the InnoDB Cluster metadata cache to dynamically configure host information.

By executing the start.sh script we can start the MySQL router daemon:

# ./start.sh

# PID 1684 written to '/var/lib/mysqlrouter/mysqlrouter.pid'

logging facility initialized, switching logging to loggers specified in

configuration

Now, we can observe the process running:

# ps -ef | grep -i mysqlrou

root      1683     1  0 17:36 pts/0    00:00:00 sudo

ROUTER_PID=/var/lib/mysqlrouter/mysqlrouter.pid /usr/bin/mysqlrouter -c

/var/lib/mysqlrouter/mysqlrouter.conf --user=mysql

mysql     1684  1683  0 17:36 pts/0    00:00:17 /usr/bin/mysqlrouter -c

/var/lib/mysqlrouter/mysqlrouter.conf --user=mysql

root      1733  1538  0 17:41 pts/0    00:00:00 grep --color=auto -i

mysqlrou

And the ports open:

# netstat -tulnp | grep -i mysqlrouter

tcp        0      0 0.0.0.0:64470           0.0.0.0:*

LISTEN      1684/mysqlrouter

tcp        0      0 0.0.0.0:8443            0.0.0.0:*

LISTEN      1684/mysqlrouter

tcp        0      0 0.0.0.0:64460           0.0.0.0:*

LISTEN      1684/mysqlrouter

tcp        0      0 0.0.0.0:6446            0.0.0.0:*

LISTEN      1684/mysqlrouter

tcp        0      0 0.0.0.0:6447            0.0.0.0:*

LISTEN      1684/mysqlrouter

All right, we have configured the MySQL router with the InnoDB Cluster . Now we can test this with read and read/write connections. First, we will connect to the writer port:

# mysql -uroot -psecret -h 127.0.0.1 -P 6446 \

-e "create database learning_mysql;

# mysql -uroot -psecret -h 127.0.0.1 -P 6446 \

-e "use learning_mysql; select database()"

+----------------+

| database()     |

+----------------+

| learning_mysql |

+----------------+

As you can see, it is possible to execute reads and writes in the writer port (6446).

Now we will check the read port (6447) using a SELECT statement:

# mysql -uroot -psecret -h 127.0.0.1 -P 6447 \

-e "use learning_mysql; select database()"

+----------------+

| database()     |

+----------------+

| learning_mysql |

+----------------+

And it’s working. Let’s try to execute a write:

# mysql -uroot -psecret -h 127.0.0.1 -P 6447 \

-e "create database learning_mysql_write;"

ERROR 1290 (HY000) at line 1: The MySQL server is running with the

--super-read-only option so it cannot execute this statement

So, the read port only accepts reads. It is also possible to see the router load-balancing the reads:

# mysql -uroot -psecret -h 127.0.0.1 -P 6447 -e "select @@hostname"

+-----------------------+

| @@hostname            |

+-----------------------+

| vinicius-grippa-node1 |

+-----------------------+

# mysql -uroot -psecret -h 127.0.0.1 -P 6447 -e "select @@hostname"

insecure.

+-----------------------+

| @@hostname            |

+-----------------------+

| vinicius-grippa-node2 |

+-----------------------+

In this way, if any downtime occurs in one of the MySQL nodes, MySQL router will route the queries to the remaining active nodes.

Chapter 16. Miscellaneous Topics

The idea of this chapter is to go beyond troubleshooting a query, system overload, or setting up different MySQL topologies. We want to show the arsenal of tools available for you to make daily tasks easier or investigate complex issues. Let’s start with MySQL Shell.

MySQL Shell

MySQL Shell is an advanced client and code editor for MySQL. It expands the functionality of the traditional MySQL client that most DBAs worked with in MySQL 5.6 and 5.7. MySQL Shell supports programming languages such as Python, JavaScript, and SQL. It also extends functionalities using an API command syntax. For example, it is possible to customize scripts to administer an InnoDB Cluster. From MySQL Shell, you can also start and configure MySQL sandbox instances.

Installing MySQL Shell

For supported Linux distributions, the easiest way to install MySQL Shell on Linux is to use the MySQL APT repository or MySQL Yum repository. Let’s see how to install it for Ubuntu Focal Fossa and RHEL/Centos versions.

Installing MySQL Shell on Ubuntu 20.04 Focal Fossa

Installing MySQL Shell in Ubuntu is relatively easy since it is part of the regular repositories.

First, we need to configure the MySQL repository. We need to download it to our server:

# wget  https://dev.mysql.com/get/mysql-apt-config_0.8.16-1_all.deb

# dpkg -i mysql-apt-config_0.8.16-1_all.deb

Once installed, we will make sure to update our package information:

# apt-get update

And we just execute the install command:

# apt-get install mysql-shell

We can now start MySQL Shell using the command line:

# mysqlsh

MySQL Shell 8.0.23

Copyright (c) 2016, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates.

Other names may be trademarks of their respective owners.

Type '\help' or '\?' for help; '\quit' to exit.

MySQL  JS >

Installing MySQL Shell in Centos 8

To install MySQL Shell in Centos 8, we need to follow the same logic as described for Ubuntu. But first we need to make sure the default MySQL package present in Centos 8 is disabled:

# yum remove mysql-community-release -y

No match for argument: mysql-community-release

No packages marked for removal.

Dependencies resolved.

Nothing to do.

Complete!

# dnf erase mysql-community-release

No match for argument: mysql-community-release

No packages marked for removal.

Dependencies resolved.

Nothing to do.

Complete!

Next, we are going to configure our yum repository. We need to get the correct OS version from the download page:

# yum install https://dev.mysql.com/get/mysql80-community-release-el8-1.noarch.rpm -y

With the repository installed, we will install the MySQL Shell binary:

# yum install mysql-shell -y

And we can validate by running it:

# mysqlsh

MySQL Shell 8.0.23

Copyright (c) 2016, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates.

Other names may be trademarks of their respective owners.

Type '\help' or '\?' for help; '\quit' to exit.

MySQL  JS >

Deploying a Sandbox InnoDB Cluster with MySQL Shell

MySQL shell automates the deployment of sandbox instances with AdminAPI that provides the dba.deploySandboxInstance(*port_number*) command.

By default, the sandboxes instances are placed in a directory named $HOME/mysql-sandboxes/port. Let’s see an example changing the directory:

# mkdir /var/lib/sandboxes

# mysqlsh

MySQL  JS > shell.options.sandboxDir=/var/lib/sandboxes

/var/lib/sandboxes

A prerequiste to deploy a sandbox instance is to install MySQL binaries. If necessary, review Chapter 1. During the deploy we are asked to type the root password in order to complete the deploy:

MySQL  JS > dba.deploySandboxInstance(3310)

A new MySQL sandbox instance will be created on this host in

/var/lib/sandboxes/3310

Warning: Sandbox instances are only suitable for deploying and

running on your local machine for testing purposes and are not

accessible from external networks.

Please enter a MySQL root password for the new instance: ******

Deploying new MySQL instance...

Instance localhost:3310 successfully deployed and started.

Use shell.connect('root@localhost:3310') to connect to the instance.

We are going to deploy two more instances:

MySQL  JS > dba.deploySandboxInstance(3320)

MySQL  JS > dba.deploySandboxInstance(3330)

The next step is to create the InnoDB Cluster while connected to the seed MySQL Server instance. The seed instance is the instance we are connected to via MySQL Shell and that we want to replicate to the other instances. In this example, the sandbox instances are blank instances, so we can choose any instance. In a production setup the seed instance would be the one that contains the existing dataset and would be replicated to the other instances in the cluster.

We will connect MySQL Shell to the seed instance, in this case the one at port 3310:

MySQL  JS > \connect root@localhost:3310

Creating a session to 'root@localhost:3310'

Please provide the password for 'root@localhost:3310': ******

Save password for 'root@localhost:3310'? [Y]es/[N]o/Ne[v]er (default No): Y

Fetching schema names for autocompletion... Press ^C to stop.

Your MySQL connection id is 12

Server version: 8.0.21 Source distribution

No default schema selected; type \use <schema> to set one.

Subsequently, we will use the createCluster() method to create the InnoDB cluster with the currently connected instance as the seed:

MySQL  localhost:3310 ssl  JS > var cluster = dba.createCluster(learning_mysql)

A new InnoDB cluster will be created on instance 'localhost:3310'.

Validating instance configuration at localhost:3310...

NOTE: Instance detected as a sandbox.

Please note that sandbox instances are only suitable for deploying test clusters for use within the same host.

This instance reports its own address as 127.0.0.1:3310

Instance configuration is suitable.

NOTE: Group Replication will communicate with other members using '127.0.0.1:33101'. Use the localAddress option to override.

Creating InnoDB cluster 'learning_mysql' on '127.0.0.1:3310'...

Adding Seed Instance...

Cluster successfully created. Use Cluster.addInstance() to add MySQL instances.

At least 3 instances are needed for the cluster to be able to withstand up to

one server failure.

As we can see in the output, three instances are capable of keeping the database online with one server failure, which is why we deployed three sandbox instances.

The next step is to add secondary instances to our learning_mysql InnoDB cluster. Any transactions that were executed by the seed instance are re-executed by each secondary instance as it is added.

The seed instance in this example was recently created, so it is nearly empty. Therefore, there is little data that needs to be replicated from the seed instance to the secondary instances. If necessary to replicate data, MySQL will use the clone plugin to configure the instances automatically.

Let’s add one secondary to see the process in action. To add the second instance to the InnoDB cluster:

MySQL  localhost:3310 ssl  JS >  cluster.addInstance(root@localhost:3320)

NOTE: The target instance '127.0.0.1:3320' has not been pre-provisioned (GTID set is empty). The Shell is unable to decide whether incremental state recovery can correctly provision it.

The safest and most convenient way to provision a new instance is through automatic clone provisioning, which will completely overwrite the state of '127.0.0.1:3320' with a physical snapshot from an existing cluster member. To use this method by default, set the 'recoveryMethod' option to 'clone'.

The incremental state recovery may be safely used if you are sure all updates ever executed in the cluster were done with GTIDs enabled, there are no purged transactions and the new instance contains the same GTID set as the cluster or a subset of it. To use this method by default, set the 'recoveryMethod' option to 'incremental'.

Please select a recovery method [C]lone/[I]ncremental recovery/[A]bort (default Clone):

Validating instance configuration at localhost:3320...

NOTE: Instance detected as a sandbox.

Please note that sandbox instances are only suitable for deploying test clusters for use within the same host.

This instance reports its own address as 127.0.0.1:3320

Instance configuration is suitable.

NOTE: Group Replication will communicate with other members using '127.0.0.1:33201'. Use the localAddress option to override.

A new instance will be added to the InnoDB cluster. Depending on the amount of

data on the cluster this might take from a few seconds to several hours.

Adding instance to the cluster...

Monitoring recovery process of the new cluster member. Press ^C to stop monitoring and let it continue in background.

Clone based state recovery is now in progress.

NOTE: A server restart is expected to happen as part of the clone process. If the

server does not support the RESTART command or does not come back after a

while, you may need to manually start it back.

* Waiting for clone to finish...

NOTE: 127.0.0.1:3320 is being cloned from 127.0.0.1:3310

** Stage DROP DATA: Completed

** Clone Transfer

FILE COPY  ############################################################  100%  Completed

PAGE COPY  ############################################################  100%  Completed

REDO COPY  ############################################################  100%  Completed

NOTE: 127.0.0.1:3320 is shutting down...

* Waiting for server restart... ready

* 127.0.0.1:3320 has restarted, waiting for clone to finish...

** Stage RESTART: Completed

* Clone process has finished: 59.62 MB transferred in about 1 second (~59.62 MB/s)

State recovery already finished for '127.0.0.1:3320'

The instance '127.0.0.1:3320' was successfully added to the cluster

Add the third instance:

MySQL  localhost:3310 ssl  JS >  cluster.addInstance(root@localhost:3320)

At this point weu have created a cluster with three instances: a primary, and two secondaries. We can see the status by running the following:

MySQL  localhost:3310 ssl  JS > cluster.status()

{

"clusterName": "learning_mysql",

"defaultReplicaSet": {

"name": "default",

"primary": "127.0.0.1:3310",

"ssl": "REQUIRED",

"status": "OK",

"statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",

"topology": {

"127.0.0.1:3310": {

"address": "127.0.0.1:3310",

"mode": "R/W",

"readReplicas": {},

"replicationLag": null,

"role": "HA",

"status": "ONLINE",

"version": "8.0.21"

},

"127.0.0.1:3320": {

"address": "127.0.0.1:3320",

"mode": "R/O",

"readReplicas": {},

"replicationLag": null,

"role": "HA",

"status": "ONLINE",

"version": "8.0.21"

},

"127.0.0.1:3330": {

"address": "127.0.0.1:3330",

"mode": "R/O",

"readReplicas": {},

"replicationLag": null,

"role": "HA",

"status": "ONLINE",

"version": "8.0.21"

}

},

"topologyMode": "Single-Primary"

},

"groupInformationSourceMember": "127.0.0.1:3310"

Assuming MySQL Router is already installed (see “MySQL Router”), the only required step is to bootstrap it with the location of the InnoDB Cluster metadata server. The following does this using all default settings:

# mysqlrouter --bootstrap root@localhost:3310 --user=mysqlrouter

And we observe the router being bootstrapped:

# mysqlrouter --bootstrap root@localhost:3310 --user=mysqlrouter

Please enter MySQL password for root:

# Bootstrapping system MySQL Router instance...

- Creating account(s) (only those that are needed, if any)

- Verifying account (using it to run SQL queries that would be run by Router)

- Storing account in keyring

- Adjusting permissions of generated files

- Creating configuration /etc/mysqlrouter/mysqlrouter.conf

Existing configuration backed up to '/etc/mysqlrouter/mysqlrouter.conf.bak'

# MySQL Router configured for the InnoDB Cluster 'learning_mysql'

After this MySQL Router has been started with the generated configuration

$ /etc/init.d/mysqlrouter restart

or

$ systemctl start mysqlrouter

or

$ mysqlrouter -c /etc/mysqlrouter/mysqlrouter.conf

the cluster 'learning_mysql' can be reached by connecting to:

## MySQL Classic protocol

- Read/Write Connections: localhost:6446

- Read/Only Connections:  localhost:6447

## MySQL X protocol

- Read/Write Connections: localhost:64460

- Read/Only Connections:  localhost:64470

MySQL Shell Utilities

As we’ve said, MySQL shell is a powerful, advanced client and code editor for MySQL. Among its many functionalities are utilities to create a logical dump and do a logical restore for the entire database instance, including users. The advantage, compared to mysqldump for example, is that the utility has parallelization capacity, improving the dump and restore speed by many times.

Here are the utilities to execute the dump and restore process:

- util.dumpInstance(): Dump an entire database instance, including users

- util.dumpSchemas(): Dump a set of schemas

- util.loadDump(): Load a dump into a target database

- util.dumpTables(): Load specific tables and views

util.dumpInstance()

The dumpInstance() utility will dump all the databases that are present in the MySQL data directory (“The Contents of the MySQL Directory”). It will exclude the information_schema, mysql_, _ndbinfo, performance_schema, and sys schemas while taking the dump.

In this utility, there is a dry run option to inspect the schemas and view the compatibility issues, and then run the dump with the appropriate compatibility options applied to remove the issues. Let’s look at how to execute it, examine possible errors, and see the options for the dump utility.

To start the dump, run the following command:

MySQL  JS > shell.connect(root@localhost:48008);

MySQL  localhost:48008 ssl  JS > util.dumpInstance("/home/vinicius.grippa/sandboxes/backup",

> {ocimds: true, compatibility: ["strip_restricted_grants"], dryRun: true})

And the output will be as follows:

Acquiring global read lock

Global read lock acquired

Gathering information - done

All transactions have been started

Locking instance for backup

Global read lock has been released

Checking for compatibility with MySQL Database Service 8.0.23

NOTE: User 'root'@'localhost' had restricted privileges (AUDIT_ADMIN, BACKUP_ADMIN, BINLOG_ADMIN, BINLOG_ENCRYPTION_ADMIN, CLONE_ADMIN, CREATE TABLESPACE, ENCRYPTION_KEY_ADMIN, FILE, FLUSH_OPTIMIZER_COSTS, FLUSH_STATUS, FLUSH_TABLES, FLUSH_USER_RESOURCES, GROUP_REPLICATION_ADMIN, INNODB_REDO_LOG_ARCHIVE, INNODB_REDO_LOG_ENABLE, PERSIST_RO_VARIABLES_ADMIN, PROXY, RELOAD, REPLICATION_SLAVE_ADMIN, ROLE_ADMIN, SERVICE_CONNECTION_ADMIN, SESSION_VARIABLES_ADMIN, SET_USER_ID, SHOW_ROUTINE, SHUTDOWN, SUPER, NOTE: User 'vgrippa'@'%' had restricted privileges (AUDIT_ADMIN, BACKUP_ADMIN, BINLOG_ADMIN, BINLOG_ENCRYPTION_ADMIN, CLONE_ADMIN, CREATE TABLESPACE, ENCRYPTION_KEY_ADMIN, FILE, FLUSH_OPTIMIZER_COSTS, FLUSH_STATUS, FLUSH_TABLES, FLUSH_USER_RESOURCES, GROUP_REPLICATION_ADMIN, INNODB_REDO_LOG_ARCHIVE, INNODB_REDO_LOG_ENABLE, PERSIST_RO_VARIABLES_ADMIN, RELOAD, REPLICATION_SLAVE_ADMIN, ROLE_ADMIN, SERVICE_CONNECTION_ADMIN, SESSION_VARIABLES_ADMIN, SET_USER_ID, SHOW_ROUTINE, SHUTDOWN, SUPER, SYSTEM_USER, SYSTEM_VARIABLES_ADMIN, TABLE_ENCRYPTION_ADMIN) removed

NOTE: Database test had unsupported ENCRYPTION option commented out

ERROR: Table 'test'.'sbtest1' uses unsupported storage engine MyISAM (fix this with 'force_innodb' compatibility option)

Compatibility issues with MySQL Database Service 8.0.23 were found. Please use the 'compatibility' option to apply compatibility adaptations to the dumped DDL.

Util.dumpInstance: Compatibility issues were found (RuntimeError)

With the ocimds option set to true, the dump util will check the data dictionary and index dictionary. Encryption options in CREATE TABLE statements are commented out in the DDL files, to ensure that all tables are located in the MySQL data directory and use the default schema encryption. strip_restricted_grants removes specific privileges that are restricted by MySQL Database Service that would cause an error during the user creation process. dryRun is self-explanatory: it will perform a validation only, and no data will be actually dumped.

So we have a MyISAM table in the test database. The dry run option clearly throws the error.

To fix the error, we are going to use the force_innodb option, which will convert all unsupported engines to InnoDB in the CREATE TABLE statement:

MySQL  localhost:48008 ssl  JS > util.dumpInstance("/home/vinicius.grippa/sandboxes/backup",

> {ocimds: true, compatibility: ["strip_restricted_grants","force_innodb"], dryRun: true})

Now the dry run does not throw any error and there are no exceptions. Let’s run the dump instance command to take an instance backup. The target directory must be empty before the export takes place. If the directory does not yet exist in its parent directory, the utility creates it.

We are going to process the dump in parallel. For this, we will use the option threads and set a value of 10 threads:

MySQL  localhost:48008 ssl  JS > util.dumpInstance("/home/vinicius.grippa/sandboxes/backup",

> {ocimds: true, compatibility: ["strip_restricted_grants","force_innodb"],threads : 10 })

And if we observe the last part of the output:

1 thds dumping - 100% (10.00K rows / ~10.00K rows), 0.00 rows/s, 0.00 B/s uncompressed, 0.00 B/s compressed

Duration: 00:00:00s

Schemas dumped: 1

Tables dumped: 10

Uncompressed data size: 1.88 MB

Compressed data size: 598.99 KB

Compression ratio: 3.1

Rows written: 10000

Bytes written: 598.99 KB

Average uncompressed throughput: 1.88 MB/s

Average compressed throughput: 598.99 KB/s

If we were using mysqldump, we would have a single file. As we can see, there are multiple files in the backup directory:

@.done.json

@.json

@.post.sql

@.sql

test.json

test@sbtest10@@0.tsv.zst

test@sbtest10@@0.tsv.zst.idx

test@sbtest10.json

test@sbtest10.sql

...

test@sbtest1@@0.tsv.zst

test@sbtest1@@0.tsv.zst.idx

test@sbtest1.json

test@sbtest1.sql

test@sbtest9@@0.tsv.zst

test@sbtest9@@0.tsv.zst.idx

test@sbtest9.json

test@sbtest9.sql

test.sql

- The @.json file contains server details and list of users, database names, and their character sets.

- The test.json file contains view, SP, and function names along with the list of tables.

- The @.sql and @.post.sql files contain MySQL server version details.

- The @.users.sql file contains a list of database users.

- The test@sbtest1.sql file contains a table structure. There will be one for each dumped table.

- The test.sql file contains a database statement.

- The test@sbtest1.json file contains column names and character sets. There will be one for each dumped table.

- The test@sbtest1@@0.tsv.zst.idx file is a binary file. It stores table indexes stats. There will be one for each dumped table.

- The test@sbtest1@@0.tsv.zst file is a binary file and it stores data. There will be one for each dumped table.

- The @.done.json file contains backup end time and data files sizes in KB.

util.dumpSchemas()

Similar to dumpInstance, however, using this util we can specify schemas to dump. Also, it supports the same options:

MySQL  localhost:48008 ssl  JS > util.dumpSchemas(["test"],"/home/vinicius.grippa/sandboxes/backup",

> {ocimds: true, compatibility: ["strip_restricted_grants","force_innodb"],threads : 10 , dryRun: true})

If we want to specify multiple schemas, we can easily do that:

MySQL  localhost:48008 ssl  JS > util.dumpSchemas(["test","percona","learning_mysql"],"/home/vinicius.grippa/sandboxes/backup",

> {ocimds: true, compatibility: ["strip_restricted_grants","force_innodb"],threads : 10 , dryRun: true})

util.dumpTables

If we want do extract more detailed data, like specific tables, we can use the dump tables util. Again, the bigger advantage compared to mysqldump is the parallel potential to extract data from MySQL:

MySQL  localhost:48008 ssl  JS > util.dumpTables("test", [ "sbtest1", "sbtest2" ],"/home/vinicius.grippa/sandboxes/backup",

> {ocimds: true, compatibility: ["strip_restricted_grants","force_innodb"],threads : 2 , dryRun: true})

util.loadDump(url[, options])

Now that we’ve seen all the utils to extract data, there is one remaining: the one to load the data into MySQL.

The dump loading utility provides data streaming to remote storage, parallel loading of tables or table chunks, progress state tracking, resume and reset capability, and the option of concurrent loading while the dump is still taking place.

Note the the dump loading utility uses the LOAD DATA LOCAL INFILE statement, so we need to enable this local_infile parameter globally while importing.

The dump loading utility checks whether the sql_require_primary_key system variable is set to ON, and if it is, returns an error if there is a table in the dump files with no primary key:

MySQL  localhost:48008 ssl  JS > util.loadDump("/home/vinicius.grippa/sandboxes/backup",

> {progressFile :"/home/vinicius.grippa/sandboxes/backup/restore.json",threads :12})

And the last part of the output will be similar to this:

[Worker006] percona@sbtest7@@0.tsv.zst: Records: 400000  Deleted: 0  Skipped: 0  Warnings: 0

[Worker007] percona@sbtest4@@0.tsv.zst: Records: 400000  Deleted: 0  Skipped: 0  Warnings: 0

[Worker002] percona@sbtest13@@0.tsv.zst: Records: 220742  Deleted: 0  Skipped: 0  Warnings: 0

Executing common postamble SQL

23 chunks (5.03M rows, 973.06 MB) for 23 tables in 3 schemas were loaded in 1 min 24 sec (avg throughput 11.58 MB/s)

0 warnings were reported during the load.

Be sure to check the warnings reported in the end in case any shows up.

Flame Graphs

Quoting Brendan Gregg, determining why CPUs are busy is a routine task for performance analysis, which often involves profiling stack traces. Profiling by sampling at a fixed rate is a coarse but effective way to see which code paths are hot (busy on the CPU). It usually works by creating a timed interrupt that collects the current program counter, function address, or entire stack back trace, and translates these to something human readable when printing a summary report. Flame graphs are a visualization for sampled stack traces that allows hot code paths to be identified quickly.

A stack trace (also called stack backtrace or stack traceback) is a report of the active stack frames at a certain point in time during the execution of a program. There are many tools available to collect stack traces. These tools are also known as CPU profilers. The CPU profiler we are going to use is perf.

perf is a profiler tool for Linux 2.6+ based systems that abstracts away CPU hardware differences in Linux performance measurements and presents a simple command-line interface. perf is based on the perf_events interface exported by recent versions of the Linux kernel.

perf_events is an event-oriented observability tool that can help solve advanced performance and troubleshooting functions. Questions that can be answered include these:

- Why is the kernel on-CPU so much? What code paths?

- Which code paths are causing CPU level 2 cache misses?

- Are the CPUs stalled on memory I/O?

- Which code paths are allocating memory, and how much?

- What is triggering TCP retransmits?

- Is a certain kernel function being called, and how often?

- Why are threads leaving the CPU?

Note that in this book, we are only scratching the surface of perf capabilities. We highly recommend checking out Brendan Gregg’s page, which contains much more detailed information about perf and other CPU profilers.

To produce flame graphs we need to start collecting the stack trace report with perf in the MySQL server. This operation needs to be done in the MySQL host. We will collect data for 60 seconds:

# perf record -a -g -F99 -p $(pgrep -x mysqld) -- sleep 60;

# perf report > /tmp/perf.report;

# perf script > /tmp/perf.script;

And if we check the /tmp directory we will see perf files:

# ls -l /tmp/perf*

-rw-r--r-- 1 root root  502100 Feb 13 22:01 /tmp/perf.report

-rw-r--r-- 1 root root 7303290 Feb 13 22:01 /tmp/perf.script

The next step it is not necessary to be executed in the MySQL host. We can copy the files to another Linux host or even macOS.

To produce the Flame Graphs we can use Brendan’s GitHub repository. For this example, we will clone the Flame Graph repository in the same directory that our perf report is located.

# git clone https://github.com/brendangregg/FlameGraph

# ./FlameGraph/stackcollapse-perf.pl ./perf.script > perf.report.out.folded

# ./FlameGraph/flamegraph.pl ./perf.report.out.folded > perf.report.out.svg

We’ve produced a file named perf.report.out.svg. This file can be opened in any browser to be visualized. The figure Figure 16-1 is an example of a Flame graph:

Figure 16-1. An example of Flame Graph

Flame Graphs show the sample population across the x-axis, and stack depth on the y-axis. Each function (stack frame) is drawn as a rectangle, with the width relative to the number of samples. So the bigger the bar, more CPU time was spent on that. Also, the x-axis spans the stack trace collection. It does not show the passage of time, so the left-to-right ordering has no special meaning. The left-to-right ordering of stack traces is performed alphabetically on the function names, from the root to the leaf of each stack.

The file created is interactive, so we can explore where kernel CPU time is spent. In the previous example, an INSERT operation is consuming 44% of the CPU time. The figure Figure 16-2 shows an example:

Figure 16-2. 44% of CPU time is used for an INSERT operation

Building MySQL from Source

As Chapter 1 explained, MySQL has a distribution available for most common operating systems. Some companies have compiled their own MySQL versions, such as Facebook, which worked on the RocksDB engine and integrated it to MySQL. RocksDB is an embeddable, persistent key-value store for fast storage that has several advantages compared with InnoDB with regard to space efficiency.

Despite its advantages, RocksDB does not support replication or an SQL layer. This led the Facebook team to build MyRocks, a new open source project that integrates RocksDB as a new MySQL storage engine. With MyRocks, it is possible to use RocksDB as backend storage and still benefit from all the features in MySQL. The project is open source and available in GitHub.

Another motivation to compile MySQL is the ability to customize its build. For example, in a very specific problem, we can always try to debug MySQL to gather extra information. To do this, we need to configure MySQL with the -DWITH_DEBUG=1 option.

Building MySQL for Ubuntu Focal Fossa and ARM processors

Because ARM processors are currently gaining traction (particularly thanks to Apple’s M1 chip), we will compile MySQL for Ubuntu Focal Fossa running in ARM.

First, we are going to create our directories. We will create one directory that will have the source code, another one to place the compiled binaries, and the last one is where we are going to place the boost library.

# cd /

# mkdir compile

# cd compile/

# mkdir build

# mkdir source

# mkdir boost

# mkdir basedir

# mkdir /var/lib/mysql

Next, we need to install the additional Linux packages required to compile MySQL:

# apt-get -y install dirmngr

# apt-get update -y

# apt-get -y install cmake

# apt-get -y install lsb-release wget

# apt-get -y purge eatmydata || true

# apt-get -y install psmisc pkg-config

# apt-get -y install libsasl2-dev libsasl2-modules libsasl2-modules-ldap || apt-get -y install libsasl2-modules libsasl2-modules-ldap libsasl2-dev

# apt-get -y install dh-systemd || true

# apt-get -y install curl bison cmake perl libssl-dev gcc g++ libaio-dev libldap2-dev libwrap0-dev gdb unzip gawk

# apt-get -y install lsb-release libmecab-dev libncurses5-dev libreadline-dev libpam-dev zlib1g-dev

# apt-get -y install libldap2-dev libnuma-dev libjemalloc-dev libeatmydata libc6-dbg valgrind libjson-perl  libsasl2-dev

# apt-get -y install libmecab2 mecab mecab-ipadic

# apt-get -y install build-essential devscripts libnuma-dev

# apt-get -y install cmake autotools-dev autoconf automake build-essential devscripts debconf debhelper fakeroot *

# *apt-get -y install libcurl4-openssl-dev patchelf

# apt-get -y install libeatmydata1

# apt-get install libmysqlclient-dev -y

# apt-get install valgrind -y

These packages are related to the CMAKE flags that we will run. If we remove or add certain flags, some packages are not necessary to install (for example, if we don’t want to compile with valgrind, we don’t need this package).

Next, we will download the source code. For this, we will use MySQL repository in GitHub:

# cd source

# git clone https://github.com/mysql/mysql-server.git

The output will be similar to this:

Cloning into 'mysql-server'...

remote: Enumerating objects: 1639611, done.

remote: Total 1639611 (delta 0), reused 0 (delta 0), pack-reused 1639611

Receiving objects: 100% (1639611/1639611), 3.19 GiB | 42.88 MiB/s, done.

Resolving deltas: 100% (1346714/1346714), done.

Updating files: 100% (32681/32681), done.

To check which version we will compile, we can run the following:

# cd mysql-server/

# git branch

Next, we will go to our build directory and run CMAKE with our chosen flags:

# cd /compile/build

# cmake ../source/mysql-server/  -DBUILD_CONFIG=mysql_release \

-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE:-RelWithDebInfo} \

-DWITH_DEBUG=1 \

-DFEATURE_SET=community \

-DENABLE_DTRACE=OFF \

-DWITH_SSL=system \

-DWITH_ZLIB=system \

-DCMAKE_INSTALL_PREFIX="/compile/basedir/" \

-DINSTALL_LIBDIR="lib/" \

-DINSTALL_SBINDIR="bin/" \

-DWITH_INNODB_MEMCACHED=ON \

-DDOWNLOAD_BOOST=1 \

-DWITH_VALGRIND=1 \

-DINSTALL_PLUGINDIR="plugin/" \

-DMYSQL_DATADIR="/var/lib/mysql/"\

-DWITH_BOOST="/compile/boost/"

- DBUILD_CONFIG configures a source distribution with the same build options as for MySQL releases (we are going to override some of them).

- DCMAKE_BUILD_TYPE with RelWithDebInfo enables optimizations and generates debugging information.

- DWITH_DEBUG enables the use of the --debug="d,parser_debug" option when MYSQL is started. This causes the Bison parser used to process SQL statements to dump a parser trace to the server’s standard error output. Typically, this output is written to the error log.

- DFEATURE_SET indicates we are going to install community features

- DENABLE_DTRACE includes support for DTrace probes. The DTrace probes in the MySQL server are designed to provide information about the execution of queries within MySQL and the different areas of the system being utilized during that process.

- The DWITH_SSL option adds support for encrypted connections, entropy for random number generation, and other encryption-related operations.

- DWITH_ZLIB enables compression library support to the COMPRESS() and UNCOMPRESS() functions, and compression of the client/server protocol.

- _DCMAKE_INSTALL_PREFIX sets our installation base directory.

- DINSTALL_SBINDIR tells where to install the mysqld server.

- DWITH_INNODB_MEMCACHED generates memcached shared libraries (libmemcached.so and innodb_engine.so).

- DDOWNLOAD_BOOST makes CMAKE download the boost library and place it in the DWITH_BOOST specified location.

- DWITH_VALGRIND enables Valgrind, exposing the Valgrind API to MySQL code. It is useful to analyze memory leaks.

- DINSTALL_PLUGINDIR defines where the compiler will place the plugin libraries.

- DMYSQL_DATADIR defines the location of the MySQL data directory.

- DWITH_BOOST defines the directory where CMAKE will download boost library.

NOTE

If by mistake you miss one step and the CMAKE process fail, to prevent old object files or configuration information from being used in the next tentative, we need to clean the build direcoty and the previous configuration. The following commands will need to run in the build directory on Unix before re-running CMake:

# cd /compile/build

# make clean

# rm CMakeCache.txt

After we run CMAKE, we are going to compile MySQL de facto, using the make command. To optimize the compiling process we will use the -j option, which how many threads we are going to use to compile MySQL. Since in our instance we have 16 ARM cores, we are going to use 15 threads (leaving one for OS activities):

# make  -j 15

# make install

This process may take a while, and it is a very verbose process. After it’s finished, we can see the binaries in the basedir directory:

# ls -l /compile/basedir/bin

Note that we are not going to find a mysqld binary in the /compile/build/bin/, but instead we will see a mysqld-debug. This is because of the DWITH_DEBUG option we set previously.

# /compile/build/bin/mysqld-debug --version

/compile/build/bin/mysqld-debug  Ver 8.0.23-debug-valgrind for Linux on aarch64 (Source distribution)

Now, we can test our binary. For this we are going to manually create the directories, configure the permission and my.cnf:

# mkdir /var/log/mysql/

# mkdir /var/run/mysqld/

# chown ubuntu: /var/log/mysql/

# chown ubuntu: /var/run/mysqld/

And add these settings to /etc/my.cnf:

[mysqld]

pid-file        = /var/run/mysqld/mysqld.pid

socket          = /var/run/mysqld/mysqld.sock

datadir         = /var/lib/mysql

log-error       = /var/log/mysql/error.log

Next, we are going to initialize the MySQL data dictionary:

# */compile/basedir/bin/mysqld-debug --defaults-file=/etc/my.cnf --initialize --user ubuntu

And now, MySQL is ready to be started:

# /compile/basedir/bin/mysqld-debug --defaults-file=/etc/my.cnf  --user ubuntu &

A temporary password will be created, and we can extract from the error log:

# grep "A temporary password" /var/log/mysql/error.log

2021-02-14T16:55:25.754028Z 6 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: yGldRKoRf0%T

Now we can validate connecting using the MySQL client of our preference:

# mysql -uroot -p'yGldRKoRf0%T'

mysql: [Warning] Using a password on the command line interface can be insecure.

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 8

Server version: 8.0.23-debug-valgrind

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Analyzing a MySQL Crash

We say that MySQL crashes when the mysqld process dies without the proper shutdown command. MySQL can crash for a variety of reasons, including these:

- Hardware failure (memory, disk, processor)

- Segmentation fault (invalid memory access)

- Bugs

- Being killed by the oom process

- Others, such as cosmic rays.

The MySQL process can receive a number of signals from Linux. The following are among the most common:

- Signal 15 (SIGTERM): Causes the server to shut down. This is like executing a SHUTDOWN statement without having to connect to the server (which for shutdown requires an account that has the SHUTDOWN privilege). For example, the following two commands result in a regular shutdown:

# systemctl stop mysql

# kill -15 -p $(pgrep -x mysqld)

- Signal 1 (SIGHUP): Causes the server to reload the grant tables and to flush tables, logs, the thread cache, and the host cache. These actions are like various forms of the FLUSH statement:

mysql> FLUSH LOGS;

Or:

# kill -1 -p $(pgrep -x mysqld)

- Signal 6 (SIGABRT): Happens because something went wrong. It is commonly used by libc and other libraries to abort the program in case of critical errors. For example, glibc sends a SIGABRT in case of a detected double-free or other heap corruption. SIGABRT will write the crash details in the MySQL error log:

18:03:28 UTC - mysqld got signal 6 ;

Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

Thread pointer: 0x7fe6b4000910

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 7fe71845fbc8 thread_stack 0x46000

/opt/mysql/8.0.23/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x2066f1e]

/opt/mysql/8.0.23/bin/mysqld(handle_fatal_signal+0x323) [0x1032cc3]

/lib64/libpthread.so.0(+0xf630) [0x7fe7244e5630]

/lib64/libc.so.6(gsignal+0x37) [0x7fe7224fa387]

/lib64/libc.so.6(abort+0x148) [0x7fe7224fba78]

/opt/mysql/8.0.23/bin/mysqld() [0xd52c3d]

/opt/mysql/8.0.23/bin/mysqld(MYSQL_BIN_LOG::new_file_impl(bool, Format_description_log_event*)+0x493) [0x1c2b483]

/opt/mysql/8.0.23/bin/mysqld(MYSQL_BIN_LOG::rotate(bool, bool*)+0x35) [0x1c2bb65]

/opt/mysql/8.0.23/bin/mysqld(MYSQL_BIN_LOG::rotate_and_purge(THD*, bool)+0x52) [0x1c37d22]

/opt/mysql/8.0.23/bin/mysqld(handle_reload_request(THD*, unsigned long, TABLE_LIST*, int*)+0x51f) [0xf30e9f]

/opt/mysql/8.0.23/bin/mysqld(signal_hand+0x2ea) [0xe101da]

/opt/mysql/8.0.23/bin/mysqld() [0x25973dc]

/lib64/libpthread.so.0(+0x7ea5) [0x7fe7244ddea5]

/lib64/libc.so.6(clone+0x6d) [0x7fe7225c298d]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0): Connection ID (thread ID): 0

Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

2021-02-14T18:03:29.120726Z mysqld_safe mysqld from pid file /home/vinicius.grippa/sandboxes/rsandbox_8_0_23/master/data/mysql_sandbox48011.pid ended

- Signal 11 (SIGSEGV): A segmentation fault, bus error, or access violation issue. This is generally an attempt to access memory that the CPU cannot physically address, or access violation. When MySQL receives a SIGSEGV, a core dump will be generated if it is enabled.

- Signal 9 (SIGKILL): Probably the most famous, the SIGKILL signal that is sent to a process causes it to terminate immediately (kill). In contrast to SIGTERM and SIGINT, this signal cannot be caught or ignored, and the receiving process cannot perform any cleanup upon receiving this signal. Besides the chance of corrupting MySQL data, SIGKILL will also force MySQL to perform a recovery process when restarted to bring MySQL to an operational state.

# kill -9 -p $(pgrep -x mysqld)

Also, the Linux OOM process executes a SIGKILL to finish with the MySQL process.

How do we analyze a crash? Let’s analyze a Percona crash with SIGSEGV(11):

11:47:47 UTC - mysqld got signal 11 ;

Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

Build ID: Not Available

Server Version: 8.0.22-13 Percona Server (GPL), Release 13, Revision 6f7822f

Thread pointer: 0x7f0e46c73000

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 7f0e664ecd10 thread_stack 0x46000

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x210c1cd]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(handle_fatal_signal+0x3c3) [0x1260d33]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0) [0x7f0e7acd58a0]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(Item_splocal::this_item()+0x14) [0xe36ad4]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(Item_sp_variable::val_str(String*)+0x20) [0xe38e60]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(Arg_comparator::compare_string()+0x27) [0xe5c127]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(Item_func_ne::val_int()+0x30) [0xe580e0]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(Item::val_bool()+0xcc) [0xe3ddbc]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(sp_instr_jump_if_not::exec_core(THD*, unsigned int*)+0x2d) [0x106f64d]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(sp_lex_instr::reset_lex_and_exec_core(THD*, unsigned int*, bool)+0x63b) [0x10717bb]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(sp_lex_instr::validate_lex_and_execute_core(THD*, unsigned int*, bool)+0xa1) [0x1072231]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(sp_head::execute(THD*, bool)+0x5c7) [0x1068e37]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(sp_head::execute_trigger(THD*, MYSQL_LEX_CSTRING const&, MYSQL_LEX_CSTRING const&, GRANT_INFO*)+0x29a) [0x106973a]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(Trigger::execute(THD*)+0x10b) [0x12288cb]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(Trigger_chain::execute_triggers(THD*)+0x18) [0x1229c98]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(Table_trigger_dispatcher::process_triggers(THD*, enum_trigger_event_type, enum_trigger_action_time_type, bool)+0x46) [0x1223046]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(fill_record_n_invoke_before_triggers(THD*, COPY_INFO*, mem_root_deque<Item*> const&, mem_root_deque<Item*> const&, TABLE*, enum_trigger_event_type, int, bool, bool*)+0x1c5) [0x1088b35]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(Sql_cmd_update::update_single_table(THD*)+0x1e98) [0x11ec138]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(Sql_cmd_update::execute_inner(THD*)+0xd5) [0x11ec5f5]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(Sql_cmd_dml::execute(THD*)+0x6c0) [0x116f590]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(mysql_execute_command(THD*, bool)+0xaf8) [0x110e588]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(mysql_parse(THD*, Parser_state*, bool)+0x4ec) [0x111327c]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x1be5) [0x1115325]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(do_command(THD*)+0x204) [0x1116554]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld() [0x1251c20]

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld() [0x2620e84]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f0e7acca6db]

/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f0e78c95a3f]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (7f0e46cb4dc8): update table1 set c2_id='R', c3_description='testing crash' where c1_id=1

Connection ID (thread ID): 111

Status: NOT_KILLED

Please help us make Percona Server better by reporting any

bugs at https://bugs.percona.com/

The stack trace is analyzed from bottom-up. We can see from the crash that this is a Percona Server 8.0.22. Next, we see the thread being created at the OS level at this point:

/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f0e7acca6db]

Following the way up, the code path enters in MySQL and starts executing a command:

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(do_command(THD*)+0x204) [0x1116554]

And the code path that crashes is the Item_splocal function:

/home/lalit/mysql_tar/percona/8.0.22/bin/mysqld(Item_splocal::this_item()+0x14) [0xe36ad4]

With a bit of investigation in the MySQL code, we discover that Item_splocal is part of the stored procedure code. If we look at the end of the stack trace, we will see a query:

Query (7f0e46cb4dc8): update table1 set c2_id='R', c3_description='testing crash' where c1_id=1

Triggers can also use the stored procedure path when they contain variables. If we check whether this table has triggers, we see this:

CREATE DEFINER=`root`@`localhost` TRIGGER `table1_update_trigger` BEFORE UPDATE ON `table1` FOR EACH ROW BEGIN

DECLARE vc1_id VARCHAR(2);

SELECT c2_id FROM table1 WHERE c1_id = new.c1_id INTO vc1_id;

IF vc1_id <> P THEN

INSERT INTO table1_hist(

c1_id,

c2_id,

c3_description)

VALUES(

old.c1_id,

old.c2_id,

new.c3_description);

END IF;

END

;;

With all this information, we can create a test case and report the bug:

USE test;

CREATE TABLE `table1` (

`c1_id` int primary key auto_increment,

`c2_id` char(1) NOT NULL,

`c3_description` varchar(255));

CREATE TABLE `table1_hist` (

`c1_id` int,

`c2_id` char(1) NOT NULL,

`c3_description` varchar(255));   insert into table1 values (1, T,  test crash);

delimiter ;;

CREATE DEFINER=`root`@`localhost` TRIGGER `table1_update_trigger` BEFORE UPDATE ON `table1` FOR EACH ROW BEGIN

DECLARE vc1_id VARCHAR(2);

SELECT c2_id FROM table1 WHERE c1_id = new.c1_id INTO vc1_id;

IF vc1_id <> P THEN

INSERT INTO table1_hist(

c1_id,

c2_id,

c3_description)

VALUES(

old.c1_id,

old.c2_id,

new.c3_description);

END IF;

END

;;

And we will run multiple commands simultaneously in the same table until the error happens:

$ mysqlslap --user=msandbox --password=msandbox --socket=/tmp/mysql_sandbox37515.sock \

--create-schema=test --port=37515 \

--query="update table1 set c2_id=R, c3_description=testing crash where c1_id=1" \

--concurrency=50 --iterations=200

This bug is relatively easy to reproduce, and we recommend you test it. You can find more details about this bug in Percona JIRA system.

Also, we can see that Oracle fixed this bug at version 8.0.23 thanks to the release notes:

“Prepared statements involving stored programs could cause heap-use-after-free memory problems. (Bug #32131022, Bug #32045681, Bug #32051928)”

Sometimes bugs are not easy to reproduce and can be really frustrating to investigate. Even experienced engineers have problemas with that, specially when investigating memory leaks. We hope we have sparked your curiosity to investigate crashes.

Index

About the Authors

Sergey Kuzmichev is a Senior Support Engineer working for Percona and has a Master of Science degree. He represents Percona in open source conferences and meetups all around the globe. He contributes to the community writing articles and blog posts about MySQL and open source databases. Sergey started his career as an Oracle DBA, later venturing into the DevOps side of things, but then returning to what he likes most: databases.

Vinicius Grippa is a Senior Support Engineer working for Percona and an Oracle Ace Associate. Vinicius has a Bachelor’s degree in Computer Science and has been working with databases for 13 years. He has experience in designing databases for mission-critical applications and, in the last few years, has become a specialist in MySQL and MongoDB ecosystems. Working in the Support team, he has helped Percona customers with hundreds of different cases featuring a vast range of scenarios and complexities. Vinicius is also active in the OS community, participating in virtual rooms like Slack, and speaking at MeetUps, and presenting conferences in Europe, Asia, North and South America.

Colophon

The animals on the cover of Learning MySQL are blue spotted crows (Euploea midamus), butterflies found in India and Southeast Asia.

Many of the animals on O’Reilly covers are endangered; all of them are important to the world.

The cover illustration is by Karen Montgomery, based on a black and white engraving from FILL IN CREDITS. The cover fonts are Gilroy Semibold and Guardian Sans. The text font is Adobe Minion Pro; the heading font is Adobe Myriad Condensed; and the code font is Dalton Maag’s Ubuntu Mono.

1. Preface

1. Who This Book Is for

2. How This Book Is Organized

1. Starting with MySQL

2. Using MySQL

3. Conventions Used in This Book

4. Using Code Examples

5. O’Reilly Online Learning

6. How to Contact Us

7. Acknowledgments

1. From Vinicius Grippa

2. From Sergey Kuzmichev

2. 1. Installing MySQL

1. MySQL Forks

1. MySQL community

2. Percona Server for MySQL

3. MariaDB Server

4. MySQL Enterprise

2. Installation Choices and Platforms

1. 1. Download the distribution that you want to install

2. 2. Install the distribution

3. 3. Perform any necessary post-installation setup

4. 4. Run Benchmarks

3. Installing MySQL on Linux

1. Installing MySQL on Centos 7

2. Installing MySQL on Centos 8

3. Installing MySQL on Ubuntu 20.04 LTS (Focal Fossa)

4. Installing MySQL on macOS Big Sur

5. Installing MySQL on Windows 10

6. The Contents of the MySQL Directory

1. MySQL 5.7 default files

2. MySQL 8.0 default files

7. Using the Command-Line Interface

8. Using Docker

1. Installing Docker

9. Using Sandboxes

10. Upgrading MySQL Server

3. 2. Modeling and Designing Databases

1. How Not to Develop a Database

2. The Database Design Process

3. The Entity Relationship Model

1. Representing Entities

2. Representing Relationships

3. Partial and Total Participation

4. Entity or Attribute?

5. Entity or Relationship?

6. Intermediate Entities

7. Weak and Strong Entities

4. Database Normalization

5. Normalizing an Example Table

1. First normal form: No repeating groups

2. Second normal form: Eliminate redundant data

3. Third normal form: Eliminate data not dependent on key

6. Entity Relationship Modeling Examples

7. Using the Entity Relationship Model

1. Mapping Entities and Relationships to Database Tables

2. Creating a Bank Database ER Model

3. Converting the EER to a MySQL Database Using Workbench

4. 3. Basic SQL

1. Using the Sakila Database

2. The SELECT Statement and Basic Querying Techniques

1. Single Table SELECTs

2. Choosing Columns

3. Selecting Rows with the WHERE Clause

4. ORDER BY Clauses

5. The LIMIT Clause

6. Joining Two Tables

3. The INSERT Statement

1. INSERT Basics

2. Alternative Syntaxes

4. The DELETE Statement

1. DELETE Basics

2. Using WHERE, ORDER BY, and LIMIT

3. Removing All Rows with TRUNCATE

5. The UPDATE Statement

1. Examples

2. Using WHERE, ORDER BY, and LIMIT

6. Exploring Databases and Tables with SHOW and mysqlshow

5. 4. Working with Database Structures

1. Creating and Using Databases

2. Creating Tables

1. Basics

2. Collation and Character Sets

3. Other Features

4. Column Types

5. Keys and Indexes

6. The AUTO_INCREMENT Feature

3. Altering Structures

1. Adding, Removing, and Changing Columns

2. Adding, Removing, and Changing Indexes

3. Renaming Tables and Altering Other Structures

4. Deleting Structures

1. Dropping Databases

2. Removing Tables

6. 5. Advanced Querying

1. Aliases

1. Column Aliases

2. Table Aliases

2. Aggregating Data

1. The DISTINCT Clause

2. The GROUP BY Clause

3. The HAVING Clause

3. Advanced Joins

1. The Inner Join

2. The Union

3. The Left and Right Joins

4. The Natural Join

5. Constant expressions in joins

4. Nested Queries

1. Nested Query Basics

2. The ANY, SOME, ALL, IN, and NOT IN Clauses

3. The EXISTS and NOT EXISTS Clauses

4. Nested Queries in the FROM Clause

5. Nested Queries in JOINs

6. User Variables

7. 6. Transactions and Locking

1. Isolation Levels

1. REPEATABLE READ

2. READ COMMITTED

3. READ UNCOMMITTED

4. SERIALIZABLE

2. Locking

1. Metadata Locks

2. Row Locks

3. Deadlocks

3. MySQL Parameters Related to Isolation and Locks

1. transaction_isolation

2. innodb_lock_wait_timeout

3. innodb_print_all_deadlocks

4. lock_wait_timeout

5. innodb_deadlock_detect

8. 7. Doing More with MySQL

1. Inserting Data Using Queries

2. Loading Data from Comma-Delimited Files

3. Writing Data into Comma-Delimited Files

4. Creating Tables with Queries

5. Updates and Deletes with Multiple Tables

1. Deletion

2. Updates

6. Replacing Data

7. The EXPLAIN Statement

8. Alternative Storage Engines

1. InnoDB

2. MyISAM and Aria

3. MyRocks and TokuDB

4. Other Table Types

9. 8. Managing Users and Privileges

1. Understanding Users and Privileges

2. root user

3. Creating and Using New Users

4. Grant tables

5. User and Privileges Management Commands, Logging and Replication

6. Modifying and Dropping a User

1. Modifying the User

2. Dropping the User

7. Privileges

8. Static Versus Dynamic Privileges

9. SUPER privilege

10. Privilege Management Commands

11. Checking Privileges

12. The GRANT OPTION Privilege

13. Roles

14. Changing root’s password and insecure startup

15. Some ideas for secure setup

10. 9. Using Option Files

1. Structure of the Option File

2. Scope of Options

3. Search Order for Option Files

4. Special Option Files

1. Login Path Configuration File

2. Persistent System Variables Configuration File

5. Determining the Options in Effect

11. 10. Backups and Recovery

1. Physical and Logical Backups

1. Logical Backups

2. Physical Backups

3. Overview of Logical and Physical Backups

2. Replication as a Backup Tool

1. Infrastructure Failure

2. Deployment bug

3. The mysqldump Program

1. Bootstrapping Replication with mysqldump

4. Loading Data from an SQL Dump File

5. mysqlpump

6. mydumper and myloader

7. Cold Backup and Filesystem Snapshots

8. Percona XtraBackup

1. Backing up and Recovering

2. Advanced Features

9. Incremental Backups with XtraBackup

10. Other Physical Backup Tools

1. MySQL Enterprise Backup

2. mariabackup

11. Point-in-Time Recovery

1. Binary Logs Technical Background

2. Preserving Binary Logs

3. Identifying a PITR Target

4. Example Point-in-Time-Recovery: XtraBackup

5. Example Point-in-Time-Recovery: mysqldump

12. Export and Import InnoDB Tablespaces

1. Technical Background

2. Export Tablespace

3. Import Tablespace

4. XtraBackup Single-Table Restore

13. Testing and Verifying Your Backups

14. Database Backup Strategy Primer

12. 11. Configuring and Tuning the Server

1. The MySQL Server Daemon

2. MySQL Server Variables

1. Checking Server Settings

2. Best Practices

13. 12. Monitoring MySQL Servers

1. Operating System Metrics

1. CPU

2. Disk

3. Memory

4. Network

2. MySQL Server Observability

1. Status Variables

2. Basic Monitoring Recipes

3. The Slow Query Log

4. InnoDB Engine Status Report

3. Investigation Methods

1. USE Method

2. RED Method

4. MySQL Monitoring Tools

5. Incident/diagnostic and Manual Data Collection

1. Gathering System Status Variable Values Periodically

2. Using pt-stalk to Collect MySQL and OS Metrics

3. Extended Manual Data Collection

14. 13. High Availability

1. Asynchronous Replication

1. Basic Parameters to Set in the Source and the Replica

2. Create a Replica Using the Percona Xtrabackup Tool

3. Create a Replica Using the Clone Plugin

4. Create a Replica Using mysqldump

5. Create a Replica Using mydumper and myloader

6. Group Replication

2. Synchronous replication

1. Galera Cluster (Percona XtraDB Cluster)

15. 14. MySQL in the Cloud

1. Database as a service (DBaaS)

1. Amazon RDS for MySQL/MariaDB

2. Google Cloud for MySQL

3. Azure SQL

2. Amazon Aurora RDS

3. MySQL in the Cloud Instances

4. MySQL in Kubernetes

1. Deploying Percona XtraDB Cluster in Kubernetes

16. 15. Load Balancing MySQL

1. Load Balancing with Aplication Drivers

2. ProxySQL Load Balancer

1. Installing and Configuring ProxySQL

3. HAProxy Load Balancer

1. Installing and Configuring HAProxy

4. MySQL Router

17. 16. Miscellaneous Topics

1. MySQL Shell

1. Installing MySQL Shell

2. Installing MySQL Shell on Ubuntu 20.04 Focal Fossa

3. Installing MySQL Shell in Centos 8

4. Deploying a Sandbox InnoDB Cluster with MySQL Shell

5. MySQL Shell Utilities

2. Flame Graphs

3. Building MySQL from Source

1. Building MySQL for Ubuntu Focal Fossa and ARM processors

4. Analyzing a MySQL Crash

18. Index

Table of Contents

Preface

Who This Book Is for

How This Book Is Organized

Starting with MySQL

Using MySQL

Conventions Used in This Book

Using Code Examples

O’Reilly Online Learning

How to Contact Us

Acknowledgments

From Vinicius Grippa

From Sergey Kuzmichev

1. Installing MySQL

MySQL Forks

MySQL community

Percona Server for MySQL

MariaDB Server

MySQL Enterprise

Installation Choices and Platforms

1. Download the distribution that you want to install

2. Install the distribution

3. Perform any necessary post-installation setup

4. Run Benchmarks

Installing MySQL on Linux

Installing MySQL on Centos 7

Installing MySQL on Centos 8

Installing MySQL on Ubuntu 20.04 LTS (Focal Fossa)

Installing MySQL on macOS Big Sur

Installing MySQL on Windows 10

The Contents of the MySQL Directory

MySQL 5.7 default files

MySQL 8.0 default files

Using the Command-Line Interface

Using Docker

Installing Docker

Using Sandboxes

Upgrading MySQL Server

2. Modeling and Designing Databases

How Not to Develop a Database

The Database Design Process

The Entity Relationship Model

Representing Entities

Representing Relationships

Partial and Total Participation

Entity or Attribute?

Entity or Relationship?

Intermediate Entities

Weak and Strong Entities

Database Normalization

Normalizing an Example Table

First normal form: No repeating groups

Second normal form: Eliminate redundant data

Third normal form: Eliminate data not dependent on key

Entity Relationship Modeling Examples

Using the Entity Relationship Model

Mapping Entities and Relationships to Database Tables

Creating a Bank Database ER Model

Converting the EER to a MySQL Database Using Workbench

3. Basic SQL

Using the Sakila Database

The SELECT Statement and Basic Querying Techniques

Single Table SELECTs

Choosing Columns

Selecting Rows with the WHERE Clause

ORDER BY Clauses

The LIMIT Clause

Joining Two Tables

The INSERT Statement

INSERT Basics

Alternative Syntaxes

The DELETE Statement

DELETE Basics

Using WHERE, ORDER BY, and LIMIT

Removing All Rows with TRUNCATE

The UPDATE Statement

Examples

Using WHERE, ORDER BY, and LIMIT

Exploring Databases and Tables with SHOW and mysqlshow

4. Working with Database Structures

Creating and Using Databases

Creating Tables

Basics

Collation and Character Sets

Other Features

Column Types

Keys and Indexes

The AUTO_INCREMENT Feature

Altering Structures

Adding, Removing, and Changing Columns

Adding, Removing, and Changing Indexes

Renaming Tables and Altering Other Structures

Deleting Structures

Dropping Databases

Removing Tables

5. Advanced Querying

Aliases

Column Aliases

Table Aliases

Aggregating Data

The DISTINCT Clause

The GROUP BY Clause

The HAVING Clause

Advanced Joins

The Inner Join

The Union

The Left and Right Joins

The Natural Join

Constant expressions in joins

Nested Queries

Nested Query Basics

The ANY, SOME, ALL, IN, and NOT IN Clauses

The EXISTS and NOT EXISTS Clauses

Nested Queries in the FROM Clause

Nested Queries in JOINs

User Variables

6. Transactions and Locking

Isolation Levels

REPEATABLE READ

READ COMMITTED

READ UNCOMMITTED

SERIALIZABLE

Locking

Metadata Locks

Row Locks

Deadlocks

MySQL Parameters Related to Isolation and Locks

transaction_isolation

innodb_lock_wait_timeout

innodb_print_all_deadlocks

lock_wait_timeout

innodb_deadlock_detect

7. Doing More with MySQL

Inserting Data Using Queries

Loading Data from Comma-Delimited Files

Writing Data into Comma-Delimited Files

Creating Tables with Queries

Updates and Deletes with Multiple Tables

Deletion

Updates

Replacing Data

The EXPLAIN Statement

Alternative Storage Engines

InnoDB

MyISAM and Aria

MyRocks and TokuDB

Other Table Types

8. Managing Users and Privileges

Understanding Users and Privileges

root user

Creating and Using New Users

Grant tables

User and Privileges Management Commands, Logging and Replication

Modifying and Dropping a User

Modifying the User

Dropping the User

Privileges

Static Versus Dynamic Privileges

SUPER privilege

Privilege Management Commands

Checking Privileges

The GRANT OPTION Privilege

Roles

Changing root’s password and insecure startup

Some ideas for secure setup

9. Using Option Files

Structure of the Option File

Scope of Options

Search Order for Option Files

Special Option Files

Login Path Configuration File

Persistent System Variables Configuration File

Determining the Options in Effect

10. Backups and Recovery

Physical and Logical Backups

Logical Backups

Physical Backups

Overview of Logical and Physical Backups

Replication as a Backup Tool

Infrastructure Failure

Deployment bug

The mysqldump Program

Bootstrapping Replication with mysqldump

Loading Data from an SQL Dump File

mysqlpump

mydumper and myloader

Cold Backup and Filesystem Snapshots

Percona XtraBackup

Backing up and Recovering

Advanced Features

Incremental Backups with XtraBackup

Other Physical Backup Tools

MySQL Enterprise Backup

mariabackup

Point-in-Time Recovery

Binary Logs Technical Background

Preserving Binary Logs

Identifying a PITR Target

Example Point-in-Time-Recovery: XtraBackup

Example Point-in-Time-Recovery: mysqldump

Export and Import InnoDB Tablespaces

Technical Background

Export Tablespace

Import Tablespace

XtraBackup Single-Table Restore

Testing and Verifying Your Backups

Database Backup Strategy Primer

11. Configuring and Tuning the Server

The MySQL Server Daemon

MySQL Server Variables

Checking Server Settings

Best Practices

12. Monitoring MySQL Servers

Operating System Metrics

CPU

Disk

Memory

Network

MySQL Server Observability

Status Variables

Basic Monitoring Recipes

The Slow Query Log

InnoDB Engine Status Report

Investigation Methods

USE Method

RED Method

MySQL Monitoring Tools

Incident/diagnostic and Manual Data Collection

Gathering System Status Variable Values Periodically

Using pt-stalk to Collect MySQL and OS Metrics

Extended Manual Data Collection

13. High Availability

Asynchronous Replication

Basic Parameters to Set in the Source and the Replica

Create a Replica Using the Percona Xtrabackup Tool

Create a Replica Using the Clone Plugin

Create a Replica Using mysqldump

Create a Replica Using mydumper and myloader

Group Replication

Synchronous replication

Galera Cluster (Percona XtraDB Cluster)

14. MySQL in the Cloud

Database as a service (DBaaS)

Amazon RDS for MySQL/MariaDB

Google Cloud for MySQL

Azure SQL

Amazon Aurora RDS

MySQL in the Cloud Instances

MySQL in Kubernetes

Deploying Percona XtraDB Cluster in Kubernetes

15. Load Balancing MySQL

Load Balancing with Aplication Drivers

ProxySQL Load Balancer

Installing and Configuring ProxySQL

HAProxy Load Balancer

Installing and Configuring HAProxy

MySQL Router

16. Miscellaneous Topics

MySQL Shell

Installing MySQL Shell

Installing MySQL Shell on Ubuntu 20.04 Focal Fossa

Installing MySQL Shell in Centos 8

Deploying a Sandbox InnoDB Cluster with MySQL Shell

MySQL Shell Utilities

Flame Graphs

Building MySQL from Source

Building MySQL for Ubuntu Focal Fossa and ARM processors

Analyzing a MySQL Crash

Index