Full Stack Testing

A Practical Guide for Delivering High Quality Software

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

Gayathri Mohan

Full Stack Testing

by Gayathri Mohan

Copyright © 2021 Gayathri Mohan. All rights reserved.

Printed in the United States of America.

Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.

- Acquisitions Editor: Melissa Duffield

- Development Editor: Jill Leonard

- Production Editor: Daniel Elfanbaum

- Interior Designer: David Futato

- Cover Designer: Karen Montgomery

- Illustrator: Kate Dullea

- March 2022: First Edition

Revision History for the Early Release

- 2021-07-13:First Release

- 2021-08-24:Second Release

See http://oreilly.com/catalog/errata.csp?isbn=9781098108137 for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Full Stack Testing, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-098-10806-9

[LSI]

Chapter 1. Introduction to Full Stack Testing

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 1st chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at jleonard@oreilly.com.

In today’s world, ‘digitalization’ is inevitable to sustain and grow any business. Many businesses are leading the way in this aspect, while some are in the early phases of modernizing their existing digital platforms.

Digitalization is key to broadening a business’s reach from a local community to a global scale, translating to more adoption and more revenue. Almost all small and large-scale enterprises in various sectors, like health care, retail, travel, academics, social media, banking, entertainment, devise plans to advance their digital strategies as a critical measure to yield higher profits.

In this journey towards digitalization and modernization, innovation becomes a crucial driver. We can see the businesses that innovate constantly continue to thrive over many decades. Netflix is a classic example. It started as an online DVD rental portal in the 1990s. It ventured into movie streaming, making its own DVD rental business obsolete in the 2000s. It later started producing original content called Netflix Originals. As of 2021, Netflix is the largest online media subscription platform with over 200 million subscribers.

The technology space has been evolving in parallel to these innovative businesses to accommodate their increasingly advanced needs. Gone are the days where we were willing to wait in line to buy movie tickets, visit a store in a remote location to buy a specialty product, or carry around a hand-written shopping list searching for specifics. Technology eases everyday tasks extensively. We can sit at home and stream our favorite entertainment programs in one touch, try a new dress virtually, schedule our shopping list to be delivered regularly, and do even more—brew a coffee with voice command!

With such rapid evolution in technology, product strategy must be versatile, catering to different customer needs to survive the ongoing competition in their respective sectors. It has to expand its horizons beyond just building websites. Consider the ride-hailing companies, such as Uber and Lyft, which provide their services at our fingertips in various ways—the web, Android, and iOS mobile platforms. In some regions, they even provide tele-booking services through interactive voice response systems (IVR) and call centers. This kind of versatile product strategy has helped them expand across the globe and outgrow competitors.

With such versatile product strategy and innovation, the businesses become successful in acquiring a critical mass of customers. But now, the next challenge is to thrive further—earn more revenue and more customers. We can see the giants in the industry, such as Amazon, leveraging their existing customer base to cross-sell services & products as a strategy to expand beyond. Amazon, which started as an online bookstore, now cross-sells various products ranging from fresh pantry items to electronics to retail products and more, meeting customer demand for consumer goods in nearly every market segment.

Today’s software industry caters to all these business needs—innovate new product ideas, bring them to life, and scale them to reach new customer segments across the globe. To aid the process, they continuously learn new technologies. Unquestionably, the software development teams are standing on the edge, especially when the need for the hour is to deliver all of that with high quality!

Software quality is a non-negotiable criterion when it comes to sustaining a business in the long term. There are many examples of how a glitch in software quality can affect reputation and result in big loss for the businesses. For example, in October 2014, on “Big Billion Day” sale,1 the two Indian eCommerce giants SnapDeal and Flipkart went head to head on their seasonal sale. Flipkart’s website crashed multiple times during the day due to overwhelming demand, losing their sales to Snapdeal.

We can see many such examples across the globe, where businesses albeit their novel product ideas, slip down a steep slope as customers quickly move on to more reliable competitors when quality is compromised. As a result, we can firmly say that quality is quintessential for sustaining a business in the long term.

Such high software quality can be reaped only by a combination of skillful development and meticulous testing. In particular, when I say meticulous testing, I refer to paying attention to every aspect of the application throughout its stack in various facets. This chapter will introduce the breadth of such full stack testing, which has to be performed for a typical web or mobile application.

Full Stack Testing for High Quality

So, to begin with, let us come together on a common understanding of software quality.

Software quality was once equated to a bug-free application. Anyone in the software world today will agree that it’s not just that anymore. If you ask end-users to define quality, we will hear them speak of ease of use, look and feel, data privacy, swiftness in rendering information, and availability of services 24/7. If you ask businesses to define quality, we will hear them speak about reusable code, real-time analytics, zero downtime, no vendor lock-in, scalable infrastructure, data security, legal compliance, and more. All of these are what make an application a high-quality software. A slack in any of these aspects would affect the quality in some way or the other, which is why we need to test for them meticulously!

Though the list of quality requirements looks tall, we have tools and methodologies to cater to most of these needs. So, the bridge to high quality is the knowledge around the tools and, more importantly, the skill to apply them in a given context, both from a development and a testing perspective. This book aims to serve as that bridge between learning the testing skills needed and delivering high-quality software.

Testing, in a nutshell, is a practice to validate the behavior of the application to be as expected throughout. For testing to be successful, it needs to be practiced at the micro and macro levels. It has to be entwined with the granular aspects of the application, such as testing every method in a class, every input data, log message, error code, etc. Similarly, it has to focus on the macro aspects such as testing features, integration between features, end-to-end workflows, etc. But we cannot stop testing there! We need to further test at micro and macro levels for the holistic quality aspects of the application such as security, performance, accessibility, usability, and more from the list mentioned earlier to achieve the end goal of delivering high-quality software. We can encapsulate that to say we need to do full stack testing! As represented in Figure 1-1, full stack testing is to test for different quality aspects of the application at each stack and application as a whole.

Figure 1-1. A representation of full-stack testing

Full stack testing and development should be inseparable indeed, as the two tracks of a railway line. We must build both tracks simultaneously to build quality-in;2 otherwise, we are guaranteed to derail. For example, suppose you are writing a small block of code to calculate the total order amount for an e-commerce application. Parallelly, you need to test if the code is computing the right amount and if it is a secure code. If you do not test, then you have a broken rail line. When you continue to develop on top of such a fractured rail-line, you will have poor integration and broken functionality.

To ingrain testing at such elementary levels, teams need to stop conceiving testing as a siloed post-development activity, as it has been traditionally done. Full stack testing needs to begin parallel to development and practiced throughout the delivery cycle, giving faster feedback. The practice of starting testing early in the delivery cycle is referred to as shift-left testing, and for full stack testing to yield the required results, you need to follow shift-left testing.

Shift-Left Testing

If we were to write down the sequence of activities in a traditional software development lifecycle (SDLC), they’d read: requirements analysis, design, development, and testing, where testing comes at the end. As seen in Figure 1-2, ‘shift left testing’ suggests shifting the testing activities to the beginning of the cycle instead to yield high-quality results.

Figure 1-2. Shift left testing

Let us take an analogy to understand this concept better. Let us say your team is building a new house. Does it seem sensible to complete the construction fully and then check for quality at the end of construction? What if you find out the rooms are not of the correct measurements then? What if there is no space to place the door? Those are the issues shift-left testing tries to overcome. The approach believes the quality checks should start right from the planning stage and continue throughout the development phase for the end product to be of the highest quality.

When it says quality checks should continue throughout the development phase, it means the quality checks have to be repeated iteratively for every small chunk of work so that the changes can be incorporated smoothly. In the house construction analogy, it means to do quality checks for each wall when it is getting built so that the issues are corrected immediately.

To perform such extensive testing, shift-left testing heavily relies on automated testing and Continuous Integration (CI), where the quality checks are automated and continuously run in the CI for every small chunk of work. This ensures the application is continuously tested at less cost and effort than manually testing every small chunk of work for multiple quality aspects.

To elaborate this further, some of the steps teams take while shifting left are:

- Writing unit tests as part of each story and integrating with CI to be run for every commit.

- Including linting tools and plugins for static code analysis during development and also integrating them with CI.

- Adding functional tests for every story and integrating with CI.

- In general, running all types of tests (unit, service, integration, end-to-end tests, etc.) for every commit on the local machine and CI to ensure frequent feedback.

Doing them will ensure almost 50% of the testing tasks are completed even before the newly developed code is deployed to the testing environment. In other words, testing is shifted to the left.

This concept of incorporating testing earlier in the delivery cycle is not restricted just to functional application testing. It can be applied to testing in general, including security testing, performance testing, and others. For example, one of the many ways to shift security testing to the left is to add a pre-commit scanning tool like Talisman, which scans the commit for secrets and alerts, even before checking in the code. In each of the upcoming chapters, you will get to see practical approaches to do shift-left testing.

Overall, shift left testing embodies the aphorism, ‘Quality is the team’s responsibility’ as performing quality checks at every phase of the software development life cycle—starting from performing quality checks on application design prototypes, requirements, and so on, requires every team member to own testing for different quality aspects in collaboration. A few examples of how collaboration between team members has proven to work well while shifting left are: the developers and testers together design and implement the test strategy; analysts and testers together refine the quality of the requirements; analysts and designers refine the quality of the prototypes and so on. So building relevant testing skills to perform various quality checks is vital for all roles in a team to deliver high-quality software successfully.

Ten Full Stack Testing Skills

When we think of testing skills, we tend to consolidate them as two broad skills—manual and automated testing. But technology has evolved over many decades, and these broad terms mask the essential new skills that one has to learn to perform various quality checks and deliver high-quality software. Let’s break down these two monolith terms and look at the ten testing skills that will enable us to perform full stack testing efficiently! Figure 10-3 shows the ten full stack testing skills.

Figure 1-3. Ten full stack testing skills

I will explain briefly what these ten skills are and why we should care to learn them:

Manual Exploratory Testing

Manual testing refers to the activity where someone verifies a given list of requirements. But manual exploratory testing is the ability to think/analyze different test cases beyond the given requirement list with the intention to break the system and find issues. For instance, the requirement states the application should give an error when it cannot create an order. The exploratory mindset is to think of all possible technical and business scenarios that will result in an error and how the error affects the integrating systems. This is the most preliminary skill to create a bug-free application.

Automated Functional Testing

In simple terms, this is the skill to write code to automate feature requirements and test cases discovered during exploratory testing. It includes the skill to strategize the automation approach and also the knowledge of various tools to automate web applications, APIs, and desktop applications in some cases. This is a core skill for shift-left testing, as we discussed earlier. It reduces the manual testing efforts significantly as the application grows with more features.

Continuous Testing

Continuous Delivery (CD) is a practice where features are delivered incrementally to customers in short cycles instead of a single big-bang release. CD allows businesses to see profits early and keep their customers engaged. To power CD, you have to build and test the application for each release continuously. The wise way is to automate and integrate tests with the CI pipelines and run them frequently to ease the release process. Continuous testing is the skill to integrate the tests to CI/ CD tools for delivering high-quality releases.

Data Testing

You may have heard the sayings, ‘data is money’ and ‘data is the new oil,’ which shows how important is testing for data quality. This skill is the knowledge about various types of databases, query languages, fundamentals about consistency models in distributed systems, and testing for data quality.

Cross-Functional Requirements (CFR) Testing

There are so many cross-functional requirements for any given application, such as availability, scalability, maintainability, observability, and so on. A common practice is to do a prioritization exercise to call out the critical ones and the nice to haves. Teams have to focus extensively on testing the CFR; failing to test for them will sometimes result in a massive business loss. (Data testing and the upcoming four are a few major items among the many CFR).

Visual Testing

The UI look and feel of the application is a major contributor to brand quality. It is essential to test the visual quality, especially for the B2C (business to customers) applications, such as Amazon, as it is directly exposed to millions of customers. The knowledge to strategize and execute the manual and automated tests for validating the UI look and feel of the application is included in the visual testing skill. It may seem like it overlaps with automated functional testing, but we will explore the stark differences in Chapter 7.

Security Testing

Even giants such as Facebook and Twitter have encountered security issues. Indeed, security bugs tend to be too costly in terms of legal penalties and brand reputation. Security testing is the skill to think like a hacker and attack the system to find issues early. We should also be able to automate security test cases and integrate them with CI pipelines for frequent feedback.

Performance Testing

Even a slight slack in web performance leads to a huge revenue and reputation loss for businesses—recall the Flipkart example. Performance testing is the skill to measure the key performance indicators at different application layers and automate and integrate performance tests in CI. Shifting performance testing left is the best path forward for software teams to avoid major performance mishaps. I’ll discuss this more in Chapter 9.

Accessibility Testing

Websites have become everyday commodities. Making the web accessible to everyone is not only mandated by legal regulations in many countries but also proves lucrative for businesses. The accessibility testing skill includes knowledge about recommended accessibility standards, various accessibility testing tools, and automating them to run as part of CI.

Mobile Testing and Automation

As per the data, there are more mobile consumers than desktop across the globe.3 So every product has a mobile variant these days, and learning to test mobile applications has become mandatory. Mobile testing skill encompasses the ability to strategize testing across multiple mobile platforms, test for mobile-specific CFR, and knowledge of tools to automate and integrate with CI pipelines.

The ten full stack testing skills will enable us to test for the holistic quality aspects of the application. As mentioned earlier, every role in the team has to upskill themselves to a varied degree of competency in each of these skills. The book dedicates one chapter for each skill, where we explore how to shift-left and also apply them at various layers of the application with practical examples.

Key Takeaways

Here are the key takeaways from this chapter:

- Software quality cannot be equated to just bug-free software anymore. Quality dimensions have grown to include various facets like security, performance, etc.

- To ensure high quality, we need to test for all the quality aspects.

- Teams should focus on testing and development equally to deliver high-quality software.

- For testing to be successful, it has to be practiced at the micro and macro levels of the application.

- Shift-left testing calls for automation and CI/CD practices to build quality-in.

- Shift-left testing embodies the aphorism “Quality is the team’s responsibility.”

- All team members have to possess testing skills at a certain competency level.

- The two monolith terms, manual and automation testing, mask a vast set of testing skills in them.

- Full stack testing is to apply a set of ten different testing skills at different layers of the application individually and holistically while shifting testing to the left.

1   The Hindu: Technical glitches mar Flipkart’s ‘Big Billion Day’, thehindu.com, October 06, 2014

2   Scaled Agile Framework (SAFe): Built-in Quality, scaledagileframework.com, February 10, 2021

3   Statcounter GlobalStats: Mobile vs. Desktop usage, gs.statcounter.com

Chapter 2. Security Testing

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 8th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at jleonard@oreilly.com.

“A chain is only as strong as its weakest link.”

Thomas Reid, “Essays on the Intellectual Powers of Man” (1786)

We live in a world where we are more gullible to cybercrimes than ever before, especially when we have social media! Cybercrime is an umbrella term referring to all the illegal activities that can be performed with a computer and network, including financial theft, theft of private assets such as sales documents and research reports, exploitation of sensitive information such as an individual’s biological data, and more. A report in 2018 quoted the revenue generated by the cybercrime industry to be $1.5 trillion, and social media enabled cybercrimes alone to have generated $3.25 billion.1 That is unquestionably a huge amount, and sadly, it may include some of our and our friends’ hard-earned money!

Further, the numbers imply that cybercrimes are more prevalent than we would have imagined. Daily news also shows us proof that cybercrimes are not restricted just to banking or social media sites but to all kinds of websites—hotel booking, flight booking, adult dating sites get attacked too. We will discuss such real-life attacks later in the chapter to get more insights. This poses a crucial question to the software teams: What measures can we take to protect an application from such attacks?

To build a strong, secure system, the industry recommended principle is to build your defenses at depth, i.e., build security measures in multiple layers of the application and not just restrict to only a single outer solid layer. This is analogous to how kingdoms protected themselves - with a moat, then strong iron gates, armed guards, and so on. Each layer has to be strong enough, as breaking through one layer gives access to more and more resources every time.

Also, remember the security of your system is only as strong as your weakest link—be it your internal admins or hacked passwords. Thus exploring for weak links is a crucial step in the effort to build strong impenetrable systems. And security testing is the primary activity that unveils such weak links!

Security testing is the skill to think like a hacker and identify the potential vulnerabilities, threats, and risks in the system that opens gates to cybercrimes. Professional security testers or pen testers have developed such a skill over many years, and they can script different attacks on the application to expose the vulnerabilities. However, you don’t have to be a pen tester to use some good automated security testing tools and techniques in your day-to-day team activities to prevent significant security issues.

Security bugs, just like any functional software bug, are incredibly costly to fix when found at later stages of the development cycle. As a result, waiting to engage pen testers until the end of the delivery cycle can leave considerable gaps in your application’s defense systems. Hence, you should practice shift-left security testing.

Indeed, the recommended best practice is to start thinking about a feature’s security aspects right from the beginning of the requirements gathering phase. For example, the requirement to hide all account transactions from anyone other than the account holder and bank administrator is an obvious and necessary security feature. Similarly, the requirement to have a two-factor authentication feature provides an additional layer of security, much like having armed guards in addition to a stone wall protecting a castle. So inculcating security best practices throughout analysis, development, and testing phases, frequently and continuously, will help us build strong, secure systems.

This chapter will cover the necessary topics needed to upskill yourself in security testing so that you can shift your security testing to the left in the delivery cycle. You will learn about real-life attacks, STRIDE model, security vulnerabilities, threat modeling, security testing strategy, automated security testing tools, and integrating them to the CI.

NOTE

Becoming a professionally trained security tester is not the expected outcome of this chapter. As mentioned, it takes years of practice. But that does not wash away the responsibilities of a software team to build secure systems. Hence this chapter focuses on the best practices and testing tools the software teams can leverage to build secure applications early on.

COMMONLY USED SECURITY-RELATED TERMS

The following are some commonly used security-releated terms:

- Assets are the critical entities in the application that need to be guarded by building appropriate defense mechanisms.

- A security compromise happens when the defense mechanisms of the system fail to protect the assets.

- Vulnerability is a term to refer to the potential gaps in your system which can be leveraged to compromise the security of the system.

- Threats are the potential negative actions or events that can leverage the underlying vulnerabilities to compromise the security of your system.

- An attack is performing an unauthorized malicious action on the system and compromising the security of the system.

- Encryption is a technique to scramble information in such a way that only the intended recipient who has the key to decipher can understand the information.

- Hashing is a technique to map any length of data to a fixed size output using algorithms. This is used to verify the authenticity of the information itself. Hashing is a one-way technique, i.e., it is hard to get back the original content. Hashes are immutable, i.e., each unique input produces the same unique output.

Common Cyber Attacks

Let’s begin with understanding some of the widespread attacks with real-life examples so that you can start understanding how a hacker would think! Later we will discuss a thinking framework for all possible threats and vulnerabilities in a system to further tune your mindset towards the hacker’s way of thinking.

Web Scraping

The easiest way to abuse a system is to exploit the publicly available data on the website, especially the user’s personal data. It’s almost equivalent to placing your valuables at your window. A web scraping attack is a method to automatically crawl a website and gather crucial information, which is further used for malicious intent. Especially, social media has a lot of personal data like the user’s location, phone numbers, and more. From the report mentioned at the beginning of the chapter, the personal data scraped from social media makes a revenue of $630 million per year.

An example of a web scraping attack is that in 2019, 419 million Facebook user records that had their phone numbers were found in an unprotected database server on the internet, as reported by TechCrunch.2 Though Facebook had removed the feature to display phone numbers on user’s profiles by then, it seemed that the information was scraped when the feature was available.

Exposed data is always an avenue for exploitation, whether it be on the website or anywhere else. When you are thinking like a hacker, you should look for such exposed data in the system. Another example of unintentional exposure of data is, in 2018, Twitter found a bug in their internal tool, which logged the passwords in plain text without hashing 3 (we never know where our weak link is!). Hence Twitter asked its 330 million users to change their passwords though fortunately, there was no evidence of compromises.

Brute Force

If you had to guess your friend’s password, how would you go about it? You might try their date of birth, favorite color, spouse’s name, or a combination of such things, correct? When you extend the same trial and error method with a big organized list of all possible key combinations, it’s called brute force attack.

For example, the brute force attack on the AdultFriendFinder site in 2016 exposed 412 million user records with passwords and other sexual preferences.4 It is said that the usernames and passwords were encrypted using the SHA-1 cryptography algorithm in their database. With modern brute force techniques and the availability of computing power, SHA-1 failed to provide a good defense.

Social Engineering

Social Engineering is the psychological manipulation of individuals into giving away their confidential information. You might have also received such phone calls from what seemed like a reputed company, and a friendly customer service person would have requested your credit card details in exchange for their service! You are not alone. In 2019, a UK-based energy company’s CEO was manipulated by a phone call from someone who sounded like his boss (later found out to be a trained AI program) to transfer a lump sum of $243,000 to the hacker’s account!5

Phishing

A phishing attack is a type of social engineering where an attacker sends malware to the user via emails to get their personal data. It would be a surprise if you didn’t receive one such email in today’s world. In 2021, Microsoft 365 users received a phishing email with a scripted HTML attachment which on opening was designed to abduct their personal data.6

Cross-site Scripting

Cross-site scripting is when the attackers take advantage of an unsecured website and inject code to manipulate the application’s behavior. For instance, hackers can inject code into the website to redirect the customer’s payment details to their servers. In 2018, British Airways had a cross-site scripting attack on their website, exposing 500,000 customers’ credit card details.7 British Airways was fined heavily due to a lack of a proper defense system.

Ransomware

Ransomware attacks are where a malware blocks the system until a ransom is paid to the attacker. The Weather Channel went offline for an hour in 2019 due to a malicious ransomware attack on their network.8 Since they had backup servers, they were able to get past the episode successfully.

Cookie Forging

Cookie forging is a technique to manipulate the cookies that store user information on a website and get access to the user’s account. In 2016, around 32 million Yahoo! user accounts were breached by forging the cookies.9 It was stated that the hackers learned the internal login functionality of Yahoo! from their proprietary code and forged the cookies to gain access to accounts.

Cryptojacking

Cryptojacking has become a widespread attack these days. It is the activity of secretly mining cryptocurrencies in an unauthorized machine. Usually, bots are set to crawl the public Github repositories to find infrastructure (e.g., AWS) access keys. Once found, the instances are exploited within seconds, costing huge loss for the infrastructure owners. In 2018, Tesla Inc. was a victim of such illegal mining.10

Figure 2-1. Common threats to software systems

The attacks discussed in this section are only the tip of the iceberg. As seen in Figure 2-1, many more attacks happen at different layers - application, infrastructure, and network. Hackers continue to create new kinds of attacks every day. It is the software teams’ paramount responsibility to be ahead of the game to save the customers and the businesses. Governments, parallelly, develop legal regulations like GDPR and PSD 2 to enforce security and data privacy across countries.

Threats Framework

The real-life examples we discussed in the previous section elicit the attackers’ motives clearly—they seek to abuse the user’s personal data, money, infrastructure, access to services, and brand reputation. These are the primary assets that we need to protect in any application. I worked on a project where we designed the central security system for a bank. We discussed so many security threats (it was such a stand-out experience!) that could lead to a compromise of these five assets.

Similarly, when you are brainstorming the security threats for your application, you need to think of all possible threats that could compromise your application’s assets. To help you navigate the vast breadth of threats, you can adopt a threats thinking framework called the STRIDE. STRIDE is an acronym for spoofed identity, tampering with inputs, repudiation of actions, information disclosure, denial of service, and escalation of privileges. You can take one at a time and discuss all possible threats under that category for your application.

Let’s take a closer look at each of these classifications of security threats.

Spoofed Identity

Spoofed identity attacks are those where a hacker assumes the identity of an authorized person to access the assets. Recall the social engineering example from earlier where the AI impersonated the CEO’s boss’s identity to launder money. Identity thefts are prevalent and made possible by social engineering, phishing attacks, malware, and shoulder surfing (the method of spying over someone’s shoulder when they’re entering their personal information, such as a password or ATM pin), just to name a few.

Some of the defense mechanisms adopted for this type of threat are multi-factor authentication, strong password recommendations, strong encryptions while storing data, and so on.

Tampering with Inputs

The threat of tampering with inputs involves modifying something on the application (e.g., code, data, memory, etc.), which violates integrity. This could happen by injecting malicious code, such as a script, into the UI or other layers. As we saw in the British Airways example from earlier, the hacker changed the behavior of the website by injecting scripts to get the customer’s credit card details.

The defense to this threat includes adding appropriate validations (e.g., add validations on the input fields to prevent someone from sending SQL queries), authentication and authorization mechanisms, and other standard security practices while coding to avoid vulnerabilities resulting in a code injection.

Repudiation of Actions

Repudiation occurs when the actions of a malicious user cannot be proved or tracked back to them. For example, a customer could deny receiving the item after delivery if there is no systemic proof. This is a critical aspect to consider while designing a functionality as it will result in loss of goods, reputation, money, and sometimes legal suits to compromise. To prevent this threat, the system should have adequate logs and auditing mechanisms to ensure non-repudiation.

Information Disclosure

The information disclosure class of threats involves unauthorized entities having access to the application’s assets. As we saw in the Twitter example earlier, the employees could see the exposed user passwords, which they were not supposed to have access to. The Twitter example is an unintentional consequence, and fortunately, there were no security compromises. However, such attacks to gain unauthorized information can be conducted by setting up malware to listen in the background and relay the information from a legitimate site to the hacker. This is referred to as a man-in-the-middle attack. To provide a proper defense against this kind of threat, you should build a strong authorization mechanism, encrypt all secrets, and have secure transmission protocols.

Denial of Services

The denial of services threat deals with an attacker bringing the application’s services down, causing loss of revenue and reputation for the company. One way this threat could manifest is by a Distributed Denial of Services (DDoS) attack. The system is intentionally overloaded with millions of requests so that it becomes slow and finally starts failing.

The defense to this kind of threat is to add load balancers, throttle requests per IP address, create backups for systems, auto-spin new machines on load, set up monitoring systems to alert on sudden surges in request volumes, to list a few among others.

Escalation of Privileges

An escalation of privileges threat happens when a malicious unauthorized user gains higher privileges than necessary to an application causing loss of private data, denial of services, loss of money, and so on. In my opinion, this is the worst kind of threat to deal with. Imagine a hacker getting super admin privileges to a system! A best practice is to follow the principle of least privilege, which is to give only the least necessary privilege to all the system’s users and nothing more. We can also apply it to our everyday activities, like granting code-committing privileges only to developers on the team and grant access to others only when necessary. Some useful defense techniques to this threat are frequent access token refreshes, multi-signature features for authorizing transactions, secrets stored in vaults, etc.

At the end of brainstorming security threats for your application using the STRIDE model, you need to first think of solutions to prevent the threats from becoming attacks and then think of confining the impact radius if the attack happens, unfortunately.

Application Vulnerabilities

In the effort towards learning to think like a hacker, we have discussed some of the widespread attacks, potential assets that hackers try to abduct, and a framework to think of all possible threats to your application. The next step is to get closer to the application code to learn about the different security vulnerabilities that can be leveraged by the threats. Understanding these vulnerabilities will help us to add defense code and test for them.

Code or SQL Injection

An attacker could inject malicious commands or SQL queries to alter the behavior of the website if it is unprotected. Example 2-1 shows the code to retrieve the student’s record by name.

Example 2-1. SQL query in code where it takes an input variable

//SQL query in the code to fetch student’s record by name

SELECT * FROM Students WHERE name = '$name’

As we can see, the query takes an input $name from the user. The query is written in such a way that when a malicious user enters a SQL query to drop the table instead of a student’s name, it will work perfectly fine, as seen in Example 2-2.

Example 2-2. The injected SQL query will drop the entire table

//Malicious user inputs this as ‘student name’ in UI

Name: Alice’; DROP TABLE Students; --

//The application will execute this command:

SELECT * FROM Students WHERE name = 'Alice'; DROP TABLE Students; --'

Cross-Site Scripting (XSS)

As discussed earlier, cross-site Scripting (XSS) allows attackers to execute scripts in the victim’s browser and take control of user sessions which is then used to redirect the users to malicious sites or deface websites. It is prevalent when there is no validation and proper sanitization of the user input data.

For example, a Twitter user posted a sample harmless Javascript to reveal the XSS vulnerability in the Tweetdeck application, as seen in Figure 2-2. It is a code to auto-retweet itself whenever it appears on someone’s timeline, popping an alert dialog box after. This could have been prevented if the app had properly validated the post’s text for the presence of scripts. But since the app didn’t, the browser innocently executed the script.

Figure 2-2. Tweet with XSS

Unhandled Known Vulnerabilities

If an application is dependent on several third-party softwares (e.g., OS, libraries, frameworks, tools), a vulnerability in any of these could be exploited to gain access to the system. Such vulnerabilities are often found, fixed, and patches are sent for an update by the tool maintainers. But the teams do not regularly update the vulnerable components continuing to expose the application.

Authentication and Session Mismanagement

Sometimes, authentication mechanisms in the websites are not foolproof, leaving space for attackers to steal the session tokens and exploit the privileges. If you store session IDs and sensitive user data in the session cookie, you need to refresh them very often and invalidate the older cookies. Also, you need to watch out for vulnerabilities like exposing session IDs in the URLs, using unencrypted connections to send sensitive authentication data, and so on.

Unencrypted Private Data

Many times, we fall prey to the common vulnerability of unencrypted private data. We saw how unintentional exposure of sensitive data such as the Facebook users’ phone numbers from the example earlier resulted in exploitation. You must ensure private data is not laid open without encryption in logs, databases, code repositories, project documentations, publicly hosted services, etc. Also, choosing high-end cryptographic algorithms like the AES, HMAC, MD5, SHA-256, etc., with dynamic salt and pepper techniques help to protect data in transit and rest.

Application misconfigurations

It is a frequent mistake to give blanket admin permissions everywhere and for everyone using the application simply because it saves some effort in maintenance. Misconfiguring the relevant permissions to users, folders, systems, etc., will result in unauthorized access to application content such as the database and admin endpoints, which can be easily abused without much effort.

Application Secrets Exposure

A common practice that leads to compromise is hardcoding the application secrets such as the environment credentials, superuser credentials, etc., in code and configuration files as plain text. An appropriate measure is to use secret management services such as the vaults and access secrets only from there. This applies to application code, CI/CD pipelines, configuration files, and all the places you might have to access secrets.

The list in this section shows a set of vulnerabilities that have to be carefully dealt with during development and testing. To further expand your hacker-like thinking skills, continue to read about exposed vulnerabilities and various real-life attacks. Open Web Application Security Project (OWASP), a community-led non-profit organization, has identified the top ten common vulnerabilities on the web, which you might be interested in reading.11

Threat Modeling

Having come this far, you might be thinking about the security threats and vulnerabilities your application might be exposed to right now. In this section, I’ll show you how to approach threat modeling in a methodical way.

A general best practice is to do the threat modeling exercise as a recurrent activity for a small scope of the application, such as a 15-minute threat modeling exercise per user story. Once you model your security threats, you can prioritize them based on impact and probability of risk, which then can be solutionized as part of the story or as a new feature. When you prioritize the threats, use the general rule of thumb: the cost of building security measures to handle a potential threat should not be higher than the value of the asset that you are trying to protect.

For example, let’s say your team is developing a blogging platform. Before building the home page, you do a 15-minute threat modeling exercise to discover a potential threat that a ransomware attack could bring down the page. The team proposes to have a security monitoring system as a solution. The monitoring system is estimated to cost $400K per year. Do you need to fix the ransomware threat? Maybe not, as the cost to fix the threat might be higher than the profit per year for the company. Also, how often does a ransomware attack happen on a blogging platform? The probability is very low. On the other hand, a threat like a code injection attack on the website that leads to loss of credit card details can be considered a high impact and a high probability threat.

Once you have identified the prioritized threats, address the solutions in the same user story or, if needed, create abuser/evil user stories—user stories that address the solutions to prioritized threats.

For example:

“As an abusive user, I cannot inject code to redirect the content of the website.”

You can also derive security-related test cases from the threat modeling exercise and abuser stories’ acceptance criteria for development and testing purposes.

Threat Modeling Steps

Let’s take a closer look at the framework for completing a threat modeling exercise. It is recommended to do this exercise as a team with all roles represented. Whiteboarding with colored stickies works well as you can capture your team’s thoughts and categorize them quickly. In a remote world, choose tools like Mural to conduct the exercise. Once your team is assembled, navigate through the following milestones:

Define the feature

The first step is to define the scope of the feature. Then, start by drawing the user flows and the different types of users or actors in the system. Once this is clear, understand the flow of data from one component to another. This way, you would have covered user flows, actors, data flow, and the integration between components in the system.

Define the assets

The second step is to identify the assets in the feature that need to be protected. Also, discuss the impact of losing that asset and capture the severity.

Evil brainstorming

Next, open the floor for evil brainstorming. Here, the mindset of the team should be - ‘Let’s break the system’! Capture all possible ways to attack the application’s assets. Use the STRIDE model to structure this discussion. Allow the imagination to flow freely without debating if it’s really a threat or not, for now.

Prioritize the threats and capture stories

Analyze the probability and the impact of threats and prioritize them. Capture these as abuser stories so the team can act on them after the threat modeling brainstorming session.

Now that you know the basics let’s complete a sample threat modeling exercise.

Threat Modeling Exercise

For our threat modeling exercise, consider that we have an application that creates, views, edits, and cancels orders for a retail store.

NOTE

This exercise is to get you familiar with the threat modeling steps and does not intend to give an accurate threat model for an order management system (OMS).

To begin with, let us define the feature. The application has a Web UI and a backend REST service to perform business operations on the order data stored in the database. The users of the system are:

- The store assistant who places, edits, and cancels orders.

- The system administrator who manages the infrastructure, configurations, and deployments.

- The customer service executive who uses the application to answer queries related to order statuses over the phone.

The user flow is simple - the store assistants and the customer service executives login to the application and see the latest order list with options to manage the orders. Figure 2-3 shows the integration of components and data flow between the components as well.

Figure 2-3. User flows and data flows for the sample order management system.

Similarly, as shown in Figure 2-4, the system admin has to login to the VMs to run any scripts or configure the infrastructure.

Figure 2-4. User flow of the system admin

Let’s discuss the assets that we need to protect next. Here are a few:

- Order information is a critical asset to the business as the customers will be dissatisfied if the order has been tampered with, resulting in a loss of reputation.

- Orders have private details of customers such as names, phone numbers, payment details, and home addresses. Any exposure of confidential information will result in major lawsuits and also create harm to customers. So, customer details are another essential asset.

- The database has the complete sales information of the business. A breach there will be hazardous to customers and the business, as that could be sold in the black market or competitors.

- The infrastructure that hosts the application is crucial to be protected as any downtime will lead to failure in order transactions and loss in sales.

Now, let’s get to evil brainstorming! Take a moment to think. Recall the user and data flows and think about how the hackers could abduct the assets. Use the STRIDE model to structure your thinking.

When you are done thinking, here is a list of few possible threats:

Figure 2-5. OMS exercise threats

Spoofed identity:

1. Social engineering tricks could be played on the system admin to get their login credentials, or just shoulder surfing or malware could do the trick. Since the system admin credential has superpowers, anyone can use it to bring the infrastructure down.

2. The store assistant could forget to logout, and anyone in the store can use the browser to tamper the order with an already logged-in session.

Tampered inputs:

3. The attacker can get hold of the order service endpoints from any open browser session and tamper with the orders.

4. While placing the orders, customer payment details could be attacked from the UI using code injection.

Repudiation of actions:

5. The system admin, when they find out there are no logs for their actions, could create bulk orders for their families and friends by directly inserting records in the DB.

Information disclosure:

6. If the DB is attacked via backdoors, all the information will be exposed as they are stored in plain text.

7. Stealing the access token from unencrypted logs or other storage will enable the attacker to tamper with the order data.

8. The customer service executive doesn’t have any restrictions on the operations. So they can edit the order when the only need for them is to relay the current order statuses to the customers. Hence they can work with an accomplice to abuse their access tokens.

9. The /viewOrders endpoint allows any number of records to be returned. Once compromised, the endpoint could be used to view all orders. We should at least think of reducing the blast radius.

Denial of service:

10. With any stolen access token, the attacker could perform a DDoS attack on the order service.

Escalation of privileges:

11. If the admin credentials are hacked, new users could be added or escalated in privileges to create order records without anyone noticing as there are no logs for system admin actions.

As we can see in Figure 2-5, even in a small system with a few handful of components and users, there are so many attack points. Imagine how many attack points will be there for a real system with many components and users!

Therefore, to discover all the threats to your application, as mentioned earlier, continue performing threat modeling exercises for small scope frequently throughout your development cycle. You will find new threats to older features as you incrementally brainstorm new features’ threats.

Finally, let us prioritize the threats and capture stories. Based on likelihood and impact, we can add new security-related user and abuser stories like the following:

- “As an abusive user, I should not be able to see the customer details even if I gain access to the database.”

- “As an abusive user, I should not be able to get access to unused sessions.”

- “As an abusive user, if I get access to system administrator’ & customer executive’s access tokens, I should not be able to edit the order.”

- “As a store assistant, I should be the only authorized person to make edit requests to the order service.”

- “As a store assistant, I should be frequently prompted to change my password to a strong password.”

Security Test Cases from the Threat Model

The threat model gives insights into the many ways an attack can happen on the application. Now, that should provide us with an idea about what test cases to look for. Recollect the approach of testing in parts from Chapter 2 to arrive at the security test cases for every application layer. For instance, here are some security test cases for the order management system example:

NOTE

Zero trust is a principle that suggests not to place your trust on any entity—be it a person or a component even within your secure perimeter. Zero trust architectures verify the authenticity of every request before executing the request. OAuth2 is one such authentication and authorization protocol that uses bearer tokens for verifying authenticity.12

UI layer:

- Verify if the website has a session timeout, after which the user is prompted to login again.

- Verify user credentials are locked after a set number of failed login attempts.

- Verify if input fields on the UI have data validation for javascript code, SQL queries, and other illegitimate inputs.

- Verify if access tokens expire after a short period. However, a refresh token call should be made from the UI to keep the user logged in until the session times out.

- Verify if the system admin and the customer executive do not see an editing option in the UI.

API layer:

- Verify if reusing an expired access token on the order service throws a 401 Unauthorized response (though 400 is usually issued to abstain any further knowledge to the attacker).

- Verify if inputs to API parameters are validated for malicious input data and throw a 404 error.

- Verify the /editOrder endpoint throws a 401 Unauthorized response using a system admin and customer executive access tokens.

DB layer:

- Verify if the password is stored as a hash (with a dynamic salt) in the DB.

- Verify if the customer details are encrypted in the DB.

Application logs:

- Verify the passwords are not logged as plain text in application logs.

- Verify user-sensitive information is not logged as plain text in application logs.

- Verify there are appropriate application logs for all actions performed on the system with timestamps.

These are just a few of the possible test cases. There may be more! I hope this exercise helped you understand how teams can bring security into the software development lifecycle, starting from analysis to solutionizing security threats and testing security aspects.

Security Testing Strategy

The steps we discussed so far—conducting a 15 minutes threat modeling exercise per user story, writing abuser stories, building security measures in layers, deriving security-related test cases, and so on, will significantly help strengthen your defense systems. One more critical step we need to do is add mechanisms to give continuous feedback on the code being developed for potential vulnerabilities so that it can be fixed at the earliest. This is where we need to think of shifting security testing to the left. We will discuss a security testing strategy that implements shift-left testing here.

Figure 2-6 shows a shift-left security testing strategy across the different stages of the path to production. Let’s discuss them one by one now.

Figure 2-6. Shift Left Security Testing

Static Application Security Testing (SAST)

SAST refers to the technique of analyzing the static application source code, byte code, and assembled code for known vulnerabilities. For example, it scans the application code for unencrypted secrets. SAST tools come in various forms—plugins, libraries, Saas solutions (e.g., Snyk IDE, Checkmarx, Security code scan) and can be integrated with CI pipelines to run against every commit. SAST is a huge shift-left step as it helps discover issues during development.

Further, Talisman, though not exactly a SAST tool, is specifically designed to scan the application code for secrets such as private keys, environment credentials, and so on, which can be integrated as a pre-commit hook. This prevents secrets from even being pushed to the repository.

Source Composition Analysis (SCA)

SCA is the technique that identifies the vulnerabilities in the application’s third-party dependencies. Especially when you are using a lot of open source libraries, these tools (e.g. OWASP Dependency Check, Snyk Open source) will add a lot of value. They also give feedback during development and can be integrated with CI for every commit. SCA combined with SAST will help uncover the static application code’s vulnerabilities during the development phase itself.

Image Scanning

Containers have become a widely adopted way to package and deploy applications. Hence testing for vulnerabilities in the container images, if you are using them, is critical. Tools like Snyk Container, Anchore, etc., can be used to do image scanning and can be integrated with CI. Docker has an inbuilt command docker scan to perform vulnerability scanning on docker images. Similarly, Amazon Elastic Container Registry (ECR) offers image scanning capabilities for the images pushed to ECR. When you are writing your infrastructure code using Terraform and Kubernetes, tools like Snyk Iac, terraform-compliance, etc., can be used to enforce security best practices as well.

Dynamic Application Security Testing (DAST)

DAST is a black-box testing technique. It finds security issues by analyzing how the application responds to specially crafted requests that mimic actual attacks. For example, DAST tools (e.g. OWASP ZAP, Burp Suite) try to inject malicious scripts into the application to check for code injection vulnerability. They can be integrated with CI pipelines too. They might take longer to run based on the application. So you can choose to integrate them as nightly regression in CI.

These days, the interactive application security testing (IAST) technique tries to combine both the SAST and DAST techniques to analyze the application behavior during run time. It works through software instrumentation and scans for security vulnerabilities in real-time. This space is evolving, and some example IAST tools are Contrast security and Acunetix.

Functional Security Tests Automation

You can add automated tests using Selenium or Cypress, as discussed in Chapter 3, to cover the functional security test cases. For instance, in the OMS example from earlier, you can automate the functional security test case – the system admin should not have access to edit the order.

Manual Exploratory Testing

Also, during manual exploratory testing, derive security-related test cases from threat modeling exercises across all layers, i.e., the UI, services, DB. Chrome DevTools and Postman provide various flexibilities to execute security test cases.

Pen Testing

Based on the criticality of the application and the development team’s competency with respect to security, choose to involve a professional security tester near the end of the delivery cycle so that the application doesn’t suffer from security issues later.

Runtime Application Self Protection (RASP)

RASP is the technique to monitor the application for potential attacks in the production environment. The RASP method takes the traditional firewall approach further. The RASP tools (e.g., Twistlock, Aqua Security) listens to the application processes, builds a model of what is and not an expected behavior, and automatically takes protective measures such as terminating a crypto mining process and so on.

And that’s how you distribute security testing throughout the delivery phase, starting from the left and also in production.

Security Testing Tools

We will explore some of the open-source SAST, SCA, and DAST tools mentioned earlier. We will also look at integrating them into CI.

Snyk IDE plugin

Snyk Jetbrains IDE plugin combines both SCA and SAST capabilities together. It is entirely free for users with any JetBrains IDE (e.g., IntelliJ IDEA, Webstorm, Pycharm). The biggest advantage is it is so close to development. You can trigger scans to check for vulnerabilities in both application code and their dependencies while the code is being developed. It also shows options to fix them, making it easier for developers to build security-in.

To try this:

1. Install the plugin in your IDE. In IntelliJ, Preferences -> Plugins -> Search for ‘Snyk’ -> Install the vulnerability scanner option.

2. After you do that, a Snyk tab will appear in the IDE bottom pane beside the ‘Terminal’ tab.

3. Open the tab and trigger your scan, which will show the project’s scanned report in a few seconds, as seen in Figure 2-6. (It might ask you to authenticate against your Snyk account. You can create a free account and complete the step.)

Figure 2-7. Snyk IDE example scan results

As we can see from Figure 2-7, the results have the remediation as well. Snyk also comes as a CLI option but only with SCA capabilities to be integrated with CI. Other Snyk services are currently paid services.

Talisman Pre-commit Hook

Talisman, an open-source tool, scans the source code for secrets and sensitive information, such as passwords, SSH keys, tokens, etc., when you commit to the version control system and raises alerts. This is very helpful in preventing development teams from committing secrets in a hurry.

To use Talisman:

1. Install Talisman as a global pre-commit hook by running the following command:

curl --silent https://raw.githubusercontent.com/thoughtworks/talisman/master/global_install_scripts/install.bash > /tmp/install_talisman.bash && /bin/bash /tmp/install_talisman.bash

2. Once installed, whenever you run git commit, talisman automatically scans for secrets and generates reports, as seen in Example 2-3.

Example 2-3. Talisman sample scan results

$ git commit

Talisman Report:

+-----------------+-------------------------------------------------------------

------------------+

|     FILE        |                                    ERRORS                                     |

+-----------------+-------------------------------------------------------------

------------------+

| sampleCode.pem  | The filename "sampleCode.pem"

|

|                 | failed checks against the

|

|                 | pattern ^.+\.pem$

|

+-----------------+-------------------------------------------------------------

------------------+

| sampleCode.pem  | Expected file not to contain hex-encoded texts

such as:                     |

|                 |

awsSecretKey=c99e0c79ddcf5ddb02f1274db2d973f363f4f553ab1692d8d203b4cc09692f79 |

+-----------------+-------------------------------------------------------------

------------------+

As we can see in Example 2-3, it has identified an “awsSecretKey” in the code.

OWASP Dependency-Check

As we saw earlier, one of the common threats is to have dependencies with vulnerabilities. OWASP Dependency-Check is an open-source SCA tool that scans for known vulnerabilities in the project’s libraries and external dependencies. It can be used either via command line or as a plugin to Jenkins, among other ways.

To try the command-line option:

1. Install on Dependency-Check on Mac, using the command:

$ brew install dependency-check

In Windows, download the dependency-check zip file from the official website.

2. Once installed, run the scan on the project using the command:

//In Mac

$ dependency-check --project project_name -s project_path --prettyPrint

//In Windows. The dependency-check.bat is inside the bin folder when you unzip the folder you downloaded in the previous step.

dependency-check.bat --project "Project Name" --scan "Project path"

3. This would have generated an HTML scan results report, as seen in Figure 2-8.

Figure 2-8. OWASP Dependency check scan results

As we can see in Figure 2-8, it has reported a vulnerability in the jquery-1.8.2 library. It has referred to the vulnerability ID CVE-2012-6708 with the explanation that “jQuery before 1.9.0 is vulnerable to Cross-site Scripting (XSS) attacks” to help us update the library appropriately.

The same can be integrated with CI to make it fail if vulnerabilities are detected.

OWASP ZAP

OWASP Zed Attack Proxy (ZAP) is an open-source tool that does DAST in a deployed environment. It uses preconfigured automated scripts to expose a known set of vulnerabilities on an application. This makes it easier for teams new to security fundamentals to find security issues easily. ZAP operates as a ‘man-in-the-middle’ tool between the browser and the application. It sniffs the messages exchanged between them for known vulnerabilities and also modifies them to conduct various attacks.

ZAP has an exhaustive list of configurations and add-ons to support multiple functionalities using which even security professionals can add their advanced scripts. There is good documentation to explore all these options. ZAP can be integrated with other tools like Selenium making it easier to run on CI.

To install ZAP on Mac:

$ brew install cask owasp-zap

There are installation binaries for other platforms like Windows, Linux, etc., too, on their official website.

ZAP comes with a desktop UI to play with the attacks, as seen in Figure 2-9. Once the installation is done, you should be able to open ZAP from Applications. (On a Mac, you may have to give permissions under ‘Security preferences’ since it’s not from the Apple store.)

Figure 2-9. ZAP Desktop UI

First, ZAP needs to know the existing URLs and UI components within the application in order to attack them. To do this, click on the ‘Manual Explore’ option and enter the application URL as seen in Figure 2-10. It allows us to use Firefox or Chrome. Once the browser opens your app, manually walk through the user flow once, and ZAP will scan the application in the background to record the relevant details.

Figure 2-10. ZAP Manual Explore option

WARNING

It is illegal to conduct security testing on unauthorized websites.

OWASP gives us a guinea pig website, the ‘OWASP Juice shop,’ to learn security testing. I have used the same URL here. To avoid toggling between the desktop app and the browser during attacks, ZAP gives the Heads Up Display (HUD) option. HUD is a browser overlay that comes on top of your application to execute attacks from the browser itself. There is an option to turn it off, as seen in Figure 2-10. You can see the HUD on top of the Juice shop website with the right and left panels in Figure 2-11.

Figure 2-11. HUD on JuiceShop website

If you explore the site manually, you’ll see that the ‘Sites’ icon on the right panel has the site tree, and the History tab on the bottom has the list of URLs visited. These details will be used by ZAP when it actively attacks the application. There is also an automated way to gather these details using the ZAP spider feature.

ZAP spider will automatically scan the website for all possible URLs and UI components. It uses Selenium to drive through the website by clicking buttons, links, etc. The simple ‘spider’ option might not be able to navigate Javascript components; hence there is an ‘AJAX spider.’ So first select the simple spider - the ‘grey’ spider icon on the right-side panel and then the ‘red’ spider icon, which is the AJAX spider, to complete the entire application scan.

Scanning

There are two types of scanning with ZAP: passive and active.

Passive scanning

As the spider crawls the website, ZAP automatically does ‘passive scanning’ in the background. Passive scanning is when the messages between the browser and web application are only read and inspected for vulnerabilities and not modified, i.e., not attacked. So we will see alerts being shown in the right and left side panels when the spider is crawling. The alerts are prioritized based on severity as ‘High,’ ‘Medium,’ ‘Low’ and grouped under red, orange, and yellow flags. Refer to Figure 2-11.

On clicking those flags, we can see details of those vulnerabilities. Also, the ZAP Desktop UI will have detailed logs. For example, a passive scan found a private IP exposed in the demo website, as seen in Figure 2-12.

Figure 2-12. Passive Scanning Results in ZAP Desktop UI

Active scanning

Once ZAP gets the site information, it can now attack the application by intercepting the requests, modifying them, and bouncing them back and forth for known vulnerabilities like SQL injection and many more. The ‘Active scan’ option is the 4th icon from the top in the right-side panel of the HUD. During an active scan, you will see it navigates through the website across various pages one by one, mimicking different attacks. This takes a lot of time to complete. Once completed, we can see the vulnerabilities under the colored flags. Figure 2-13 shows a SQL injection vulnerability found on the demo website.

Figure 2-13. SQL Injection vulnerability found in Active scan.

Integrating ZAP with CI

Analyzing all the vulnerabilities and making decisions about fixing them takes a lot of time and expertise. It is advisable to do the exercise continuously instead of piling it up at the end—what better option than integrating with CI to get continuous feedback. Please be aware that the scan can take a lot of time to complete depending on your application, and sometimes it even takes hours. If it is, you can choose to integrate it with CI as nightly regression or as a manual trigger which you could use for every story.

The easiest option to integrate with CI is to use the predefined Github Actions - OWASP ZAP Baseline Scan, OWASP ZAP Full Scan. Github Actions can be used to define workflows in the software development cycle. For example, every time there is a commit event, then run the specific action. So the above two predefined Github actions do the ZAP scan and add issues to Github.

Another way is to create scripts that will trigger the scan using ZAP APIs and execute from the ZAP command-line interface (CLI). Create a script file in python or Javascript with the following ZAP APIs:

1. Open the URL with zap.urlopen(target)

2. Spider the web with zap.spider.scan(target)

3. Trigger active scan using zap.ascan.scan(target)

4. Print the alerts from zap.core.alerts()

Pass the script file to ZAP to trigger the scan using the below command:

/Applications/OWASP\ZAP.app/Contents/Java/zap.sh -script <script_file_path>

The third option to integrate with CI is to integrate the ZAP APIs in functional tests using Selenium Webdriver and run like a typical functional test. The webdriver, here, is used to login to the website, which the scanner might not be able to do on its own. Example 2-4 shows a sample ZAP scanning test that will fail if any vulnerability is found. You will have to add appropriate waits in between in the code for the scan to complete.

Example 2-4. ZAP scan as part of Selenium tests

@Test

public void testSecurityVulnerabilities() throws Exception {

zapScanner = new ZAProxyScanner(ZAP_PROXYHOST, ZAP_PROXYPORT, ZAP_APIKEY);

login.loginAsUser();

// Step 1- Spider the app using ZAP API

zapSpider.spider(BASE_URL)

// Step 2 - Enable passive scanning

zapScanner.setEnablePassiveScan(true);

// Step 3 -Start Active scan. Add wait methods.

zapScanner.scan(BASE_URL);

// Step 4 - Log the alerts and assert the count of alerts

List<Alert> alerts = filterAlerts(zapScanner.getAlerts());

logAlerts(alerts);

assertThat(alerts.size(), equalTo(0));

}

ZAP produces HTML reports of the vulnerabilities, as seen in Figure 2-14, that can be saved as output artifacts in CI.

Figure 2-14. ZAP HTML Report

ZAP has many other helpful features that enable various kinds of exploratory security testing on the application. A few are listed below:

- ZAP can take Open API specifications to do security testing on APIs.

- It has a feature called ‘Breaks’ which will help us insert specific test data to a request and observe the behavior. For example, to test if the API validates the input parameters for SQL injection, we can use the Break feature.

- It allows replaying a request on the browser.

- There is the ability to highlight specific hidden keywords in the HTML.

- It has a feature to disclose all hidden input fields on the application.

- There are add-ons with prewritten scripts crafted by experts which can be used to play specific types of attacks if needed.

Overall, ZAP is an excellent place to try and learn many things about security.

Chrome DevTools & Postman

For performing manual exploratory security testing around functional use cases, such as the different test cases that result from a threat modeling exercise, Chrome and Postman tools are pretty handy. Postman allows us to configure API tokens to test for scenarios like tampered or expired access tokens, as seen in Figure 2-15.

Figure 2-15. Postman access token setup

Similarly, the Security tab in Chrome Dev tools tells us if the page is properly served over HTTPS. Refer to Figure 2-16. It highlights when resources from third-party sites are not served securely, which can potentially lead to man-in-the-middle attacks.

Figure 2-16. Security tab in Chrome Dev tools.

Security as a habit

We may shell out our ultimate efforts in all the activities discussed till now to create secure software, but unless we make security a habit, we may still leave unexpected weak links that could be exploited. For example, several day-to-day practices in a software team could lead to a security compromise. Have you thought about the security aspects of the tools we use to assist in development and testing? Is it storing project data in their private cloud? Are you leaving the architecture diagram of the project with environment details in an online portal? Did you share your production system credentials with all your team members? Was it shared in Slack in plain text? And many, many such simple acts that could result in a compromise very easily.

Hence we need to make security a habit – the same as we analyze a food item before eating it or become conscious when someone is following us. We need to start questioning if our simple, unintentional, harmless act could result in security breaches.

Key Takeaways

Here are the key takeaways from this chapter:

- Cybercrimes are more prevalent than we would imagine leading to an underworld revenue of $1.5 trillion. So, let us not consider security a ‘nice to have’ feature.

- Security should be embedded throughout the SDLC from analysis to testing in order to build strong impenetrable systems.

- The STRIDE model gives a structured lens to explore security threats for an application.

- Do threat modeling exercises with the entire team frequently and for a small scope. Capture abuser stories and derive security test cases from the threat model.

- Shifting security testing to the left using automated security testing tools and integration with CI should be our security testing strategy.

- Most importantly, make security a habit.

- Continue to read about exposed vulnerabilities and real-life attacks to hone your security testing skills.

1   Casey Crane: Alarming Cybercrime Statistics in 2019, securityboulevard.com, November 15, 2019.

2   Zack Whittaker: Facebook phone numbers breach, techcrunch.com, September 5, 2019.

3   Chaim Gartenberg: Twitter’s internal tooling bug, theverge.com, May 3, 2018

4   Steve Ragan: AdultFriendFinder attack, csoonline.com, November 3, 2016.

5   Catherine Stupp: AI voice fraud, wsj.com, August 30, 2019.

6   Mayank Sharma: Phishing attack on Microsoft employees, techradar.com, March 2021.

7   Lily Hay Newman: British Airways hack details, wired.com, November 9, 2018.

8   Lee Mathews: The Weather channel Ransomware Attack, forbes.com, April 19, 2019.

9   Eduard Kovacs: Cookie forging on Yahoo! Users, securityweek.com, March 2, 2017.

10   Lily Hay Newman: Cryptojacking Tesla’s Amazon cloud, wired.com, November 20, 2018.

11   OWASP: Top 10 Vulnerabilities, owasp.org.

12   Oauth 2.0: Bearer Tokens, oauth.net.

Chapter 3. Performance Testing

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 9th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at jleonard@oreilly.com.

“Time is money!”

Benjamin Franklin

Sometimes, our favorite websites suddenly become as slow as sloths, leaving us wondering - ‘Is there a problem with my internet?’! Remember waiting an eternity during the Cyber Monday sales for the website to load? Or staring at the loading icon for the train tickets to show up when you are dying to book your Christmas vacation? Or left hanging on the booking page of a blockbuster movie? These are some day-to-day instances where we, as customers, feel intense frustrations due to poor website performance!

If you want to save your application’s end-users from such frustrations, you have to continuously measure and improve the performance of your application. This chapter aims to equip you with the essentials around measuring or testing performance—specifically topics like performance KPIs, API performance testing, front-end performance testing and shift-left performance testing are covered.

Performance, Sales, and Weekends Off Are Correlated!

We spoke about customers being frustrated because of application performance earlier. We need to understand what it leads to. How does a few seconds delay really matter? Indeed, there is a quantitative indicator called the ‘Bounce rate’, which tells how customer’s frustrations lead to loss in business. Bounce rate captures the number of customers that leave the website after viewing just one page!

Among all the other factors that would increase bounce rate, website performance has been the major contributor. Statistics published by Google (see Table 3-1) show the correlation between page load time and user bounce rates.1 The data confirms that with every additional second delay, businesses lose their customers to their competitors.

Table 3-1. Google statistics correlating page load time and bounce rates

Page Load Time

Increases the probability of bounce by

1 - 3 seconds

32%

1 - 5 seconds

90%

1 - 6 seconds

106%

1 - 10 seconds

123%

Google also ranks the slower websites lower in their SEO (Search Engine Optimisation) algorithms. So that pushes your website further down into the abyss if it is not performant enough! According to Maile Ohye in her online webmasters class in 2010, Google, one of the best-in-class websites, aims for under a half-second load time.2

Losing customers translates to losing sales. Indeed businesses lose a heavy cut; like in 2018, when Amazon failed to handle traffic loads on their Prime Day sales, the estimated loss was around $72 million! Lousy performance can lead to a loss of reputation for the brand as well. Especially, when the world is tightly connected over social media, bad reviews spread so rapidly.

On the flip side, a slight increase in performance can result in significant improvement in sales. For instance, in 2016, The Trainline, a train operating company in the UK, reduced load time by 0.3s, and revenue increased by an extra £8 million ($11 million) a year. Similarly, Mobify has observed that every 100-millisecond decrease in homepage load time increased the annual revenue by $380,000.3 The correlation between sales and performance makes it clear that the first step to improving sales for an online business is to look at their application’s performance. This means we need to build and test for performance early and frequently i.e., shift your performance testing to the left!

My motivation to focus on website performance early is straightforward - I love my weekends and want to spend them leisurely. Since performance issues are very costly and directly affect the brand reputation, software development teams are under high pressure to fix them ASAP! Forrester’s research showed that software teams that don’t focus on performance at the earliest end up fixing 100% of performance issues directly in production.4 So, if you do not incorporate performance testing early and often during the SLDC, you can expect to pay for it by working on weekends (and long hours) later to fix the performance issues!

Simple Performance Goals

Performance, in simple terms, is the ability of the application to serve a vast number of concurrent users without significant degradation in its behavior compared to when it is serving only a single user, provided overall behavior stays within the end-user acceptable limits. So, to test for performance, firstly you need to seal the expected number of peak-time users for your application and then ensure if the performance is within the end-user acceptable limits.

There are studies to tell us what the end-user acceptable limits are:

- According to Jakob Nielsen, when the response time of the site is 0.1 second, the user feels the behavior was instantaneous.5

- Similarly, when the response time is 1 second, they feel the delay but are still in control of the navigation on the website.

- And, from the Google statistics we saw earlier, beyond 3 seconds we are at a risk of losing the majority of the customers. Google, indeed, recommends to keep the page load time less than 2 seconds at least from the second visit onwards.

These are your performance goals. To achieve such good results, a lot of infrastructure tuning and code optimization needs to happen in many iterations before going live—yet another reason to shift your performance testing to the left.

Factors Affecting Application Performance

Achieving the performance goals is not that easy. There are so many factors in an application that challenges us in the path. Let us look at some of the factors that affect application performance now.

Architecture design

Architecture design plays a vital role in contributing to the performance of a website. For instance, when the services are not sliced appropriately, it will lead to numerous service calls delaying the response time. Similarly, when appropriate caching mechanisms are not implemented at the right levels, it will affect the website performance too.

Choice of tech stack

Different stacks of the application need a varied set of tools. If not considered keenly, the tools may fail to work together coherently, affecting the performance. An example of tech stack impacting performance is how different runtime environments (e.g Java, RUST, Go, C#) have a minor difference in AWS Lambda cold startup time.

Code Complexity

Bad code often leads to performance issues due to cyclic dependencies, long operations, missed validations, duplicate validations, etc. Consider the case where a search is done with empty values. What would be optimal is for the search service to do a simple input data validation and fail the request quickly. If not, the service searches the database for empty values and returns an error later, delaying the response time unnecessarily.

Database performance

Every database comes with its SLAs for processing time. If your application requires very high performance, choosing suitable databases and the proper format for storing data will be critical. For instance, storing the order data in parts across multiple tables will require consolidation and delay retrieving the final order. So depending on the domain, structuring the data properly with performance in mind is essential.

Network Latency

The central nervous system for any application is the network. All components access each other via some kind of network communication - WAN, LAN, etc. So, having good connectivity between components in the data center where they are hosted is crucial. The end-users across the globe reach the application using the network as well. We cannot take network connectivity for granted. Indeed, most users will be on mobile networks with weak signals. So, if your website is designed to be heavy with images or involves substantial data transfers, better to test the system with various network throttling conditions early.

GeoLocation of the application and users

If the users of your website are only from a particular region, then having the website hosted physically closer to the region will reduce the network hops and hence the latency. For example, if the website is for European customers but is hosted in Singapore, there will be multiple to-and-fro network hops to connect to the system. If a website intends to serve customers across the globe, there should be a strategy to replicate the application’s hosting in closer geolocations (or use CDNs). If you use cloud infrastructure, you should remember to request physically closer machines to the end customers, because a common mistake is getting a machine closer to the development team’s location.

Infrastructure

Infrastructure is the skeleton that supports all the muscles of a system. The power of infrastructure in terms of CPU, memory, etc., will directly impact the system’s ability to take the load. Designing infrastructure to deliver a high-performing system is an art in itself. Infrastructure engineers continuously collect the results of the performance tests as one of the parameters to plan the infrastructure needs of the application.

Third-party integrations

When there are integrations with third-party components, the application is dependent on that component’s performance. Any latency in that component will eventually add to our application’s performance. For example, a typical retail application integrates with third-party payment services, vendor’s product information management systems, warehouse management systems, etc. So choosing a performant accomplice is vital.

In this section, you saw a high-level view of multiple factors that add up to the application’s performance. The factors are also variables i.e., they change over the course of delivery. Sometimes, they tend to be mutually exclusive, forcing us to make tough decisions. Shifting your performance testing to the left will guide the team as a north star in handling the volatile nature of the application performance. Also, it is equally important to consider these factors during performance testing. For instance, have a performance testing environment that is very similar to the production environment in terms of network, infrastructure, etc., as otherwise, you may not have an accurate measure of performance!

Key Performance Indicators

Measuring the application performance means to capture the quantitative key performance indicators (KPIs) that will help the team to course-correct their way throughout development. The earlier you measure them, the minor the corrections are. The following are three such quantitative KPIs.

Response time

Response time refers to the time taken by the application to answer any query made by the user. For example, what is the exact time taken to show the results of a product search to the customer? The average response time for web applications is at 3 seconds, beyond which we face the risk of losing the majority of our customers.

Concurrency/Throughput

Usually, there will be more than a single user accessing the application at any given time. Indeed, some of the high-speed applications such as the stock exchange sites will have millions of transactions happening within a second. As previously mentioned, measuring the total number of concurrent users the application can support successfully will help to scale the application. For example, the application can respond within 3 seconds for 500 concurrent users.

Although ‘concurrent users’ is a commonly used term by the businesses and teams, when we think from the system’s perspective, it receives various requests from end-users and other components, which are queued and picked for processing one after the other by a few parallel threads. Hence, the number of concurrent users indicator doesn’t sit well while thinking from the system’s perspective. A better indicator to measure here will be the ‘throughput.’ Throughput measures the number of requests the system can support during an interval of time.

To understand this better, consider the analogy of cars crossing a very short bridge on a highway in respective car lanes. Let’s say there are four car lanes. Given the type of car, it can swiftly pass the bridge in the range of a few hundred milliseconds. So in a second, the total number of cars crossing the bridge will be 30 to 40. This value of 30 cars per second is the throughput.

Concurrency and throughput are both helpful in server capacity planning and are often used in different contexts to make wise decisions.

Availability

‘Availability’ means the system responds to the users at the same expected performance for a given continuous period. Typically, the websites are expected to be available 24 / 7 except for planned maintenance. Availability is an essential criterion to test because the applications may perform well for the first half-hour. Then responses could degrade over time due to memory leakage or parallel batch jobs consuming the infrastructure capacity or any such unpredicted reasons.

The next step is to understand how to measure these key performance indicators.

Types of Performance Tests

To measure the KPIs, you need to design performance tests accordingly. The following are the three of the most common types or designs of performance tests.

Load/Volume tests

We need to validate if the system performs well for a given volume of users expected by the business. For instance, the search functionality should respond within 2 seconds for a volume of 300 users. The performance test to simulate the volume and validate if the system meets the required targets is called ‘volume tests.’ We may have to repeat such tests multiple times to observe consistency and measure the average to benchmark the application.

Stress tests

The system’s performance will start degrading when more users are stressing it. We need that exact measure of load when the system gets stressed. This number is vital for scaling the application in scenarios like expanding to new countries or peak sales. Normal system behavior is that it performs well for X users. Beyond X users, delays creep into the response time. Finally, at X+n requests, the system responds with errors. We need to know these figures. To get that, we can design the performance test to slowly increase the load on the system, beyond the volume test limits, in small steps and study precisely where the system breaks down. This process of stressing the system to find the breaking point is called ‘stress testing.’

Soak Tests

When the system runs with good volume for a while, there may be degradation in response time due to infrastructure issues or memory leakage, etc. The performance test designed to keep the system under a constant volume of load for a prolonged period and measuring KPIs is called the ‘soak test.’

While designing these tests, a key point to note is to keep them realistic to avoid overheating the application with extreme situations that may never occur. For instance, not all users are logging into the application at the same instant of time. A more realistic use case is that the users log in gradually with a few milliseconds gap. This behavior is called the ‘Ramp-Up.’ Our test cases should include such a realistic design; let’s say, 100 users are ramped up in 1 minute.

Further, users are not robots who perform a login, search for a product, and buy a product within milliseconds. But test cases might end up being designed that way unfortunately. Practically, users take at least a few seconds to think between their actions and take minutes to complete a user flow like buying a product after logging in. This is called the ‘think time’ in performance testing terms. We need to include appropriate think time in our test cases to spread the user actions apart a few seconds or minutes and make it realistic.

Types of Load Patterns

The performance test cases are nothing but simulating different load patterns on the application. Different load patterns can be generated using these three key parameters: ramp-up time, think time, and the number of concurrent users. Following are some of the commonly tested load patterns:

Steady Ramp-Up Pattern

In the steady ramp-up pattern, the users are steadily ramped up within a given period, and then the load is maintained constantly for a sustained period to measure performance. Refer to Figure 3-1. This is typically the scenario in most real-world applications, such as during the Black Friday sales, where the users gradually but steadily come into the system and stay there for some time before dropping out progressively.

Figure 3-1. Steady Ramp-Up of users

Step Ramp-Up Pattern

With the step ramp-up pattern, users are ramped up in batches periodically. For example, 100 users every 2 minutes. Refer to Figure 3-2. This is to observe and measure the system’s performance for each step count of users, which will help benchmark the application for different loads. Step Ramp-Up pattern will be used in performance tuning and infrastructure capacity planning.

NOTE

Benchmarking is to record the response time with a single user and measure the average from repeated runs.

Figure 3-2. Step Ramp-Up of users

Peak-Rest Pattern

The peak-rest pattern is when the system is ramped up to reach peak load and then ramped down to complete rest in repeated cycles. Refer to Figure 3-3. This scenario could be true in some applications like social networking, where the peak comes and goes in cycles.

Figure 3-3. Peak-Rest Load Pattern

As you’ll see later in this chapter, performance testing tools will allow you to configure these patterns to fit your needs.

Performance Testing Steps

Now that we’ve discussed the importance of performance, KPIs, and load patterns, next is to know the performance testing steps. Use these steps to plan the necessary time and capacity for shifting your performance testing left.

Define the Target KPIs

The first step is defining the target KPIs based on business needs. The best way to start thinking about the target numbers is to think about them qualitatively and then translate them into numbers.6 For instance, qualitative thinking about performance could lead to goals such as:

- We should be able to scale the application to one more new country.

- We should be able to perform better than our competitor X.

- We want to do better than the last version of the application.

These qualitative goals naturally lead towards the next steps. If the goal is to do better than the last version of the application, we need to measure the performance numbers of the earlier version and see if our current numbers are better. Similarly, if we know the competitor’s performance numbers, we need to validate that our numbers are better than them.

NOTE

TIP: Business folks tend to call-out performance numbers which may not be the actual usage pattern. Derive the target KPIs from data always.

- If there is an existing application, analyze the production data to arrive at KPIs and load patterns.

- If you are building a new application, ask for competitors’ data.

- If the application is completely new with no reference data, still use data around country wise internet usage, probable peak duration, etc., to work out your target KPIs.

Define the Test Cases

The second step is to use the load patterns and the performance test types to define the different performance testing cases. Our test cases should be around measuring availability, throughput, and response time for all critical endpoints. We need to include both positive and negative scenarios for performance testing as sometimes negative scenarios take more time to respond. The test cases will reveal the test data setup needed to run the test cases.

Prepare the Performance Testing Environment

The performance testing environment should be as close to the production environment so that you can get realistic results. It will also help you to identify any performance bottlenecks in environment configurations.

When we say as close to the production environment, we mean:

- The respective tiers/components need to be deployed in a similar fashion as in production.

- Machine configurations like the number of CPUs, memory capacity, OS version, etc., should be similar.

- The machines should be hosted in the respective geolocation in the cloud.

- Network bandwidth between machines should be similar.

- Application configurations like rate limiting should be precisely the same.

- If there will be batch jobs running in the background, those should be in place. If there are emails to be sent, those systems should be in place too.

- Load balancers, if any, should be in place.

- Third-party software should be available at least in a mocked capacity.

Getting a production-like environment is often challenging due to additional costs involved, though we get cheaper cloud provisions. It is a matter of cost vs. value conversation. If you don’t win that battle, prepare to make meaningful tradeoffs on the performance environment setup. Also, it is worth raising to the respective stakeholders early that the performance numbers measured with such tradeoffs might not be foolproof.

NOTE

A best practice is to request the performance testing environment aside from the QA environment right at the beginning of the project so it’s available when you need it.

Apart from the performance test environment, we also need a separate machine to be the test runner i.e., to run the performance tests. Plan to have individual test runners hosted in different geolocations (this is possible with cloud providers) to observe the respective performance behaviors with network latencies from multiple countries, if your application is intended to serve a global audience.

Prepare the Test Data

Just like how the performance testing environment should be as close to the production environment, the test data should be as reflective as the production data. The performance numbers that you will measure will greatly depend on the test data quality and hence this is a critical step. An ideal situation will be to use the same production data after anonymizing the sensitive user information as it will reflect the actual database size and complexity of data. However, you may find blockers in getting the real production data due to security issues in certain situations. In such cases, prepare the test data that exactly mimics the production data.

Few pointers around creating test data are: -

- Estimate the size of the production database (e.g., 1 GB or, 1 TB) and set up scripts to populate the test data. It may be necessary that for each test run, the test data must be cleaned and repopulated. So, having the test data creation scripts will be handy.

- Create variety in test data similar to production. Instead of ‘Shirt1’, ‘Shirt2’ etc., use actual production-like values such as ‘Van Heusen Olive Green V-Neck Tshirt.’

- Populate a fair share of erroneous values like addresses with spelling mistakes, blank spaces, etc.

- Have similar distribution of data representing factors like age, country, etc., closer to production.

- You may have to create a lot of unique data like credit card numbers, login credentials, etc., to run volume tests with concurrent users.

Yes, preparing the test data can be a tedious job! These activities need to be planned well ahead of time in the release cycle. It’s impossible to squeeze it in later as an afterthought and if you did, the test data might not be of good quality resulting in inaccurate performance numbers.

Integrate APM Tools

The next step is to integrate application performance monitoring (APM) tools such as New Relic so that we can see how the system behaved during the performance tests. These tools greatly help while debugging performance issues. For instance, requests may fail during performance test runs due to insufficient memory in the machine, which the APM tools will expose easily.

Script and Run the Performance Tests Using Tools

The last step is to script the performance test cases using tools and run them against the performance testing environment. There are many tools to script and run the performance test cases in a single click and also integrate them with CI to help us shift left. JMeter, Gatling, k6, Apache Benchmark (ab) are some of the popular kids in this playground. These are open-source tools. There are also commercial cloud-hosted tools like Blazemeter, Neoload, etc. Some of these tools provide simple user interfaces to configure the performance tests and don’t require coding. We can get test run reports with graphs, while commercial tools even offer a dashboard view. I will detail how to create test scripts and integrate them with CI using open-source tools in the next section. Performance test runs may actually take time from a few minutes to a few hours depending on the test. So, do a dry run of the scripts with a lesser user count before starting the full fledged test run.

Those are the six performance testing steps. The key to successfully execute all the six steps is to plan capacity for them adequately. Also, plan time and capacity to run the tests, collect reports, debug and fix performance issues, and do server capacity tuning which will complete the entire performance testing cycle!

Performance Testing Exercise

Let us go over the performance testing steps starting from defining the target KPIs to scripting the tests using an example application. For our discussion, we will be using an online library management system where admin users can add, delete books, and customers can view all the books and search a book by its ID. The Rest APIs /createBooks, /deleteBooks, /books and /viewBookByID perform the respective actions in the backend. Let’s get started!

Define the Target KPIs

To arrive at the target KPIs for our library application, assume we got the following data from the business and their in-house marketing team:

- They are campaigning aggressively for the launch in two European cities and expect 100,000 unique users to join in the first year.

- They have a study that said users spend 10 minutes on average searching for books, viewing similar books, etc., in a single session.

- The study also said that a typical user might borrow a book twice every month on average. Hence might access the site twice a month.

- In Europe, the users are active on the internet between 10 a.m. and 10 p.m. (12 hours) daily.

With that data, we can calculate the following:

- Total users accessing the site monthly = 100,000 users *2 accesses per month = 200,000 monthly users

- Average users per day = 200,000 monthly users ÷ 30 days per month = 6,667 daily users. (Note, there could be more users during weekends than weekdays, but we are calculating average users)

- Average users per hour = 6,667 average daily users ÷ 12 hours per day = 555 hourly users. (Similarly, hourly users could be more during lunch or evening times)

- To include peak performances, we can be generous and round up to 1000 hourly users.

- Each user uses the website for a session time of 10 minutes, which is 0.166667 hours.

- Number of concurrent users = 1000 peak hourly users * 0.166 = 166 concurrent users!

Based on our calculations, these are our target KPIs:

- For 166 concurrent users, the system should respond within 3 seconds.

- System throughput has to be 1000 users per hour.

We can get a consensus with the client management on these numbers before we proceed. We can also probe the business to think beyond the first year and check again for target numbers.

NOTE

This is only a sample calculation to give an idea of how to work out target KPIs. However, the first place to dig, as mentioned earlier, is the existing application’s production data or the competitor’s data, which will give a more accurate picture of KPIs and load patterns.

Define the test cases

Now that we know our target KPIs, we can define appropriate performance test cases based on the library application’s features. Let us recollect the factors we discussed earlier momentarily. The test cases for the library application could include:

- Benchmark the response times for all four endpoints - /createBooks, /deleteBooks, /viewBookById and /books.

- Benchmark the response times with negative test cases in all of the four endpoints. Examples of negative test cases here could be deleting a book that doesn’t exist in the library. As discussed earlier, error cases might take more time due to missed validations compared to positive responses.

- Volume test with 166 - 200 concurrent users for /viewBookById and /books endpoints and validate response time to be less than 3 seconds. Admins only access the other two endpoints; hence volume testing may not be necessary for them.

- Stress test the customer-facing endpoints with ramp-up steps of 100 users and find the breaking point.

- Validate throughput of 1000 users in one hour. The user flow could be to select a book from the page where all the books are listed. After skimming the contents of the chosen book, they go back to the page where all books are listed to select a new book again. Users will continue doing this for a total session time of 10 minutes. Include a think time of 45 seconds between each of these actions. Ramp up 10 users every 30 seconds.

- Soak test for continuous 12 hours to validate that the system is available to users continuously. We could reuse the above throughput test design to run for 12 hours, too, if it is successful.

The remaining steps

I have created and deployed my sample library management APIs in the performance testing environment. I have also functionally tested them once to make sure they are working smoothly. I have set up 50 books as test data. I did it by writing a script in JMeter which I will explain in the tools section. Assume that I have integrated an APM tool too. Last step is to script and run the tests using tools. I will show how to script the library app’s test cases using tools like JMeter, Gatling and Apache Benchmark in the next section.

Performance Testing Tools

As mentioned earlier, the performance testing tools help us to script the different performance test cases. They give us the essential handles like the number of concurrent users, ramp-up time, and think time to configure the load patterns as per the test case. Each tool provides a different way to configure these handles—for instance, JMeter gives GUI, Gatling uses domain specific languages (DSL), and Apache Benchmark (ab) takes these handles as simple command-line arguments. Let us explore each of them to script our library app’s test cases.

JMeter

JMeter is a well-evolved and popular tool in the performance testing space. It is entirely open-source with provisions to integrate with CI and generate beautiful graph reports. It integrates with Blazemeter, a cloud-hosted performance analytics tool if you want to be free from infrastructure management tasks. JMeter is based on Java, and there is a community of active developers who contribute to different valuable plugins. The figures in the “Types of Load Patterns” section are from one of the plugins as well. There is good documentation and tutorials on many use cases, too, for beginners. Let us install the tool and write test scripts for our library application.

To install JMeter, download the zip file from JMeter’s official site. Make sure your local Java version is compatible with the required version for JMeter. Also, ensure the JAVA_HOME variable is set in your environment bash_profile. To open JMeter GUI, run the shell script .jmeter.sh, inside the folder /apache-JMeter-version/bin from the terminal.

The JMeter plugins are pretty helpful as well.You can download the plugins manager from the official site and place the jar under /apache-JMeter-version/lib/ext. After that, restart JMeter. You should then see the menu Options->Plugins Manager.

NOTE

Conducting load tests on public APIs with a vast load can be considered a DDoS attack! Most performance testing tools give a test site to try performance testing. Refer to their official site to get the test site URL and hit the test sites with only a minimum prescribed load.

JMeter test skeleton

We can see how to set up a basic JMeter test skeleton using our library app. Let us add a simple test to measure our library app’s response time for /books endpoint, which our customers will access to view all the books in the library. The following are the steps to do the same:

1. In the JMeter GUI, right-click on Test Plan -> Add -> Threads (Users) -> Thread Group on the left-hand pane. Name the thread group as ‘ViewBooks.’ Configure parameters Number of Threads = 1, Ramp up = 0, Loop Count = 10 (see Figure 3-4) to record the response time of the endpoint ten different times and average them.

2. Next, we can create the HTTP request sampler. Right-click on Thread Group -> Add -> Sampler -> HTTP Request. Enter HTTP Request data for /books endpoint like server, path, request body (see Figure 3-5) and name the sampler as ‘ViewBooksRequest.’

3. Next, we can add listeners, which will record every request and response during the test run. Right-click on HTTP Request ->Add ->Listeners -> View Results Tree. Similarly, add the listener ‘Aggregate Report’ as well.

Figure 3-4. Thread Group configuration to run one request ten times

Figure 3-5. ViewBooks HTTP Request configuration

Now, ‘Save’ the basic test skeleton to measure the response time and click the ‘Run’ button. The results will now be available in the two listeners sections.

View results tree listener

Here, we will see the list of individual requests made by JMeter with success or failure indication. JMeter takes Status code 200 as ‘success’; otherwise, it considers the request a failed request. A point to note is that there can be situations in your application where the service will return a 200 status code to indicate that the operation had been executed but may not have produced the intended results. For example, /createBooks endpoint could return 200 status code for duplicate books with a message indicating it is a duplicate. So, in such cases, we need to add explicit assertions on the results (Assertions, like Listeners, are components of JMeter too). The View Results Tree view will also show request and response data on clicking each request for further debugging, as shown in Figure 3-6.

Figure 3-6. View Results Tree Listener Output

Aggregate report listener

Similarly, when we click on Aggregate Report, we will see a table with metrics like Average time, Median, Throughput, etc. For our endpoint, the average response time for 10 samples is 379ms, see Figure 3-7, i.e., the best-case response time is 379ms when the system is not under load.

Figure 3-7. Aggregate Report view for the response time of /books endpoint

Next, we can add load on the /books endpoint and check the response time for 166 concurrent users.

Ways to configure load in JMeter

JMeter offers many controllers to configure different load patterns. Here I present three simple controllers in Jmeter to show how we can use them in our library app’s context.

ThreadGroup Parameters:

Thread Group is a basic JMeter element to configure load. All listeners and controllers have to be placed inside the thread group. Thread group is the place where you configure basic structure of your load tests such as number of parallel threads, ramp up period and the number of times the test has to repeat.

As we saw in Figure 3-4, we configured the thread group parameters to run a single request in loop 10 times to measure response time. Now we can change parameters to Number of Threads = 166, Ramp up = 0, Loop Count = 5. JMeter will spin up 166 concurrent threads with no ramp-up time and loop 5 times to get the average response time.

Concurrency Controller:

Concurrency controller, which comes as a plugin, is a useful controller to design step ramp up load patterns.

Open Options->Plugins Manager. Search for ‘Custom Thread Groups’ under the ‘Available Plugins’ tab and install them. After restarting JMeter, the concurrency controller will be available now. Add the controller to our test by doing a right-click on Test Plan -> Add -> Threads (Users) -> bzm -> Concurrency Thread Group. Here, we can configure the load parameters as Target Concurrency = 166, Ramp-Up time (min) = 0.5, Hold Target Rate Time (min) = 2. Refer to Figure 9-8. By this, we are asking JMeter to ramp up 166 users in 30 seconds and hold them for 2 minutes in the system. Add the HTTP Request sampler like before under this controller, run the test, and view the results in the listeners.

Figure 3-8. Refer 9-8. Concurrency Controller to volume test /books Endpoint

Ultimate Thread Group:

The ultimate thread group controller comes as part of the same Custom Thread Groups plugin we installed earlier. As the name suggests, the controller gives detailed configurability provisions like initial delay before test run, shutdown time after test run to tailor your load pattern.

To add this controller, right-click on Test Plan -> Add -> Threads (Users) -> jp@gc Ultimate Thread Group. For the library app, as seen in Figure 3-9, we can mention the load parameters as Start threads count = 166, Initial Delay = 0, Startup Time = 10, Hold Load for = 60, Shutdown time = 10. These configurations instruct JMeter to spin up 166 concurrent requests within 10 seconds and hold the load for 1 minute; post which ramps down the users within 10 seconds. We can add more rows appropriately to generate the peak-rest pattern, too, here. To run the test, add the HTTP Request sampler like before and view results in the listener.

Figure 3-9. Ultimate Thread Group Controller to volume test /books Endpoint

The results using the simple thread group option (first option) with 166 concurrent users and zero ramp-up time, averaged over five loop counts, is ‘Average = 801 ms and 90% Line = 1499ms’. Refer to Figure 9-10. In other words, 90% of 166 concurrent users get their response back in ~1.5s, and on average, all 166 concurrent users get back their response within 0.8s. The average is lesser because, as we can see from the table, the minimum time to get a response for some users has been just 216ms.

Figure 3-10. Figure 9-10. Volume test results for /books Endpoint

Designing other performance test cases

You have seen various ways to distribute the load on the system using JMeter till now. This knowledge about the tool is a great start to simulate other performance test cases like stress tests, soak tests, throughput validation, etc. To do stress testing, you can use the Concurrency controller to introduce load in steps of x users, until a maximum limit of users, with each step running for a given time. The objective here is to find the load at which the response time slows down and ultimately throws errors.

To do soak testing, you can simulate constant load for an extended time using one of the above controllers, such as the Ultimate thread group. To validate hourly throughput, use ‘Parallel controllers’ to run multiple HTTP requests parallelly while pausing between requests using ‘Timer’ components, i.e., think time.

There are more components in JMeter to help model application-specific use cases. The If, Loop, Random controllers enable to include conditions in tests. There are provisions to provide user credentials if the application needs a login. The login credentials can be fed to the test from an external source like a CSV file to perform concurrent user scenarios. This is called Data-Driven Performance Testing, a feature I used to create test data for our library application which we will see next.

Data-Driven Performance Testing

Our library application has the /createBooks endpoint, which takes request data like name, author, language, and ISBN of the book. I created a CSV file with the request data parameters - name, author, language, and ISBN for 50 books. In JMeter, as usual, add a thread group with HTTP Request sampler for /addBooks endpoint to run 50 times. To wire the CSV file to the HTTP Request, right-click on Thread Group -> Add -> Config Element -> CSV Data set Config. In the CSV Data Set Config window (see Figure 9-11), give the CSV file path and the variables to read from the file. These variables can be accessed as ${variable_name} in the HTTP request body like in Figure 9-12.

Figure 3-11. Figure 9-11.Configuring CSV Dataset Input for Data-driven testing

Figure 3-12. Figure 9-12. Referencing variables from CSV file

That is how I created 50 books in less than a second! Variables in JMeter can be extracted and referred to using ${variable_name} notation in other places too.

Run as part of CI pipelines

Having seen how to add test cases in JMeter and wire data using external sources, the last step is to run them as part of CI pipelines and shift the performance testing left. For that, save the tests first. It gets saved as .jmx files. To run the tests from the command line:

jmeter -n -t <library.jmx> -l <log file> -e -o <Path to output folder>

We can also configure JMeter to provide exhaustive dashboard reports as needed with further extensions.

Gatling

Gatling provides Scala-based DSL to configure the load pattern. It is also an open-source tool with the option to record user flows. The tests can be integrated with CI pipelines. If you are game to explore Scala, this is a robust tool to simulate load patterns with nuances. We can see the sample Scala script demonstrating how to induce load with think time for one of our library application API in Example 9-1.

Example 3-2. Example 9-1. Sample Scala scripts for load testing

package perfTest

import scala.concurrent.duration._

import io.gatling.core.Predef._

import io.gatling.http.Predef._

class BasicSimulation extends Simulation {

//Defining the HTTP Request

val httpProtocol = http

.baseUrl("https://library.herokuapp.com/")

.acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")

.doNotTrackHeader("1")

.acceptLanguageHeader("en-US,en;q=0.5")

.acceptEncodingHeader("gzip, deflate")

.userAgentHeader("Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0")

//Defining a single user flow with think time

val scn = scenario("BasicSimulation")

.exec(http("request_1")

.get("/books"))

.pause(5) // Think time

//Configuring load of 166 concurrent users to do the above user flow

setUp(

scn.inject(atOnceUsers(166))

).protocols(httpProtocol)

}

Apache Benchmark:

If you want to quickly get some numbers on the application performance without the hassle of learning, configuring tools, Apache Benchmark (ab) is your go-to tool! It is a simple open-source command-line tool. If you are on Mac, ab comes as part of the OS, so you don’t have to worry about installation. To get performance numbers for 200 concurrent users accessing our library app, we can run the following command from the terminal:

ab -n 200 -c 200 https://library.herokuapp.com/books

The results will be printed as follows:

Concurrency Level:      200

Time taken for tests:   5.218 seconds

Complete requests:      200

Failed requests:        0

Total transferred:      1389400 bytes

HTML transferred:       1340800 bytes

Requests per second:    38.33 [#/sec] (mean)

Time per request:       5217.609 [ms] (mean)

Time per request:       26.088 [ms] (mean, across all concurrent requests)

Transfer rate:          260.05 [Kbytes/sec] received

Connection Times (ms)

min  mean[+/-sd] median   max

Connect:      869 2074  97.6   2064    2289

Processing:   249 1324 299.4   1303    1783

Waiting:      249 1324 299.5   1303    1781

Total:       1192 3398 354.3   3370    4027

Percentage of the requests served within a certain time (ms)

50%   3370

66%   3483

75%   3711

80%   3776

90%   3863

95%   3889

98%   4016

99%   4022

100%   4027 (longest request)

And that’s how you use tools to script your test cases and measure the application’s performance KPIs.

We have dealt with the performance testing topic at good depth until here, but it is not complete yet. We have to turn our focus to front-end performance testing next!

Front-End Performance Testing

Though performance testing tools allow you to mimic application behavior during peak time, there is a gap between the measured performance numbers and the actual user experienced performance. This is because the tools are not actual browsers and they don’t do all the tasks a typical browser does!

To understand the gap, let us explore a bit about browser behaviors. The front-end code is getting rendered on the browsers, and it has three parts to it:

- HTML code, which is the bare bone structure of the website

- CSS code which styles the page

- Scripts to create logic on the page

A typical browser first downloads the HTML code entirely from the server then gets to downloading the stylesheet, images and executing the scripts as per the sequence in the HTML. There is parallelization to an extent when it is downloading images from different hosts etc. But the browser stops parallel processing completely when executing a script as it is possible for the script to change the way the page is made visible entirely. Since there could be scripts at the end of the HTML, the page becomes visible for the user only when the entire document is fully executed.

Performance testing tools don’t do most of these jobs. They hit the page directly and get the HTML code, but they don’t render the page while executing the performance tests. So even when we have measured the services’ response time to be within milliseconds, the end-user will see the page appear only after a further delay because of the additional rendering tasks, which the browser does. According to Yahoo!, this front-end rendering takes almost 80% of the entire page load time!7 Isn’t it shocking?

For example, if you navigate to the CNN home page, the browser will carry out 90 tasks before the page appears to us. Figure 3-13 shows the first 33 of these tasks. If you had been thinking optimizing the web service’s response time alone would create an impact on website performance, here is a piece of evidence for changing that view!

Figure 3-13. CNN Front end Browser tasks during page load

However, let us not question the importance of the KPIs we measured in the previous section. Those are still critical to plan the system’s capacity and troubleshoot performance issues in application code. In other words, KPIs help us to answer questions such as “Will the application support a peak load of 5,000 transactions during Black Friday sales?”. But if we find from our KPIs that the peak response time for our application is 600 milliseconds, it still may not reflect what an end-user experiences. Here is where we have to evaluate the front-end performance metrics additionally. That is what we will see in this section.

Factors Affecting Front-End Performance

There are several factors that contribute to front-end performance. As such, we need to consider these factors when we test for performance metrics.

Front-end code complexity

Best practices such as minifying the Javascript, reducing the number of HTTP requests made per page, proper caching techniques, etc., which, when not followed properly, will lead to lower performance. For instance, each HTTP request takes at least a few milliseconds to complete and when there are many, the time for each request adds up to the total page load time.

Content Delivery Networks (CDNs)

A Content Delivery Network (CDN) is a collection of servers hosted across multiple locations to deliver web content, such as images, to users more efficiently. As we discussed earlier, the geolocation of the server and the user has an impact on the network latency. So usually, the images are stored in CDNs which will have a server that is closer to the user to reduce the network latency. This is much simpler than replicating the application in different geolocations. But the performance of the CDN itself will affect the page load time.

DNS lookups

It typically takes 20–120 milliseconds for a browser to look up the IP address for a given hostname. The browser caches the IP addresses once resolved for subsequent calls. While trying to resolve the DNS, even the Internet Service Provider caches the IP addresses for a while, improving the user’s experience from the second time.

Network latency

The user’s internet bandwidth will call the shots on the overall page load time. The data suggests that the end-users prefer mobile usage over desktop across the globe and the mobile bandwidth tends to be very low at times in different urban and rural areas. Some sites overcome this by serving a ‘lite’ version of their website when the user’s network bandwidth is identified to be very low. On the other hand, it is studied that the users who usually operate with low bandwidth like 3G are used to the slowness and don’t complain unless the performance is jarringly bad.

Browser caching

The browser caches many contents like images, cookies, IP addresses, etc., after the first visit. Due to this, the page load time significantly varies from the first time it renders to the subsequent usages. Caching of elements on the page can be made intentional via code to improve page load time.

Data transfers

If there are volumes of data transfer to and fro between the user and the application, that will affect the overall front-end performance, with the network’s effects adding on.

All these factors could make us feel like they are beyond the team’s control to even think about optimizing them, leaving a puzzle in our heads about where to start! Many folks in the software industry have also felt the pain of dealing with this challenge. That’s where the RAIL model comes in.

RAIL Model

The RAIL model is a way to structure the thought process around front-end performance.8

It is designed to keep the user’s experience at the core of your software development. It defines specific front-end performance goals as well. It is suitable for us to see the front-end performance through this lens and integrate the goals as part of our testing.

The RAIL model breaks down the user’s experience with a website into four key actions: response, animation, idle, and load. All user interactions can be measured according to these aspects.

Response

Have you ever clicked on a button, but it neither changed color nor popped a loading icon, making you wonder if you imagined clicking that button in the first place? That is the input latency. The ‘response’ aspect of RAIL defines the goals for input latency. When a user does some action, RAIL prescribes the website’s response time to the action to be less than 100 milliseconds; failing which the user will sense the lag! This is applicable for all user actions like clicking a button, toggling an element, selecting a checkbox, etc.

Animation

Animation includes visual animations like loading indicators, scrolling, drag, and drop, etc. The user will perceive a lag in animation when each frame is not completed within 16 milliseconds.

Idle

One of the general front-end design patterns is to group non-critical tasks like beaconing back analytics data, bootstrapping a comments box, etc., to later when the browser is idle. This grouping of tasks can be bundled to make blocks of 50 milliseconds, as recommended by RAIL and not more so that when the user comes back to interact, we can respond within the 100-milliseconds window.

Load

This refers to the page loading time. A high-performing website should aim to render the page within 1 second; only then the user feels they are in complete control of the navigation, as per the research mentioned earlier.

The RAIL model guides us to think about what to test for from a usability perspective. It also gives a concrete language to communicate within teams instead of expressing vague feelings like the ‘page seems slow’!

Front-End Performance Metrics

The RAIL model gives high-level goals for different front-end aspects. These high-level goals, in practice, are broken down into smaller, tinier metrics that will help debug issues. A few standard metrics are as follows

First Contentful Paint

It is the time taken by the browser to render the first element of DOM content like images, non-white elements, SVGs, etc. after a user navigates to the page. This helps us understand how long the user has to wait to see some action on the website after opening it.

Time to Interactive

It is the time taken for the page to become interactive. In the urge to make the page performant, the elements are made visible quickly, but they won’t respond to user actions resulting in customer dissatisfaction. Hence just like measuring the time to see the first content on the website, the ‘time to interactive’ helps us understand if the information presented is helpful or just noise.

Largest Contentful Paint:

The time taken for the most prominent element like a big blob of text or image on the web page to become visible.

Cumulative Layout Shift

This is the measure of visual stability, i.e., it helps quantify how often the user will face an unexpected change in page layout. Have you ever started reading an article and the page automatically shifted down, making you lose track of what you were reading? It is frustrating. And this metric measures precisely that. The lower the number, the better is the performance.

First Input Delay

Between the first contentful paint and time to interactive, when the user clicks on a link or makes any interaction on the web page, there will be a delay which is more than the usual delay because the page is still loading. This metric gives that time delay for the first interaction.

Max Potential First Input Delay

This is the worst-case scenario of the first input delay where it measures the time taken by the most prolonged task that occurs between the first contentful paint and time to interactive to complete.

Google classified Largest Contentful Paint, First Input Delay, and Cumulative Layout Shift as the ‘Core Web Vitals’ to help the business folks to understand their site performance in simple terms.9 Most of the front-end performance testing tools capture these three metrics specifically. We can use the tools to capture these metrics as part of CI and hence shift front-end performance testing to the left.

Front-End Performance TestingTools:

WebPageTest, Google Lighthouse, PageInsights, chrome dev tools, etc., are tools that enable us to measure front-end performance metrics. You might have to capture the above metrics for various test cases to understand how different users experience your website in terms of performance. Consider building your test cases along these lines:

- Users with different types of devices like desktop, mobile, tablet, etc., and which device manufacturers are significant players in the region you wish to serve. This is important because each device will have its CPU, battery, and memory capacity, which affects the user experience.

- Users with varying network bandwidths - WiFi, 3G, 4G, etc. The internet bandwidth is different in different countries too. A report published in 2021 on internet speeds says Singapore has an average broadband speed of 226.6 Mbps, Spain has 157.22 Mbps, Germany has 114.66Mbps, for example.10

- Target users distribution based on geolocation.

Plenty of this usage data is available on the internet only a click away. Alternatively, Google Analytics will give the site’s real-time usage analytics if there is an existing application.

I suggest you jot down a list of test cases specific to your application to capture the front-end performance metrics. Here is an example test case for which we can measure the metrics using different tools - ‘A user from Milan, who has a Samsung Galaxy S5, is accessing the site home page using 4G network connection’.

WebPageTest

WebPageTest is a free online tool publicly hosted for everyone around the world to assess their website performance. It is a powerful tool since it evaluates the website performance on real browsers hosted on machines across geolocations. It also can assess performance on their real mobile browsers. The tool can’t get closer than this to replicate a real end-user behavior.

The usage is simple. Enter your application URL on the WebPageTest site, choose the end-user location, browser type, mobile device type, network bandwidth, run the audit, and view the reports with metrics. Since it’s a free publicly available tool, you may have to wait in the queue for a few minutes to view the report. To avoid the wait, you can choose to set it up privately in a local test environment for a fee.

See Figure 3-14, where we have configured our example test case - ‘A user from Milan who has a Samsung Galaxy S5 accessing the Amazon site over 4G’.

Figure 3-14. WebPagetest configuration

We have provided two more inputs for our test case, as seen in Figure 3-14,

Number of tests to Run

As results from just one trial may be faulty due to glitches in network bandwidth, we request to run the test case a few times to observe the average.

Repeat View

To capture performance metrics separately for the first visit and subsequent visits since the metrics could vary for the first visit due to lack of caching.

Click ‘Start Test,’ and the report will be presented shortly. The report has many valuable sections that enable detailed debugging. Each report can be retrieved using a unique ID for 30 days. Let’s discuss a couple of important sections which capture the performance metrics from our report.

Performance metrics table

The performance metrics table (See Figure 3-15) has the Core Web Vitals for the first view and repeat view for all test runs. To benchmark the page load time for this test case, we can take the median of document complete time from all the runs. Specifically, we can notice the ‘Time’ under the ‘Document Complete’ is 3.134s and ‘Largest Contentful Paint’ is 2.105s for the first-time view, which tells us that the user experience is within acceptable limits. The fully loaded time includes the time taken to load all the secondary content, i.e., deferred tasks by the load event, which is substantial ~14s with 230 requests and will not certainly affect user experience.

Figure 3-15. Performance metrics table from WebPagetest report

Waterfall view

Figure 3-16. WaterFall view from WebPagetest report

The Waterfall view, like in Figure 3-16, shows a beautiful timeline view of how long each task, like DNS resolution, connection initiation, downloading Html, images, run time for scripts, etc., takes to enable further optimization.

You can change the view of the report to show the different domains accessed by the website, which is helpful to trigger optimization ideas further.

WebPageTest provides facilities to pass authentication credentials too. Note that any test credentials you provide will be visible to whoever has access to the report as it is publicly hosted. WebPageTest also exposes its APIs to get the reports programmatically. There is a node module variant to run the tests directly from the command line too. These two options enable integration with the CI pipelines. Both of these options require an API Key which is to be bought for a fee. If you have decided to purchase it, refer to Example 9-2 and Example 9-3 for CLI commands and API usage, respectively.

Example 3-5. Example 9-2. WebPageTest CLI commands to install , run test cases, and view results

//Step 1: Install using npm

npm install webpagetest -g

//Step 2: Run our sample scenario via command line

webpagetest test http://www.example.com --key API_KEY --location

ec2-eu-south-1:Chrome --connectivity 4G --device Samsung Galaxy S5 --runs 3 --first --video --label

"Using WebPageTest" --timeline

//Step 3: Reading test results from the report ID 2345678 generated from the above command

webpagetest results 2345678

Example 3-6. Example 9-3. APIs run WebPageTest test cases and view results

//Step 1: Run our sample scenario via API

http://www.webpagetest.org/runtest.php?url=http%3A%2F%2Fwww.example.com&k=API_KEY&location=ec2-eu-south-1%3AChrome&connectivity=4G&runs=3&fvonly=1&video=1&label=Using%20WebPagetest&timeline=1&f=json

//Step 2: Reading test results from report ID 2345678, returned as the response by the above API

http://www.webpagetest.org/jsonResult.php?test=2345678

Lighthouse

Lighthouse comes as part of Google Chrome and it is also available as a Firefox extension. It audits your website from multiple dimensions, including security, accessibility, and front-end performance. The performance audit report has an overall score and all the page metrics.

One of the advantages of Lighthouse is that it is not publicly hosted and hence no queuing or wait time. Since it is in your browser, there are no security concerns as well. As you are accessing the application on your local Chrome browser, the geolocation of the end-user cannot be altered. You can still throttle your network and CPU and resize to mobile browser resolutions in Chrome to simulate different test cases and obtain metrics using Lighthouse.

It is also available as a CLI tool, thus making it easier to integrate with CI for continuous performance feedback. Zalando, a leading European retail chain, had stated that they could reduce their front-end performance feedback time from 1 day to 15 minutes with Lighthouse CI.11 The tool is entirely free and open-sourced.

Lighthouse is simple to use. If you want to try,

1. Open Chrome Dev tools using shortcut Cmd + Option + J on Mac. Or choose Inspect option from Chrome right-click menu and you will see Lighthouse in other OS.

2. Choose your network throttling preferences under the ‘Network’ tab.

3. Choose your CPU throttling preferences under the ‘Performance’ tab. Default options are 4x, 6x for middle and low-tier mobile devices.

4. Choose your window size from the ‘Responsive’ dropdown

5. Now open the ‘LightHouse’ tab. Select the Performance checkbox and ‘Generate Report.’

In Figure 3-17, I chose ‘Slow 3G’, ‘4x slowdown’, ‘Galaxy S5’ options to measure performance metrics for Amazon.

Figure 3-17. LightHouse Window with Network, CPU, Resolution configurations

As seen in Figure 3-18, the results tell us that Amazon does a pretty good job. The ‘Time to interactive’ in such skewed situations is still 3.8s!

Figure 3-18. LightHouse Performance report

We can use Lighthouse to test various test cases as early as during development itself. To integrate this with CI, we can use the LightHouse node module. Install node module using the command,

npm install -g lighthouse

To run a performance audit,

lighthouse https://www.example.com/ --only-categories=performance

We can see the reports in the current directory. We can use optional parameters to configure the network and CPU throttling and choose device screen sizes.

We can write a wrapper to fail the pipeline if the performance score is less than a threshold; for instance, fail the build if the score is less than 90! We can also define performance budgets (upper threshold value) for each of the web vitals using the Lighthouse Wallet feature.12 This will assert the Lighthouse’s performance results against the defined threshold values for each metric and raise alerts when it is overboard. Cypress-Audit tool enables integration of Lighthouse with Cypress functional test suite, which is another way to plug them into CI.

PageInsights

The previously discussed tools allow us to simulate test cases like in a lab, where we set pre-conditions and observe results. But there are many variations in the pre-conditions in real life as every user has minor differences in network bandwidth, device configurations, etc., which we can’t predict. The only way to know our users’ experience is to do Real-time User Monitoring (RUM) after the application has gone live. Google provides free monitoring services which record the core web vitals and other metrics as and when the users worldwide access the live applications. This data is called the ‘Field data’ or ‘RUM data.’

PageInsights tool tries to give a holistic view of front-end performance by presenting the RUM/field data from across the globe and the lab data by fetching the audit report from Lighthouse, as seen in Figure 3-19. Try entering your live application URL on the PageInsights homepage to see this.

Figure 3-19. PageInsights field data and lab data report

PageInsights also exposes APIs to monitor and alert constantly.

Chrome DevTools

Another handy tool for front-end performance debugging is the performance profiler available under the ‘Performance’ tab in Chrome DevTools. It gives detailed analysis reports around network stack, frames per second, screenshots, GPU consumption, memory, script run time, etc., for the developers to save the extra milliseconds lost in performance. Since it is embedded in the browser itself, it is development-friendly. The profiler allows for throttling the network and CPU while debugging.

Figure 3-20. A sample report from the performance profiler of Chrome dev tools

Here is how it works: Let’s say that you want to find out how the auto-populate drop-down of your application performs on the UI. First, we’d record the action while manually searching something on the drop-down. The record option is available in the ‘Performance’ tab of the browser. Then, once the recording is stopped, the performance analysis reports are shown in the same tab, like in Figure 3-20.

Performance Testing Strategy

You are now equipped with the required knowledge to conduct end-to-end performance testing for your application successfully. The last part is putting them all together to form your performance testing strategy. Like all other testing, shift-left should be your performance testing strategy as well. Shifting left should start from designing the architecture based on performance numbers to integrating performance tests in CI pipelines for frequent and continuous feedback. Recall that this will not only be useful for the business and the application’s end-users but will also save your weekends. Throughout the chapter, we have learned how to do shift-left performance testing with various tools. Figure 3-21 shows an overview of shift-left performance testing.

Figure 3-21. Shift Left Performance Testing

To shift the performance testing to the left:

- Arrive at a consensus on performance KPIs with all the application stakeholders from the business, marketing, and technical side before the project starts. Include these numbers as acceptance criteria in stories.

- Try to get a performance testing environment at the beginning of the project. If not close enough to production, at least have an environment to begin.

- Include response time validations and if possible, load tests of critical endpoints as part of CI pipelines to get frequent feedback and catch performance issues early. This will also help us see how the degradation happens gradually with additional features and will help in debugging. Based on the time taken to run the load tests, integrate them as part of every commit, if not at least part of nightly regressions.

- Include front-end performance tests for the frequently visited pages as part of CI pipelines, just like web services.

- Include various front-end performance test cases (like network, geolocation, etc.) as part of every story acceptance criteria. Validate them during the development and manual testing of stories and features.

- Plan for application performance testing and debugging activities as part of release cycles. Perform release-level performance testing on a proper performance testing environment with an appropriate load of test data. If the previous list items are executed accurately, this phase should run the tests on the performance testing environment without many surprises.

Performance testing is not complex but takes significant effort!

Key Takeaways

- Web performance has a steep impact on the sales and profit of the business.

- Factors affecting the performance of an application could be many ranging from application architecture, third-party service performance, network bandwidth, user geolocation, etc.

- Availability, concurrency/throughput, and response time are three crucial performance KPIs.

- Use tools like JMeter, Gatling, or Apache Benchmark, etc., that are suitable for your team to measure the KPIs early and continuously.

- Front-end performance testing is essential as 80% of the application’s page load time is from the front-end code.

- Google developed the RAIL model as a thought process for front-end performance.

- Design your front-end performance test cases with the end-user experience at the core and include different end-user variables like network bandwidth, geolocation, device capabilities to test for user experience.

- Shift your performance testing to the left and save yourselves from big-bang performance surprises when you have to release tomorrow!

- Like any other skill, performance testing develops with practice and seeing through many real-time edge cases.

1   Daniel An: New industry benchmarks for mobile page speed, thinkwithgoogle.com, February 2018.

2   Maile Ohye: Site performance for Webmasters, youtube.com, May 2012.

3   Diego Lo Giudice: Shift Left Performance Testing, go.forrester.com, April 2019.

4   Ian Molyneaux: The Art of Application Performance Testing, 2nd Edition, O’Reilly Media Inc., 2014.

5   Jakob Nielsen: Website Response Times, www.nngroup.com, June 20, 2010.

6   Scott Barber: Get performance requirements right—think like a user White paper, perftestplus.com, 2007.

7   Steve Souders: High Performance Web Sites, O’Reilly Media Inc., 2007.

8   Paul Irish: RAIL Model, smashingmagazine, October 2, 2015.

9   Philip Walton: Core Web Vitals, web.dev, April 2020.

10   World Population Review: Internet Speeds By Country 2021, worldpopulationreview.com, 2021.

11   Jeremy Colin: Zalando reduced performance feedback time from 1 day to 15 minutes with Lighthouse CI, web.dev, April 2021.

12   Katie Hempenius: Performance Budgets with LightHouse Wallet, web.dev, June 14, 2019.

Chapter 4. Moving Beyond in Testing

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 13th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at jleonard@oreilly.com.

Practitioners follow directions; experts understand principles.

So far, I’ve covered all of the necessary testing skills an engineer should possess in order to successfully deliver high-quality software. We’ve established that testing is a broad and growing space that has evolved over decades to include new processes, tools, and methodologies. While there are 10 different lenses that testing skills can be viewed through today, tomorrow, there could be more. However, even in such a dynamic environment, the foundational principles in testing will remain unchanged irrespective of the technology or the domain. Understanding these first principles in testing will provide you with the framework and knowledge you need to succeed regardless of how the testing space continues to grow in the future.

In this chapter, I would like to brief on these first principles in testing, their critical benefits, and look at how the existing tools and team practices have evolved based on these principles. We will also look at how an individual’s soft skills add to their technical skills in contributing to their team’s overall success in delivering high-quality software.

First Principles in Testing

Figure 4-1 shows the seven first principles in testing. Let’s deep dive into them now!

Figure 4-1. First-principles in testing

Defect Prevention Over Defect Detection

Even though testing mainly intends to find issues in the application, this principle should be considered the core of testing. An apparent reason why we should avoid defects is the cost of fixing them. We can compare defect fixes to painting a broken patch in an otherwise seamlessly painted wall — sometimes, the newly painted patch doesn’t fit well, and we have to paint the whole wall all over again! Similarly, software defects could lead to significant architectural changes and that is a huge rework and cost. Hence, this core principle suggests adopting practices, tools, and methods that will allow defects prevention rather than defect detection.

Some practices in today’s software world that aim to fulfill this principle are the Three Amigos process,1 Iteration Planning Meetings (IPMs), Story Kick-offs, Architecture Decision Records (ADRs), Shift Left Testing, Test-Driven Development (TDD), and so on. The three amigos process is where the business analysts, developers, and testers mull out the feature thoroughly during the analysis phase. The process aims to collect all three roles’ perspectives so that integrations, edge cases, business requirements in the features are not missed out. Similarly, the story kickoff aims to repeat the same process just before the story development begins. It is also a common practice in shift-left testing for the testers to capture and discuss the test cases during story kickoffs.

ADRs and test strategies are discussed and documented to align the team towards the project’s end goal. IPMs are conducted at the beginning of an iteration/sprint to discuss the stories in detail. It is an open space for teams to brainstorm missing integrations and edge cases in stories. During development, TDD helps in thinking of all the edge cases for a small piece of code. Pair programming is another practice aimed to prevent bad code and not to miss edge cases during development. Similarly, linting software catches bad coding practices even while creating them.

As we can see, there are so many practices that focus on defect prevention. This could be applied to any new domain, like data, even when roles in the team are different.

Empathetic Testing

You may be working with your team day in and out. But when you are wearing the tester’s hat, your closest friend should be the application’s end-user. Testing is all about placing yourself into the role of end-users. While we can get completely taken away by the business needs and technical implementation details, we should operate with the end-user’s interest at heart while testing. We cannot restrict ourselves to verify the story’s acceptance criteria and move on simply. We have to explore the application, like how a typical end-user would navigate the application. As a result, understanding, and empathizing with the different end user personas we are targeting our application to is critical before starting to test. Often, the teams tend to make trade-offs on end-user needs against factors like the complexity of development and timelines. However, the role of a tester is to bring the end-user perspective to the forefront and negotiate the trade-offs.

Micro & Macro Level Testing

As discussed in Chapter 1, testing must be entwined at both micro and macro levels to deliver high-quality software. To quickly recall, micro-level testing is to dive closer to a small piece of functionality and test that in detail — for example, testing a small functionality such as the order total calculation with various boundary conditions. Macro-level testing is to use a broader lens to cover the functional flows, data propagation between modules, integration between components, and so on. For example, testing the order creation flow by testing the database, third-party integrations, UI flows, failures in order creation, and so on.

We saw an elaborate automated functional testing strategy with various types of micro and macro-level tests in Chapter 3. In summary, unit, integration, and contract tests focus on micro-level testing, while API tests, UI functional tests, visual tests, and so on focus on macro-level testing. We also discussed how an imbalance in such distribution of micro and macro-level testing would delay feedback, leading to a delay in delivery.

Another critical consequence of such an imbalance will be unexpected issues in production. This is because, in general, teams that focus only on macro-level testing discards details. For example, they would have tested macro-level scenarios such as the order getting created successfully, the order creation failing when there is no item availability, and so on. But when the item prices in production are negative or have an unexpected number of decimals, order creation would fail as they did not focus on micro-level testing. So, zooming in and out constantly on the micro and macro details while testing is crucial.

Fast Feedback

This principle is about finding defects early so that the defect fixing cycle and, as a result, the release cycle can be faster. Defects tend to become costlier when they are found later in the delivery cycle. For instance, imagine a high-priority bug found two weeks post the feature development. First, there is the time and effort in creating bug cards, triaging them, tracking them in iterations, and finding the right developer’s time to fix them. Second, in some worst-case scenarios, we will find out that it is impossible to fix the defect without a major refactoring, resulting in a delay in release. This is the ultimate cost to pay for a defect!

There is another notable correlation between the time taken to fix a defect and how late it is found. When a feature is in development, the developer has all the required context on the code, can easily understand bugs’ root causes, and quickly fix them. Now, when the developer moves on to other features and the codebase continues to grow every day with refactoring, context is lost, and debugging the root cause becomes a longer and costlier process. As a result, we need faster feedback.

How early can we test a piece of code to give fast feedback? Shift left testing is all about faster feedback, and we have seen how to implement this in each of the earlier chapters. Some of the team practices that yield faster feedback, as discussed in Chapter 2, Chapter 3, and Chapter 4, are dev-box testing, automated tests run on the developer’s machine, and CI, coverage metrics, and implementing the test pyramid, which primarily suggests a way to reduce the test run time. Also, story sign-offs by product owners (PO) and showcases after every sprint completion to all stakeholders will get the team faster feedback on missing business cases.

Testing to give faster feedback is equivalent to harvesting at the right time. When the timing is delayed, we have to settle for a lower-quality harvest.

Continuous Feedback

Fast feedback should be backed by continuous feedback. It is not enough to just test a feature once and then leave it idle until release. We have to continue performing regression testing on the feature to see if the integrations are still intact and the refactorings have not hampered the existing functionalities. Using these continuous feedback mechanisms helps in fixing regression defects early and prevents disruption of release timelines.

One of the prominent ways teams achieve this is by integrating all the automated tests to the CI, the implementation of which we discussed in Chapter 4. As we saw, the ideal practice is to run all the tests for every commit. If the tests take too long to run, we have to adopt parallelization techniques. Some teams also take the nightly regression route to get feedback at least once daily. But following the test pyramid strategy as discussed in Chapter 3 should ideally curb the need to separate the tests as smoke and nightly regression, allowing us to get continuous feedback for every commit.

Another team practice that yields continuous feedback is the bug bash. Bug-Bash is a type of mob testing — a practice where a diverse set of individuals try the application and provide feedback. A bug bash typically has the team members and extended stakeholders participate in it to find regression issues. Bug-bashes can be scheduled regularly, for example, once a month, to get such diverse feedback continuously.

Measure Quality Metrics

Anything that is measured tends to improve! The whole purpose of having KPIs in any field is so that we track the indicators and iteratively improve them by taking the right measures. So when we are trying to achieve high-quality results, we should measure quality as well. Some of the quality metrics that will help when tracked regularly are the following:

Defects caught by automated tests in all layers

Automated tests create a safety net for the team, and when a majority of defects are found in the early stages, teams feel more confident while making new changes. It also reflects the strength of the safety net.

Time taken from commit to deployment

Like we saw earlier, faster feedback is critical to making progress. When the developer commits, the new changes should be quickly tested by the automated tests in the CI pipelines and deployed to the QA environment for kickstarting exploratory testing. I have seen teams where the CI pipelines take a long time to generate a green build due to unstable tests and environment issues, delaying feedback, and resulting in productivity loss.

Number of automated deployments to testing environments

This and the previous parameter will show the tempo of the team in making new changes. Ideally, we need the team to be set up with a good safety net that will yield faster and stable deployments. So if you find out that the number of automated deployments to the testing environments is very less due to multiple infrastructure, tests, or other failures, it shows that your feedback cycle is hampered.

Regression defects caught during story testing

Regression defects caught during the story testing phase indicate a missed business use case or a missing automated test. For example, a refactoring of the SQL query to use ‘equals’ instead of ‘like’ will be missed by the automated tests in the CI if the input data was designed as such. We discussed in Chapter 3 that when such regression defects are found during story testing, it can be a symptom of teams following antipatterns in automated testing. Hence, teams should immediately reflect on the root causes of such regression defects and improve their processes regularly.

Automation coverage based on the severity of test cases

Keep a detailed record of your automation coverage with the goal of having no backlog. Tracking this will help plan your iterations in advance to cover the backlog if any.

Production defects and their severity

Tracking production defects shows us the bigger picture of missing business use cases, missing configuration, data mismatch, or anything else that the team may have overlooked. Identify their root cause and automate tests around them. Also, build a living test strategy and keep evolving it as the application and team venture into new grounds.

Usability scores with end-users

Collect feedback on the overall user experience during the development phase itself with end-users. This will help determine design metrics for your application, like the number of clicks to obtain any information, text over icons, and more.

Failures due to infrastructure issues

Track the infrastructure issues like services being down intermittently in testing environments, CI pipelines issues, mismatch in testing and dev environments’ configurations, etc. Sometimes, the infrastructure code may need the technical debt card to make it scalable and stable.

Metrics around cross-functional aspects

Measure the performance, security, accessibility, and other cross-functional metrics discussed in previous chapters and report them as part of quality metrics.

Many of the above metrics tie back to the ‘4-key metrics’ from the book Accelerate,2 which measure quality in terms of stability of the code and delivery tempo of the team. For instance, one of the 4-key metrics, ‘lead time for change’ (the time to make a change in the application) for a high-quality performer, is expected to be ‘less than an hour.’ When there is a good safety net of automation coverage, the team can make changes confidently.

Similarly, the ‘deployment frequency’ metric needs to be ‘on demand’ for a high-quality performer. When we measure the time taken from commit to deployment and the number of deployments in a day to testing environments, we get the team’s delivery tempo. Production defects will tell us about the ‘change fail percentage’ (percentage of changes made to production that fail), which should be 0-15% for a high performer. When tracked and discussed consistently, these metrics let the team chase the goal of high-quality software.

Communication & Collaboration Is Key To Quality

Testing cannot be done as a siloed activity. We need proper communication about business requirements, domain knowledge, technical implementation, environment details, and so on to do appropriate testing. This requires consistent collaboration and good communication amongst all roles within a project team. Communication could be via agile ceremonies like stand-ups, story kick-offs, IPMs, dev box testing, and proper documentation like story cards, ADRs, test strategies, test coverage reports, etc. While we cannot expect the communication to be synchronous in today’s world with distributed teams, we should ensure to hand over things via asynchronous mediums like video recordings, documentation, emails, and other innovative ways.

To summarize, these seven principles should help the teams to drive their testing strategies even as they venture into new territories of the technology space! I have applied these principles in projects with new technology stacks and new domains and have observed them consistently yield high quality. We should be able to apply the same testing principles in emerging technologies like AI, Blockchain, IoT, and anything that pops up tomorrow as well to reap high quality.

Soft Skills Aid in Building Quality First Mindset

At this point, we have established enough that quality is multi-faceted at the end of the day, and testing contributes to it one-dimensionally. Every aspect in software development, starting from design, analysis, development, infrastructure, and so on, together, produce high-quality products. Quality can be achieved only when all team members work towards it. No one person can completely own quality, and neither can no single person not own quality—not dissimilar to how a relay team cannot win a race even if one runner slows.

To build such a quality-first mindset in the team, soft skills become extremely critical. If you are a tester by profession or someone who owns the testing perimeter at work, here is a list of soft skills I would like to throw light on to build a collaborative quality first mindset within teams.

Driving outcomes

A high-quality product being the key outcome for a team, you, the tester, primarily should drive the testing-related activities that will contribute to high-quality. It starts from the simple exercise of getting all the high-priority defects fixed before the code freeze to getting the automation backlog prioritized and pushing the release timelines if the quality is not great.

Collaboration

Inculcating the mindset, ‘quality is the team’s responsibility,’ can only happen through good collaboration with all team members and customers. If we are rigid in our ideas and apathetic towards team members, we will not achieve quality in tasks. For example, owning the test strategy collaboratively with the developers will go a long way in achieving the right quality.

Effective communication

Sometimes, the way we communicate is the difference between the task being completed or not, agreed? Effective communication also means choosing the suitable medium and the right time to communicate. The communication regarding the overall product quality and the risks of achieving it should continuously go out to the team.

Prioritization

Testing can become a never-ending activity if we don’t prioritize efficiently. As we gain experience, we will realize that testing activities must be balanced within the time available. We wouldn’t have time to test on all the six different browsers for every story. Similarly, we wouldn’t test with 100 different user profiles for testing a login functionality. We automatically learn to prioritize during testing. We should keep honing this skill to become efficient. Efficient prioritization will help plan the testing activities in a team holistically.

Stakehold er management

Your customers, managers, teammates, tech lead, and anyone who can change the course of actions of the project are your stakeholders. We have to manage the stakeholders’ expectations about quality consistently. Your customers can be expecting the automation coverage to be 100%, which may not be realistic. Sometimes, the managers might be interested in meeting the release timelines more than the quality. Managing all these expectations upfront through collaboration, effective communication, and planning will lead to collective success.

Coaching / Mentoring

It is a usual scenario in teams to have new members joining them. We cannot expect the new joiners to know all the team’s practices and tools. To achieve the goal, ‘quality is the team’s responsibility,’ everyone should be on the same page regarding the testing practices and tools. We should pair with them, conduct whiteboarding sessions, and help them in ways they can ramp up quickly.

Influence

Building influence becomes very important when you start working with large teams and new clients. You might lay out a wise test strategy, but the teams may not adopt them like you wish them to. Most of the time, it’s the influence that makes people believe in the test strategy. Similarly, for customers to invest in new testing tools and practices, they need an influential voice.

Testing is a continuous learning journey for all of us, and there are many testing tools, processes, and best practices that keep evolving day by day. The rapid growth may be overwhelming sometimes, and if it is, you should remember that they fundamentally fall under one of the first principles; learning how and where they fit in is only a small step away. Overall, when you focus on blending the full stack testing skills with your soft skills, you will find yourself sailing through a smooth lane in achieving your goal of delivering high-quality software!

Key Takeaways

Here are the key takeaways from this chapter:

- As you venture into new territories of the technology space, the first principles in testing will guide you to deliver high-quality software.

- We can observe the current set of testing best practices and tools abide by these first principles as well.

- Quality is multi-faceted while testing contributes to it one-dimensionally. Hence, we should build the team’s mindset to own quality as a collective responsibility.

- Your soft skills are critical in building the ‘quality as a collective responsibility’ mindset in teams.

- When you blend your full-stack testing skills with your soft skills, you can smoothly contribute to your team’s goal of delivering high-quality software!

As we have reached the end of the book, I would like to thank you for traveling this long path. It only shows your commitment to delivering high-quality software, which is truly commendable! I hope the book gave you the necessary learnings and insights that you can put to use at work immediately.

Until we meet again in our testing journey, all the very best, and thank you!

1   Agile Alliance: The three amigos process, agilealliance.com.

2   Gene Kim, Jez Humble, and Nicole Forsgren: Four Key metrics, Accelerate, IT Revolution Press, 2018.

About the Author

Gayathri Mohan is a passionate technology leader with expertise across multiple software development roles and technical and industrial domains. Gayathri has proven her mettle by successfully managing large quality assurance (QA) teams for clients at ThoughtWorks, where she is now Principal Consultant at Thoughtworks Chennai. While working as the company’s Global QA SME, she defined career pathways and the desired skill development structure for QAs at Thoughtworks. As Office Tech Principal, Gayathri cultivated local tech communities, organized technical events and developed thought leadership across technical themes.

Gayathri is also coauthor of Perspectives of Agile Software Testing, released by ThoughtWorks on Selenium’s 10th anniversary.

1. 1. Introduction to Full Stack Testing

1. Full Stack Testing for High Quality

2. Shift-Left Testing

3. Ten Full Stack Testing Skills

4. Key Takeaways

2. 2. Security Testing

1. Common Cyber Attacks

1. Web Scraping

2. Brute Force

3. Social Engineering

4. Phishing

5. Cross-site Scripting

6. Ransomware

7. Cookie Forging

8. Cryptojacking

2. Threats Framework

1. Spoofed Identity

2. Tampering with Inputs

3. Repudiation of Actions

4. Information Disclosure

5. Denial of Services

6. Escalation of Privileges

3. Application Vulnerabilities

1. Code or SQL Injection

2. Cross-Site Scripting (XSS)

3. Unhandled Known Vulnerabilities

4. Authentication and Session Mismanagement

5. Unencrypted Private Data

6. Application misconfigurations

7. Application Secrets Exposure

4. Threat Modeling

1. Threat Modeling Steps

2. Threat Modeling Exercise

3. Security Test Cases from the Threat Model

5. Security Testing Strategy

1. Static Application Security Testing (SAST)

2. Source Composition Analysis (SCA)

3. Image Scanning

4. Dynamic Application Security Testing (DAST)

5. Functional Security Tests Automation

6. Manual Exploratory Testing

7. Pen Testing

8. Runtime Application Self Protection (RASP)

6. Security Testing Tools

1. Snyk IDE plugin

2. Talisman Pre-commit Hook

3. OWASP Dependency-Check

4. OWASP ZAP

5. Chrome DevTools & Postman

7. Security as a habit

8. Key Takeaways

3. 3. Performance Testing

1. Performance, Sales, and Weekends Off Are Correlated!

2. Simple Performance Goals

3. Factors Affecting Application Performance

1. Architecture design

2. Choice of tech stack

3. Code Complexity

4. Database performance

5. Network Latency

6. GeoLocation of the application and users

7. Infrastructure

8. Third-party integrations

4. Key Performance Indicators

1. Response time

2. Concurrency/Throughput

3. Availability

5. Types of Performance Tests

1. Load/Volume tests

2. Stress tests

3. Soak Tests

6. Types of Load Patterns

1. Steady Ramp-Up Pattern

2. Step Ramp-Up Pattern

3. Peak-Rest Pattern

7. Performance Testing Steps

1. Define the Target KPIs

2. Define the Test Cases

3. Prepare the Performance Testing Environment

4. Prepare the Test Data

5. Integrate APM Tools

6. Script and Run the Performance Tests Using Tools

7. Performance Testing Exercise

8. Performance Testing Tools

1. JMeter

2. Gatling

3. Apache Benchmark:

9. Front-End Performance Testing

1. Factors Affecting Front-End Performance

2. RAIL Model

3. Front-End Performance Metrics

4. Front-End Performance TestingTools:

10. Performance Testing Strategy

11. Key Takeaways

4. 4. Moving Beyond in Testing

1. First Principles in Testing

1. Defect Prevention Over Defect Detection

2. Empathetic Testing

3. Micro & Macro Level Testing

4. Fast Feedback

5. Continuous Feedback

6. Measure Quality Metrics

7. Communication & Collaboration Is Key To Quality

2. Soft Skills Aid in Building Quality First Mindset

3. Key Takeaways

Table of Contents

1. Introduction to Full Stack Testing

Full Stack Testing for High Quality

Shift-Left Testing

Ten Full Stack Testing Skills

Key Takeaways

2. Security Testing

Common Cyber Attacks

Web Scraping

Brute Force

Social Engineering

Phishing

Cross-site Scripting

Ransomware

Cookie Forging

Cryptojacking

Threats Framework

Spoofed Identity

Tampering with Inputs

Repudiation of Actions

Information Disclosure

Denial of Services

Escalation of Privileges

Application Vulnerabilities

Code or SQL Injection

Cross-Site Scripting (XSS)

Unhandled Known Vulnerabilities

Authentication and Session Mismanagement

Unencrypted Private Data

Application misconfigurations

Application Secrets Exposure

Threat Modeling

Threat Modeling Steps

Threat Modeling Exercise

Security Test Cases from the Threat Model

Security Testing Strategy

Static Application Security Testing (SAST)

Source Composition Analysis (SCA)

Image Scanning

Dynamic Application Security Testing (DAST)

Functional Security Tests Automation

Manual Exploratory Testing

Pen Testing

Runtime Application Self Protection (RASP)

Security Testing Tools

Snyk IDE plugin

Talisman Pre-commit Hook

OWASP Dependency-Check

OWASP ZAP

Chrome DevTools & Postman

Security as a habit

Key Takeaways

3. Performance Testing

Performance, Sales, and Weekends Off Are Correlated!

Simple Performance Goals

Factors Affecting Application Performance

Architecture design

Choice of tech stack

Code Complexity

Database performance

Network Latency

GeoLocation of the application and users

Infrastructure

Third-party integrations

Key Performance Indicators

Response time

Concurrency/Throughput

Availability

Types of Performance Tests

Load/Volume tests

Stress tests

Soak Tests

Types of Load Patterns

Steady Ramp-Up Pattern

Step Ramp-Up Pattern

Peak-Rest Pattern

Performance Testing Steps

Define the Target KPIs

Define the Test Cases

Prepare the Performance Testing Environment

Prepare the Test Data

Integrate APM Tools

Script and Run the Performance Tests Using Tools

Performance Testing Exercise

Performance Testing Tools

JMeter

Gatling

Apache Benchmark:

Front-End Performance Testing

Factors Affecting Front-End Performance

RAIL Model

Front-End Performance Metrics

Front-End Performance TestingTools:

Performance Testing Strategy

Key Takeaways

4. Moving Beyond in Testing

First Principles in Testing

Defect Prevention Over Defect Detection

Empathetic Testing

Micro & Macro Level Testing

Fast Feedback

Continuous Feedback

Measure Quality Metrics

Communication & Collaboration Is Key To Quality

Soft Skills Aid in Building Quality First Mindset

Key Takeaways