Mastering Test Data Management: Unleashing the Potential of Test Data in Test Automation Frameworks — Part 1: Test Data Pipelines

Test Data is one of the most common dilemmas in the Test Automation puzzle. It remains a neglected component within the early steps of developing and selecting the Test Automation Framework. While test data plays a significant role in uncovering issues, it can also lead to some failure and bugs under certain circumstances when it’s not covering happy, negative, and edge scenarios. Statistics highlight how test data can cause failure in Software Systems. In a case study published by the SEI (Software Engineering Institute), a software system failure occurred due to incomplete test data that didn’t adequately represent real-world usage scenarios. As a result, critical bugs went unnoticed until the system was already deployed, leading to major failures.

Test Data is the contextual components

Building a Test Automation Framework is not an easy task. It may require some effort to ensure flexibility, extendability, and well-structured layers for the framework. Moreover, Building a robust Test Data workflow for generating, maintaining, and tailoring test data is crucial nowadays in Test Automation Framework for increasing coverage and identifying system state at any moment.

Now let’s dig deep into key challenges and considerations in managing test data in Automation Frameworks.

Frequently most of the Test Automation Frameworks — and as recommended — are being built at the early stages which leads to insufficient test data for test execution. Generating Test Data is one of the most commonly used techniques for enriching the amount of Test Data which will be used for boosting Test Coverage. There are different ways for generating test data and how it can be generated

1. Dummy Test Data

Generating random Test Data is one of the strategies that can be used to simulate Monkey Testing. It provides raw random test data to fill some fields or select random values in AUT (Application Under Test) forms. Faker Library is one of the most commonly used libraries to generate random values and types of test data. Although it can be used directly in Test classes or suites, this strategy only works when no validation is required therefore random test data will be efficient.

2. Generating valid Test Data

Whenever a system has some rules and validations — which is almost all the time — it needs valid and invalid test data so that test methods/keywords can be used for the execution. In this manner, Software Automation Engineers can follow different approaches:

• Using constant Test Data: As easy as it sounds, it will cause dozens of problems such as flaky tests, a pre-manual action needed to let the scenario act normally, inadequate test coverage, or false positive results.
• Generate Test Data on the fly: While the tests are being executed, some test data will be generated so that it can be used. One of the most important matters is to tailor it enough to fit for test case either by connecting to datasources to insert new test data or creating formulas for creating test data on the fly. However, it’s better than constant or dummy test data, it takes more time to generate test data in the execution phase. This may lead to false insights about execution time and system behavior especially when connecting to datasources is included.

3. Fetching Test Data from datasources

One of the most commonly used strategies is to fetch data from datasources like databases, excel sheets,..etc. Software Automation engineers are involved in developing some SQL queries to fetch test data from the database. It’s very helpful by adding conditions for getting valid test data from the database and ensuring test data is suitable for tests. As accurate as it looks, it has multiple cons, like the dependency on database status — is it up or down — also it needs a lot of time to connect to databases, fetch data then close the connection. It may lead to bad performances in test execution while starting and closing database connections. Imagine the case when hundreds of thousands of test cases have to get test data and be used for execution, looks like a mess, Doesn’t it?

Introducing Test Data Pipeline

A Test Data pipeline is one of the most accurate and independent ways of generating and maintaining test data. It ensures high-quality test data using full control before, while, and after executing Test cases in Test Automation Frameworks.

Test Data Pipelines can be scheduled too.

Now, let’s list the most recommended steps for creating a test data pipeline to enrich the test automation Framework with high-quality test data:

1. Add rules for Test Data — Be decisive with Data

All test data especially when it comes from datasources, can contain some inaccurate, dummy, or misleading test data. Therefore, it’s crucial to add general rules to make sure that all test data is validated properly. This can be done by filtering and segmenting test data. Purifying test data can be achievable in many ways:

• Create a cloning database from the App’s database for Test Automation purposes only
• Specify general rules for selecting data from the database
• Divide test data according to general scenarios

However, all the above-mentioned approaches can lead to successful results, so it’s recommended to keep the original data — in Stage or Test Environments — clean. This can be achieved by cleansing test data from one time to another.

So cleansing test data is a very recommended step using some scripts that can filter and segment test data. Some important rules to apply to test data can include the following:

• Fresh test data; which can be assured by the data creation date
• Valid test data; which can be assured by some conditions and isolating test data to a table — or isolated component— which assures test data validity

2. Test Data Characterizing — Data Labeling

Data Labeling leads to better coverage

For high test coverage sake, it’s very important to label test data. As most scenarios and cases have many aspects, some scenarios need different test data, thus test data will need to be labeled according to the scenario. This can be achieved by adding some flags to the database or marking test data to be suitable for some scenarios. This step should be done previously while preparing test data for test methods and keywords in the Test Automation Framework.

3. Test Case Parameterization — Add Test Data Provider

Allowing test cases to be executed through different test data is very important and has to be taken into consideration while building a test automation framework. This prevents test cases from being repeated over different test data, which saves time for scripting in test automation as well as increasing test coverage. DataProvider Annotation in Java can be used to feed test methods with different test data. On the other hand, Robot Framework has a Test Template which does pretty much the same.

4. Acceleration of Test Data fetching

Before the test starts to execute, it will get test data before execution therefore this may lead to a load of time for fetching test data to each test case. Test Cases have to read this data faster in an accelerating manner. JSON files are one of the good examples to read test data faster and feed methods/keywords with test data. Instead of opening and closing connections with the database, getting test data from JSON files is much faster and rarely causes I/O Issues if best practices follow; such as `with` in Python or try-with-resources in Java.

5. Test Data Isolation — Prepare Test Data

Quarantining the preparation of Test Data a step away from Test execution is very important to ensure accurate insights about test execution time and performance. Pipeline jobs can do this magic, by creating some scripts to be executed at nightly builds before test execution.

6. Cleansing Test Data — Maintenance and Control

Maintaining and controlling Test Data is very important to make sure that tests will be fed by suitable Test Data. Controlling test data by dropping invalid values — or not anymore valid — and enriching test data with suitable new ones is very important to ensure test data still fitting. Test Data can be controlled by applying the same rules in the first two steps to keep high-quality data.

7. Monitoring Test Data — Set Alerts

Logs are very helpful for Test Data Monitoring and Performance Analysis. It allows us to track the behavior of the test data pipeline, identify database bottlenecks, and analyze resources as well as its insights about test data. Adding logs for pipeline jobs can be very helpful for corrective and proactive actions whenever new behavior is introduced or some bugs produce unsuitable test data.

In conclusion, effective test data management is a critical aspect of a robust and efficient test automation framework. By implementing a Test Data Pipeline, organizations can overcome data quality and availability challenges and maximize the value of test automation coverage. In the ever-evolving landscape of software development and testing, test data management remains an ongoing process that requires continuous evaluation, monitoring, and adaptation.

In the next two Parts, we will cover Test Data Pipeline Implementation using :

• Part 2: Test Data Pipeline — Using Robot Framework
• Part 3: Test Data Pipeline — Using Java

Stay Tuned!