In 2018 on Amazon Prime Day, one of the company’s biggest sale days, the Amazon website went down several times, and at a certain point had to be taken down for maintenance. This was due to a massive amount of site traffic, which Amazon lacked the resources and capacity to support. Amazon confiscated all spot instances that were being used by other AWS customers and used backup data centers to handle the load. The company’s financial losses from the downtime were estimated at $1 billion, and the effect on Amazon’s reputation was significant. Yet this incident could have been completely prevented if Amazon had conducted proper tests.

Performance testing—generating higher-than-expected traffic on servers for an extended period of time—along with monitoring, give R&D perspective about how a system functions under stress or transaction spikes. But performance is only one piece of the quality assurance agenda. Functionality tests, security tests, and other validations are an integral part of every agile-oriented CI/CD process. Today, almost every R&D organization implements tests and verifications inside the build process and executes automatic tests for every successful deployment.

This article will explore how tests can be integrated into the CI/CD process and how Kaholo easily assimilates testing and validation steps in different scenarios.

The Challenges of Testing

Testing can sometimes seem endless. Quality assurance engineers can spend days planning and executing tests for a given feature and still find there are additional test types and scenarios that need to be covered. Even for an experienced QA engineer, 100% coverage is an unattainable objective. Still, most QA engineers will agree that if a feature passes the main types of tests —functionality, performance, security, and usability—this is enough for approving its release to customers.

As the Agile Manifesto grew and became cannon for R&D organizations, code releases to production became more rapid, and quality assurance had to be performed on shorter cycles and, more importantly, faster. Automation was introduced for the different testing types—on both the API and user interface levels. Every component can and should be tested as part of the build step, and then in an integration environment.

A test of the basic functionality of the service—the happy path test—can be performed before the service is released to QA, and the integration test suite should be executed on a working environment after the new code is deployed. Here is where Kaholo helps to simplify the process.

Automating Testing with Kaholo

With Kaholo’s UI, different automated tests can be added to every step of the CI process. Developers, QA, and DevOps can easily add a step that executes unit, component, and other API tests after the build process is successfully finished. Each test can have an external validation step, allowing verifications to be done after the test has passed, which can give another perspective on the functionality of the API or service.

Kaholo enables you to deploy simultaneously to different environments and to perform different tests on different environments independently. This helps to shorten test cycles by allowing you to perform tests in parallel on different environments, execute different user scenarios, and examine several parts and levels of the service functionality. This also simplifies R&D support if there are several cloud providers with the same code. The ability to perform tests in parallel in different environments also helps you get quick responses and views of service quality, which is step one of the defect cost measurement methodology.

But executing tests does not just mean running them. Services can be used in different ways, and each scenario needs to be verified. Kaholo gives you the ability to provide multiple variables for a single test—meaning the test will be executed several times. Every execution runs with separate configurations, testing the system in diverse permutations. A dedicated report is generated at the end of every test cycle, with information about all the tests executed and their status.

More Automation? More Kaholo

One of the main challenges QA and DevOps engineers face is gaining control of a variety of test suites, each with different dependencies and configurations. UI tests done with Selenium or other tools, like Microfocus UFT and Ranorex, require a variety of browsers or, for mobile testing, the relevant device emulator installed on the host or container. API tests, on the other hand, may require that unusual ports be opened so the relevant GET or POST requests can go through.

Automatic security tests require access to infrastructure components, and performance tests require high resources in order to create the desired load. At some point, maintaining test suites becomes a burden and receives less attention and less budget. This can lead to the organization missing out on the value of these tests, and later, becoming lax about testing. Kaholo makes executing tests easy, helps with configuration of each test suite, and allows you to execute several test suites in the same process, which provides a broad picture of the service/code quality.

Kaholo also simplifies test maintenance. Tests can be broken into smaller, dedicated chunks, and validation steps can be added for every small test, making it easier to identify failures and analyze root causes. Every “Check Tests” step in the flow below performs the relevant validations and verifications for the specific tests executed prior to its run. This allows QA to focus testing on specific customer use cases, and in some instances, to remove the verifications from the test code.

If a test fails, these external validation steps provide information about the reason for the failure: Was it due to execution problems, such as inaccurate code, UI element identification issues, or other technical problems, or was the failure due to business logic, meaning that the test passed but the service doesn’t function as it should?

CI flows with tests on Kaholo
CI flows with tests on Kaholo

 

Kaholo helps with collaboration as well. Every test failure requires that relevant personnel be notified, according to a specific logic. No matter who needs to be informed—feature owner or team leader, developer or automation engineer who developed the test, product or project managers who want to track development progress, or security engineers who wants to verify the system’s security as part of the CI process (as the DevSecOps methodology recommends)—Kaholo has the ability to alert the right person, with a full report and required information packaged inside the message, helping to shorten the defect investigation and fixing process.

Summary

Kaholo makes the whole R&D group more agile, enabling it to release more rapidly and facilitating development and deployment. Kaholo addresses challenges that DevOps face daily, but R&D and QA can also gain from using it. Kaholo allows them to upgrade automation tests to a whole new level, and for QA to focus less on test stability and maintenance and more on adding coverage and improving the customer experience.

Kaholo’s user interface helps make automation test planning and execution more accessible to less-technical stakeholders while allowing developers and code-oriented people to manage and improve these steps using the js code that controls the flow, inside the web IDE implemented inside Kaholo.

Kaholo code module- Controlling the flow with JavaScript
Kaholo code module: Controlling the flow with JavaScript

 

Kaholo provides numerous benefits across the entire DevOps ecosystem. It allows you to take the flow, break it into smaller pieces, and sit back and enjoy while Kaholo executes the steps for you.

Simplify your DevOps processes. Get started with Kaholo today.