Book Description

This dissertation makes contributions to four areas of performance testing - the test process itself, monitoring, automation and end-to-end performance evaluation of backend applications. The first contribution deals with the testing process. Performance testing is a key element of industrial software development, but we still encountered the following two problems. (1) While testing textbooks prescribe writing tests against performance goals, we find that it is impractical to gather from business analysts performance goals that are detailed enough for finding subtle performance bugs. (2) Once performance tests are conducted, we were asked questions such as - how do you make sure that the results collected during testing are a good indication of how the code will function in production? To enable practitioners to address these problems, we introduce two additional performance testing process components, which we call release certification and test data correlation. Our key idea to address problem (1) is to run two different versions of the same subject application side-by-side in the same test environment and (2) is to correlate the performance measurements of the test and production environments. Running individual soak, load and stress tests to assess different aspects of an application is a tedious process. So we introduce a new test - impulse test that not only combines the characteristics of multiple existing tests but also enables testing for engineering aspects such throttles, alerts and timeouts. The second contribution is in the area of instrumentation. Monitoring the performance of distributed applications is an important task in practical software engineering. Current monitoring tools are often limited in the range of computing platforms they support, which limits their utility in several business monitoring scenarios. Current monitoring tools can also impose a significant performance overhead. We describe our in house built EI Enterprise Instrumentation monitoring tool that addresses these issues. We compare EI with a state-of-the-art monitoring tool on a real online shopping application and describe our experience with EI as well as feedback we received from EI users. The third component of the study deals with test automation and scripting. Scarcity of commercially available testing tools that could support all native or application specific message formats as well as those that cater to non GUI or non-web based backend applications led to creating our own customized traffic generator and scripts. This study provides (1) the general design principles for a test script that can be used to generate traffic for any request format as well as (2) specific factors to keep in mind when creating a script that will work in a test environment that uses a mock. It will also address the (3) design and properties of a test harness. It provides a simple framework that can be easily used to complete an end-to-end testing process: pre test, traffic generation and post test activities. Last but not the least the dissertation proposes a solution to assess performance for backend applications. State space models specifically Markov models are used extensively to predict anomalies, capacity and throughput in computer systems. In most cases existing solutions depend on underlying architecture and historical data to create the models and then detect states that differ from the created profile. In this study we will use Markov models to proactively predict the performance of a software system by using client side input parameters and application attributes without the need to know the underlying architecture. The goal is to find pressure points/bottlenecks for the application when it interacts with other components. Pressure points are any application resource that can become exhausted thereby restricting or degrading service level performance such as CPU, memory, disk, network and so on. All solutions proposed in this study have been implemented on real world travel applications, in most cases an airline shopping application. The data gathered and all measurements/plots in this study are from travel related pieces of code that are live and used by customers today.