Performance Evaluation and Benchmarking with Realistic Applications


Book Description

The book discusses rationales for creating and updating benchmarks, the use of benchmarks in academic research, benchmarking methodologies, the relation of SPEC benchmarks to other benchmarking activities, shortcomings of current benchmarks, and the need for further benchmarking efforts. Performance evaluation and benchmarking are of concern to all computer-related disciplines. A benchmark is a standard program or set of programs that can be run on different computers to give an accurate measure of their performance. This book covers a variety of aspects of computer performance evaluation, with a focus on Standard Performance Evaluation Corporation (SPEC) benchmarks. SPEC is a nonprofit organization whose members represent industry, academia, and other organizations. The book discusses rationales for creating and updating benchmarks, the use of benchmarks in academic research, benchmarking methodologies, the relation of SPEC benchmarks to other benchmarking activities, shortcomings of current benchmarks, and the need for further benchmarking efforts. Contributors Brian Armstrong, Frederica Darema, Edward S. Davidson, Sylvia Dieckmann, Jozo J. Dujmovic, Rudolf Eigenmann, J. Kelly Flanagan, Greg Gaertner, Jonathan Geisler, John Gustafson, Urs Hölzle, Shih-Hao Hung, Kathryn S. McKinley, Reinhard Riedl, Faisal Saied, Frank Sorenson, Mark Straka, Valerie Taylor, Olivier Temam, Rajat Todi, Reinhold Weicker




Performance Evaluation and Benchmarking


Book Description

Computer and microprocessor architectures are advancing at an astounding pace. However, increasing demands on performance coupled with a wide variety of specialized operating environments act to slow this pace by complicating the performance evaluation process. Carefully balancing efficiency and accuracy is key to avoid slowdowns, and such a balance can be achieved with an in-depth understanding of the available evaluation methodologies. Performance Evaluation and Benchmarking outlines a variety of evaluation methods and benchmark suites, considering their strengths, weaknesses, and when each is appropriate to use. Following a general overview of important performance analysis techniques, the book surveys contemporary benchmark suites for specific areas, such as Java, embedded systems, CPUs, and Web servers. Subsequent chapters explain how to choose appropriate averages for reporting metrics and provide a detailed treatment of statistical methods, including a summary of statistics, how to apply statistical sampling for simulation, how to apply SimPoint, and a comprehensive overview of statistical simulation. The discussion then turns to benchmark subsetting methodologies and the fundamentals of analytical modeling, including queuing models and Petri nets. Three chapters devoted to hardware performance counters conclude the book. Supplying abundant illustrations, examples, and case studies, Performance Evaluation and Benchmarking offers a firm foundation in evaluation methods along with up-to-date techniques that are necessary to develop next-generation architectures.




Performance Evaluation and Benchmarking of Intelligent Systems


Book Description

To design and develop capable, dependable, and affordable intelligent systems, their performance must be measurable. Scienti?c methodologies for standardization and benchmarking are crucial for quantitatively evaluating the performance of eme- ing robotic and intelligent systems’ technologies. There is currently no accepted standard for quantitatively measuring the performance of these systems against user-de?ned requirements; and furthermore, there is no consensus on what obj- tive evaluation procedures need to be followed to understand the performance of these systems. The lack of reproducible and repeatable test methods has precluded researchers working towards a common goal from exchanging and communic- ing results, inter-comparing system performance, and leveraging previous work that could otherwise avoid duplication and expedite technology transfer. Currently, this lack of cohesion in the community hinders progress in many domains, such as m- ufacturing, service, healthcare, and security. By providing the research community with access to standardized tools, reference data sets, and open source libraries of solutions, researchers and consumers will be able to evaluate the cost and be- ?ts associated with intelligent systems and associated technologies. In this vein, the edited book volume addresses performance evaluation and metrics for intel- gent systems, in general, while emphasizing the need and solutions for standardized methods. To the knowledge of the editors, there is not a single book on the market that is solely dedicated to the subject of performance evaluation and benchmarking of intelligent systems.




Health Care Benchmarking and Performance Evaluation


Book Description

This new edition continues to emphasize the use of data envelopment analysis (DEA) to create optimization-based benchmarks within hospitals, physician group practices, health maintenance organizations, nursing homes and other health care delivery organizations. Suitable for graduate students learning DEA applications in health care as well as for practicing administrators, it is divided into two sections covering methods and applications. Section I considers efficiency evaluations using DEA; returns to scale; weight restricted (multiplier) models; non-oriented or slack-based models, including in this edition two versions of non-controllable variable models and categorical variable models; longitudinal (panel) evaluations and the effectiveness dimension of performance evaluation. A new chapter then looks at new and advanced models of DEA, including super-efficiency, congestion DEA, network DEA, and dynamic network models. Mathematical formulations of various DEA models are placed in end-of-chapter appendices. Section II then looks at health care applications within particular settings, chapter-by-chapter, including hospitals, physician practices, nursing homes and health maintenance organizations (HMOs). Other chapters then explore home health care and home health agencies; dialysis centers, community mental health centers, community-based your services, organ procurement organizations, aging agencies and dental providers; DEA models to evaluate provider performance for specific treatments, including stroke, mechanical ventilation and perioperative services. A new chapter then examines international-country-based applications of DEA in health care in 16 different countries, along with OECD and multi-country studies. Most of the existing chapters in this section were expanded with recent applications. Included with the book is online access to a learning version of DEA Solver software, written by Professor Kaoru Tone, which can solve up to 50 DMUs for various DEA models listed in the User’s Guide at the end of the book.




Performance Evaluation and Benchmarking


Book Description

This book constitutes the reviewed proceedings of the first Conference on Performance Evaluation and Benchmarking, TPCTC 2009, held in Lyon, France, August 24-28,2009. The 16 full papers and two keynote papers were carefully selected from 34 submissions. This book considers issues such as appliance, business intelligence, cloud computing, complex event processing, database performance optimizations, green computing, data compression, disaster tolerance and recovery, energy and space efficiency, hardware innovations, high speed data generation, hybrid workloads or operational data warehousing, unstructured data management, software management and maintenance, virtualization and very large memory systems




OpenMP Shared Memory Parallel Programming


Book Description

This book contains the presentations given at the Workshop on OpenMP App- cations and Tools, WOMPAT 2001. The workshop was held on July 30 and 31, 2001 at Purdue University, West Lafayette, Indiana, USA. It brought together designers, users, and researchers of the OpenMP application programming int- face. OpenMP has emerged as the standard for shared memory parallel progr- ming. For the rst time, it is possible to write parallel programs that are portable across the majority of shared memory parallel computers. WOMPAT 2001 s- ved as a forum for all those interested in OpenMP and allowed them to meet, share ideas and experiences, and discuss the latest developments of OpenMP and its applications. WOMPAT 2001 was co-sponsored by the OpenMP Architecture Review Board (ARB). It followed a series of workshops on OpenMP, including WOMPAT 2000, EWOMP 2000, and WOMPEI 2000. For WOMPAT 2001, we solicited papers formally and published them in the form of this book. The authors submitted extended abstracts, which were reviewed by the program committee. All submitted papers were accepted. The authors were asked to prepare a nal paper in which they addressed the reviewers comments. The proceedings, in the form of this book, were created in time to be available at the workshop. In this way, we hope to have brought out a timely report of ongoing OpenMP-related research and development e orts as well as ideas for future improvements.




Benchmarking Peer-to-Peer Systems


Book Description

Peer-to-peer systems are now widely used and have become the focus of attention for many researchers over the past decade. A number of algorithms for decentralized search, content distribution, and media streaming have been developed. This book provides fundamental concepts for the benchmarking of those algorithms in peer-to-peer systems. It also contains a collection of characteristic benchmarking results. The chapters of the book have been organized in three topical sections on: Fundamentals of Benchmarking in P2P Systems; Synthetic Benchmarks for Peer-to-Peer Systems; and Application Benchmarks for Peer-to-Peer Systems. They are preceded by a detailed introduction to the subject.




Handbook of Parallel Computing


Book Description

The ability of parallel computing to process large data sets and handle time-consuming operations has resulted in unprecedented advances in biological and scientific computing, modeling, and simulations. Exploring these recent developments, the Handbook of Parallel Computing: Models, Algorithms, and Applications provides comprehensive coverage on a




Performance Evaluation, Prediction and Visualization of Parallel Systems


Book Description

Performance Evaluation, Prediction and Visualization in Parallel Systems presents a comprehensive and systematic discussion of theoretics, methods, techniques and tools for performance evaluation, prediction and visualization of parallel systems. Chapter 1 gives a short overview of performance degradation of parallel systems, and presents a general discussion on the importance of performance evaluation, prediction and visualization of parallel systems. Chapter 2 analyzes and defines several kinds of serial and parallel runtime, points out some of the weaknesses of parallel speedup metrics, and discusses how to improve and generalize them. Chapter 3 describes formal definitions of scalability, addresses the basic metrics affecting the scalability of parallel systems, discusses scalability of parallel systems from three aspects: parallel architecture, parallel algorithm and parallel algorithm-architecture combinations, and analyzes the relations of scalability and speedup. Chapter 4 discusses the methodology of performance measurement, describes the benchmark- oriented performance test and analysis and how to measure speedup and scalability in practice. Chapter 5 analyzes the difficulties in performance prediction, discusses application-oriented and architecture-oriented performance prediction and how to predict speedup and scalability in practice. Chapter 6 discusses performance visualization techniques and tools for parallel systems from three stages: performance data collection, performance data filtering and performance data visualization, and classifies the existing performance visualization tools. Chapter 7 describes parallel compiling-based, search-based and knowledge-based performance debugging, which assists programmers to optimize the strategy or algorithm in their parallel programs, and presents visual programming-based performance debugging to help programmers identify the location and cause of the performance problem. It also provides concrete suggestions on how to modify their parallel program to improve the performance. Chapter 8 gives an overview of current interconnection networks for parallel systems, analyzes the scalability of interconnection networks, and discusses how to measure and improve network performances. Performance Evaluation, Prediction and Visualization in Parallel Systems serves as an excellent reference for researchers, and may be used as a text for advanced courses on the topic.




Computer Performance Evaluation and Benchmarking


Book Description

This book constitutes the proceedings of the SPEC Benchmark Workshop 2009 held in Austin, Texas, USA on January 25th, 2009. The 9 papers presented were carefully selected and reviewed for inclusion in the book. The result is a collection of high-quality papers discussing current issues in the area of benchmarking research and technology. The topics covered are: benchmark suites, CPU benchmarking, power/thermal benchmarking, and modeling and sampling techniques.




Recent Books