Performance testing is a kind of test to determine quality from performance (speed or effectiveness) of an application or product. Performance testing is not to find a bug, but to eliminate bottlenecks.
The comparing between performance and stress testing
Performance = the limit of performance (positive testing)
Stress = unpredictable process or try to break the system (negative testing)
Example, a new product (hand phone) of NOKIA
Performance testing:
1. Accept a call for long time (a few days)
2. Accept calls from many people
3. Run many application (all application)
Stress testing:
1. During the call, shutdown the phone or turn out the battery
2. Run many program and then shutdown the phone
3. Shake the phone
How about we drop the phone on the floor during the call, is it a performance testing or stress testing?
~it is stress testing, because the test try to destroy it and find the condition after that~
What the difference with load test? Load test is a kind of test for a system which is a system is operated with the largest tasks. You can constantly increase the load on the system to find the bottlenecks. So, the load test is a part of the process of performance testing
The methodology of performance testing
According to BBST overview by Cem Kaner, the fundamental question in software testing. Once after we have a mission of our performance testing, the next level we design methodology performance testing. Remember that performance testing to determine how speed and how effectiveness a product is.
A product consists of 2 main parts, the application/functionalities and the devices. The methodology of performance testing is combination of those parts. Test the functionality which includes the specific individual tests, test the devices with as well as configuration setting of the devices it selves, and between them we increase the load testing.
The strategy of performance testing
To test the performance of a product we need a strategy to know where we want to go in term of performance testing. There might be a lot of strategy, and this is the one of them. We can start with create a test scenario, and then test metrics (collect the test result along with hits/sec or other service server resource monitors), and the last is stability (endurance) test.
Test Scenario, create a scenario and identify which is the main functionalities, the largest tasks, and the potentials a system is being down/crash (according the requirement or the fact in client). Or we can create a good scenario testing according on “Cem Kaner On Scenario Testing” (www.stqemagazine.com, September/October 2003).
Test Metrics, there could be many components involve in a functionality, such as what the network connection is used, how many hits, users, and other service server resource monitors.
Stability (endurance) test, an application is run a few hours (10-12 hours) or a few days to identify the memory bottlenecks. This test needs to be run the largest tasks at all time, such as generate report by many users for 1 day (24 hours).
Automation Testing Tool
Automation Testing Tool; is a tool to test an application/product by automatically. It is easy and very helpful for a tester, but it needs more effort to create the script.
These are the reason why automation testing tool is very helpful.
1. Every time a tester reports a defect, she can create a script with the same of procedures. If a tester forgets how to replicate the defect, she just can replay from automation testing tool
2. Retest all defects that have been closed for the new build/version (Regression testing), and cover a sample test for a new build (Smoke Test)
3. Performance testing
We can test the performance of a product with 2 ways, by using manual testing and by using automation testing tool. If we use manual testing to test the performance, it is hard to find the exactly condition of user/client. We need many users, we need many computers, and we need specific network configuration of each user. But, if we use automation testing tool, we can set how many user we need, what/how scenario is run, and which network configuration is set. Automation testing tools (not all) also provide the report/result of testing.
OpenSTA and JMeter are kinds of automation tool testing. OpenSTA is an automation testing tool for performance testing Web Application Environment. Simple words, OpenSTA helps us to test a product automatically, and provide a performance testing from component of the system under test, such as number of virtual user, and run for multiple task (concurrent or sequence). OpenSTA also provide test results to analyze the performance of an application. JMeter is an automation testing tool like OpenSTA, but it is developed under Java Environment. JMeter is designed to load test functional behavior and measure performance.
2. http://www.kaner.com/pdfs/ScenarioSTQE.pdf
3. http://hosteddocs.ittoolbox.com/Questnolg22106java.pdf
4. http://devcentral.f5.com/LinkClick.aspx?link=http%3A%2F%2Fdevcentral.f5.com%2Fdownloads%2Ff5%2Fcreating-performance-test-methodology.pdf&tabid=73&mid=935
5. http://portal.opensta.org/modules.php?op=modload&name=phpWiki&file=index&pagename=SclStruts&action=diff&previous=major
6. http://jakarta.apache.org/jmeter/
7. http://agiletesting.blogspot.com/2005/02/performance-vs-load-vs-stress-testing.html
8. http://searchsoftwarequality.techtarget.com/sDefinition/0,,sid92_gci1259932,00.html
9. http://ramya-moorthy.blogspot.com/2008/02/tips-to-define-performance-test.html
How to use them;
- we just record what we test on the product/application
- set any configuration (if it is needed)
- we can replay it to find the result/report of testing.
9 months ago