The technique of testing the software system against the external functional behavior of the software is called black box testing. The external system behavior is typically defined in the form of requirement specifications, functional use cases and API docs. etc.
The focus is completely on the inputs or data that goes into the software and the output that comes out of the software. The data processing, internal structure, the code, the development methods, server logic and everything else is not considered or looked into by the tester.
Since this class of testing is absolutely function centric and is focused on the behavior of the software, it is also called functional testing and behavioral testing. Few of the other terms used for this type of testing are called opaque testing and closed box testing.
Black box testing techniques are applicable to most of the testing phases/types in the overall software development life cycle and is the primary method of software verification by end users.
The other sibling of the black box testing is white box testing but there is significant difference in both these methods of testing the software. We will talk very briefly about white box testing, later in this article. At a very high level, white box testing techniques require the tester to look into the internal workings of the code and software structure.
An very simple example is to test the functions of a calculator that does a+b=c. User enters various ranges of inputs like 1+2, 1+2+5, 1+9+7+667 etc. and verify the result without getting into how the calculator does it. We will talk about more complex examples, later in this article.
Black Box Testing – How To Do?
Tester provides inputs to the software in the form of data, clicks, scrolls, touch, voice, feeds and using any other means of inputs applicable to the software and records the output. The key focus is to ensure that software is capable of taking all types and ranges of inputs and provide desired output. How it does, is not a point of concern here.
Testing Preparation
The preparation for testing ideally starts as soon as software requirement specifications are finalized. Testing team can start preparing the test cases for each of the functional use case by critically analyzing various inputs and outputs applicable to the individual use case.
In the agile development, test cases keep on getting revised and updated with every new iteration or addition/modification of the requirement or use case.
When an independent module or feature of the software is ready, one can execute test cases to validate the requirements. Software design and build can continue in parallel.
Testing Types and Black Box Testing
There are multiple types and phases of testing that are mainly divided into unit testing, integration testing, functional testing, system testing, acceptance testing, beta testing and regression testing. Further categorization in the form of load testing, performance testing etc. is also prominent and used much often as separate categories.
Black box testing technique is applicable to most of the above categories but the applicability varies, unit testing for example is done as white box testing for most part of specific software systems.
Advantages/Disadvantages
Black box testing is a necessity and for that reason it is must and cannot be avoided. There are many advantages and the most notable one’s are as listed below –
- Tester doesn’t need to know about the internal structure, code and tools etc. used to build the software
- The test cases can be created as soon as the functional specifications are ready. No dependency on code readiness.
- Testing team is totally independent of development team and provides unbiased inputs with a clear focus on finding issues in the software.
- No programming skills required for testing team.
- Testing is done from end user perspective and ensures software acceptance by end users.
A key disadvantage of this technique is the difficulty to cover all scenarios, specifically when software business logic becomes complex or the software deals with multiple ranges and types of inputs.
The testing techniques becomes highly cost ineffective and should be supplemented with white box testing or other modes of software validation and verification methods. There are various strategies to keep this cost low, as explained in the next few sections of this article.
Black Box Application Security Testing
Application security is non-negotiable, specifically when we talk about web applications. InfoSec testing in itself has become a separate discipline over the last few years and is central to any software development these days. For application security testing also, the key technique used is black box testing where the testers ask for details around application access and various user roles possible. The black box security testing is typically performed with help of various tools to identify one or more of below vulnerabilities –
- SQL injection attacks
- Cross Site Scripting
- Input data validation
- Sessions expiry and management
- Folder structure access control checks.
- Buffer overflow etc.
- ..and many more
Security testing is a wide area in itself and would be covered in a separate article. Follow on Facebook to stay updated.
Black Box Testing Techniques
Testing the software system with whole range of inputs/output combinations is near to impossible, and for that reason, various testing techniques are required to ensure maximum coverage with minimum effort and cost. Furthermore running all the test cases when the software changes a bit too is impractical, and for that reason test case prioritization techniques to choose most relevant test cases are required to be adopted.
There are multiple techniques and strategies to conduct black box testing and the key things that every testing strategy for sure takes into account are as listed below –
- Focus on features expected to be used the most by end customer
- Maximum defects detection with minimum number of test cases execution
- Avoiding redundant/duplicate test cases
- Revealing various modes of failure
- Testing with extreme input/output combinations
- Testing the flow of events and actions.
Combination of various strategies is generally applied to get maximum benefit. Equivalence partition and boundary value analysis techniques for example can be applied to test the inputs and UI validations etc. whereas state transition testing and decision table testing techniques focus more on validation of business rules and logic. Some of the most effective black box testing techniques are as listed below.
Equivalence Partitioning
The evolution of various testing techniques is to ensure that overall testing times and costs are as low as possible. Equivalence partitioning is the technique that focuses on reducing the number of test cases required to cover testing of maximum functionality. In equivalence partitioning technique, sets of inputs or ranges of inputs are grouped into different classes. Each class of inputs is expected to derive same output from the software.
Boundary Value Analysis
This is one of the most critical and useful black box testing techniques and builds further on top of equivalence partitioning. Boundary value Analysis (BVA) deals with testing the software with extreme or boundary values. Talking about defining extreme input values, in equivalence partitioning various groups of input classes are created and extreme values from each group can be taken for creating test cases for Boundary value analysis. Boundary value analysis doesn’t only deal with extreme input values but extreme/edge output values too should be considered and tested.
To make boundary values analysis further effective, boundary values plus the immediate outer value should also be tested.
State Transition Testing
In state transition testing, the inputs, outputs and state of the system at a given point of time are taken into consideration. Depending on the type of software under consideration, software or system can behave one way in one state and another way in another state even when the inputs are same.
The test cases for state transition testing are developed by looking at the sequence of events or states and transitions between those states. The complete set of test cases would include traversal of all states and expected output values.
Error Guessing
They say experience counts. Yeah! Error guessing is technique where software tester guesses various input, outputs that can be erroneous and needs fixing. This is purely based on the perception and judgement of the tester based on the past experience.
Graph based Testing (Cause and Effect Diagrams)
As the name goes, the testing involves drawing of a graph that shows a relationship between the effects and the causes that produce those effects. In simple terms, causes can be looked at as inputs and effect as output from the software system.
For the purpose of testing, various combinations of causes (inputs) are tried and the effects (output) are recorded. This also happens to be a very useful technique for understanding the functional workings of the software, since inputs, outputs and flow gets to life in a visual manner.
Decision Table Testing
Sometimes the combination of inputs results in different outputs and it becomes really complex to track various possibilities. Decision tables are generally used in such complex scenarios since these give systematic view to testers on combination of inputs and desired output. This technique of black box testing is very similar to graph based (cause and effect diagrams) testing, the main difference is one uses graphs/diagrams and the other uses tables.
There are many other popular techniques used for testing software as black box including orthogonal array testing, all pair testing and fuzzing.
Black Box Testing Example
Let us consider a simple shopping scenario –
- Shop for $1000 and get 5% discount
- Shop for $2000 and get 7% discount
- Shop for $3000 and above and get 10% discount.
Talking about Equivalence partitioning – We can divide the input into four partitions, amount<0, 0 to 1000, 1001 to 2000, 2001 to 3000 and then 3001 and above.
We will keep things simple for now and will not go into details like the max amount one can shop for, or what about cents and all that stuff.
Adding the flavor of boundary value to the above partitions – the boundary values would be 0, 1000, 1001, 2000, 2001, 3000 and 3001. In boundary value analysis, immediate upper and lower values too are recommended to be tested, so we will include -1, 1 and 999. These are the values that can tell behavior the software for almost entire range of input values, but there can be other issues in software. For that reason, other input values too can be considered.
Now to understand state transition technique – Let us say when you shop above 3000 twice a month, your status changes from Silver to Gold (and you get a gift) and if you do not shop further for next 3 months your status goes back to Silver. It is becoming a little complex for the tester and further test cases are required to track the various states here.
State connected on Facebook for more refined and detailed black box testing examples.
Summary
Black box testing is the way to test software against requirements but it is quite impractical to do extremely exhaustive testing. To cut down on the testing time and effort various techniques like boundary value analysis, equivalence partitioning, error guessing, state transition testing and decision table testing etc. are applied.
Also, since this type of testing doesn’t depend on the way software is build, the test case preparation for it can start as early as requirement specifications are ready and continue to evolve or iterate throughout the development life cycle.
Another noteworthy point is the timing for the testing, it is always advisable to start the testing as early as possible in the development life cycle since the time and cost to fix the defects early in the development cycle is always the minimum.
Last but not the least is that the testing can be automated for most of the testing phases like functional testing, integration testing, system testing and regression testing etc. But during acceptance testing, one should still target to get the system tested by the end users manually, so that subjective inputs etc. specifically around UI/UX can be taken care of.
Hope you liked the article! Do share your experience and knowledge of black box testing via comments! Cheers.
