JUnit - What's next?
Document History:
$Revision: 1.12 $ | $Date: 2007/10/03 12:41:16 $ | CVS
Table of Contents:
I. Introduction
II. Competitive Analysis
III. Proposed New Features
I. Introduction
There is no doubt that unit testing is regular part of developement cycle for
all serious developers and IT companies all over the world. A lot of typographical work
has been already done about unit testing and its reasons, positives, constraints,
etc. Let us point you to one which is related to Netbeans project -
The Netbeans Q-Evangelism.
This document tries to describe which support the Netbeans IDE provides for
developing, running and debugging unit tests and compare it to other major
IDEs available in the market. At the end after analysing strenghts and weaknesses of current
unit testing support in Netbeans IDE and looking for RFEs filed by our users
we give a proposal of what other features should be implemented in next versions
of Netbeans.
II. Competitive Analysis
In order to get the idea what new we should implement in Netbeans regarding unit
testing support we tried to look around for what features provide our competitors.
This should give us an overall picture and show strenghts and weaknesses of Netbeans
unit testing support.
All compared products build their support for unit testing on top of the JUnit
framework which is de facto standard for writing unit tests in Java world. You can
find the JUnit open source project on http://www.junit.org.
Compared Products
We examined three major products available on the market and compared them to Netbeans.
They were:
- Netbeans -- developers version of Netbeans IDE and JUnit module from 08-Dec-2003
- Idea -- release version of IntelliJ Idea 3.0.5
- Eclipse -- stable build of developers version of Eclipse 3.0M5
- JBuilder -- release version of JBuilder 9 (Personal) with installed documentation
Examined features
Here is the list of features we found in compared products together with their
brief description.
- General Overview
- Tests generation - the possibility to generate test suites and/or test cases
- Tests execution - the possibility to execute tests (suites, cases or even test methods)
- Tests debugging - the possibility to run tests and debug both test and tested code
- Integrated results presenter - IDE captures results from tests executor and
shows them integrated in its GUI
- Private methods testing support - IDE allows to test private methods of classes (generaly
access private class members from inside the test class
- Online documentation - IDE contains documentation about JUnit integration support
- Tests Generation Wizard
- Single test suites - the possibility to create single test suite class
- Single test cases - the possibility to create single test case class
- Single test methods - IDE allows to create skeleton of test method for one tested method
- Test cases/suites for given scope (project, package) - IDE automaticaly generates test
suites/cases for all classes in some scope (e.g. package, project, filesystem)
- Custom test case/suite templates - user can customize existing templates for test
suites and test cases and/or create her own templates
- Test Case Generation
- Select methods - when generating test case class user can select which methods will
be tested
- Test inherited methods - during tested methods selection user cas choose also from
methods inherited by tested class
- Test method skeletons - test case class generator is able to create skeletons of
test methods
- Basic test - test case class generator is able to create bodies of test methods performing
simple test of tested method
- Javadoc for test methods - generated test methods have reasonable javadoc
- Test case setUp/tearDown - generator is able to generate setUp/tearDown methods which
instaniate tested class and allows test method to access this instance through the
field member of test case class
- Test Classes Placement
- Along tested classes - test suites/cases can reside in the same folder as tested classes
- Under special source root - test suites/cases are generated in the same package structure
as tested classes but under the special source root
- Multiple source roots - IDE support multiple source roots with tests
- Source Code Navigation
- Tests refering to given class - the possibility to find all test cases which test
(instantiate and call some methods) given class
- Tests refering to given method - the same as above, but it seeks for tests which
call specific method from given class
- Exec/Debug Granularity
- Single test suite - the possibility to start execution/debugging of specific test suite
- Single test case - the possibility to start execution/debugging of specific test case
- Single test method - the possibility to start execution/debugging of specific test method
- All tests in given context (project, package) - the possibility to start execution/debugging
of all tests in some context (e.g package, project, filesystem)
- Test-bags - user is able to create specify set of test suites/cases/methods and execute/debug
them; tests can specified in several ways (e.g. one-by-one, using mask, regexp or ant-like
path mask)
- Auto search for tests - the possibility to find test suites/cases in given context; this
can be used on various places where user is supposed to enter the test suite or test case class
- Results Presentation
- Integrated GUI presenter - there is a graphical presenter of test results smoothl
integrated in IDE's window
- Integrated textual presenter - the textual output of tests execution is captured and
shown in integrated output window
- Progress indication - an indication of percentage of test already performed is shown during
the tests execution/debugging
- Failures/errors presentation - all failures/errors encountered during tests
execustion/debugging are shouwn to the user
- Jump from failures/errors to source code - it is possible to navigate from failure/error
report to test source code
- Full tests hierarchy - the tree of all test suites and test cases and their methods which
have executed is shown to the user for browsing
- Jump from full tests hierarchy to source code - it is possible to navigate from
above described tests hierarchy to source code
- Stacktraces filtering - IDE allows to filter out irrelevant pieces of stacktrace in
failure/error reports
Results
The table below sumarizes results of competitive analysis. It shows features in the left
column and groups them in logical categories. The rest of columns is dedicated to particular
products and shows whether given product supports the feature or not. In that case cells contain
'yes' or 'no' values respectively. If the cell contains any other value please refere to
table's legend.
The results in Netbeans' column are colored according the following rules. The red color
signalizes that given feature is not supported by the Netbeans IDE, but there is at least
one of our competitors who provides such support. The green color is used for opposite
situation when Netbeans is the only IDE providing support for given feature.
| General Overview |
| Feature \ Product |
Netbeans |
Idea |
Eclipse |
JBuilder |
| Tests generation |
yes |
no |
yes |
yes |
| Tests execution |
yes |
yes |
yes |
yes |
| Tests debugging |
yes |
yes |
yes |
yes |
| Integrated results presenter |
yes |
yes |
yes |
yes |
| Private methods testing support |
no |
no |
no |
no |
| Online documentation |
yes |
yes |
yes |
install |
| Tests Generation Wizard |
| Feature \ Product |
Netbeans |
Idea |
Eclipse |
JBuilder |
| Single test suites |
no |
no |
yes |
yes |
| Single test cases |
yes |
no |
yes |
yes |
| Single test methods |
no |
no |
no |
no |
| Test cases/suites for given scope (project, package) |
yes |
no |
no |
no |
| Custom test case/suite templates |
yes |
no |
no |
no |
| Test Case Generation |
| Feature \ Product |
Netbeans |
Idea |
Eclipse |
JBuilder |
| Select methods |
no |
no |
yes |
yes |
| Test inherited methods |
no |
no |
yes |
yes |
| Test method skeletons |
yes |
no |
yes |
yes |
| Basic test |
no |
no |
no |
yes |
| Javadoc for test methods |
yes |
no |
no |
no |
| Test case setUp/tearDown |
yes |
no |
yes |
yes |
| Test Classes Placement |
| Feature \ Product |
Netbeans |
Idea |
Eclipse |
JBuilder |
| Along tested classes |
yes |
yes |
yes |
yes |
| Under special source root |
yes |
yes |
yes |
yes |
| Multiple source roots |
yes |
yes |
yes |
yes |
| Source Code Navigation |
| Feature \ Product |
Netbeans |
Idea |
Eclipse |
JBuilder |
| Tests referring to given class |
primitive |
no |
yes |
no |
| Tests referring to given method |
no |
no |
yes |
no |
| Exec/Debug Granularity |
| Feature \ Product |
Netbeans |
Idea |
Eclipse |
JBuilder |
| Single test suite |
yes |
yes |
yes |
yes |
| Single test case |
yes |
yes |
yes |
yes |
| Single test method |
no |
yes |
no |
no |
| All tests in given context (project, package) |
no |
yes |
yes |
enterprise |
| Test-bags |
no |
no |
no |
enterprise |
| Auto search for tests |
no |
yes |
yes |
?? |
| Results Presentation |
| Feature \ Product |
Netbeans |
Idea |
Eclipse |
JBuilder |
| Integrated GUI presenter |
no |
no |
yes |
yes |
| Integrated textual presenter |
yes |
yes |
yes |
yes |
| Progress indication |
no |
no |
yes |
yes |
| Failures/errors presentation |
yes |
yes |
yes |
yes |
| Jump from failures/errors to source code |
yes |
yes |
yes |
yes |
| Full tests hierarchy |
no |
no |
yes |
yes |
| Jump from full tests hierarchy to source code |
no |
no |
yes |
yes |
| Stacktraces filtering |
no |
no |
yes |
yes |
|
Table 1. Results of competitive analysis.
Primitive - product contains very simple version of the feature; Install - the feature
is available after installing additional packages; Enterprise - the feature is available
in enterprise version of product; ?? - it's not known whether the feature is supported
III. Proposed New Features
This section contains a list of features we propose to add in JUnit module in the
near future. The list is based on results of competitive analysis and
RFEs and feature requests filed in
IssueZilla.
Testing private methods
This feature would allow to access private members of tested classes from written tests.
Currently the JUnit framework allows to test public, protected and package private
class members, but doesn't provide any way for testing private members. It is not good
coding practice to use package private access modifier for class members just to allow
their testing.
In Netbeans there is an abandoned module called innertestres which provides support
for testing private members. This support could be merged to JUnit module.
See also: 14678.
Template for test suite
There should be template for test suite class allowing to create single test
suite. Currently there is only template for test cases. Also the test suite template
wizard should allow to define contents of created test suite. It should
allow to add existing test cases to the suite; alternatively it could also allow to add
all existing test cases from some package or project.
See also: 21977.
Better wizard for test case
There is a test case class template which allows to create single test case. However
its wizard doean't allow to specify any additional information needed by test case
generator. That's why the created test case is empty and nearly useless. The wizard
should allow to specify following things:
- Tested class - the class for which the test case is generated
- Tested methods - methods which will be tested by the test case and which test
skeletons shall be generated; such list should contain both methods declared by tested
class and also inherited methods
- The setUp/tearDown flag - checkbox specifying whether setup/tearDown methods should
be generated
- Super class - it should be possible to change test case's super class (default is
junit.framework.TestCase, but can be e.g. org.netbeans.junit.NbTestCase)
Generaly the wizard should allow to control all features available in test
case code generator.
Generate test for single method
During the developement cycle it can happen that new methods are added to class which
test case is already written. Then it is neccessary to extend the test case and additional
tests for newly created methods. We should simplify this somehow. The feature should
allow user to right click on the method name in some class and generate test method
in the test case class. There should also be a editor shortcut assigned to that feature.
Alternatively we could allow user to synchronize test case with tested class. This action
when invoked would seek for all untested methods and generated skeleton methods for
their testing.
See also: 21985.
Improve test case code generator
There is a couple of things which could be improved in generator of test cases.
- Basic tests - the generator should generate reasonable test code instead of
dummy test method bodies making the test to fail by default; the test code should simply
call the tested method passing dummy arguments to it (empty strings, null, 0, etc.)
- Customized tests - users should be able to customize test code generated in
test method bodies. They could write the code snippet with some parameters
(e.g. %n for tested method name) which will be used as a template for generating test
methods bodies.
- setUp/tearDown - allow to generate setUp/tearDown methods which create instance
of tested class and assign it to some private field of test case class and then release
that instance when the test is finished.
See also: 30575.
Better navigation in tests
Often the developer needs to know which tests excersise given class or method. Currently
there is very primitive support for this search based on naming convetion used for
test case classes names. We should improve it to search for all test cases which realy
excersise (instantiate and call) given class and/or method.
Improved test execution
The execution/debugging in Netbeans IDE suffers from missing support for exec/debug
profiles or configurations. There is no way to define the way how things should be
executed/debugged; save this definition and reuse it anytime in the future. Once this
will be allowed we should reuse this for tests exceution/debugging. We could then
think about features like:
- Test-bag support - it allows user to define a bag of tests which should be exec/debugged;
tests can be defined either one-by-one or user can specify mask (regexp, ant-like file mask)
to define set of tests; also the set of tests can be defined as 'all tests' in some context
(e.g. package, source root, project)
- Auto search for tests - whenever user is supposed to choose some test cases/suites
she has not been offered by ordinary classes (non-tests)
Single test method exec/debug
Allow to execute/debug the single test method. Currently it is possible to
exec/debug either whole test suite or test case which is an overkill in many situations.
Developers often need to run just single test method to diagnose the problem.
There could be action on test method allowing to exec/debug it.
Integrated GUI presenter
It would be beneficial if Netbeans would provide a GUI presenter of test results smoothly
integrated in the main window. This GUI should allow user to run tests, see the progress
(percentage of tests already exercised), visualize test suites/cases and their methods which
failed, allow to navigate to test's source code and tested code, provide automatic
(customizable) stack trace filtering, etc. In current version there is only textual output
displayed in output pane. Often this output is messy and hard to read. The GUI presenter
would provide for better and easier understanding of which tests failed,
if some tests are still running and so no.
Improve test suite code generator
Also in test suite code generator there is couple of things to imporve.
- Code readability - we should sort "suite.addTest" lines alphabetically
according to names of added classes for better code readability.
- Use guarded blocks - there is a code generated in test suites which shouldn't be edited by the user, because all user's
changes done in this code will be lost when test suite is regenerated. In current version there
is just comment explaining this to the user. We should consider using guarded blocks to protect
generated code from being edited by the user.
See also:
32309,
15304.
