JUnit: development directions outline
- Authors
- Ondrej Rypacek, Arseniy Kuznetsov
Started on: 21/06/04 | Version: 1.0
- Status
- Reviewed version, open for feedback
- Document History
- $Revision: 1.2 $ | $Date: 2005/04/25 16:13:55 $ | CVS
Contents
I. Introduction
II. Development directions
III. Proposed new features
IV. Releases
I. Introduction
This document is a logical continuation of the JUnit -
proposed new features document. It
outlines future development direction and proposes new features that
are being considered for mid-term implementation.
II. Development Directions
Although the JUnit module available in NetBeans 4.0 is fully competitive
with what can be seen in other major IDEs, in terms of the future development it is believed that
unit testing support in a modern IDE should go beyond mere integration with
the
JUnit testing framework.
Although heavily promoted by engineering literature, writing unit
tests is still not an integral part of programming in many engineering
teams and is often considered an extra work as opposed to the 'real'
coding. Unit tests are typically logically separated
from the code, kept in another source tree and displayed separately in
an IDE, thus often being simply overlooked by developers. Additionally,
with the curent level of development tools support, unit test tend to get
easily outdated throughout the development cycle.
At very high level, proposed approach to improving unit tests suport in NetBeans is viewed as:
- The IDE makes it easy to create and update existing
tests, unobtrusively reminding to the user whenever an appropriate
action is needed
- Tests are closely integrated with the source code
- Unit tests are
treated as a integral specification of the source code, just as types and method signatures are. Ideally, unit tests should
be viewed by users as a next logical step following compilation (i.e.
automic states of sources would be uncompilable / compilable but test-failed / compilable and tested)
- The IDE offers an option for compilation and running tests together with the
source code.
The
ultimate goal in this area is seen as
promoting unit testing as a
standard development methodology by making creation and usage of tests
in the IDE as easy and appealing as possible
.
The rest of this document provides more detailed description of
specific features that are intended to progressively contribute to this
goal.
III. Specific Features
This section provides description of specific features for middle-range outlook.
1. Test Creation
1.1 Improved
wizard for creation of test cases
In NetBeans 4.0, there is a test case class template which allows
creation of a single test case. It needs to be updated as follows:
- Unified UI -- currently, the UI for unit tests is somewhat diverse
and depends on the invoked action. Specifically, the wizard initiated
by the 'File->New->Unit Test' menu item is substantially
different from the dialog invoked from the context menu on a Java class
- Super class -- it should be possible to change test case's super
class (default is junit.framework.TestCase, but can be e.g.
org.netbeans.junit.NbTestCase)
- Tested individual methods -- methods which will be tested by the test case
and which test skeletons shall be generated; such list should contain
both methods declared by tested class and also inherited methods
1.2 Support for common Unit Test patterns
To faciliate test creation, it is possible to identify and support several most common patterns
of tests. Specific examples
coud be:
- Invoke and Verify -- applicable for all kind of tests, a method is run on a test
data and the returned result is compared to an
expected "golden" value stored in the
test. There are several variables in this basic pattern reasonably supportable in UI:
- input can be sent in
arguments or state
- the result can be returned in a return value
or in a state
- the predicate on result can vary,
e.g. equals, not equals
- Expect an Exception -- a method is run and a
specific exception value is expected to be
thrown
- Test Series -- several other tests of the
same kind are executed in succession. The tests are
intentionally not run in isolation but within a
single test-case.
- Golden Data Reference -- when a regression test
is created, a function known to be correct is run and
the result is persisted in the code for later reference. When the function is broken later, the regression
test fails as the persisted data and actual results
don't match
The variations listed above can be of course combined
arbitrarily, e.g. the result can be checked in a
return value and state at the same time. However, the
wizard UI should intentionally be simple to cover only the most
common cases, leaving more complex variants for the 'manual' coding.
2. Integrated Graphical Tools
The following set of advanced features is intended to simplify unit
test development, allowing most typical scenarious to be done with a few
clicks and keystrokes.
2.1 Unit Tests Logical View
Unit Tests Logical view is intended as a central point for handling unit tests in a project. It is to be based on the the following general
requirements:
- Efficient tests presentation -- test should be presented
in a comprehensive tree-like structure with
test-suites and test-cases easily navigable and
manageable.
- Advanced test creation -- integrated support for common Unit Tests patterns
- Test data management - support for running a test on different data sets
- Golden data management - the expected
results of tests can be managed, easily checked, added
and verified
Notice: detailed specification is to be descried in a dedicated UI design document
2.2 Test Results GUI presenter
In NetBeans 4.0,
tests results are displayed as a simple textual output in the output
pane. The GUI presenter is intended to provide enhanced tests results
handling, by e.g. visualizing the running tests progress (percentage of
tests already exercised),
showing failed test methods with ability to
navigate to test's source code and tested code, provide automatic
(customizable) stack trace filtering, etc.
Notice: detailed specification is to be descried in a dedicated UI design document
3. Test
Coverage
3.1 Test Coverage Detection
Test
coverage should be automatically detected so that the user
can immediately learn which classes, methods and
code-lines are covered by tests. The coverage should be
indicated in the class view by bullets / flags.
Note that test coverage does not simply mean that every
class have its corresponding test class and every method
have its corresponding test method in the test class. More
sophisticated methods must be employed to achieve reliable
coverage detection, although that would be nontrivial in
the general case.
JDK contains a basic command line support - jcov
- for evaluating code coverage. Files must be first
compiled with an option that inserts appropriate source
code references to the byte-code. Another runtime option
turns on the evaluation on execution. Results are output
into a textual dump file. Using jcov would thus require writing an appropriate
analysis and GUI for the coverage dump.
Another potential option is Cenqua Clover,
which is a code coverage tool with existing NetBeans
plugin (beta stage) and free license for open-source
products. More
info and screenshots can be found on the website.
An example of a commercial product with java api is JCover.
One of the design decision is specifying at which point in time the
covarage analysis should be run. For example, Clover integrates the
covarage analysis with the compile-action and then gathers coverage
data on every execution. That might not be the best solution for
performance reasons, so one of the alternatives would be a separate
action to update the coverage data and produce the coverage suggestions
(described bellow). It could be also possible to integrate
coverage analysis with running tests on a project.
3.2 Create Test Suggestions (integration with the Tasklist module)
The test coverage tool is a good candidate for being integrated with
the TaskList module, which should contain suggestions on tests
creation. The suggestion's action would invoke an appropriate wizard or
alternatively create the test right away.
3.3 Update Test Suggestion (integration with the Tasklist module)
When a covered method changes signature, the previously created test
must be detected and offered to a user for update.
4. Integration
with source code
This section describes features related to close integration of the
source code with associated tests.
4.1 Navigation in Tests
Currently, NetBeans 4.0 version provides just a very simple
navigation based on test-class
name convention and is available for test classes, not for individual
methods. In terms of the future, it should be possible to easily
navigate in editor from a
class/method to the corresponding test with a single shortcut.
Notice:
a single class/method can be covered by more then one test case
or test method.
4.2 Editor integration (show Test/Code in Code/Test)
It would be possible to bring tests even closer to code by
displaying (on shortcut) the corresponding test in a popup when
writing a code. And similarly, to display the tested code when writing
a test. The role of such a popup is similar to that of the javadoc
popup -- to provide documentation with the code being developed.
Notice 1: More research is needed to identify
the right, unobtrusive ways of presenting the functionality to the user
Notice 2: For all of the above cases, a detailed and
accurate test coverage analysis is necessary to reliably pair code
with tests.
5. Compilation
and
Running
5.1 Compile tests with the code
Test could be set up to compile along with the classes. The results of such
a test compilation would not be presented as errors but probably just
as a single suggestion ("correct your tests") and by a glyph in the
class view. The tests could also be compiled explicitly to yield full
compilation reports.
5.2 Run tests during compilation
If all classes and test compile well, tests could be set up to run
silently in the background, with found failures indicated by
suggestions and glyphs in
the class view. The tests could also be run explicitly to yield full
test reports.
5.3 Single test method exec/debug
Allow to execute/debug a single test method. Currently it is
possible to exec/debug the whole test suite for a class (Ctrl+Alt+L), which
is an overkill in many situations. Developers often need to run just
single test method to diagnose a problem.
IV. Releases
This section provides mapping of the above described ideas to
implementation plans for specific releases.
NetBeans 4.2 targets
Notice: For releases beyond NB 4.2,
the JUnit planning considerations are heavily influenced by the TaskList
dev. directions -- specifically, by the overall concept and
available integration infrastructure in the TaskList module.
