We have evaluated RefactoringCrawler on three real-world components. We chose the components that had good release notes describing the API changes; these notes helped us to manually check the correctness of the logs of refactorings produces by RefactoringCrawler. In general, it is easier to spot the false positives (refactorings erroneously reported by RefactoringCrawler) by comparing the refactoring against the source code than it is to detect the false negatives (refactorings that RefactoringCrawler missed). Sometimes the description of the change in release notes would map 1-to-1 with refactorings defined in refactoring catalogs. Other times, the release notes would be vague like “we eliminated duplication in class X”. For these cases, we had to look in the source code to find out what the exact type of change was. This extensive manual analysis allowed us to build a repository of refactorings that happened between the two versions. We compare these manually found refactorings against the refactorings that were found by Refactoring Crawler to determine the false negatives. For each component we chose for comparison two major releases that span large architectural changes. There are two benefits to choosing major releases as comparison points. First, it is likely that there will be lots of changes between the two versions. Second, it is likely that those changes will be documented thus providing some starting point for a detailed analysis of the changes.
For each case study we provide the log of expected refactorings found by digging through the release notes. To this log we added those confirmed refactorings that were found by the RefactoringCrawler but were not documented. The log of actual refactorings contains those refactorings found by RefactoringCrawler (including false positives). One can download the same versions that we used for each case study component and import them as Eclipse projects. Once you donwloaded the archives below, use File/Import.../Existing Projects Into Workspace/Select Archive File/Finish. Repeat the process to import the second version of the component as an Eclipse project. Now you can start RefactoringCrawler from the RefactoringCrawler menu/ Detect Refactorings. First select the projects that represent the two versions of the component you want to analyze. By selecting the "Change Default Values" you can play with different threshold values for the detection strategies. For most applications the default values should be good enough, that is the reason why they are displayed initially greyed. The results for the evaluations below were obtained using these settings values:
You can download the Eclipse projects containing JHotDraw version 5.2 and version 5.3
The expected results are here. The actual results are here. Both the precission and recall for this case study are 100%.
Download the Eclipse projects containing Struts version 1.1 and version 1.2.4.
The expected results are here. The actual results are here. Using the above settings, precision is 100% while recall is 86%.
Download the Eclipse projects containing EclipseUI component version 2.1.3 and version 3.0
For this case study you will need to enable the "Use Javadoc Comments" option in the left pane of the settings window, since EclipseUI comes with many changes to its interfaces. The expected results are here. The actual results are here. Using the above settings, precision is 90% and recall is 86%.