BuildFast

BuildFast: History-Aware Build Outcome Prediction for Fast Feedback and Reduced Cost in Continuous Integration

Long build times in continuous integration (CI) can greatly increase the cost in human and computing resources, and thus become a common barrier faced by software organizations adopting CI. Build outcome prediction has been proposed as one of the remedies to reduce such cost. However, the state-of-the-art approaches have a poor prediction performance for failed builds, and are not designed for practical usage scenarios. To address the problems,we first conduct an empirical study on 2,590,917 builds to characterize builds times in real world projects, and a survey with 75 developers to understand their perceptions about build outcome prediction. Then, motivated by our study and survey results,we propose a newhistory-aware approach,named BuildFast, to predict CI build outcomes cost-efficiently and practically. It can help to obtain fast integration feedback and reduce integration cost. In particular, we introduce multiple failure-specific features from closely related historical builds via analyzing build logs and changed files, and propose an adaptive prediction model to switch between two models based on the build outcome of the previous build. We also investigate a practical online usage scenario of BuildFast, where builds are predicted in chronological order, and measure the benefit from correct predictions and the cost from incorrect predictions. Our experiments on 20 projects have demonstrated that BuildFast can improve the state-of-the-art approach by 47.5% in F1-score for failed builds.

Survey

You can get the survey in details in here.

Features

Features about the Current Build

In this table, for fine-grained feature extractions such as class-, method-, field- and import-level changes, we use the ClDiff tool

Features about the Previous Build

Features about Historical Builds

Code and Dataset

You can download the code to extract features and train the model on github

Feature selections and different classifiers

Use different feature selections

These feature selection methods can be got on scikit-learn

BuildFast_IG&Chi2 is our feature selection approach, we adopted Chi-Squared Testing to select the top 30 features for our first model, and Information Gain to select the top 25 features for our second model.

Select From model : when training the model, we select features where the features feature importance is larger than the mean feature importances.

BuildFast_IG : we adopted Information Gain to select 30 features and 25 features for the two models respectively.

BuildFast_Chi2 : we adopted Chi-Squared Testing to select 30 features and 25 features for the two models respectively.

BuildFast_Mutual : we adopted Mutual information to select 30 features and 25 features for the two models respectively.

BuildFast_Fpr : we select the pvalues below alpha(0.01) based on a FPR test.

BuildFast_Fdr : we select the p-values for an estimated false discovery rate.

Compared with other approaches, BuildFast_IG&Chi2 improved the precision, recall and F1-score for failed builds by 4% and 2% and 3% in most cases; and for other metrics, BuildFast_IG&Chi2 slightly improved 1%-4%. We can see that BuildFast_IG&Chi2 get the most stable result for all the metrics compared with other method such as Select From model , BuildFast_Mutual. For example, Select From model slightly improved the recall-fail by 1.5% but it has a much lower pre-fail, reduced by 6.6%. For benefit, cost and gain, there was no statistically significant differencedue to the minority of failed builds and the variance of build times. Still, BuildFast_IG&Chi2 had a total gain of 2,131 hours for all projects from one-fourth of the builds (i.e., testing data) with its benefit exceeding its cost. Thus, BuildFast is cost-efficiency and can save CI cost.

Use different classifier

Compared with Randomforest , Xgboost improved the recall and F1-score for failed builds by 5.1% and 4%, thus Xgboost model contributes to the improved recall and F1-score for failed builds.