Draft:Streaming Gradient Boosting (SGB)

• Comment: Please do not use Wikipedia to promote your latest research paper. The entire article exists only to promote Gunasekara et al. (2024). Caleb Stanford (talk) 00:26, 1 September 2025 (UTC)

---

Streaming Gradient Boosting (SGB) is a family of online machine learning algorithms that extend gradient boosting to evolving data streams. Unlike batch boosting methods, which require multiple passes over static datasets, streaming variants update incrementally under strict memory and time constraints, while adapting to concept drift.

Background

Gradient boosting was introduced by Friedman (2000) as a stage-wise additive model that combines weak learners to minimize a differentiable loss function.. In batch learning, implementations such as XGBoost achieve state-of-the-art results by optimizing both accuracy and computational efficiency

In the data stream setting, ensemble methods such as OzaBag, Adaptive Random Forest (ARF), and Streaming Random Patches (SRP) became widely used. Boosting, however, historically underperformed for streaming tasks until recent advances in gradient-based formulations.

Streaming Gradient Boosted Trees (SGBT)

Streaming Gradient Boosted Trees (SGBT) were introduced for streaming classification by Gunasekara et al. (2024). They employ second-order Taylor approximation of the loss function, as in XGBoost, to guide tree construction incrementally.

At boosting step S, the model $\phi$ is trained to predict ${\hat {y}}_{i}$ :

${\hat {y}}_{i}=\phi (x_{i})=\sum _{s=1}^{s}f_{s}(x_{i}),\quad f_{s}\in {\mathcal {F}}$.

Here, ${\mathcal {F}}$ is the space of regressors. The loss at the $s$ th boosting step can approximated by:

${\mathcal {L}}^{(s)}\approx \sum _{i=1}^{n}\left[l(y_{i},{\hat {y}}_{i}^{(s-1)})+g_{i}f_{s}(x_{i})+{\tfrac {1}{2}}h_{i}f_{s}^{2}(x_{i})\right]+\Omega (f_{s})$

where $g_{i}$ and $h_{i}$ are the first- and second-order derivatives of the loss with respect to $s-1$-th prediction. Here, ${\mathcal {n}}$ is the size of the dataset or the stream.

Furthermore, the above loss at $s$ can be rewritten to explain it as a weighted squared loss with weight $h_{i}$ and target $g_{i}/h_{i}$:

${\mathcal {L}}^{(s)}\approx \sum _{i=1}^{n}{\frac {1}{2}}h_{i}(f_{t}(x_{i})-g_{i}/h_{i})^{2}+\Omega (f_{s})+constant.$

Above weight $h_{i}$ and target $g_{i}/h_{i}$ can be used to train an incremental regression tree at $s$.

Empirical studies show that SGBT surpasses random subspace and random patch ensemble methods in streaming classification under different drift conditions.

Streaming Gradient Boosted Regression (SGBR)

Regression with streaming gradient boosting poses additional challenges. Vanilla SGBT with squared loss exhibits high variance, leading to poor predictive performance. To address this, Gunasekara et al. (2025) proposed Streaming Gradient Boosted Regression (SGBR), which integrates bagging-based streaming regressors to reduce variance.

Two main designs are described:

• SGB(Bag): bagged streaming regressors are used as base learners within the boosting framework. Variants include:
  • SGB(Oza) – employs the Oza Bag Regressor
  • SGB(ARF) – employs Adaptive Random Forest Regressor
  • SGB(SRP) – employs Streaming Random Patches Regressor

• Bag(SGBT): ensembles of boosted models are combined via bagging (e.g. Oza(SGBT), SRP(SGBT)).

Advantages of SGB(Oza)

Among SGBR variants, SGB(Oza) has shown the best performance. Key advantages include:

• Variance reduction: Bagging mitigates boosting’s tendency toward variance inflation in regression.
• Predictive accuracy: On 11 benchmark datasets with concept drift, SGB(Oza) achieved higher adjusted $R^{2}$ scores than state-of-the-art regressors such as Self-Optimising k-Nearest Leaves (SOKNL).
• Computational efficiency: Unlike Bag(SGBT) variants, SGB(Oza) achieves these gains without added time complexity.
• Drift robustness: Maintains stable performance across abrupt, gradual, and recurrent drift scenarios.

Applications

Streaming gradient boosting methods are applicable to real-time regression tasks such as:

• financial forecasting
• sensor network monitoring
• online energy demand prediction
• traffic volume prediction
• large-scale retail demand forecasting

Software Implementations

Software implementations of SGBT and SGBR are available in MOA (Java) and CapyMOA (Python) stream learning platforms.