Skip to main content
SpringerLink
Log in

Adaptive spline fitting with particle swarm optimization

  • Original paper
  • Published:
Computational Statistics Aims and scope Submit manuscript

Abstract

In fitting data with a spline, finding the optimal placement of knots can significantly improve the quality of the fit. However, the challenging high-dimensional and non-convex optimization problem associated with completely free knot placement has been a major roadblock in using this approach. We present a method that uses particle swarm optimization (PSO) combined with model selection to address this challenge. The problem of overfitting due to knot clustering that accompanies free knot placement is mitigated in this method by explicit regularization, resulting in a significantly improved performance on highly noisy data. The principal design choices available in the method are delineated and a statistically rigorous study of their effect on performance is carried out using simulated data and a wide variety of benchmark functions. Our results demonstrate that PSO-based free knot placement leads to a viable and flexible adaptive spline fitting approach that allows the fitting of both smooth and non-smooth functions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  • Akaike H (1998) Information theory and an extension of the maximum likelihood principle. In: Selected papers of Hirotugu Akaike. Springer, Berlin, pp 199–213

  • Bratton D, Kennedy J, (2007) Defining a standard for particle swarm optimization. In: Swarm intelligence symposium (SIS). IEEE, pp 120–127

  • Burchard HG (1974) Splines (with optimal knots) are better. Appl Anal 3(4):309–319

    MathSciNet  MATH  Google Scholar 

  • Claeskens G, Krivobokova T, Opsomer JD (2009) Asymptotic properties of penalized spline estimators. Biometrika 96(3):529–544

    MathSciNet  MATH  Google Scholar 

  • Craven P, Wahba G (1978) Smoothing noisy data with spline functions. Numer Math 31(4):377–403

    MathSciNet  MATH  Google Scholar 

  • Curry HB, Schoenberg IJ (1947) On spline distributions and their limits—the polya distribution functions. Bull Am Math Soc 53(11):1114–1114

    Google Scholar 

  • de Boor C (1972) On calculating with b-splines. J Approx Theory 6(1):50–62

    MathSciNet  MATH  Google Scholar 

  • de Boor C (2001) A practical guide to splines (applied mathematical sciences). Springer, Berlin

    MATH  Google Scholar 

  • Denison D, Mallick B, Smith A (1998) Automatic Bayesian curve fitting. J R Stat Soc Ser B (Stat Methodol) 60(2):333–350

    MathSciNet  MATH  Google Scholar 

  • DiMatteo I, Genovese CR, Kass RE (2001) Bayesian curve-fitting with free-knot splines. Biometrika 88(4):1055–1071

    MathSciNet  MATH  Google Scholar 

  • Donoho DL, Johnstone IM (1995) Adapting to unknown smoothness via wavelet shrinkage. J Am Stat Assoc 90(432):1200–1224

    MathSciNet  MATH  Google Scholar 

  • Eilers PH, Marx BD (1996) Flexible smoothing with b-splines and penalties. Stat Sci 11:89–102

    MathSciNet  MATH  Google Scholar 

  • Eilers PH, Marx BD, Durbán M (2015) Twenty years of p-splines. SORT: Stat Oper Res Trans 39(2):0149–186

    MathSciNet  MATH  Google Scholar 

  • Engelbrecht AP (2005) Fundamentals of computational swarm intelligence, vol 1. Wiley, Chichester

    Google Scholar 

  • Friedman JH (1991) Multivariate adaptive regression splines. Ann Stat 19(1):1–67

    MathSciNet  MATH  Google Scholar 

  • Friedman JH, Silverman BW (1989) Flexible parsimonious smoothing and additive modeling. Technometrics 31(1):3–21

    MathSciNet  MATH  Google Scholar 

  • Gálvez A, Iglesias A (2011) Efficient particle swarm optimization approach for data fitting with free knot b-splines. Comput Aided Des 43(12):1683–1692

    Google Scholar 

  • Goepp V, Bouaziz O, Nuel G (2018) Spline regression with automatic knot selection. arXiv preprint arXiv:1808.01770

  • Golub GH, Heath M, Wahba G (1979) Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 21(2):215–223

    MathSciNet  MATH  Google Scholar 

  • Green PJ (1995) Reversible jump markov chain monte carlo computation and bayesian model determination. Biometrika 82(4):711–732

    MathSciNet  MATH  Google Scholar 

  • Härdle W (1990) Applied nonparametric regression, no. 19. Cambridge University Press, Cambridge

  • Huo X, Duncan M, Levi O, Buckheit J, Chen S, Donoho D, Johnstone I, (2000). http://statweb.stanford.edu/~wavelab/Wavelab_850/AboutWaveLab.pdf

  • Jupp DL (1978) Approximation to data by splines with free knots. SIAM J Numer Anal 15(2):328–343

    MathSciNet  MATH  Google Scholar 

  • Kang H, Chen F, Li Y, Deng J, Yang Z (2015) Knot calculation for spline fitting via sparse optimization. Comput Aided Des 58:179–188

    MathSciNet  Google Scholar 

  • Kennedy J (1999) Small worlds and mega-minds: effects of neighborhood topology on particle swarm performance. In: Proceedings of the 1999 congress on evolutionary computation-CEC99 (Cat. No. 99TH8406), vol 3. IEEE, pp 1931–1938

  • Kennedy J, Eberhart RC (1995) Particle swarm optimization. In: Proceedings of the IEEE international conference on neural networks, Perth, WA, Australia, vol 4. IEEE, p 1942

  • Krivobokova T (2013) Smoothing parameter selection in two frameworks for penalized splines. J R Stat Soc Ser B (Stat Methodol) 75(4):725–741

    MathSciNet  MATH  Google Scholar 

  • Krivobokova T, Crainiceanu CM, Kauermann G (2008) Fast adaptive penalized splines. J Comput Graph Stat 17(1):1–20

    MathSciNet  Google Scholar 

  • Lee T (2002) On algorithms for ordinary least squares regression spline fitting: a comparative study. J Stat Comput Simul 72(8):647–663

    MathSciNet  MATH  Google Scholar 

  • Leung C (2015) Estimation of unmodeled gravitational wave transients with spline regression and particle swarm optimization. SIAM Undergraduate Research Online (SIURO) 8

  • Li M, Yan Y (2006) Bayesian adaptive penalized splines. J Acad Bus Econ 2:129–141

    Google Scholar 

  • Lindstrom MJ (1999) Penalized estimation of free-knot splines. J Comput Graph Stat 8(2):333–352

    MathSciNet  Google Scholar 

  • Liu Z, Guo W (2010) Data driven adaptive spline smoothing. Stat Sin 20:1143–1163

    MathSciNet  MATH  Google Scholar 

  • Luo Z, Wahba G (1997) Hybrid adaptive splines. J Am Stat Assoc 92(437):107–116

    MathSciNet  MATH  Google Scholar 

  • Luo J, Kang H, Yang Z (2019) Knot calculation for spline fitting based on the unimodality property. Comput Aided Geom Des 73:54–69

    MathSciNet  MATH  Google Scholar 

  • Lyche T, Mørken K (1988) A data-reduction strategy for splines with applications to the approximation of functions and data. IMA J Numer Anal 8(2):185–208

    MathSciNet  MATH  Google Scholar 

  • Matlab Release (2018b) The MathWorks, Inc., Natick, MA, United States. http://www.mathworks.com/

  • Mitchell M (1998) An introduction to genetic algorithms by Melanie Mitchell. MIT Press, Cambridge

  • Miyata S, Shen X (2003) Adaptive free-knot splines. J Comput Graph Stat 12(1):197–213

    MathSciNet  Google Scholar 

  • Mohanty SD (2012) Particle swarm optimization and regression analysis I. Astron Rev 7(2):29–35

    Google Scholar 

  • Mohanty SD (2017) Spline based search method for unmodeled transient gravitational wave chirps. Phys Rev D 96:102008. https://doi.org/10.1103/PhysRevD.96.102008

    Article  Google Scholar 

  • Mohanty SD (2018) Swarm intelligence methods for statistical regression. Chapman and Hall/CRC, London

    MATH  Google Scholar 

  • Normandin ME, Mohanty SD, Weerathunga TS (2018) Particle swarm optimization based search for gravitational waves from compact binary coalescences: performance improvements. Phys Rev D 98(4):044029

    Google Scholar 

  • Park H, Lee J-H (2007) B-spline curve fitting based on adaptive curve refinement using dominant points. Comput Aided Des 39(6):439–451

    Google Scholar 

  • Pittman J (2002) Adaptive splines and genetic algorithms. J Comput Graph Stat 11(3):615–638

    MathSciNet  Google Scholar 

  • R Core Team, R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria (2019). https://www.R-project.org/

  • Ramsay J, Silverman BW (1997) Functional data analysis. Springer series in statistics. Springer, New York

    MATH  Google Scholar 

  • Reinsch CH (1967) Smoothing by spline functions. Numer Math 10(3):177–183

    MathSciNet  MATH  Google Scholar 

  • Ruppert D (2002) Selecting the number of knots for penalized splines. J Comput Graph Stat 11(4):735–757

    MathSciNet  Google Scholar 

  • Ruppert D, Carroll RJ (2000) Theory and methods: spatially-adaptive penalties for spline fitting. Austr N Z J Stat 42(2):205–223

    Google Scholar 

  • Ruppert D, Wand MP, Carroll RJ (2003) Semiparametric regression, vol 12. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  • Scheipl F, Kneib T (2009) Locally adaptive bayesian p-splines with a normal-exponential-gamma prior. Comput Stat Data Anal 53(10):3533–3552

    MathSciNet  MATH  Google Scholar 

  • Smith PL (1982) Curve fitting and modeling with splines using statistical variable selection techniques. Technical report, NASA, Langley Research Center, Hampton, VA, report NASA, 166034

  • Solis FJ, Wets RJ-B (1981) Minimization by random search techniques. Math Oper Res 6(1):19–30

    MathSciNet  MATH  Google Scholar 

  • Stone CJ, Hansen MH, Kooperberg C, Truong YK et al (1997) Polynomial splines and their tensor products in extended linear modeling: 1994 Wald memorial lecture. Ann Stat 25(4):1371–1470

    MATH  Google Scholar 

  • Storlie CB, Bondell HD, Reich BJ (2010) A locally adaptive penalty for estimation of functions with varying roughness. J Comput Graph Stat 19(3):569–589

    MathSciNet  Google Scholar 

  • Ülker E, Arslan A (2009) Automatic knot adjustment using an artificial immune system for b-spline curve approximation. Inf Sci 179(10):1483–1494

    Google Scholar 

  • Wahba G (2002) In discussion Wavelet shrinkage: Asymptopia? with discussion and a reply by Donoho DL, Johnstone IM, Kerkyacharian G, Picard D, J R Stat Soc Ser B 57: 545–564

  • Wahba G (1990) Spline models for observational data, vol 59. SIAM, Philadelphia

    MATH  Google Scholar 

  • Wand MP (2000) A comparison of regression spline smoothing procedures. Comput Stat 15(4):443–462

    MathSciNet  MATH  Google Scholar 

  • Wang X, Du P, Shen J (2013) Smoothing splines with varying smoothing parameter. Biometrika 100(4):955–970

    MathSciNet  MATH  Google Scholar 

  • Wegman EJ, Wright IW (1983) Splines in statistics. J Am Stat Assoc 78(382):351–365

    MathSciNet  MATH  Google Scholar 

  • Wold S (1974) Spline functions in data analysis. Technometrics 16(1):1–11

    MathSciNet  MATH  Google Scholar 

  • Yang L, Hong Y (2017) Adaptive penalized splines for data smoothing. Comput Stat Data Anal 108:70–83

    MathSciNet  MATH  Google Scholar 

  • Yoshimoto F, Harada T, Yoshimoto Y (2003) Data fitting with a spline using a real-coded genetic algorithm. Comput Aided Des 35(8):751–760

    Google Scholar 

  • Zhou S, Shen X (2001) Spatially adaptive regression splines and accurate knot selection schemes. J Am Stat Assoc 96(453):247–259

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The contribution of S.D.M. to this paper was supported by National Science Foundation (NSF) Grant PHY-1505861. The contribution of E.F. to this paper was supported by NSF Grant PHY-1757830. We acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin (www.tacc.utexas.edu) for providing HPC resources that have contributed to the research results reported within this paper.

Author information

Authors and Affiliations

  1. Department of Physics and Astronomy, The University of Texas Rio Grande Valley, One West University Blvd., Brownsville, TX, 78520, USA

    Soumya D. Mohanty

  2. Department of Physics and Astronomy, The University of Rochester, 500 Wilson Blvd., Rochester, NY, 14627, USA

    Ethan Fahnestock

Authors
  1. Soumya D. Mohanty
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ethan Fahnestock
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Soumya D. Mohanty.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mohanty, S.D., Fahnestock, E. Adaptive spline fitting with particle swarm optimization. Comput Stat 36, 155–191 (2021). https://doi.org/10.1007/s00180-020-01022-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00180-020-01022-x

Search

Navigation