Urban water managers rely heavily on forecasts of water consumption to determine management decisions and investment choices. Typical forecasts rely on simple models whose criteria for selection has little to do with their performance in predicting out-of-sample consumption levels. We demonstrate this issue by comparing forecast models selected on the basis of their ability to perform well in-sample versus out-of-sample. Our results highlight the benefits of developing out-of-sample evaluation criteria to ascertain model performance. Using annual data on single-family residential water consumption in Southern California we illustrate how prediction ability varies according to model evaluation method. Using a training dataset, this analysis finds that models ranking highly on in-sample performance significantly over-estimated consumption (10% − 25%) five years out from the end of the training dataset relative to observed demands five years out from the end of the training dataset. Whereas, the top models selected using our out-of-sample criteria, came within 1% of the actual total consumption. Notably, projections of future demand for the in-sample models indicate increasing aggregate water consumption over a 25-year period, which contrasts against the downward trend predicted by the out-of-sample models.