Buildings are responsible for the majority of energy consumption and carbon dioxide emissions in urban areas. With the rapid urbanization and population growth, solving the intensive energy issues in the built environment is becoming increasingly crucial in achieving a more sustainable world. To this end, an increasing number of cities have adopted energy benchmarking and disclosure policies to reduce energy consumption. However, although a large volume of data has been collected, the temporal variation of the buildings' energy performance has yet to be fully analyzed. There is limited research that analyzes the panel energy disclosure data for predicting the long-term energy demand of a large scale of buildings and considers neighborhood features in the prediction. Towards addressing these knowledge gaps, this paper proposes a data-driven annual building energy demand prediction methodology. A set of building physical features and neighborhood features were selected and used for the prediction; and four ensemble learning algorithms were tested. The actual energy consumption data for more than 16,000 residential and commercial buildings in New York City were analyzed. The prediction models were evaluated in terms of coefficient of variation, mean absolute percentage of error, and R-squared. The paper discusses the proposed approach and the performance results, and identifies the potential that the long-term prediction has for informing municipal officials in making better energy supply strategies and policy decisions.