Not only does the destruction caused by natural disasters impair human lives, but it can also result in devastating damages to the community infrastructure and possibly cause the loss of historic structures as well as vital documents. Technological advances in remote sensing survey tools such as satellite images and aerial photographs have allowed emergency responders to rapidly and remotely conduct a comprehensive assessment of the damages caused by a disaster event. Most of the previously proposed research in the automatic identification and prediction of building damage assessments from optical remote sensing data depends on the availability of accurate geometric footprints of the affected area’s structures. However, the available building footprints may rapidly become outdated as new infrastructures are built while old ones are demolished or renovated. We propose an end-to-end weakly-supervised damage assessment model where the assumption is that the building footprint is unknown during training.Instead, there is a rough estimate of the building’s location and the level of damage it sustained. Ablation tests are conducted on both a large-scale satellite imagery set and a smaller set of aerial photographs prepared and curated by our team to demonstrate our proposed model’s performance.