Multi-modal fake news detection has drawn great attention for their promising potential of preventing the spread of fake information in social media, and it has become an important task to be addressed due to many negative effects, such as reader misdirection, economic recession, and political volatility. In this paper, we propose a new event-categorizing neural networks (ECNN) framework, which combines a multi-modal feature extractor and an event categorizer to extract transferable features of different events for fake news detection. Moreover, a residual network is introduced to enrich the features extracted by the feature extractor. The experimental results show that the proposed ECNN model improves the accuracy and F1 scores compared to the baseline approach.