Neural radiance fields (NeRFs) have recently attracted significant attention in the realm of wireless channel prediction, primarily due to their capacity for high-fidelity reconstructions of complex wireless measurement environments. However, the ray-tracing component of NeRF-based techniques faces obstacles in realistically representing wireless scenarios, largely because the expressive power of multilayer perceptrons (MLP) remains limited. To address this challenge, in this paper, we propose NeRF-APT, an encoder-decoder architecture integrated into the NeRF-based wireless channel prediction framework. This architecture leverages the strengths of NeRF-like models in learning environmental features while capitalizing on encoder-decoder modules' capacity to extract critical information. Furthermore, we ultilize the attention mechanism into the skip connections between encoder-decoder structures, significantly improving contextual understanding across layers. Extensive experimental evaluations on several realistic and synthetic datasets demonstrate that our approach outperforms state-of-the-art methods in wireless channel prediction.