The transformation of the legacy electric grid to Smart Grid (SG) poses numerous challenges in the design and development of an efficient SG communications network. While there has been an increasing interest in identifying the SG communications network and possible SG applications, specific research challenges at the network protocol have not been elaborated yet. This dissertation revisited each layer of a TCP/IP protocol stack which basically was designed for a wired network and optimized their performance in IEEE 802.11s-based Advanced Metering Infrastructure (AMI) communications network against the following challenges: security and privacy, AMI data explosion, periodic simultaneous data reporting scheduling, poor Transport Control Protocol (TCP) performance, Address Resolution Protocol (ARP) broadcast, and network interoperability. To address these challenges, layered and/or cross-layered protocol improvements were proposed for each layer of TCP/IP protocol stack. At the application layer, a tree-based periodic time schedule and a time division multiple access-based scheduling were proposed to reduce high contention when smart meters simultaneously send their reading. Homomorphic encryption performance was investigated to handle AMI data explosion while providing security and privacy. At the transport layer, a tree-based fixed Retransmission Timeout (RTO) setting and a path-error aware RTO that exploits rich information of IEEE 802.11s data-link layer path selection were proposed to address higher delay due to TCP mechanisms. At the network layer, ARP requests create broadcast storm problems in IEEE 802.11s due to the use of MAC addresses for routing. A secure piggybacking-based ARP was proposed to eliminate this issue. The tunneling mechanisms in the LTE network cause a downlink traffic problem to IEEE 802.11s. For the network interoperability, at the network layer of EPC network, a novel UE access list was proposed to address this issue. At the data-link layer, to handle QoS mismatch between IEEE 802.11s and LTE network, Dual Queues approach was proposed for the Enhanced Distributed Channel Access. The effectiveness of all proposed approaches was validated through extensive simulation experiments using a network simulator. The simulation results showed that the proposed approaches outperformed the traditional TCP/IP protocols in terms of end to end delay, packet delivery ratio, throughput, and collection time.