Despite their high energy consumption, rural residential houses in the hot summer and cold winter (HSCW) zone still have a generally poor indoor thermal environment. The objective of this study was to understand the current status of the indoor thermal environment for rural residential houses in the HSCW zone and analyze its cause in order to develop some strategies for improvement through passive design of the building envelope. Face-to-face questionnaires and interviews, air-tightness testing, and temperature and humidity monitoring were conducted to understand the building envelope, energy consumption, and indoor thermal environment. Then, some passive design strategies were simulated, including the application of functional interior materials such as hygroscopic and phase change materials. An overall passive design for the building envelope can increase the room temperature by 3.6 °C, reduce the indoor relative humidity by 12% in the winter, and reduce the room temperature by 4.4 °C in the summer. In addition, the annual energy-saving rate can reach ~35%.