Human respiratory viruses are of vastly different virulence, giving rise to symptoms ranging from common cold to severe pneumonia or even death. Although this most likely impacts molecular evolution of the corresponding viruses, the specific differences in their evolutionary patterns remain largely unknown. By comparing structural and nonstructural genes within respiratory viruses, greater similarities in codon usage bias (CUB) between nonstructural genes and humans were observed in weakly virulent viruses, whereas in strongly virulent viruses, it was structural genes whose CUBs were more similar to that of humans. Further comparisons between genomes of weakly and strongly virulent coronaviruses revealed greater similarities in CUBs between strongly virulent viruses and humans. Finally, using phylogenetic independent contrasts, dissimilation of viral CUB from that of humans was observed in SARS-CoV-2. Our work revealed distinct CUB evolutionary patterns between weakly and strongly virulent viruses, a previously unrecognized interaction between CUB and virulence in respiratory viruses.