With the development of large-thrust liquid rocket engines, the behavior of liquid in supercritical conditions arouses increasing public interest. Due to the high pressure and temperature of the combustion chamber, fuel reaches its critical point much more easily, and enters supercritical conditions. Due to the drastic changes in the physical properties of the fluid near the critical point, it is usually difficult to simulate the fluid motion using traditional computational fluid dynamic methods; but molecular dynamics (MD) can simulate fluid motion at the molecular level. In view of the engineering application, the physical properties of a binary system consisting of argon and nitrogen, and the stability of subcritical jets sprayed into supercritical environment, has been studied here using the MD method. First, the molecular dynamic simulation of the equation of state (EOS) of the mixture was put forward. Four conditions, with different mixing ratios of nitrogen, were designed. The results showed that the mixing ratio of nitrogen noticeably affected the results; these results were compared with the Soave-Redich-Kwong (SRK) EOS. Second, a simulation was conducted of subcritical nitrogen jet sprayed into a supercritical argon environment. After analyzing the results, the jet density and temperature distributions were obtained and the disturbance growth rate of the shear layer was analyzed.