Abstract Sparse representation is considered an important coding strategy for cortical processing in various sensory modalities. It remains unclear how cortical sparseness arises and is being regulated. Here, unbiased recordings from primary auditory cortex of awake adult mice revealed salient sparseness in layer (L)2/3, with a majority of excitatory neurons exhibiting no increased spiking in response to each of sound types tested. Sparse representation was not observed in parvalbumin (PV) inhibitory neurons. The nonresponding neurons did receive auditory-evoked synaptic inputs, marked by weaker excitation and lower excitation/inhibition (E/I) ratios than responding cells. Sparse representation arises during development in an experience-dependent manner, accompanied by differential changes of excitatory input strength and a transition from unimodal to bimodal distribution of E/I ratios. Sparseness level could be reduced by suppressing PV or L1 inhibitory neurons. Thus, sparse representation may be dynamically regulated via modulating E/I balance, optimizing cortical representation of the external sensory world.