When one very short empty time interval follows right after another, the second one can be underestimated considerably, but only if it is longer than the first one. We coined the term "time-shrinking" for this illusory phenomenon in our previous studies. Although we could relate our finding to some studies of rhythm perception, we were not able to explain the illusion. The present article presents our attempt to understand the mechanism that causes the time-shrinking. Four experiments are reported. The first one ruled out the possibility that the illusion results from a difficulty in resolving the temporal structure. The second experiment showed that the listener was not inadvertently judging the duration of the first interval instead of that of the second one. In addition, this experiment yielded more information about the time window within which the illusion occurs. The third experiment showed that forward masking of the sound markers, delimiting the empty durations, could not explain the illusion either. Furthermore, this experiment revealed a clue to the mechanism of time-shrinking: competition between expected and observed temporal positions. The fourth experiment further examined the temporal conditions that give rise to the illusion and showed that categorical perception plays a crucial role in the formation of the illusion. In the general discussion, we argue that the illusion is due to an asymmetric process of temporal assimilation.