If no enemy character steps forward to meet the challenge, one of them must retire in ignominy. This character is nominated by the challenger (though they may not nominate a character that could not have accepted the challenge). The retiring character slinks off to the back ranks and is not allowed to attack that round – move the model into a rank where they are not in base contact with the enemy. Another model will step up and fight in their place, just as if they had been slain. Furthermore, the model's Leadership cannot be used for any Leadership tests that take place that turn. The character may return to the front rank in the next round of combat if you wish, or stay in the rear rank until the unit reforms.
Once a challenge has been refused, the issuer can fight normally in that round of combat.