The challenge-response mode of the Yubikey transforms a password into another one using the hmac-sha1 algorithm, with a (secret) key stored inside.
This is the command line way, using openssl. The result is the same as that of yubikey-personalization-gui.