The challenge-response mode of the Yubikey transforms a password into another one using the hmac-sha1 algorithm, with a (secret) key stored inside.
This is the command line way, using openssl. The result is the same as that of yubikey-personalization-gui.

#!/bin/sh
echo -n "challenge: " ; read challenge
echo -n "hexkey: " ; read hexkey
echo
echo $challenge
echo -n "$challenge" | openssl dgst -sha1 -mac HMAC -macopt hexkey:$(echo "$hexkey" | tr -d ' ') | sed 's#^(stdin)= ##'