You dont have to change anything in IMP. Your should configure your imap server to authentify via LDAP using pam_ldap.