Unchecked Input for Loop Condition

Summary

Checking excessively long invalid RSA public keys may take a long time. Applications that use the function "EVP_PKEY_public_check()" to check RSA public keys may experience long delays. Where the key that is being checked has been obtained from an untrusted source, this may lead to a Denial of Service (DOS). When function "EVP_PKEY_public_check()" is called on RSA public keys, a computation is done to confirm that the RSA modulus, n, is composite. For valid RSA keys, n is a product of two or more large primes, and this computation completes quickly. However, if n is an overly large prime, then this computation would take a long time. An application that calls "EVP_PKEY_public_check()" and supplies an RSA key obtained from an untrusted source could be vulnerable to a Denial of Service (DOS) attack. The function "EVP_PKEY_public_check()" is not called from other OpenSSL functions; however, it is called from the OpenSSL "pkey" command line application. For that reason, that application is also vulnerable if used with the "-pubin" and "-check" options on untrusted data. The OpenSSL SSL/TLS implementation is not affected by this issue. The OpenSSL 3.0.0-alpha1 through 3.0.12, 3.1.0-alpha1 through 3.1.4, and 3.2.0-alpha1 through 3.2.0 FIPS providers are affected by this issue.