Integer overflow

What is integer overflow?

Integer overflow is a vulnerability that lets a malicious hacker trick the program into performing an integer operation whose result exceeds the allocated memory space. Apart from causing unexpected program behavior, this can also lead to the much more dangerous buffer overflow.

The integer overflow vulnerability was listed by CWE (the Common Weakness Enumeration system) in 2022 at position 13, outranking several major web vulnerabilities.

Severity:		severe
Prevalence:		discovered rarely
Scope:		applications with direct memory allocation
Technical impact:		unexpected program behavior
Worst-case consequences:		buffer overflow
Quick fix:		examine the operands and results

How does integer overflow work?

In most programming languages, integer values are allocated a fixed number of bits in memory. For example, space reserved for a 32-bit integer data type may hold an unsigned integer between 0 and 4,294,967,295 or a signed integer between −2,147,483,648 and 2,147,483,647. For signed integers, the most significant (first) bit usually indicates whether the integer is a positive value or a negative value.

But what happens when you perform a calculation like 4,294,967,295 + 1 and attempt to store the result as normal even though it exceeds the maximum value for the integer type? The behavior depends completely on the language and the compiler. Most languages and compilers raise no error at all – they might perform a modulo operation, wraparound, or truncation, or have other undefined behavior. In most cases, the result of the above increment operation will be 0.

With signed integer overflows, the results can be even more unexpected than for unsigned int. When you go above the maximum value for a signed integer, the result usually becomes a negative number. For example, 2,147,483,647 +1 is usually −2,147,483,648. When you go below the minimum value of a negative number (integer underflow), the result usually becomes a positive number. For example, −2,147,483,648 − 1 would give you 2,147,483,647.

In addition to typical operations such as addition, subtraction, or multiplication, integer overflows may also happen due to typecasting. For example, one operation may treat the same integer as unsigned and another operation as signed, therefore interpreting the value incorrectly.

If you want to know more about integer overflows, we recommend the extensive Phrack article by blexim.

Example of an integer overflow attack

An excellent example of an integer overflow is a real-life vulnerability in an older version of OpenSSH (3.3). The following code snippet comes directly from this software:

nresp = packet_get_int();
if (nresp > 0) {
 response = xmalloc(nresp*sizeof(char*));
 for (i = 0; i < nresp; i++)
  response[i] = packet_get_string(NULL);
}

If nresp is 1073741824 and sizeof(char*) is 4 (which is the typical size of one character), then calculating the buffer size using nresp*sizeof(char*) gives a zero size, resulting in an overflow. Therefore, xmalloc() receives and allocates a 0-byte buffer. The subsequent loop causes a heap buffer overflow, which may, in turn, be used by an attacker to execute arbitrary code.

Another example of a known, recent, and serious integer overflow vulnerability is CVE-2022-36934 in WhatsApp, which allows the attacker to follow up with remote code execution in an established video call.

Potential consequences of integer overflow attacks

Most integer overflow conditions lead to erroneous program behavior but do not cause vulnerabilities. However, for some applications and algorithms, integer overflows may have severe consequences:

• If an integer overflow happens when you calculate the length of a buffer, you may end up with a buffer overflow. A buffer overflow lets the attacker gain shell access and attempt further privilege escalation.
• If an integer overflow happens during financial calculations, it may, for example, result in a customer receiving credit instead of paying for a purchase or may cause a negative account balance to become positive.

How to detect integer overflows?

The best way to detect integer overflow vulnerabilities depends on whether they are already known or unknown.

• If you do not develop software of your own, it may be enough to identify the exact version of the existing software you are using. If the identified version is susceptible to integer overflow, you can assume that your software is vulnerable. You can identify the version manually or use a suitable security tool, such as a software composition analysis (SCA) solution, a network scanner, or Acunetix by Invicti, which finds integer overflow vulnerabilities in web servers and more.
• If you develop your own software, you need to conduct a manual static analysis of the code and debug at runtime to find potential integer overflow vulnerabilities. However, this is not an easy process. The biggest issue with even the most basic integer overflows is that there is no error or warning, only an incorrect result for some operations. The only way to discover integer overflow is to examine the operands before the operation or examine the final result (for example, to check whether the sum of two positive number operands is smaller than the input values).

How to prevent integer overflows?

Depending on the language, you may be able to come across frameworks, libraries, or mechanisms that help you prevent integer overflows.

• The GCC compiler comes with built-in functions that check for integer overflows.
• For C++ programs, there is a library called SafeInt that performs integer operations in a safe way. The downside, as with all external libraries, is that it’s not always possible to enforce its use in all your code and for all arithmetic operations.

In languages that provide different integer types to allocate different amounts of memory based on the size of the integer, programmers can prevent integer overflows by taking care to always choose a type big enough to accommodate all potential results. Some programming languages, such as Python, make such allocations automatically, preventing most integer overflows by design.