Bobbie Smulders Freelance Software Developer

Computer vision optimization when using Java

A while back, I wrote a blog post about using computer vision techniques to play an automated game of pong. This was accomplished by scanning the screen for a white pixel and moving the mouse to this position.

In that particular blog post, I didn’t really go deep into the optimization that can be done when using Java for computer vision. So this time, I’d like to try some techniques and benchmark them to find out how to shorten the execution time.

The benchmark is prepared as follows:

  • An image is loaded from disk
  • The benchmark timer starts
  • The image is converted to grayscale one hundred times
  • The benchmark timer stops
  • The image is written back to disk to manually verify the results

For each benchmark, a couple of variables are prepared:

  • imageO containing the original image as a BufferedImage object
  • imageN containing a blank target image as a BufferedImage object

Lets try some techniques, starting with a worst-case scenario:

int width = imageO.getWidth();
int height = imageO.getHeight();

for (int col = 0; col < width; col++) {
	for (int row = 0; row < height; row++) {
		Color c = new Color(imageO.getRGB(col, row), true);
		int a = c.getAlpha();
		int r = c.getRed();
		int b = c.getBlue();
		int g = c.getGreen();
		int x = (int) ((0.30 * r) + (0.59 * g) + (0.11 * b));
		Color cn = new Color(x, x, x, a);
		imageN.setRGB(col, row, cn.hashCode());
	}
}
  • Get the image width and height
  • Use two nested for-loops to iterate through all the rows and cols
  • Use getRGB() to get the current color
  • Use a Color object to get the individual colors
  • Use multiplication to calculate the gray value
  • Use another Color object to create the new color
  • Write the new color back to the new image by using setRGB

Execution time: 31.894021 seconds


The first optimization: Ditching the color object and using bit-shifting

int width = imageO.getWidth();
int height = imageO.getHeight();

for (int col = 0; col < width; col++) {
	for (int row = 0; row < height; row++) {
		int argb = imageO.getRGB(col, row);
		int a = (argb & 0xFF000000) >> 24;
		int r = (argb & 0x00FF0000) >> 16;
		int g = (argb & 0x0000FF00) >> 8;
		int b = (argb & 0x000000FF);
		int x = (int) ((0.30 * r) + (0.59 * g) + (0.11 * b));
		int cn = (a << 24) + (x << 16) + (x << 8) + x;
		imageN.setRGB(col, row, cn);
	}
}
  • Get the image width and height
  • Use two nested for-loops to iterate through all the rows and cols
  • Use getRGB() to get the current color
  • Use bit-shifting to get the individual colors
  • Use multiplication to calculate the gray value
  • Use bit-shifting to create the new color
  • Write the new color back to the new image by using setRGB

Execution time: 29.490523 seconds


The second optimization: Convert the image into an integer array and back to an image

int width = imageO.getWidth();
int height = imageO.getHeight();
int[] pixels = imageO.getRGB(0, 0, width, height, null, 0, width);

for (int col = 0; col < width; col++) {
	for (int row = 0; row < height; row++) {
		int argb = pixels[width * row + col];
		int a = (argb & 0xFF000000) >> 24;
		int r = (argb & 0x00FF0000) >> 16;
		int g = (argb & 0x0000FF00) >> 8;
		int b = (argb & 0x000000FF);
		int x = (int) ((0.30 * r) + (0.59 * g) + (0.11 * b));
		int cn = (a << 24) + (x << 16) + (x << 8) + x;
		pixels[width * row + col] = cn;
	}
}

imageN.setRGB(0, 0, width, height, pixels, 0, width);
  • Get the image width and height
  • Convert the image object to an integer array
  • Use two nested for-loops to iterate through all the rows and cols
  • Use the integer array to get the current color
  • Use bit-shifting to get the individual colors
  • Use multiplication to calculate the gray value
  • Use bit-shifting to create the new color
  • Write the new color back to the new image by using the integer array
  • Convert the integer array to an image object

Execution time: 25.176251 seconds


A very wasteful action in the optimized code above is that the image gets converted to an integer array each and every time we run the code. It would be much fairer to convert the image object to an integer array only at the initialization of the benchmark, and to exclude this conversion of the benchmark time. For the next benchmark, we prepare two extra variables:

  • pixelsO containing the original image as an integer array
  • pixelsN containing a blank target image as an integer array

The third optimization: Convert the image object to an integer array beforehand

int width = imageO.getWidth();
int height = imageO.getHeight();

for (int col = 0; col < width; col++) {
	for (int row = 0; row < height; row++) {
		int argb = pixelsO[width * row + col];
		int a = (argb & 0xFF000000) >> 24;
		int r = (argb & 0x00FF0000) >> 16;
		int g = (argb & 0x0000FF00) >> 8;
		int b = (argb & 0x000000FF);
		int x = (int) ((0.30 * r) + (0.59 * g) + (0.11 * b));
		int cn = (a << 24) + (x << 16) + (x << 8) + x;
		pixelsN[width * row + col] = cn;
	}
}
  • Get the image width and height
  • Convert the image object to an integer array
  • Use two nested for-loops to iterate through all the rows and cols
  • Use the integer array to get the current color
  • Use bit-shifting to get the individual colors
  • Use multiplication to calculate the gray value
  • Use bit-shifting to create the new color
  • Write the new color back to the new image by using the integer array
  • Convert the integer array to an image object

Execution time: 3.691897 seconds


The fourth optimization: Using a single for-loop instead of nested loops:

for (int i = 0; i < pixelsO.length; i++) {
	int argb = pixelsO[i];
	int a = (argb & 0xFF000000) >> 24;
	int r = (argb & 0x00FF0000) >> 16;
	int g = (argb & 0x0000FF00) >> 8;
	int b = (argb & 0x000000FF);
	int x = (int) ((0.30 * r) + (0.59 * g) + (0.11 * b));
	int cn = (a << 24) + (x << 16) + (x << 8) + x;
	pixelsN[i] = cn;
}
  • Get the image width and height
  • Use one nested for-loop to iterate through all the pixels
  • Use the integer array to get the current color
  • Use bit-shifting to get the individual colors
  • Use multiplication to calculate the gray value
  • Use bit-shifting to create the new color
  • Write the new color back to the new image by using the integer array

Execution time: 1.038399 seconds


Now that we’ve got all of the large performance gains in the bag, it is time to micro-optimize. The fifth optimization is to pre-declare all the variables outside the for-loop:

int i, argb, a, r, g, b, x, cn;

for (i = 0; i < pixelsO.length; i++) {
	argb = pixelsO[i];
	a = (argb & 0xFF000000) >> 24;
	r = (argb & 0x00FF0000) >> 16;
	g = (argb & 0x0000FF00) >> 8;
	b = (argb & 0x000000FF);
	x = (int) ((0.30 * r) + (0.59 * g) + (0.11 * b));
	cn = (a << 24) + (x << 16) + (x << 8) + x;
	pixelsN[i] = cn;
}
  • Declare all variables
  • Use one nested for-loop to iterate through all the pixels
  • Use the integer array to get the current color
  • Use bit-shifting to get the individual colors
  • Use multiplication to calculate the gray value
  • Use bit-shifting to create the new color
  • Write the new color back to the new image by using the integer array

Execution time: 1.03434 seconds

This was only a slightly faster method, because the Java VM already applies a lot of optimization techniques regarding variables and their declarations.


The next three optimizations all have consequences. The first of those optimizations is to not use floats to calculate the grey value but to use integers only:

int i, argb, a, r, g, b, x, cn;

for (i = 0; i < pixelsO.length; i++) {
	argb = pixelsO[i];
	a = (argb & 0xFF000000) >> 24;
	r = (argb & 0x00FF0000) >> 16;
	g = (argb & 0x0000FF00) >> 8;
	b = (argb & 0x000000FF);
	x = (int) (((30 * (r * 100)) + (59 * (g * 100)) 
			+ (11 * (b * 100))) / 10000);
	cn = (a << 24) + (x << 16) + (x << 8) + x;
	pixelsN[i] = cn;
}
  • Declare all variables
  • Use one nested for-loop to iterate through all the pixels
  • Use the integer array to get the current color
  • Use bit-shifting to get the individual colors
  • Use integer multiplication and division to calculate the gray value
  • Use bit-shifting to create the new color
  • Write the new color back to the new image by using the integer array

Execution time: 0.856214 seconds

The large consequence of using this optimization is accuracy. Integer division only returns the integer quotient and removes the remainder. Using this optimization is a question of whether or not accuracy is important.


The second optimization that can have consequences is using a lookup table:

int i, argb, a, r, g, b, x;
Integer cn;

for (i = 0; i < pixelsO.length; i++) {
	argb = pixelsO[i];
	cn = lookup.get(argb);

	if (cn == null) {
		a = (argb & 0xFF000000) >> 24;
		r = (argb & 0x00FF0000) >> 16;
		g = (argb & 0x0000FF00) >> 8;
		b = (argb & 0x000000FF);
		x = (int) ((0.30 * r) + (0.59 * g) + (0.11 * b));
		cn = (a << 24) + (x << 16) + (x << 8) + x;
		lookup.put(argb, cn);
	}

	pixelsN[i] = cn;
}
  • Declare all variables
  • Use one nested for-loop to iterate through all the pixels
  • Use the integer array to get the current color
  • Try to get the gray value from the lookup table. If there is no mapping:
    • Use bit-shifting to get the individual colors
    • Use multiplication to calculate the gray value
    • Use bit-shifting to create the new color
    • Store the result in the lookup table
  • Write the new color back to the new image by using the integer array

Execution time: 2.80093 seconds

This optimization actually performed worse in this particular case, but with images with a lot of the same colors, it might just be faster.


The third optimization that can have consequences is using the Aparapi GPGPU API. This way, we are using the GPU to calculate the gray value:

final int[] p = pixelsO;

Kernel kernel = new Kernel() {
	@Override
	public void run() {
		int i = getGlobalId();
		int argb = p[i];
		int a = (argb & 0xFF000000) >> 24;
		int r = (argb & 0x00FF0000) >> 16;
		int g = (argb & 0x0000FF00) >> 8;
		int b = (argb & 0x000000FF);
		int x = (int) ((0.30f * r) + (0.59f * g) 
				+ (0.11f * b));
		int cn = (a << 24) + (x << 16) + (x << 8) + x;
		p[i] = cn;
	}
};
kernel.execute(pixelsO.length);

pixelsN = p;
  • Declare a new final integer array containing a copy of the image pixels
  • Create a Kernel to iterate through all the pixels
  • Use the final integer array to get the current color
  • Use bit-shifting to get the individual colors
  • Use float multiplication to calculate the gray value
  • Use bit-shifting to create the new color
  • Write the new color back to the final integer array
  • Execute the kernel
  • Write the final integer array back to the new image

Execution time: 2.718005 seconds

Again, this optimization also performed worse. This is due to the fact that my current development machine doesn’t support OpenCL, causing the API to fall back to using the CPU. The results may differ on a computer capable of using OpenCL.


These are all the optimizations I could come up with. The code belonging to this benchmark can be found at the associated Github repository. If you know of any techniques that I have missed or if you think I did something wrong, don’t hesitate to use the comments section.