~/notes/android-camera-pipeline

You create a capture session, register an ImageReader as a target, and a few frames later you are holding image bytes. Between those two points runs a path with several different owners: the sensor, the vendor driver, the Camera HAL, the Camera Service, and the framework. This note follows a single frame along that path, explaining each layer and why it exists.

The first thing that usually surprises app developers is that the app does not drive the camera. It describes what it wants (resolution, format, exposure and focus controls) and receives buffers back. Everything else is negotiated underneath.

The intuition: a pipeline of requests and frames

The HAL3 documentation describes the subsystem like this: the API models the camera as a pipeline that turns capture requests into frames on a 1:1 basis. Each request carries the configuration for that frame (resolution, pixel format, manual sensor/lens/flash controls, 3A modes, RAW-to-YUV processing, statistics generation) plus the list of destinations where the images should land.

Think of a request as a work order: “for the next frame, use these parameters and write the result into these surfaces.” You can ask for a single frame or register a repeating request. Preview is exactly that: a repeating request that keeps going until you cancel it.

Roughly speaking, one frame travels like this:

        app  (Camera2 API, or CameraX on top of it)
         |
         |  Binder: ICameraService, ICameraDeviceUser
         v
        Camera Service  (the cameraserver process)
         |
         |  HAL interface (HIDL on older versions, AIDL on newer ones)
         v
        Camera HAL  (vendor implementation)
         |
         v
        driver / kernel + ISP
         |
         v
        image sensor

Let’s go down one layer at a time.

Layer 1: from the sensor to the driver

The sensor turns light into digital values. Most phone sensors capture each pixel through a color filter (a Bayer pattern), so the raw readout is a mosaic where each point holds only one color component. That is RAW. Turning that mosaic into a color image takes color interpolation (demosaicing), white balance, noise reduction, and color correction.

Most of that work happens in the ISP (Image Signal Processor) and the display/image block of the SoC. The kernel driver configures the sensor and the ISP according to the parameters that came in with the request.

This is where a distinction that recurs throughout the stack shows up: Android does not standardize what happens below the HAL. The standardized boundary for vendors is the HAL. The driver, the ISP, and the physical bus between sensor and SoC (usually MIPI CSI-2) are hardware and firmware decisions made by the vendor. That is why two devices with the same API behave differently in image quality, latency, and which formats they support.

Layer 2: the Camera HAL

AOSP has to run on sensors from dozens of vendors. Bundling a specific driver for each one into the base system would not scale. The solution was to define a contract: the Camera HAL. It sits between the driver and the rest of the framework and is the interface that vendors implement. The contract guarantees that, from Android’s point of view, every device exposes the camera the same way.

Under HAL3, that contract is request-driven. The framework submits requests, and each request becomes a frame captured by the sensor, processed, and delivered to the configured destinations. The HAL is what translates the abstract request parameters into concrete hardware commands, and it handles frame synchronization inside the pipeline.

The buffer memory does not belong to the HAL. The framework allocates it: up to Android 9 the framework pre-allocated buffers with the request, and from Android 10 the HAL requests them on demand from the framework (requestStreamBuffers) instead of holding a reserved set.

Two points worth keeping:

• The pipeline model is virtual. The documentation is explicit that it does not correspond directly to any real ISP. It is an abstraction that keeps vendors and sensor suppliers compatible.
• The boundary between the Camera Service and the HAL is treated as a security boundary. Parameters arriving from the service are considered untrusted, and the HAL is required to validate them before use.

On older versions the HAL could be a library loaded inside another process. Starting with Android 8.0, each binderized camera HAL runs in a process separate from the Camera Service. That is for isolation: if vendor code crashes or is compromised, it neither takes down nor contaminates the rest of the stack.

Layer 3: the Camera Service

Why not let the app talk to the HAL directly? Because the camera is a shared and sensitive resource. The Camera Service (the cameraserver process) centralizes access: it enumerates devices, decides who may open the camera, and serializes use between apps. Without that layer, every app would negotiate the hardware on its own, and nothing would stop two applications from fighting over the sensor at the same time.

The app talks to the service over Binder. ICameraService is the service’s general interface, ICameraDeviceUser is the interface for an already-opened device, and there are callback interfaces so the service can notify the app about events and the app can receive results.

It is also worth knowing that this layer was not always separate. Android 7.0 moved the Camera Service out of mediaserver specifically to harden media and camera security.

One performance detail: image buffers are not copied when they cross these boundaries. They are passed by reference through a BufferQueue, which connects whoever produces the buffers (the camera) to whoever consumes them (your app, the display, the video encoder). Copying a multi-megabyte frame at every step would be far too expensive, so the system passes only the buffer descriptor and leaves the memory where it is.

Layer 4: the Camera2 API

On the app side, the entry point is CameraManager, which lists the devices. You open a CameraDevice, create a CameraCaptureSession with the list of targets, and build a CaptureRequest.

The Camera2 API is the framework’s low-level interface, and the current recommendation for most apps is CameraX, a Jetpack library that sits on top of Camera2 and hides most of the per-device configuration. Both support Android 5.0 (API 21) and higher.

A capture skeleton for YUV frames, simplified for readability:

val reader = ImageReader.newInstance(
    width, height,
    ImageFormat.YUV_420_888,
    /* maxImages = */ 2
)

reader.setOnImageAvailableListener({ r ->
    val image = r.acquireLatestImage() ?: return@setOnImageAvailableListener
    try {
        val planes = image.planes   // always Y, U, V in that order
        val y = planes[0].buffer
        // process...
    } finally {
        image.close()               // return the buffer to the queue
    }
}, handler)

val request = cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW)
request.addTarget(reader.surface)   // the target is the ImageReader's Surface

session.setRepeatingRequest(request.build(), null, handler)

Two constraints show up quickly in practice. First, only a small number of output surfaces can be configured at once (something around three). If you want preview, recording, and analysis simultaneously, you are competing for those slots. Second, the targets have to be chosen up front: each surface receives a stream of buffers at a fixed resolution.

Layer 5: Surface, ImageReader, and buffers

A Surface is the destination where the frame is written. In practice it is the producer side of a BufferQueue: the camera writes, whoever consumes the surface reads. The system allocates the memory underneath through gralloc, Android’s graphics allocator, which picks the layout based on usage (display, video, CPU).

The ImageReader is the bridge between that buffer world and your Java/Kotlin code. It hands you a Surface that you give to the capture request and, in exchange, hands you Image objects whenever a new frame is available. The value passed as maxImages caps how many images can be in flight at the same time.

That cap is the source of a classic bug. If you do not call close() on the image after processing it, the buffer never returns to the queue and within a few frames you stop receiving data. That is why acquireLatestImage() followed by close() inside a finally is the common pattern: you take the most recent frame and drop the older ones instead of accumulating lag.

Layer 6: YUV_420_888 and where the pixels live

At this point you have an Image, not an RGB array. The format most commonly coming from the camera is YUV_420_888, and understanding the name goes a long way.

YUV separates luma (Y) from chroma (U and V, also called Cb and Cr). The “420” refers to color subsampling: brightness is stored for every pixel, but color only for each 2x2 block. Since the eye is far more sensitive to brightness than to color, this cuts the data a lot with no visible loss. The “888” means 8 bits per sample.

The API exposes this format as three planes. Plane 0 is Y, plane 1 is U, plane 2 is V, in that order. A W x H frame looks roughly like this:

YUV_420_888 (4:2:0), three planes

  plane 0  Y   W x H          one byte per pixel
  plane 1  U   W/2 x H/2      chroma Cb
  plane 2  V   W/2 x H/2      chroma Cr

memory (example of a planar layout)

  Y[0] Y[1] Y[2] ...           W*H bytes
  U[0] U[1] ...                (W/2)*(H/2) bytes
  V[0] V[1] ...                (W/2)*(H/2) bytes

Here is the part that breaks naive code. YUV_420_888 is generic on purpose: it describes any 4:2:0 planar or semiplanar buffer, but not a fully interleaved one. “Semiplanar” means U and V can be interleaved in the same memory region, one byte after the other, instead of in separate blocks. In that case planes 1 and 2 point at the same memory, just with different offsets.

So you must not assume the three planes are contiguous and tightly packed. Each plane carries two mandatory pieces of information:

• rowStride: how many bytes lie between the start of one row and the start of the next. It can be larger than the width, due to alignment.
• pixelStride: how many bytes separate two neighboring pixels within a row. For Y it is always 1. For U and V, if it is 2, the two components are interleaved.

Reading a plane means stepping rowStride at a time and taking one pixel every pixelStride. Ignore those values and you get torn images or swapped colors on some devices but not others, which makes the bug especially unpleasant to reproduce.

Since most computer vision and display libraries expect RGB, the next step is usually converting YUV to RGB. CameraX already offers this in image analysis, letting you request RGBA_8888 output instead of YUV_420_888.

Where the contract ends and the vendor begins

It helps to separate what the platform guarantees from what is an implementation choice:

Part	Who defines it
A HAL and a Camera Service exist, with their Binder interfaces	Android contract
Y, U, and V as three planes, in that order	API contract
Separate processes for the HAL, from Android 8.0	Android contract
Which resolutions and formats are supported	Device capability
Real memory layout (planar or semiplanar), strides	Vendor implementation
Image quality, latency, 3A behavior	Vendor hardware, firmware, and tuning

In the same way, some behavior shifts with the Android version. The Camera Service left mediaserver in Android 7.0, and the binderized HAL got its own process in Android 8.0. If you are debugging on an older device, that changes who is who in the process stack.

Where to watch this in a real app

If you want to see this pipeline actually running, the natural place is to instrument an app and watch what it does: which surfaces it registers, what it asks CameraManager for, which system calls show up when the buffers arrive. ARTEMIS is the site’s project that does exactly this kind of observation on Android apps, with instrumentation plus syscall and network capture.

For the official reading, these are the sources behind this note:

• Camera and Camera HAL, the architecture and the HAL3 request model.
• HAL subsystem, on the virtual pipeline, 3A, and the controls.
• Camera version support, on the move out of mediaserver in Android 7.0 and the HAL separation in Android 8.0.
• BufferQueue and Gralloc, on producer, consumer, and buffer allocation.
• Buffer management API, on who allocates buffers and the change introduced in Android 10.
• Camera2 overview and CameraX overview.
• ImageReader, Image, and ImageFormat, for planes, strides, and the YUV_420_888 format.