The problem

The pi camera video encoder has a fixed maximum bitrate. Normally (by default), you set a target bitrate lower than this limit, and the encoder lowers JPEG quality to stay below this bitrate.

However, fast autofocus relies on a constant JPEG frame quality to work, and so we by default disable bitrate control for the video stream.

At our default resolution, framerate, and quality, it is possible to image a sample with enough fine (high frequency) detail that this bitrate limit is exceeded, and the GPU outputs incomplete frames.

This manifests itself as partially or completely white frames, as reported on our forum.

Forum post

Camera feed going white

Some more investigation:

10. API - picamera.camera Module — Picamera 1.10 documentation

The bitrate at which video will be encoded. Defaults to 17000000 (17Mbps) if not specified. The maximum value is 25000000 (25Mbps). Bitrate 0 indicates the encoder should not use bitrate control (the encoder is limited by the quality only).

That should say -1 disables bitrate control. This is our "maximum" bitrate value, and leads to the best fast-autofocus performance (since the JPEG frame quality is constant), and as such is the default.

What I now think may be happening is the following:

I established a while ago that the quality parameter in the video encoder has a negligible, if any, effect on the MJPEG frames. We assume that the MJPEG quality is fixed, and high by default.
Lowering the bitrate actually does cause the encoder to lower the JPEG frame quality.
Disabling bitrate control then means that the encoder will attempt to output high quality frames with no regard for its own bitrate limit. Therefore, in situations where the frame is highly detailed, the required bitrate may exceed the camera limit, resulting in incomplete or missing frames coming out of the encoder

This may be tricky to really solve long-term. I can add an interface to change the stream resolution to see if that helps. Otherwise, for now we may just have to deal with the tradeoff between bitrate and autofocus reliability.

Edit: A nice test for if this issue is occurring is to check if the end of a frame is the correct JPEG end of image bytes (0xFF, 0xD9). If it doesn't end with that then it's been truncated by bitrate limits.

Solutions

Lower the fixed MJPEG quality (nope)

We could add an option to lower the fixed stream JPEG quality, thus lowering the bitrate while maintaining a constant quality. However, contrary to the documentation, this quality option does absolutely nothing in the PiCamera MJPEG video encoder.

Limit the target bitrate

I have added an option to set a bitrate target. The setting is under the Advanced section of camera settings. It includes a message explaining that this may affect autofocus reliability.

My quick testing shows that fast-AF seems to still work well, however for detailed samples the JPEG frame quality may change during the autofocus routine, causing it to perform poorly depending on the exact sample, configuration, and conditions.

This is guaranteed to fix the white-stream issue.

Limit the stream framerate

We can reduce the stream bitrate by reducing the camera acquisition framerate. By default, we use 30fps.

I have added an option to change the camera framerate. The setting is under the Advanced section of camera settings. It includes a message explaining that this may affect autofocus accuracy.

Lowering the camera framerate means that fewer frames are collected for size analysis used in autofocus. However, the frames we do collect will be of a fixed quality and so can be reliably used. However, since we have reduced our temporal resolution, fast autofocus accuracy will be compromised without also reducing the motor speed.

This is NOT guaranteed to fix the white-stream issue but should help.

Limit the stream resolution

We don't have many options for stream resolution; however I've added an option to slightly reduce it to 640x480. Hopefully since the file size scales quadratically(?) this will actually help a lot, without significantly impacting autofocus performance.

This is NOT guaranteed to fix the white-stream issue but should help.

Automatically identifying the issue

JPEG end byte analysis

I've added extra logic to our stream buffer that checks the last 2 bytes of each frame and compares them to the standard JPEG end-image bytes. In situations where the bitrate is exceeded, the end bytes will mismatch, and we know there has been an issue.

Currently, this just logs an error once each time the frames go from being good to bad.

Future work

In the future, we should consider either watching for this error somehow, and giving the user prompts to change the relevant settings. Better would be to thoroughly investigate the effect of these settings on fast-autofocus, and automatically change settings in response to hitting the bitrate limit.

Note to maintainers

Tagging @julianstirling and @rwb27 for your records after I've left the uni.

Better stream bitrate controls