In mid-January, IC Realtime, a manufacturer of visual surveillance solutions for the residential, commercial, military and government sectors, issued a press release stating the company’s support for UL’s 2802 Performance Testing Standard for Camera Image Quality. In the works for more than two years—and released last September—the new standard defines a method to assess video image quality through a series of performance-based tests that are conducted on production digital cameras.
To help create the UL 2802 standard, IC Realtime worked closely with UL, including submitting its B-2000 2-MegaPixel IP Bullet camera to use as part of the testing development process. According to UL, the camera achieved high scores on many of the performance tests, with a certification and the results expected to be published shortly.
Residential Systems was curious about this process, and because it is new, we thought that UL could provide an explanation about the steps that a visual surveillance product would have to go through to become UL certified. Thanks to UL and IC Realtime’s PR agent Paul Muto, we were able to receive the following description of the process from Lou Chavez, UL, principal engineer, security. Here’s what Mr. Chavez told us, starting with a little science:
There are basic similarities between the operating system of a digital video camera and the human eye. Both are sensitive to light and both process light rays to interpret and view images. Image quality is generally a factor of interpreting light, lens, and processing abilities. We form images through the optic systems of our eyes and our brains process interpreted light. A digital video camera operates similarly in that electronic circuits and semiconductors mimic the brain’s functions. Challenging factors, such as light intensity, distance from an object, quick movement, glare, etc. affect our ability to discern quality images. These same factors also affect the output of a digital video camera.
In an effort to provide objective test methods to assess the image quality of a camera, UL has published a new standard UL 2802, Standard for Performance Testing of Camera Image Quality. This new standard can be used as a tool to assist in determining appropriate video cameras for specific use cases based on the camera’s performance under challenging test conditions.
Program Overview
The new UL 2802 certification program objectively assesses the image quality of individual digital video camera models. The standard includes methods for assessing image quality using a series of nine performance-based tests that are conducted on production camera samples. The nine tests result in performance scores for the following camera parameters: image resolution, distortion, relative illumination, dynamic range, maximum frame rate, grey level, sensitivity, bad pixel, and veiling glare. Each test results in achieving a score in a range of 0-100 with zero being the minimum and 100 the maximum. Additionally the individual score are conveyed in standard photographic units such as line pair per picture height (LP/PH) for resolution, lux for dynamic range, frames per second (FPS) for maximum frame rate, percent of bad pixels for bad pixel tests, etc.
The testing process begins with the manufacturer of the video camera determining what device will be subjected to the test program. Typically a manufacturer will consider the multiple configurations of devices and different combination of component and software that are available in a product line. The video cameras that are selected and tested are specific to that configuration and combination components tested. Different combinations of images sensors, image processors and related software, operating system, lens, camera housing/body and the like generally affect the overall image quality of the camera.
During the test program, the manufacture and UL collaborate to ensure that the video camera settings are optimized and the test results reflect the camera’s best abilities. The process typically involves fine-tuning resolution properties, optimizing exposure time and gain during each test, enabling or disabling default features, etc., to optimize camera performance and achieve the most favorable test results. The process of optimizing settings is no different from what is typically done during deployment of a video system.
Below are a few unique features of the UL test program that offers a new approach from traditional test methods. All test and test apparatus are comprehensively detailed in the published standard.
Electronic Test Targets
Traditional methods for testing video camera image quality involve the use of standardized, published test charts for resolution and grey level tests. UL’s program implements a different approach that incorporates calibrated light sources of known luminous levels, frequency, spatial orientation, and grey levels. Each of these factors contributes to a video camera’s ability to capture record and store an image relative to the actual object. The electronic test targets that UL uses for its test procedures eliminates some of the known potential inconsistencies associated with the use of printed test charts under very specific lighting conditions and lighting angles, which can be difficult to control. The electronic test targets are calibrated and measured during each test program, with all of the data and settings recorded for each test.
The UL2802 test lab
Circular Edge Analysis
Image resolution is a critical test in that it is a measure of how closely the digital image captured by the camera matches the actual subject. Digital images are made up of pixels, which are stored in the relative dimension of the actual image. An accurate resolution measurement requires an evaluation of a camera’s lens and sensors, as well as its imaging software, and is often presented as line pairs per picture height or LP/PH. The metric is a measure of how many distinguishable alternating colors can be represented in an image.
Modulation transfer function (MTF) is a technique used to quantify image resolution in more complex images. MTF corresponds to the spatial frequency of image line pairs per picture height (LP/PH), that is, the ability to represent the “real” object by taking the light intensity and plotting it along imaginary lines traversing the representation of the object.
Other mechanisms for measuring distortion are detailed in ISO 12233 (photography—Electronic still-picture cameras—Resolution measurements), and in the standard mobile imaging architecture (SMIA) forum specification. UL 2802 uses the same spatial frequency response (SFR) method for resolution. The primary difference is that UL 2802 uses a circular edge analysis method (see “The Circular-edge Spatial Frequency Response Test,” by Richard Baer, Agilent Laboratories, 2002) versus a more traditional slanted-edge method to detect the transition edge of a known image. Additional benefits of the circular edge methods include an accurate method of averaging SFR from all circular directions versus multiple (horizontal and vertical) measurements using slanted-edge techniques. An added benefit is that consistency of measurements is improved using circular-edge techniques, as the slanted-edge SFR assumes that the transition edge of the printed resolution chart is a straight line and factors such as lens distortion can deviate the edge from a straight line, resulting in measurement inaccuracies. The benefits of using the electronic test targets and circular edge techniques for measuring SFR are cumulative in producing accurate test data.
Circular edge vs. slanted edge. Signal around edge transition and the derivative of the edge transition signal
Performance Score
Each test results in an interpolated performance score that is calculated from traditional photographic units. UL reports the scores in two usable units of measure. For simple comparisons the UL units scores of 0-100 makes comparing one camera’s test score to another’s relatively simple without being a camera expert. For those that are more technically involved, the photographic unit scores may be more meaningful in comparing test results. Either way, the performance scores reflect the image quality achieved for each test parameter.
Performance scores will help a video system integrator make an accurate decision of what camera would be best for their particular use case based on known conditions. For example, if I need a camera that will perform well under low lighting condition and with fast moving objects, I may want to target camera parameters that specifically relate to those conditions such as sensitivity, dynamic range and frame rate.
No single number or criteria can provide a reasonably accurate and objective evaluation of a given camera therefore it is normal that an integrator will need to consider multiple parameters based on the camera’s use case application. UL 2802 is a standard that provides objective test results based on the cameras tested image quality.
