The present invention relates to a method of improving the resolution of a moving object in a digital image sequence. More in particular, the present invention relates to a method of improving the resolution of a small moving object in a digital image sequence, the object consisting mainly or exclusively of boundary pixels.
In many image processing applications the most interesting events are related to changes occurring in the scene: e.g. moving persons or moving objects. In this document we focus on multi-frame Super-Resolution (SR) reconstruction of small moving objects, i.e. objects that are comprised mainly, or even solely, of boundary pixels, in undersampled image sequences. These so-called ‘mixed pixels’ depict both the foreground (the moving object) and the local background of a scene. Especially for small moving objects, resolution improvement is useful. Multi-frame SR reconstruction improves the spatial resolution of a set of sub-pixel displaced Low-Resolution (LR) images by exchanging temporal information for spatial information.
The concept of SR reconstruction has already been in existence for more than 20 years, as evidenced by the paper by R. Y. Tsai and T. S. Huang: “Multiframe image restoration and registration,” in Advances in Computer Vision and Image Processing, JAI Press, 1984, vol. 1, pp. 317-339. However, only little attention is given to SR reconstruction on moving objects. This subject has been addressed in, for example, the paper by A. W. M. van Eekeren, K. Schutte, J. Dijk, D. J. J. de Lange, and L. J. van Vliet: “Super-resolution on moving objects and background,” Proc. IEEE 13th International Conference on Image Processing (ICIP'06), vol. 1, 2006, pp. 2709-2712. Another publication addressing SR reconstruction is the paper by M. Ben-Ezra, A. Zomet, and S. K. Nayar: “Video super-resolution using controlled sub-pixel detector shifts,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 6, pp. 977-987, 2005.
Some Prior Art techniques, such as the one disclosed in the paper by Ben-Ezra et al., apply different SR reconstruction methods, for example iterated-back-projection or projection onto convex sets, while having the use of a validity map in their reconstruction process in common. This makes these methods robust to motion outliers. These known methods perform well on large moving objects (the number of mixed pixels is small in comparison to the total number of object pixels) with a simple motion model, such as translation. Other Prior Art techniques use optical flow to segment a moving object and subsequently apply SR reconstruction to it. In these known techniques, the background is static and SR reconstruction is done solely on a masked large moving object.
In the article by Van Eekeren et al. mentioned above an algorithm was presented that performs, after segmentation, simultaneously SR reconstruction on a large moving object and background using a Prior Art SR reconstruction technique. However, in the article no SR reconstruction is applied to the boundary (mixed pixels) of the moving object because of a cluttered background.
In the paper by F. W. Wheeler and A. J. Hoogs: “Moving vehicle registration and super-resolution,” Proc. IEEE Applied Imagery Pattern Recognition Workshop (AIPR'07), 2007, super-resolution reconstruction is performed on moving vehicles of approximately 10 by 20 pixels. For object registration a trajectory model is used in combination with consistency of local background and vehicle. However, in this known SR reconstruction approach no attention is given to mixed pixels. An interesting subset of moving objects are faces. In Prior Art techniques in that area which use SR reconstruction the modelling of complex motion is a key element. However, the faces in the used LR input images are far larger than the small objects addressed by the present invention.
When a moving object is small (that is, when it consists mainly or even solely of mixed pixels) and the background is cluttered, even the most advanced pixel-based SR reconstruction methods of the Prior Art will fail. Any pixel-based SR reconstruction method makes an error at the object boundary, because it is unable to separate the space-time variant background and foreground information within a mixed pixel.
U.S. Pat. No. 7,149,262 (Columbia University) discloses a resolution enhancement algorithm for obtaining a polynomial model mapping of low resolution image data to high resolution image data. However, said patent fails to mention super-resolution and hardly mentions moving objects, and is therefore incapable of suggesting an improved SR reconstruction method.
European Patent Application EP 1 923 834 (TNO), published on 21 May 2008, discloses a method for detecting a moving object in a sequence of images captured by a moving camera. The method comprises the step of constructing a multiple number of different images by subtracting image values in corresponding pixels of multiple pairs of images. One image is a representation of a high-resolution image having a higher spatial resolution than the original captured images. This known method does not concern the identification of a moving object, only its detection.
It is an object of the present invention to overcome these and other problems of the Prior Art and to provide a method of improving the resolution of a moving object in a digital image sequence, which method has an improved resolution at the object boundary, in particular when the object consists mainly, or even entirely, of boundary pixels.
To solve the above-mentioned problems the present invention proposes to perform SR reconstruction on small moving objects using a simultaneous boundary and intensity estimation of a moving object. Assuming rigid objects that move with constant speed through the real world, a proper registration is done by fitting a trajectory through the object's location in each frame. The boundary of a moving object is modelled with a sub-pixel precise polygon and the object's intensities are modelled on a High-Resolution (HR) pixel grid.
More in particular, the present invention provides a method of improving the resolution of a moving object in a digital image sequence, the method comprising the steps of:
constructing a high resolution image background model,
detecting the moving object using the high resolution image model,
registering the object, and
producing a high-resolution object description,
wherein the step of producing a high-resolution object description involves an iterative optimisation of a function based upon an edge model of the moving object.
By using an iterative optimisation of a function and a polygonal model of the (edge of the) moving object, it is possible to produce an accurate high-resolution object description, and thereby to accurately identify the object.
The function also be based upon a high resolution intensity description, and preferably is a cost function. It is further preferred that the high-resolution object description comprises a sub-pixel accurate boundary and/or a high-resolution intensity description. The step of registering the object preferably involves a model-based object trajectory.
The step of producing a high-resolution object description may involve solving an inverse problem. Advantageously, the high resolution image background may be estimated using a pixel-based super-resolution method.
In a particularly advantageous embodiment, the iterative optimisation of a cost function involves a polygonal description parameter and/or an intensity parameter. The edge model preferably is a polygonal edge model.
In a further embodiment, the method of the present invention may comprise the further steps of:
subjecting the high-resolution object description to a camera model to produce a low resolution modelled image sequence,
producing a difference sequence from a registered image sequence and the modelled image sequence,
feeding the difference sequence to the cost function, and
minimising the cost function to produce the next iteration of the polygon description parameter and/or an intensity parameter.