HOME

ACADEMIC

INTERESTS

Texture Extraction

The texture extract project is a component of the EDVEC facade project. The project is based on Paul Debevec's facade system . EDVEC's facade software provides a semi-automatic utility for recovering scene geometry from a set of photographs. The software developed under the texture extraction project provides a mechanism for extracting surface textures from these photographs, and yields virtual objects in VRML.

The basic approach to geometry extraction is for the user to identify a set of image features in the photographs, such as edges, to create appropriate matching geometrical objects such as cuboids, and to identify correspondences between the 2D image features and the 3D objects. The facade software then optimises the position of the image features, and adjusts the geometry to find the best possible parameters describing the 3D objects. This is an iterative process during which the user may progressively refine the scene geometry and quickly build up a convincing model of the scene geometry using a set of photographs. In addition, the camera parameters are derived through optimisation, so there is no need to record the relative position and orientation of the camera when taking the photographs.

For example, a set of photos similar to the following were used to build a model of a quadrangle at Edinburgh University.

The model is mostly composed of cuboids, but demonstrates that other shape primitives such as triangular prisms, octagonal prisms and domes are also possible. In fact, arbitrary shape primitives may be designed by the user and incorporated into the optimisation process.

The texture extraction software begins with a set of photographs and a geometrical model such as those above. The first step is to assign textures to each face in the geometrical model and back project the photo images onto the faces so that the model can be rendered using the appropriate textures, from an arbitrary angle. This process is relatively straightforward since the camera parameters are given.

Since multiple photographs are used to derive the textures, and colour balancing between photographs is a notoriously awkward problem a special blending technique was used. Each pixel identified in a photograph as corresponding to a particular point on the model was assigned a weight proportional to its distance from the edge of the photograph. The weight thus trailed off to zero towards the edges of the photograph. These weights were used to evenly blend all the pixels corresponding to a single point on the model and successfully hid the photograph edges.

However, a number of other awkward problems must also be solved. Simply back projecting the photographs onto the model's surface does not take into account whether or not a give surface is obscured. Therefore, when one object occludes another, an image -like a coloured shadow- will be smeared over the surface of the occluded object. A simple approach to avoiding this unsightly artifact is to render depth images corresponding to each photograph. By comparing the depth of each pixel with the expected depth calculated from the model geometry it can be determined whether a given pixel corresponds to a given surface, or another obscuring surface.

This approach may be used to solve the occlusion problem at the 2D level, since occlusions are identified on a pixel by pixel basis and no geometric information is available regarding the geometry of occluded patches. A more comprehensive approach of mesh segmentation can be used to identify exactly which parts of the mesh are visible in each photograph. Occluded patches are implicitly identified, and the geometry of holes may be identified. Holes occur when part of a surface is not visible in any photograph, and the texture is therefore indeterminate. An occlusion segmentation was implemented based on the work of Yizhou Yu, Efficient Visibility Processing for Projective Texture-mapping . While the technique is conceptually straightforward, it involves significant computational geometry and required considerable effort to implement the necessary polygon intersection tests and clipping operations (over 2000 lines of C++ code). The method involves identifying possible occlusions using a depth buffer, and clipping all overlapping polygons into segments that are either entirely visible or entirely hidden in each photograph.

The first step in the segmentation process is to segment any intersecting faces along the line of intersection, and the for each photograph in turn segment the geometry against the view frustum of the camera and eliminate any back facing surfaces.

After the geometry has been segmented against each camera, all the surfaces that partially occlude each other when viewed from that camera are identified and clipped. Each face is marked hidden or visible, and thus faces that are outside the frustum, back-facing, or entirely occluded by another face can be marked hidden and do not need to be further segmented. The remaining faces form a non-overlapping set of polygons that appear exactly as the original model when viewed from one particular direction, but reveal their hollow structure when viewed from any other angle.

The images below show the quad model segmented according to 15 separate views, and colour coded by visibility. This yield an approximate ten-fold increase in the number of mesh faces (initially about 2.5 thousand) but fortunately, the final model may use the unsegmented geometry once the segmented geometry has been used to generate appropriate textures.

 

Last Update 10th March 2006
webmaster@joshhale.com

Home