A Model-based Approach for Plant Growth Reconstruction

Jeremy J. Loomis
Kikuo Fujimura
Hideo Ishikawa
Michael Evans
The Ohio State University

ABSTRACT

In this project we attempt to obtain three-dimensional measurements of a growing plant by reconstructing 3D time-varying models of the plant from multiple 2D image sequences.

We cast the 2D to 3D reconstruction problem as an optimizing search through template-space. Each template describes a 3D model which is viewed through "virtual cameras". These "virtual images" are compared with actual images to obtain an error value for each template.

INTRODUCTION

Motivation

The project is being conducted as joint research with Plant Biology researchers who are interested in studying the effects of various environmental conditions (primarily the type of light provided) on the growth behavior of seedlings. Some of the research questions they would like to study include:

What are the effects of red/blue light on plant growth?
How do the rates of growth of the outside/inside/spine compare?
How quickly does the hook of the plant open up?

Available Input

Since there is a need to control the types of light reaching the plant, the seedling is grown in a black box. Images are taken with CCD cameras; a red back-light that does not affect growth is the sole illumination.

Experimental Setup

The lab itself is in the Botany and Zoology building. Currently two gray-scale cameras separated by 90 degrees are used. The experimental setup has the following layout:

Figure 1. Diagram of Experimental Setup

The cameras are connected to frame grabber boards installed in IBM compatible PCs. Depending on the experimental conditions and the expected rate of growth, images are grabbed and stored with inter-picture delays ranging from 1 minute to 30 minutes.

Data Format

The images are 512x486 8-bit-grayscale files and are originally stored as .GIF files to save space. They are sent via ftp from the lab site to local CIS machines, where they are usually converted to .PGM files for easy manipulation. The following MPEG movies provide a condensed presentation of a standard sample set:

Experiment: "sstem.5"
Frames: 0-40
Time spacing: 2 minutes
"imga_xxx" series (from camera angle a)
"imgb_xxx" series (from camera angle b)

Desired Output

From the perspective of the biological researcher, it should be possible to obtain a variety of measurements of the plant at different times. To obtain measurements of the plant as a three dimensional object, it is first necessary to have an internal 3D model of the plant's structure at each time stage. Then, measurements can be taken from this model.

The types of possible measurements include:

plant height (max z-coord) vs. time
stem elongation (spine length) vs. time
inside/outside edge length vs. time
hook opening angle vs. time
plant rotation vs. time

CURRENT APPROACH

We are investigating a "top-down time-varying template-space search" approach that seems to handle noise and occlusion in a robust manner with relatively small accuracy tradeoffs.

Preprocessing

Since the plants are back-lit, shading in the images cannot be used, so the first step is to obtain pure silhouette images from the given images. The following sequence demonstrates this processing:

Figure 2. Original images - imga, imgb

Figure 3. Images after edge-detecting (Canny method) - cana, canb

Figure 4. Silhouette images (seed fill of contour) - reala, realb

Overview

Given multiple 2D images of a 3D plant, our approach is to use a priori knowledge about the sorts of shapes the plant might take on. This leads to the idea of a model which could describe the plant at any stage in any orientation. We would like to do some sort of search to determine the model that best matches the given views, but since the model is too complex, we consider a template that serves as a set of control points describing the model. With the use of an error criteria that compares the silhouettes to virtual images rendered from the model, the problem reduces to a search of template-space in the direction of monotonically-decreasing error. The following diagram highlights the important aspects of the approach:

Figure 5. System flow chart

The following sections provide further descriptions and examples.

Plant Model

The model that we use to describe the plant is a tubular object; that is, a 3D spine with a thickness that can vary along the length. In other words, the model is a set of four parametric functions:

model = { x(t), y(t), z(t), r(t) }, where
x,y,z describe the location of the spine and
r is the radius of the tube at a given point t (0<t<1) along the length.

Template

The template is a set of control points designed to represent a model using relatively few parameters. This number of parameters is important; the template needs to have sufficient degrees of freedom to cover the model-space, yet it is this number that determines the search time.

Currently, we use 28 parameters; 15 for the 5 spine control points and 13 for the radius control points. The model is generalized as follows:

P = 3n + ( n + (n-1)*k ) , where
P is number of parameters in the template,
n is the number of spine control points, and
k is the number of radius control points between them.

Creation of the model from the template is a matter of determining the number of steps to take along the spine and interpolating values between control points. Currently, we use a spline-based approach.

Rendering the Model

There are several options when it comes to rendering a tubular object defined as a spine and thickness profile:

Polygonal mesh - faceted tube, "angular" connection between sections
Raycasting - alternating frustrum/sphere components
Metaballs - "pack" tube with balls

After working with the first two methods, we realized that most standard rendering methods would be too slow (notice in the flowchart presented earlier, that the rendering stage is inside an "inner loop").So, we currently use a "quick-and-dirty" technique: project the spine to 2D, draw circles at the segment interconnections, and approximate the segment frustrums with trapezoids.

Searching Template-Space

Currently, we use a brute-force method, guided by coarse to fine refinement. Starting with the template that best described the previous frame, the search generates new possibilities by adjusting each of the parameters up or down by a certain distance: the current template precision.

Each template is tested to see if it describes the input images any better than the original. The error is calculated as the average of the two camera angle errors. Each camera angle error is calculated as: (nonmatching pixels)/(matching pixels).

The best template is kept; this continues until no better template can be discov ered using the current precision as a template-space operator. One assumption that is made is that the parameters are independent; i.e., that the best template can be reached by changing only one parameter at a time. The search is allowed to continue indefinitely, since there is a limit to the accuracy that can be achieved. Error decreases monotonically; each new template gains at least one pixel accuracy.

Results

The following images are the "virtual camera" images resulting from the best template found for input images imga and imgb shown earlier. Compare with the silhouette images (reala,realb) above.

Figure 6. "virtual camera" images (from final template) - virta, virtb

The actual result of the search process is a list of templates, one for each frame. 3D measurements are easily obtainable by using the template to create a model from which to measure plant dimensions for the frame. The next section gives some preliminary measurements obtained this way. The model can also be visualized using any appropriate rendering approach; time is not an issue once the search is completed. The picture below was generated using raytracing:

Figure 7. Ray-traced rendition of plant model

CURRENT STATUS

Preliminary Plant Measurements

Based on some early success of the template-matching approach, a few measurements have been obtained. Accuracy has not been verified, but the following graphs give a good idea of the sorts of information that can be obtained:

Hook Angle vs. Time (angle.gif)
Height vs. Time (height.gif)
Length vs. Time (length.gif)
Plant Rotation vs. Time (rotate.gif)

Research Directions

Our work can proceed along many possible fronts:

Improve the search mechanism
- experiment with genetic algorithm
- resolve problems with local minima of search-space
- try jittering initial conditions
- consider guided search (e.g. 3D morphing between key frames)
Optimize modeling
- need to simultaneously minimize error and DOF
- consider variations (incrementally defined control points)
Optimize rendering
- goal: fast and accurate
Improved measurements
- study techniques for accurate measurements from model

REFERENCES

Barron, J.L. and A. Liptay. "Optical flow to measure minute increments in plant growth." Bioimaging 2. Vol. 2 (1994). pp 57-61.

Ishikawa, H., H.K. Hasenstein, and M.L. Evans. "Computer-based video digitizer analysis of surface extension in maize roots." Planta. Vol. 2 (1991). pp 381-390.

Ohya, Jun, and Fumio Kishino. "Human Posture Estimation from Multiple Images Using Genetic Algorithm." 12th IAPR International Conference on Pattern Recognition. IEEE Computer Society (1994). pp750-753.