WO2015035450A1

WO2015035450A1 - Vision based guidance in broad-acre agriculture

Info

Publication number: WO2015035450A1
Application number: PCT/AU2014/000894
Authority: WO
Inventors: Andrew Robert ENGLISH; David Michael BALL
Original assignee: Queensland University Of Technology
Priority date: 2013-09-13
Filing date: 2014-09-12
Publication date: 2015-03-19

Abstract

A method and system enables improved vision based navigation of a vehicle. The method includes: capturing an image from the vehicle, wherein the image includes a parallel texture; estimating a roll and pitch of the vehicle using the captured image or another external source of a camera attitude; compensating for the estimated roil and pitch in the image; and calculating a heading of the vehicle relative to the parallel texture using the image in which roll and pitch is compensated.

Description

TITLE

VISION BASED GUIDANCE IN BROAD-ACRE AGRICULTURE

FIELD OF THE INVENTION

The present invention relates to autonomous navigation. In particular, although not exclusively, the invention reiates to vision based robot guidance for broad-acre agriculture.

BACKGROUND TO THE INVENTION

The growing world population, in particular in China and India, is putting more and more pressure on the world's resources. Similarly, rapid economic growth in developing regions is resulting in increased consumption. In order to meet these increased pressures, efficiency in agriculture must also increase.

Robotics and automation has been shown to be an important factor in improving efficiency in agriculture. In the fruit tree industry, for example, workers riding robotic platforms have been shown to be twice as efficient as workers employing traditional practices.

Research regarding automation of operations such as pruning, thinning, and harvesting, as well as mowing, spraying, and weed removal is popular. Additionally, certain agricultural machinery exists that is self-guided, or able to assist an operator with guidance.

However, a problem with such systems of the prior art is that they often require the use of a high accuracy Global Navigation Satellite System (GNSS) receiver. In many cases, a dedicated base station and communications link for corrections is also required to achieve sufficient accuracy, and as such the machinery is generally very expensive.

Certain attempts have been made to reduce the cost of guidance systems in agricultural machinery by utilizing image sensors. In particular, certain prior art systems utilise a 3D structure in a field, using either scanning lasers or stereo vision.

A problem with utilising a 3D structure in a field is that a distinct height difference between the crop and ground is required. As such, these systems cannot be used on very young crops. The sensors required for gathering 3d structure information are still generally very expensive or are difficult to calibrate and unreliable.

Other prior art systems use a camera to capture images of a field, and a binary segmentation of plants and soil is then performed. Template matching or line fitting is then used between sequential segmented images to generate a motion induced offset.

However, a problem with these systems of the prior art is that they are generally environment specific, and thus require both knowledge of the environment in which they are to be used, and a configuration with respect to that environment. Furthermore, such systems do not work well when, for example, artefacts in an image are present, when strong shadows are present, or when weeds are present between plant rows.

Accordingly, there is a need for an improved system and method for vision based guidance in broad-acre agriculture.

OBJECT OF THE INVENTION

It is an object of some embodiments of the present invention to provide improvements and advantages over the above described prior art, and/or overcome and alleviate one or more of the above described disadvantages of the prior art, and/or provide a useful commercial choice.

SUMMARY OF THE INVENTION

According to one aspect, the invention resides in a method for vision based navigation of a vehicle, the method comprising:

capturing an image from the vehicle, wherein the image includes a parallel texture;

estimating a roll and pitch of the vehicle using the captured image or another external source of a camera attitude;

compensating for the estimated roll and pitch in the image; and

calculating a heading of the vehicle relative to the parallel texture using the image in which roll and pitch is compensated.

Preferably, the method further comprises:

providing navigation instructions to a controller of the vehicle according to the calculated heading of the vehicle and a desired heading of the vehicle. Preferably, the controller of the vehicle includes one or more actuators adapted to control one or more controls of the vehicle. Suitably, the controls of the vehicle comprise at least one of a propulsion controller, and a steering controller.

Preferably, the image comprises a panoramic image including a horizon as well as the ground close to the vehicle.

Preferably the external source of camera attitude is an inertia! measurement unit (I U).

Preferably, the method further comprises:

estimating a location of the horizon in the image;

wherein the roll and pitch of the vehicle is detecting according to the estimated location of the horizon.

Preferably, estimating a location of the horizon comprises:

selecting a region of interest in the image;

classifying pixels in the region of interest as "sky" or "ground"; and fitting a line to a boundary between the "sky" and "ground" pixels.

Preferably, the pixels are classified as "sky" or "ground" based upon median pixel values in upper and lower portions of a region of the region of interest.

Preferably, an upper portion of the region of interest is defined by an upper 30% of the region of interest, and a lower portion is defined by a lower 30% of the region of interest.

Preferably, the line is fitted to the boundary between the "sky" and "ground" pixels using a random sample consensus (RA SAC) algorithm.

Preferably, compensating for roll comprises rotating the image such that the horizon line is parallel to a line of pixels.

Preferably, the method further comprises compensating for pitch by either cropping or translating the image to remove one or more pixels above the horizon line. Alternatively the steps of rotating the image and then translating the image can be replaced with a perspective transform that takes into account the perspective projection of a flat image plane. This is an alternative way of correcting for roll and pitch and takes into account image distortion created by projecting an image on a flat image plane as is the case in conventional cameras. Preferably, the region of interest comprises a region of the image in which the horizon is likely to appear.

Preferably, the method further comprises:

generating a simulated overhead view, wherein the simulated overhead view corresponds to a view of a virtual camera that is perpendicular to the parallel texture.

Preferably, the simulated overhead view is generated according to pixels in a further region of interest.

Preferably, the simulated overhead view is generated by a fixed homography matrix to perform a perspective transform- Alternatively the fixed homography matri may be combined with the homography matrix used for performing roll and pitch compensation (by multiplying them together), so that stabilisation and overhead view generation is performed in a single perspective transform. Yet another alternative to generating an overhead view using a perspective transform is to use some other transform, providing the transform makes the crop rows or other parallel texture being tracked parallel in the overhead image. These other transforms may have some advantages over using planar homography such as the ability to map more pixels in the input image to valid pixels in the overhead image.

Preferably, the homography matrix is generated according to a point on the horizon line.

Preferably, calculating a heading of the vehicle comprises:

skewing the simulated overhead image;

summing columns of the skewed image to generate a vector of summed columns, and

calculating a variance of the vector.

Preferably, the method furthe comprises determining a maximum variance of the vector.

Preferably, the method further comprises low pass filtering the vector. Preferably the method comprises calculating a gradient of the vector.

Preferably, the method further comprises:

determining a validity of the vehicle heading data; and

selectively using the vehicle heading data according to the validity.

Preferably, the method further comprises: determining a lateral movement of the vehicle using a function template. According to another aspect, the invention resides in a system for vision based navigation of a vehicle, the system comprising:

an image capture module, for capturing an image from the vehicle wherein the image includes a parallel texture;

a control module, for controlling movement of the vehicle;

a processor coupled to the image capture module and the control module; and

a memory coupled to the processing module, the memory including instruction code executable by the processor for:

receiving an image from the image capture module;

estimating a roll and pitch of the vehicle using the captured image;

compensating for the estimated roli and pitch in the image;

calculating a heading of the vehicle relative to the parallel texture using the image in which rail and pitch is compensated; and

providing instructions to the control module according to the calculated heading of the vehicle and a desired heading of the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

To assist in understanding the invention and to enable a person skilled in the art to put the invention into practical effect, preferred embodiments of the invention are described below by way of example only with reference to the accompanying drawings, in which;

FIG. 1 illustrates a block diagram of a system for vision based navigation in broad-acre agriculture, according to an embodiment of the present invention;

FIG. 2 illustrates an input image in which a region of interest has been seiected for horizon detection, according to an embodiment of the present invention;

FIG. 3a illustrates a further input image in which a further region of interest is defined;

FIG- 3b illustrates a simulated overhead view generated from the further region of interest of FIG. 3a;

FIG. 4a illustrates a two dimensional model defining a plurality of rows; FIG. 4b illustrates a on© dimensionai intensity function used to define the rows in the two-dimensional model of FIG. 4a;

FIG. 5 illustrates a method for vision based navigation in broad-acre agriculture, according to an embodiment of the present invention;

FIG. 6 illustrates a system for vision based navigation in broad-acre agriculture, according to a further embodiment of the present invention;

FIG. 7 illustrates an example process flow of a method for vision based navigation in broad-acre agriculture, according to an embodiment of the present invention; and

FIG. 8 diagrammatically illustrates a computing device, according to an embodiment of the present invention.

Those skilled in the art will appreciate that minor deviations from the layout of components as illustrated in the drawings will not detract from the proper functioning of the disclosed embodiments of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention comprise methods and systems for vision based navigation in broad-acre agriculture. Elements of the invention are illustrated in concise outline form in the drawings, showing only those specific details that are necessary to the understanding of the embodiments of the present invention, but so as not to clutter the disclosure with excessive detail that will be obvious to those of ordinary skill in the art in light of the present description.

In this patent specification, adjectives such as first and second, left and right, front and back, top and bottom, etc., are used solely to define one element or method step from another element or method step without necessarily requiring a specific relative position or sequence that is described by the adjectives. Words such as "comprises" or "includes" are not used to define an exclusive set of elements or method steps. Rather, such words merely define a minimum set of elements or method steps included in a particular embodiment of the present invention.

The reference to any prior art in this specification is not, and should not be taken as, an acknowledgement or any form of suggestion that the prior art forms part of the common general knowledge. According to one aspect, the invention resides in a method for vision based navigation of a vehicie, the method comprising: capturing an image from the vehicle, wherein the image includes a parallel texture; estimating a roll and pitch of the vehicle using the captured image or another external source of a camera attitude; compensating for the estimated roll and pitch in the image; and calculating a heading of the vehicle relative to the parallel texture using the image in which roil and pitch is compensated.

The term navigation can refer to relative navigation, such as navigation relative to rows of crops in a field, or absolute navigation with reference to a global coordinate system.

Advantages of certain embodiments of the present invention include an ability to efficiently and accurately provide autonomous or semi-autonomous navigation in, for example, broad-acre agriculture.

According to certain embodiments, no prior knowledge of the environment in which the system or method operates is required, in particular, for example, there is no need for crops to b present, for any crop to have a particular colour, or for any crop to have a particular size, when navigating in a field.

Certain embodiments of the invention are inexpensive, simple to implement and fast to execute, as they enable use of well optimised image processing functions.

FIG. 1 illustrates a block diagram of a system 100 for vision based navigation in broad-acre agriculture, according to an embodiment of the present invention. The system 100 can be used together with a tractor, a dedicated robot, or any other suitable vehicle that is able to navigate in a field.

The system 100 includes an image capture module 105, a processing module 110 coupled to the image capture module 105, and a control module 15 coupled to the processing module 110,

The image capture moduie 105 can, for example, comprise a camera that is adapted to capture images of a field. In particular, the camera can be adapted to capture panoramic images of a field, for example a panoramic image including the field and the horizon, as the system 100 moves across the field. As discussed in further detail below, the image capture module 105 is advantageously fixedly mounted to the system 100 and oriented in a direction substantially parallel to a typical motion of the system, such as a forward direction of the system 100.

The control module 115 can, for example, comprise a data interface, configured to provide navigation instructions to a robot, or one or more actuators adapted to control one or more controls of a vehicle, such as a propulsion controller and a steering of a tractor or other vehicle.

Finally, the processing module 110 includes a horizon detection module 110a, a transformation module 1 10b, and a direction estimation module 1 0c.

The horizon detection module 110a is used to estimate roll and pitch of the system 100 by tracking a horizon in images captured by the image capture module 105. Since agricultural fields in broad-acre cropping tend to be vast, flat and almost obstacle free, there is often a clear and unobstructed view of the horizon, which can be used to estimate a roll and pitch of the system 100.

The horizon detection module 110a is configured to select a region of interest in an input image that was captured by the image capture module 05. When the image capture module 105 is fixedly attached to the system 100, and the terrain is reasonably flat, a location of the horizon in subsequent images does not generally change a large amount.

Accordingly, a region of interest can be selected for detecting the horizon, which can reduce computational complexity when compared with searching for the horizon in an entire input image, and also reduce the chance of a "false positive" horizon detection.

Pixels in the region of interest are then classified as either "sky" or "ground" based upon median RGB values in upper and lower portions of the region of interest respectively. A straight line is then fitted along a boundary between the sky and ground pixels, the line defining an estimate of the horizon.

As an illustrative example, FIG. 2 illustrates an input image 200 in which a region of interest 205 has been selected. In particular, the input image 200 comprises a lower portion 210a comprising substantially crop image data, and an upper portion 210b that comprises sky image data and crop image data. The region of interest 205 generally comprises the upper portion 210b.

The region of interest 205 can be used to generate a region of interest image 215, in which the horizon is detected. A resulting horizon line 220 is illustrated in the region of interest image 215. According to certain embodiments, an upper portion of the region of interest 205 is defined by, but is not limited to, an upper 30% of the region of interest 205, and a lower portion is defined by, but is not limited to, a lower 30% of the region of interest 205. As the sky is above the ground in the input image in ali but very extreme cases (i.e. a vehicle that has rolled onto its roof), the upper and lower portions of the region of interest 205 provide good estimates of sky and ground pixel values.

The horizon line 220 can be fitted using any number of suitable fitting algorithms. In particular, a straight line can be fitted to the boundary between sky and ground pixels using the random sample consensus (RANSAC) algorithm of Fisch!er and Bo!les, which algorithm is weli known in the art.

The horizon line 220 is then used to compensate for roil and pitch in the image by rotating the input image 200. Additionally, the input image 200 can be cropped so that the horizon lies at the top of the image, thus resulting in a cropped image that includes almost entirely crop data.

The transformation module 110b is then used to construct a simulated overhead view of the crop rows in the input image. The crop rows appear parallel in the simulated overhead view, and equal pixel weighting then can be given to objects in an image regardless of whether the objects are close to the camera or not.

The simulated overhead view is generated by warping a rectangular region of interest using a reverse perspective transform. As the horizon position is fixed, the transform can be defined by a fixed homography matrix, and as such the warping becomes a mapping of pixels.

In particular, and with reference to FIG. 3a, a region of interest 305 is defined in a cropped and rotated input image 300 by points P P₂, Ps and P₄. The region of interest 305 is selected centrally in the cropped and rotated input image 300. However, the skilled addressee will readily appreciate that the present method can be modified to enable selection of an arbitrary region of interest.

A vanishing point Py is determined by selecting a point on the horizon line that lies centrally with respect to the region of interest 305. A trapezoid is then formed using upper and lower edges of the region of interest 305 that tapers towards the vanishing point P_Vp, as illustrated by points P₅, P_4r P₅ and P§. A 3 x 3 planar homography matrix is then determined that transforms the trapezoid into a rectangle that is defined by points P₅, Pg, P₇ and Pg. The homography matrix can then be used to transform the region of interest 305 into a simulated overhead view 310, as illustrated in FfG. 3b.

The crop rows are parallel in the simulated overhead image 310, but potentially at an angle with respect to a direction of the system 100.

The direction estimation module 110c is used to determine a direction of a dominant parallel texture in the simulated overhead image, which can in turn be used to determine a relative direction of travel of the system 100.

The field including the crop rows is modelled as a horizontally planar surface with straight parallel rows. Many agricultural fields can be efficiently modelled this way as the appearance of the crop is generally uniform in the direction of the crop rows. However, as will be readily understood by the skilled addressee, cro rows that are curved can be modelled and compensated for by the system. Similarly, parallel patterns, including parallel patterns that do not correspond to rows of crops, can be modelled using any other suitable model.

In particular, and with reference to FIG, 4a and FIG. 4b the field is modelled b a one dimensional intensity function 405 that is projected aiong a plane 410 to define a plurality of rows. In such case, the intensity at any point on the plane is defined as

Where x and y are coordinates of the plane, l{x,y) is a two dimensional intensity function, S(x) is the one dimensional intensity function and e(x,y) is a noise function that represents a component of the crop appearance that does not fit the linear parallel pattern model (e.g. texture in the crop and ground, missing plants, weeds, etc.). The noise function can be modelled as a random variable with zero mean.

The overhead image can thus be modelled as a rotated and scaled view of the intensity function l(x,y).

As ε(χ,γ) is zero mean, S(x} can be recovered by integrating the intensity function aiong the direction of the rows, thus "integrating out" the noise, as follows: Si x)

The yaw of the vehicle, i.e. the direction of the vehicle on the plane, can be determined by maximizing a variance of scaled and rotated estimates of S(x), as shown below.

where y_c is the yaw, and £_e is an estimate of Six).

According to certain embodiments, the yaw is found by iteratively skewing the image by differing amounts using an affine transform, summing the intensity values of pixels in each column of the resulting image to generate a vector, and calculating a variance of the resulting vector. The amount of skew applied to generate the largest variance is indicative of the yaw. As will be understood by a person of ordinary skill in the art, any of a number of iterative algorithms can be used to find a maximum variance, including a gradient search algorithm.

Once the yaw is determined, it can be compared to a desired yaw, and navigation of the system 100 can be adjusted by the control module 115, as discussed above.

According to certain embodiments, the vector produced by summing the columns of the image is low-pass filtered and/or a gradient is calculated before computing the variance. This can help to remove noise in the image and improve performance by rejecting frequencies that do not contain crop row information.

According to certain embodiments (not shown), the processing module 110 includes a pre-processing module, for processing the input image. In such case the pre-processing module can be used to remove distortion of the image capture module 105, such as lens distortion, and/or scale the image to a certain size.

According to certain embodiments, the pre-processing module reduces a size of an input image from 1280x720 pixels to 640x360 pixels, for improved processing speed. However, the skilled addressee will readily appreciate that any suitable input image size and scaling can be provided to images. According to other embodiments, the system 100 further includes an !nertial Measurement Unit (IMU) (not shown). In such case, the i U can be used to assist generation of attitude data to improve resilience to noise, or decrease complexity by enabling a more efficient search to be performed.

According to certain embodiments, the system 100 further comprises a validity estimation module (not shown), for determining a validity of directional data that is determined by the direction estimation module 110c. There can be sections of a field where there are no crop rows to track, for example at the ends of the field or on bare patches of field. During the iterative search for the yaw value described above, the variance values can be saved and further analysed. In particular, if the yaw is searched in a +/- 30 degree search window in 5 degree increments, up to 13 variance values are estimated. A frame is deemed to have a valid row texture if the highest variance is at least twice the average variance. If there is no directional texture in the overhead image, then all variance values will be very similar and thus the frame will not be considered to have a valid row texture. If however there is a strong directional texture in the overhead image (i.e. rows clearly present) then a strong peak in variance will be present for the correct value.

If an individual frame is not deemed to have a valid row texture, the system 100 can momentarily use earlier navigation data. If many sequential images are not deemed to have a valid row texture, the system 100 can, for example, stop or hand control over to another navigation system.

As discussed earlier, the field can be raode!!ed with an arbitrary intensity function perpendicular to the direction of the rows from the overhead image. The intensity function can be tracked directly to estimate the vehicle offset from the crop rows. In such case the intensity function is determined for an image and used as a template. Subsequent frames can be matched against the template by finding an offset that matches the previous or some other frame.

The template matching can be performed using a zero-mean normalized cross-correlation (ZNCC) algorithm. Furthermore, when an input frame does not match the template, a new template can be generated based upon an intensity function of the input frame.

F!G. 5 illustrates a method for vision based navigation in broad-acre agriculture, according to an embodiment of the present invention. In step 505, an image is captured from a vehicle, wherein the image includes a parallel texture. The vehicle can, for example, comprise an agricultural vehicle such as a tractor, and the parallel texture can comprise an image of rows of crops,

In step 510, a roll and pitch of the vehicle is estimated using the captured image. As discussed above, the roll and pitch can be estimated using a horizon line determined from the image. Alternatively the roll and pitch of the vehicle can be determined via some external source such as an Inertial Measurement Unit (IMU).

In step 515, the estimated roll and pitch is compensated for in the image.

As discussed above, compensating for the roll and pitch in the image can comprise rotating the image such that the horizon line is parallel to a row of pixels. Compensating for pitch can be achieved by translating the image so that the horizon line is located at some pre-determined location in the image. Alternatively roll and pitch may be compensated for using a perspective transform that takes into account the perspective projection of the camera.

In step 520, a heading of the vehicle is calculated relative to the parallel texture using the image in which roll and pitch is compensated. By calculating the heading of the vehicle subsequent to compensating for roll and pitch, substantial complexify reductions can be achieved compared with calculating roll, pitch and heading simultaneously.

FIG, 6 illustrates a system 600 for vision based navigation in broad-acre agriculture, according to an embodiment of the present invention. The system 100 of FIG. 1 can be similar or identical to the system 600.

The system 600 can be built based, for example, on a John Deere Gator

TE electric utility vehicle. As discussed below, the vehicle is modified for autonomous operation with the addition of, among other things, steering and braking actuators, wheel encoders, and an interface to the vehicle's motor controller.

The system 600 includes a camera 605, an IMU 610, a plurality of global positioning system (GPS) antennas 615, a wireless modem 620 and a processing module 625. The camera 605 is a forward facing camera that is fixedly attached to the system 600. An example of a suitable camera is the IDS uEye CP camera of IDS imaging Development Systems GmbH, Germany.

According to certain embodiments, the system 600 further includes a rear facing camera (not shown). In such case, either or both of the camera 605 and rear camera can be used for capturing images, tn particular, one of the camera 605 and rear camera can be selected based upon a level of directional texture in respective images. Accordingly, th rear camera can be selected when the forward camera is blinded by a sunset, or when approaching an end of the field.

The IIV1U 610 is used to estimate the system's attitude {roll, pitch and yaw), which can be used together with the vision based tracking algorithms described herein. In particular, the attitude from the llvtU 610 can, for example, be used as a starting point for an image based attitude estimation, as described above.

An example of a suitable !MU 610 is a UM6 !MU from CH Robotics LLC,

Utah, United States of America. The IMU can, for example, include one or more of a gyroscope, an aeeelerometer and a magnetometer, which can be used with an extended alman filter to estimate the system's 600 attitude.

The plurality of GPS antennas 615 can be used to estimate a motion of the system 600 across a field, and augmented with vision based data to refine an attitude estimate.

The wireless modem 620 can be used to provide Real Time Kinematic (RTK) corrections. Correction data can, for example, be provided by the SmartNet Australia Continually Operating Reference Station (CORS) network.

The processing module 625 can, for example, comprise one or more personal computers. For example, the personal computers can run the Ubuntu operating system from ubuntu.com and the Robot Operating System (ROS) middleware from ros.org.

FIG. 7 illustrates an example process flow 700 of a method for vision based navigation in broad-acre agriculture, according to an embodiment of the present invention.

An input image 705 is captured by an imaging device, such as a camera. The input image includes data corresponding to a field with parallel rows, and a horizon. The input image 705 is processed to remove camera distortion and scaled to a suitable size, forming a refined input image 710. in particular, the camera distortion can be removed using camera distortions, such as lens distortions, that are measured offline. The input image, if in colour, can also be converted to a grayscale image.

The refined input image 710 is then processed to remove roll and pitch. As discussed above, the horizon can be used to estimate a roll and pitch in the refined input image 710. The refined input image 710 can then be rotated to remove the effects of roll and pitch, and cropped to remove sky pixels, thus forming a rotated crop image 715.

An overhead image 720 can then be generated from the rotated crop image 715 using planar homography as described above, in particular, a region of interest of the rotated crop image can be used to generate the overhead image 720. Other transforms may be used to provide a simulated overhead view, providing the transform makes the crop rows or other para!iei texture being tracked parallel in the overhead image. These other transforms may have some advantages over using planar homography such as the ability to map more pixels in the input image to valid pixels in the overhead image, however their disadvantage is that they do not give equal pixel weighting to objects regardless of whether the objects are close to the camera or not.

The yaw of the camera can then be estimated from the overhead image 720. As discussed above, the yaw of the camera can be used to estimate and/or control a direction of a vehicle.

A slightly different arrangement of this genera! method is rather than estimating roll and pitch independently and before estimating yaw, is to instead modify the currently one dimensional search for yaw (maximising the variance of the sum of the columns of the skewed overhead image) to simultaneousiy estimate roil pitch and yaw in a three dimensional optimisation using the same cost function. That is, choose a candidate roll, pitch and yaw, calculate the corresponding skewed overhead view, calculate the cost function (variance of sum of columns) then repeat with slightly different roll/pttch yaw values until a camera attitude is found. This method could be used in conjunction with previous methods of independently estimating roll pitch and yaw, to for example refine the independently estimated roll and pitch. The yaw estimate can also be used to correct a heading of the camera in a corrected overhead image 725. A frame template 730 can then be generated from the corrected overhead image 725, which, as discussed above, can be used to estimate a lateral offset of the camera with respect to earlier images.

Those skilled in the art will understand that in many embodiments calculating the frame template and calculating a lateral offset is generally required to effectively control a vehicle and to avoid accumulation of lateral offset error.

FIG, 8 diagrammatically illustrates a computing device 800, according to an embodiment of the present invention. For example, the processing module 110 of FIG. 1 can be implemented using the computing device 800. Similarly, the method 500 of FiG. 5 can be implemented using the computing device 800.

The computing device 800 includes a central processor 802, a system memory 804 and a system bus 806 that couples various system components, including coupling the system memory 804 to the central processor 802. The system bus 806 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The structure of system memory 804 is well known to those skilled in the art and may include a basic input/output system (BIOS) stored in a rea only memory (ROM) and one or more program modules such as operating systems, application programs and program data stored in random access memory (RAM).

The computing device 800 can also include a variety of interface units and drives for reading and writing data. The data can include, for example, asset and user profiles as discussed above.

In particular, the computing device 800 includes a hard disk interface 808 and a removable memory interface 810, respectively coupling a hard disk drive 812 and a removable memory drive 814 to the system bus 806. Examples of removable memory drives 814 include magnetic disk drives and optical disk drives, The drives and their associated computer-readable media, such as a Digital Versatile Disc (DVD) 816 provide non-volatile storage of computer readable instructions, data structures, program modules and other data for the computer system 800. A single hard disk drive 812 and a single removable memory drive 814 are shown for illustration purposes only and with the understanding that the computing device 800 can include several similar drives. Furthermore, the computing device 800 can include drives for interfacing with other types of computer readable media.

The computing device 800 may include additional interfaces for connecting devices to the system bus 806. FIG. 8 shows a universal serial bus (USB) interface 818 which may be used to couple a device to the system bus 806. For example, an IEEE 1394 interface 820 may be used to couple additional devices to the computing device 800. Examples of additional devices include cameras for receiving images or video, or microphones for recording audio.

The computing device 800 can operate in a networked environment using

!ogical connections to one or more remote computers or other devices, such as a server, a router, a network personal computer, a peer device or other common network node, a wireless telephone or wireless personal digital assistant. The computing device 800 inciudes a network interface 822 that couples the system bus 806 to a local area network (LAN) 824. Networking environments are commonplace in offices, enterprise-wide computer networks and home computer systems.

A wide area network (WAN), such as the Internet, can also be accessed by the computing device, for example via a modem unit connected to a serial port interface 826 or via the LAN 824.

Transmission of data can be performed using the LAN 824, the WAN, or a combination thereof.

It will be appreciated that the network connections shown and described are exemplary and other ways of establishing a communications link between computers can be used. The existence of any of various well-known protocols, such as TCP/IP, Ethernet, FTP, HTTP and the like, is presumed, and the computing device 800 can be operated in a client-server configuration.

The operation of the computing device 800 can be controlled by a variety of different program modules. Examples of program modules are routines, programs, objects, components, and data structures that perform particular tasks or implement particular abstract data types. The present invention may also be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, persona! digital assistants and the like. Furthermore, the invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

While the above description focuses on agriculture and rows of crops, the present invention can be applied to other vision based navigation applications, in particular, for example, the present invention can be used to assist a robot in navigating in a room having a tiled floor (or other parallel pattern).

In summary, advantages of certain embodiments of the present invention include an ability to efficiently and accurately provide autonomous or semi- autonomous navigation in, for example, broad-acre agriculture. Certain embodiments of the invention are simple to implement and fast to execute, as they enable use of well optimised image processing functions.

The above description of various embodiments of the present invention is provided for purposes of description to one of ordinary skill in the related art. it is not intended to be exhaustive or to limit the invention to a single disclosed embodiment. As mentioned above, numerous alternatives and variations to the present invention will be apparent to those skilled in the art of the above teaching. Accordingly, while some alternative embodiments have been discussed specifically, other embodiments will be apparent or relatively easily developed by those of ordinary skill in the art. Accordingly, this patent specification is intended to embrace all alternatives, modifications and variations of the present invention that have been discussed herein, and other embodiments that fail within the spirit and scope of the above described invention.

Claims

The claims defining the invention are:

1. A method for vision based navigation of a vehicle, the method comprising: capturing an image from the vehicle, wherein the image includes a parallel texture;

estimating a roll and pitch of the vehicle using the captured image or other external source of a camera attitude;

compensating for the estimated roll and pitch in the image; and

2. The method of claim 1 further comprising:

providing navigation instructions to a controller of the vehicle according to the calculated heading of the vehicle and a desired heading of the vehicle.

3. The method of claim 2 wherein the controller of the vehicle includes one or more actuators adapted to control one or more controls of the vehicle.

4. The method of claim 1 wherein the image comprises a panoramic image including a horizon as we!f as the ground close to the vehicle.

5. The method of claim 1 further comprising:

estimating a location of the horizon in the image;

6. The method of claim 5 wherein estimating a location of the horizon comprises:

selecting a region interest in the image;

7. The method of claim 6 wherein the pixels are classified as "sky" or "ground" based upon median pixel values in upper and lower portions of a region of the region of interest.

8. The method of claim 7, wherein the upper portion of the region of interest is defined by an upper 30% of the region of interest, and the lower portion is defined by a lower 30% of the region of interest.

9. The method of claim 6, wherein the line is fitted to the boundary between the "sky" and "ground" pixels using a random sample consensus (RANSAC) algorithm.

10. The method of claim 5, wherein compensating for roll comprises rotating the image such that the horizon line is parallel to a line of pixels.

11. The method of claim 5, further comprising compensating for pitch by either cropping o transiating the image to remove one or more pixels above the horizon line.

12. The method of claim 1 ,. further comprising:

13. The method of claim 12, wherein the simulated overhead view is generated by a fixed homography matrix to perform a perspective transform.

14. The method of claim 13, wherein the homography matrix is generated according to a point on the horizon line.

15. The method of claim 12, wherein calculating a heading of the vehicle comprises;

skewing the simulated overhead image; summing columns of the skewed image to generate a vector of summed columns, and

calculating a variance of the vector.

16. The method of claim 15, further comprising;

determining a maximum variance of the vector.

17. The method of claim 11 , further comprising;

determining a validity of the vehicle heading data; and

selectively using the vehicle heading data according to the validity, to control the vehicle.

18. The method of claim , further comprising:

determining a lateral movement of the vehicle using a template and the image.

19. A system for vision based navigation of a vehicle, the system comprising; an image capture module, for capturing an image from the vehicle wherein the image includes a parallel texture;

a control module, for controlling movement of the vehicle;

a processor coupled to the image capture module and the control module; and

a memory coupled to the processing module, the memory including instruction code executable by the processor for;

receiving an image from the image capture module;

estimating a roll and pitch of the vehicle using the captured image or another external source of camera attitude;

compensating for the estimated roll and pitch in the image;

calculating a heading of the vehicle relative to the parallel texture using the image in which roll and pitch is compensated; and

20. The system of claim 19, wherein the external source of the camera attitude is an inertia! rneasurement unit (IMU)