WO2004003788A2

WO2004003788A2 - Position referenced multimedia authoring and playback

Info

Publication number: WO2004003788A2
Application number: PCT/GB2003/002794
Authority: WO
Inventors: Howard Williamson
Original assignee: Inovas Limited
Priority date: 2002-06-29
Filing date: 2003-06-30
Publication date: 2004-01-08
Also published as: WO2004003788A3; AU2003251139A1; GB0215217D0

Abstract

A system (20) and methods are provided for creating a relationship between various sequential data records (22) (e.g. representing video frames, Ground Probing Radar (GPR), thermal imagery, infra-red radar, sonar, audio commentary, text files), and linking this data with a specific position in space (e.g., GPS co-ordinates) or any other positional reference or co-ordinate. The method involves constructing and storing a geo-reference record (24) containing a co-ordinate (25) and a pointer (26) to a sequential data record. Systems and methods are provided for authoring, playback and synchronised playback of multiple sequential data sets. The authoring and playback may be synchronised with way points displayed or performed interactively on a map.

Description

Position Referenced Multimedia Authoring and Playback

This invention relates to playback of sequential data, and in particular, sequential data related to positional references and synchronised playback of multiple sequential data sets.

In the field of video recording, various schemes are used to associate positional data, such as GPS (Global Positioning System) data with video footage. For example, video footage can have each frame stamped with the GPS co-ordinates as well as the time and date at, for example, the top of each video image frame. The approaches involve encoding the positional data in the non-visible portions of the video data stream. For example GPS/Video encoders such as from HORITA Co. Inc. CA, USA provide exact time and position in SMPTE (Society of Motion Picture and Television Engineers) standard Longitudinal Time Code (LTC) and SMPTE Vertical Interval Time Code (VITC) formats for recording on videotape. In each of these products, the code is synchronised to standard time through an interface to a GPS receiver. Time synchronisation provides direct correlation of GPS position data with the video recording.

Location based video playback (spatial video browsing) and map location navigation systems have been developed, such as Visi-Data from Roadware Group Inc, Ontario, Canada, that allow users to navigate by viewing a map on- screen and clicking on locations or way points on routes and the system can play the appropriate section of video that relates to the selected co-ordinate on the map. Such systems also allow the playback of a video sequence with the concurrent update of the path on the map and can handle retrieval and playback of multiple video and data sets. However, the problem with such a system is that the data structure (i.e., having positional data within the video data stream and having databases of positional data at the video file or data set level) does not adequately facilitate the random access playback of a video stream, the synchronised playback of multiple video streams, the inserting of new sections of video or the correction of errors in the positional data, for example caused by GPS drop-out.

Another problem with stamping positional co-ordinates on video frames, is that if video footage does exist, often in a non-digital form, it is difficult to view access and integrate by marking up with positional references. Stores of VHS tapes are an excellent example of this. Many organisations are compelled to invest heavily in VHS footage. Digitising this video data into sequential data sets and then stamping the video frames with positional data is time consuming, degrades the video quality through copying, and results in a data structure which suffers from the problem identified above.

It would be advantageous to synchronise the playback of multiple sequential data files and concurrently display the related position on a map.

It is an object of the present invention to create and store a relationship between sequential data records and positional data.

It is a further object of the present invention to provide synchronised playback of multiple files of sequential data records.

According to a first aspect of the present invention, there is provided a system for linking positional references to a plurality of sequential data records, the system comprising:

• a way point modification module adapted to construct a geo-reference record comprising a co-ordinate corresponding to a positional reference and a pointer to a sequential data record; and

• a geo-reference storage module adapted to store a geo-reference record responsive to said way point modification module.

Preferably the system further comprises a data viewer module adapted to access a sequential data record and an output module adapted to output signals corresponding to said accessed sequential data record. Preferably the system further comprises a map data module for storing map data associated with positional references and a map viewer module adapted to cause the output module to display images corresponding to map data and to display way points corresponding to geo-reference records .

Preferably said system further comprises a user input module for providing user input to the map viewer module and the data viewer module.

Preferably the map viewer module is further adapted to cause the output module to highlight particular way points.

Preferably the map viewer module is further adapted to cause the output module to display lines connecting way points.

Preferably said way point modification module is further adapted to access a sequential data record.

According to a second aspect of the present invention, there is provided an authoring method for linking a sequential data record to positional reference data, including the steps of:

• selecting a sequential data record; • determining a co-ordinate corresponding to said positional reference data; • constructing a geo-reference record, comprising said co-ordinate and a pointer to said selected sequential data record; and • storing said constructed geo-reference record.

Preferably the method further includes the steps of:

• outputting signals corresponding to said sequential data record; • displaying a map; and • displaying a way point corresponding to said constructed geo-reference record.

Preferably said method further includes the step of displaying a line connecting said displayed way point to another way point.

According to a third aspect of the present invention, there is provided a method of linking a sequential data record with positional reference data, including the steps of:

• displaying a map; • displaying a way point corresponding to a stored first geo-reference record, comprising a first co- ordinate corresponding to said positional reference data and a pointer to a first sequential data record; • outputting a signal corresponding to said first sequential data record; • selecting said displayed way point; • determining a second co-ordinate or determining a second pointer to a second sequential data record; • constructing a geo-reference record comprising said second co-ordinate or said second pointer; and • replacing said first geo-reference record with said second geo-reference record.

According to a fourth aspect of the present invention, there is provided a system for synchronised playback of a first sequential data file comprising first sequential data record and a second sequential data file comprising second sequential data records, the system comprising:

• at least one geo-reference storage module adapted to store geo-reference records, each record comprising a co-ordinate and a pointer to a sequential data record; • a data viewer module adapted to access a sequential data record and an output module adapted to output signals corresponding to said accessed sequential data record; • a first matching module for matching a sequential data record to a first geo-reference record; and • a second matching module for matching a first geo- reference record to a second geo-reference record.

Preferably the system further comprises an interpolation module for selecting an intermediate sequential data record between a pair of sequential data records, the pair being referenced by said matched second geo- reference record. According to a fifth aspect of the present invention, there is provided a method of synchronised playback of a first sequential data file comprising first sequential data records and a second sequential data file comprising second sequential data records, the method including:

• selecting a first sequential data record; • outputting signals corresponding to said selected first sequential data record; • matching said selected first sequential data record to a first geo-reference record; • matching said first geo-reference record to a second geo-reference record; • selecting a second sequential data record referenced by said second geo-reference record; and • outputting signals corresponding to said selected second sequential data record

Preferably the step of selecting a second sequential data record comprises the step of selecting an intermediate sequential data record between a pair of sequential data records, the pair being referenced by said matched second geo-reference record.

Preferably said method further includes the step of displaying a way point on a map corresponding to a geo- reference record.

According to a sixth aspect of the present invention, there is provided a method of synchronised playback of a first sequential data file comprising first sequential data records and a second sequential data file comprising second sequential data records, the method including:

• determining a co-ordinate; • matching the co-ordinate to a first geo-reference record; • matching the co-ordinate to a second geo-reference record; • selecting a first sequential data record referenced by said first geo-reference record; • selecting a second sequential data record referenced by said second geo-reference record; • outputting signals corresponding to said selected first sequential data record; and • outputting signals corresponding to said selected second sequential data record.

Preferably the method further includes the step of displaying a way point on a map corresponding to a geo- reference record.

Preferably the system is implemented by computer software comprising program instructions which, when loaded onto a computer, cause said computer to perform as the system. Preferably the method is implemented by computer software comprising program instructions which, when loaded onto a computer, cause said computer to carry out the method.

In order to provide a better understanding of the present invention, an embodiment will now be described by way of example only, and with reference to the accompanying Figures, in which:

Figure 1 illustrates in schematic form the system architecture in accordance with a preferred embodiment of the present invention;

Figure 2 illustrates in schematic form the components of the system for authoring;

Figure 3 illustrates in schematic form a flow chart of the steps for authoring by adding new way points;

Figure 4 illustrates a flow chart of the steps for authoring by modifying existing way points;

Figure 5 illustrates in schematic form the components of the system used for playback of multiple sequential data files;

Figure 6 illustrates a flow chart of the steps for playback of multiple sequential data files synchronised to the playback of one sequential data file;

Figure 7 illustrates a flow chart of the steps for synchronised playback of multiple sequential data files synchronised to the user input of co-ordinates on a map; Figure 8 illustrates in schematic form the fields in the registry table and geo-reference table; and

Figure 9 illustrates a map showing way points and video frames along a route or path on the map.

The invention is systems and methods for authoring and playback of multiple sequential data files (e.g., video) which relate to positional references (e.g., map coordinates) .

The invention functions to create a relationship between various sequential data records (e.g., video, Ground Probing Radar (GPR) , thermal imagery, infra-red radar, sonar, audio commentary, text files) which can be measured against time, distance or any incrementing or decrementing scale, and further to link this data with a specific position in space or any other positional reference or co-ordinate.

With reference to Figure 1, an architecture diagram of the system 10 is illustrated. A first sequential data file 11 is indexed by a first geo-reference table 12 and a first data viewer module 13 uses the geo-reference data to access the sequential data file. The data viewer uses the geo-reference table to look up specific sequential data records (e.g., video frames) using a map co-ordinate and also to look up map co-ordinates using a pointer to a sequential data record (e.g., a video frame number).

The sequential data files and geo-reference tables may be stored using conventional relational database systems, and the data viewer can be implemented in standard programming languages, such as C++, Visual Basic. The data viewer further functions to take user input and display sequential data records. For example, a data viewer controls an output module to output signals that maintain a playback window on a computer screen with play, rewind, fast forward and pause buttons and a display area for the video frame images. The data viewer outputs co-ordinates in order to synchronise the playback of other viewers and plotting of a route by mapping software.

Multiple instances of the sequential data file, geo- reference table and data viewer are supported, i.e., 14 to 16 and 17 to 19. Each instance is created by a data view creator 110 which maintains in memory a list of currently open data views.

Mapping software 111 can be a standard GIS (Geographical Information System) mapping system, which can be connected to the rest of the system using a standard programming interface, such as a ActiveX™ components or DDE (Dynamic Data Exchange) . The mapping software maintains a map image window on the computer screen. It can capture user input such as a mouse click as a co- ordinate on the map and passes this to the synchronisation module 112 during playback.

The synchronisation module receives position reference data, such as co-ordinates or chainage (e.g., distance along a pipe) from the mapping software. It then broadcasts the co-ordinate to all of the open data views which then look up the corresponding sequential data records and display them.

Alternatively, the synchronisation module receives position reference data from a data viewer, in which case as well as broadcasting it to all of the open data views, it also send it to the mapping software so that it can display way points. A way point can be represented on a computer display as a dot with connecting lines to other way points, or just a line segment, superimposed on a map image or a site plan.

Thus, the user can synchronise the playback by either clicking on the mapping software display window or clicking on any of the data viewer windows. For example, if the user selects the first data viewer 13, it can play the video in the sequential data file 11 frame by frame, and the viewer periodically looks up and sends 113 the co-ordinate found in the geo-reference table, which corresponds to the currently displayed video frame, to the synchronisation module. The synchronisation module broadcasts the co-ordinate to the mapping software to update the way point 114 and to the other open data viewers 115, 116. The first data viewer continues playing the video by stepping through the sequential data records.

However, alternatively, the user can select the second data viewer 16 as the synchronising data screen. The second data viewer plays video from the second sequential data file 14 and uses the second geo-reference table to look up co-ordinates corresponding to the current video frame and, as before, it sends the co-ordinate 117 to the synchronisation module which broadcasts the co-ordinate to the mapping software 114, and to the other open data viewers 118, 119. Thus, the user can playback multiple video screens all synchronised together with a synchronised plot of the route on a map window, and the user can select the source of the synchronisation by clicking on the map or using the data viewers to playback, pause, wind or rewind, as desired.

At the beginning of the session, a user can select 120 the survey data that they wish to look at by accessing a survey register 121 that is a file containing a list and descriptions of each of survey that has geo-reference tables and sequential data files that are available for playback.

Although video has been used in this preferred embodiment, other sequential data may also be used, such as Ground Probing Radar (GPR) , thermal imagery data, infra-red radar data, sonar data, audio data, text data, etc. Furthermore, although a preferred embodiment uses mapping software and map data, any location referenced data could be used, including site plans or Computer Aided Design (CAD) drawings.

Figure 2 illustrates the components of the system 20 that are used in authoring, i.e., creating geo-reference tables. The sequential data file 21 contains sequential data records 22. The geo-reference table 23 stores geo- reference records 24 that can represent way points on a journey. Each record contains a co-ordinate 25 corresponding to a positional reference and a pointer 26 to a sequential data record. The co-ordinate may be one- dimensional, for example having a single variable storing a distance (D) (or chainage) along a pipe. The co- ordinate may be two-dimensional, for example an X and Y value defining a location on a plan or a latitude and longitude value defining a position on the Earth. The co-ordinate may also have three or more dimensions. For example, the third dimension can record a measure of the Z direction or height. The co-ordinate may be a combination of non-orthogonal one-dimensional, two dimensional, etc. data, e.g. X, Y, Z and D. The pointer in the geo-reference record is typically a video frame number, however it can be any integer or other variable that can be used to look up a record in a file or table or data set.

The way point modification module 27 constructs geo- reference records and stores them in the geo-reference table. It can also modify, replace or delete existing geo-reference records.

The output module 28 is a computer display screen for displaying images corresponding to sequential data records, such as video frames, GPR or text data, etc. The output module could alternatively be a sound card and loud speaker for playback of audio signals corresponding to sequential data records, such as a spoken commentary.

The data viewer 29 functions as 13 described with reference to Figure 1. The map data 210 and map viewer module 211 correspond to the data and program of the mapping software 111 described with reference to Figure 1. The user input module 212 is a computer mouse with associated software drivers. Alternatively, it might be a computer keyboard or voice input system. In the case of mouse input, then the user can provide input to the map viewer module by using the mouse to position a pointer, then clicking at a particular co-ordinate on a map presented by the map viewer module on the display. Similarly, mouse and pointer input can provide input to the data viewer during playback of e.g., video, by the user clicking on play, pause, rewind or fast forward buttons presented by the data viewer on the display.

Figure 3 illustrates a flowchart 30 of the steps in the method of authoring by adding new geo-reference records to the geo-reference table, corresponding to adding new way points, in accordance with a preferred aspect of the present invention. Most of the steps in Figure 3 are illustrated by corresponding numbers adjacent to the arrow heads on Figure 2.

In order to record a new way point and construct a geo- reference record, the way point modification module needs to determine or be provided with a pointer and a co- ordinate. Typically the data viewer displays 31 an image corresponding to sequential data record. The way point modification module selects 32 a sequential data record either during playback of the sequential data file, or by the user inputting to the data viewer a selection of a particular sequential data record. Optionally, the map viewer can read map data and display corresponding images 33 on the output module. Although in Figure 2 one output module is used for both sequential data viewing and map viewing, clearly multiple output modules could be used, e.g., several computer screens or audio systems.

The way point modification module determines 34 a co- ordinate corresponding to the selected sequential data record. Typically this can be done by the user inputting to the map viewer the selection of a co-ordinate, which is then passed to the way point modification module. Alternatively, the way point modification module can take its input positional reference directly from any other source including: • a co-ordinate logging system, such as a GPS unit; • a log file of co-ordinate data, e.g. GPS data or chainage data; and • a chainage measuring system, e.g. using electronic capture of the length of a cable.

Now having the pointer to a sequential data record and a co-ordinate, the way point modification unit constructs 35 a geo-reference record comprising the co-ordinate and the pointer. The way point modification module then stores 36 the newly constructed geo-reference record in the geo-reference table.

Finally, and optionally, the system displays 37 a way point corresponding to the constructed geo-reference record. This can be performed by the map viewer which, when fed with geo-reference records, displays corresponding way points and connecting lines on the output module.

By way of example using video playback, these steps are shown on Figure 2. First, the data viewer retrieves video frames 31A and displays them 31B. The way point modification module is either fed directly 32A with frame numbers from the video file, or the user clicks 32B on a video playback window and the data viewer sends the frame number 32C to the way point modification module. Meanwhile, the map viewer retrieves 33A map data and displays the map 33B in a separate window on the display. The way point modification module retrieves the co- ordinate of the way point when the user clicks 34A on a location in the map image, and the map viewer forwards the co-ordinate 34B to the way point modification module, which then constructs the geo-reference record from the co-ordinate and the frame number and stores 36A, 36B the new geo-reference record in the geo-reference table. Finally, geo-reference records input 37A to the map viewer are displayed as way points 37B on the map on the computer display. Lines connecting the way points may also be displayed 38.

Figure 4 illustrates a flowchart of the steps for authoring by modifying the geo-reference table corresponding to moving existing way points in accordance with the present invention. The map viewer displays 41 a map using the output module. The map viewer then displays 42 a way point corresponding to a stored geo- reference record and displays 43 a line connecting the way point to another way point. The data viewer outputs 44 a signal corresponding to the stored data recorded that is pointed to by the stored geo-reference record. The map viewer selects 45 a way point responsive to user input, for example using a mouse click. The way point modification module determines 46 a new co-ordinate for the way point. It may do this in response to user input by dragging and dropping the way point using the map viewer software. Alternatively, the user may select a way point and use a keyboard to input a new co-ordinate. Alternatively, the way point modification module determines 46 a pointer to a new sequential data record. This may be performed by the user first selecting a way point using the map viewer software, then using the data viewer to scroll or otherwise locate a sequential data record, then the data viewer forwards the pointer to that record to the way point modification module. The way point modification module constructs 47 a new geo- reference record from the new co-ordinate and the existing pointer or, alternatively, the existing co- ordinate and the new pointer. Finally, the way point modification module stores 48 the newly constructed geo- reference record in the geo-reference table. It may do this by deleting the original geo-reference record and inserting a new geo-reference record, or by modifying the original geo-reference record.

Figure 5 illustrates in schematic form the components of the system according to a preferred embodiment of the present invention for playback of multiple sequential data files. With reference to Figure 5, a first sequential data file 51 comprises sequential data records 52. A second sequential data file 53 comprises sequential data records 54. One or more geo-reference tables 55 are stored in a database or a memory table, or some other storage medium. The tables may be arranged logically as one table, or may reside as separate tables in separate files. In the first geo-reference table, a first geo-reference record 56 is stored. It comprises a co-ordinate 57 and a pointer 58 to a sequential data record. Similarly, a second geo-reference table stores a second geo-reference record 59 which comprises a co- ordinate 510 and a pointer 511.

An output module 512 is adapted to output signals corresponding to sequential data records. A first data viewer module 513 is adapted to access sequential data records and control the output module to output signals corresponding to the accessed sequential data records. The data viewer further comprises a first matching module for matching a sequential data record to a first geo- reference record. In the preferred embodiment, this first matching module is performed by a co-ordinate look- up module 514 that uses a pointer to a sequential data record to query the first geo-reference table to find the first geo-reference record. The system further comprises a second matching module for matching the first geo- reference record to a second geo-reference record. In the preferred embodiment, this is performed by the co- ordinate look-up module 514, a synchronisation module 515 (corresponding to 112 in Figure 1) and a pointer look-up module 516 in a second data viewer module 517.

The synchronisation of the playback of a first sequential data file and a second sequential data file is performed by the first data viewer, the synchronisation module and the second data viewer. The first data viewer accesses sequential data and uses the pointer to a first sequential data record to look up co-ordinates in a first geo-reference table. The found co-ordinate is broadcast by the synchronisation module to other data viewers, including the second data viewer. The second data viewer uses the co-ordinate to look up in a second geo-reference table a pointer to a second sequential data record in its own sequential data file. The second data viewer can then control the output module to output signals corresponding to this second sequential data record. Thus, the first and second sequential data files are played back in a way synchronised by the co-ordinates associated with the sequential data records.

The system further comprises a user input module 518 for providing user input to the data viewer modules and a map viewer module 519. The map viewer module accesses map data 520 and accepts input co-ordinates, for example from geo-reference tables or synchronisation module. The map viewer module is adapted to cause an output module to further cause the display of images corresponding to map data and to display way points corresponding to geo- reference records.

Figure 6 illustrates a flowchart of the steps of playback of multiple sequential data files in accordance with a preferred embodiment of the present invention, synchronised to the playback of one sequential data file. Responsive to user input, the first data viewer selects 61 a first sequential data record. Responsive to the first data viewer, the output module outputs signals 62 corresponding to the selected first sequential data record. The co-ordinate look-up module of the first data viewer matches 63 the selected first sequential data record to a first geo-reference record. The co-ordinate look-up module in the first data viewer retrieves the co- ordinate from the first geo-reference record and passes it to the synchronisation module which broadcasts it to the map viewer module and the other open data viewers, including the second data viewer. The second data viewer comprises a pointer look-up module that uses this co- ordinate to look up a second geo-reference record in a second geo-reference table. Thus, the co-ordinate look- up module in the first data viewer, the synchronisation module and the pointer look-up module in the second data viewer co-operate to match 64 the first geo-reference record to the second geo-reference record. It is clear to one skilled in the art of programming that there are alternative methods of matching a first geo-reference record to a second geo-reference record. After receiving the broadcast co-ordinate from the synchronisation module, the map viewer module may optionally display 65 a way point on the map, that is being displayed using an output module, corresponding to a geo-reference record. This may be the first geo-reference record or the second geo-reference record or an interpolation between the two.

Finally, the second data viewer having used its pointer look-up module to retrieve the pointer in the second geo- reference record, selects 66 and retrieves the corresponding second sequential data record and causes the output module to output signals 67 corresponding to the second sequential data record.

With reference to Figure 5, the arrow heads are labelled with numbers corresponding to the steps of the flowchart of Figure 6. By way of example, the playback of two video files synchronised by geo-references will be given.

A user watches a first video in a data viewer window on screen. The user clicks on the play button 61A and the data viewer retrieves a video frame 61B from the video file. As an alternative to user input, then the data viewer can automatically select video frames for synchronisation, for example choosing every tenth frame. The data viewer displays the selected video frame 62B. The data viewer looks up 63 a co-ordinate 64A which it sends to the synchronisation module 64B, which in turn broadcasts it to the other open data viewers 64C and the map viewer 65A. The map viewer displays the co-ordinate as a way point on the map on the display 65B. Alternatively, the map viewer may be triggered in an automatic way to display way points from a geo-reference table, or the data viewers may retrieve co-ordinates from a geo-reference table and forward them to the map viewer for display as way points and connecting lines.

On receiving the broadcast co-ordinate, a second data viewer looks up two frame numbers in the second geo- reference table using the co-ordinate 64D which returns the corresponding frame numbers 66A. The retrieved frame numbers are then used to interpolate an intermediate frame number 66B to access 66C a corresponding video frame for the second data viewer display window. The video frame is retrieved 67A and is displayed 67B on the screen.

With reference to Figure 7, an alternative playback method 70 is convenient for the user. This is when the user clicks on the map 71 to indicate a co-ordinate to the system which then synchronises each of the data viewer windows to play the most closely related sequential data records. The map viewer sends the co- ordinate to the synchronisation module that broadcasts it to data viewers that match 72 the co-ordinate to a first geo-reference record, match 73 the co-ordinate to a second geo-reference record, select 74 a first sequential data record referenced by the first geo-reference record and select 75 a second sequential data record referenced by the second geo-reference record. The data viewers then control or signal to the output module to output 76 signals corresponding to the selected first sequential data record and finally output 77 signals corresponding to the selected second sequential data record.

In the video playback example, and with reference to the labelled arrow heads in Figure 5, the corresponding steps to those of Figure 7 are that the user inputs a mouse click 68 into the map viewer, which then sends the co- ordinates corresponding to the location of the mouse click to a synchronisation module 69. The synchronisation module and the data viewers co-operate to match the co-ordinate to a first geo-reference record and a second geo-reference record 64D. The data viewers can then retrieve the frame numbers from the geo-reference records 66A, 66B and retrieve 66C and display 67A, 67B the appropriate video frames.

With reference to Figure 8, the fields in the registry table 81 and a geo-reference file 82 are shown. A geo- reference table 83 is accompanied by a header table 84 which contains the file identifier, a survey identifier and a description of the survey. Each of the items in the header table refer to all of the geo-reference records in the geo-reference table. The geo-reference table 83 comprises the unique table identifier, a pointer or data index (e.g., video frame number) and a co- ordinate with four dimensions, i.e., x, y, z and chainage (d=distance) .

With reference to Figure 9, a map 90 is shown with a route which connects way points 91 (solid line) and the correct route 92 (dashed line) . Four video frames are shown at different way points 93. The user can play the whole video for the route in a data viewer window. The initial route 91 does not follow the road because of GPS drop-out, therefore some of the way points have incorrect co-ordinates. The user can playback the whole video and pause at recognisable landmarks, such as junctions or bends in the road and drag and drop the highlighted way point to the correct position on the map. This corresponds to the flowchart illustrated in Figure 4.

With reference to Figure 10, the user interface presented to a user is shown schematically. The GIS software generates and updates a map window 100. Way points 101 are superimposed with connecting lines 102 on the map features 103, that in this case represent a road. Three data viewer windows for three different surveys are concurrently displayed and updated and show video filmed by walking along the road with a video camera 104, video filmed by a pig in a pipeline 105 and ground probing radar data 106 recorded while walking or driving along the route. The user initiates synchronised playback in each of the data viewer windows and the path on the map window by clicking on either video or radar windows or on the map.

The present invention adds location to moving digital imagery by integrating digital video footage of an area with site maps or plans, geographical information systems (GIS) or computer aided design (CAD) drawings. Additional data streams can also be incorporated into the software, such as ground probing radar (GPR) , thermal imagery, infra-red radar or sonar.

Thus technology can be applied to numerous applications in a diverse range of market sectors. Some examples include highways, embankments and road inspection, tunnels and pipe inspection, aerial based inspection and driver training.

An example of the use of the present invention that highlights its benefits is in the training of bus drivers.

Using a standard digital video camera, video footage of the bus route inward and outward journeys is captured using a driver's eye view perspective and position. Multiple cameras may also be used pointing in different directions to give a panoramic view of a junction or crossroad, for example.

To capture the imagery, the camera is simply mounted in the coach and the route is driven as normal. Once recorded, the video footage is recorded from mini DV format to a computer readable format and referenced to a route map of the area. Route maps can be scanned into the system supplied by a digital map supplier from a street map or by using an aerial photograph of the area. The video footage is referenced to the map using the method according to the present invention. During subsequent playback according to the present invention, a moving tracker symbol will appear on the map whenever the video is played. This shows the current position of the bus in the video being viewed, the tracker continues to move along the route showing the user on the map exactly where the vehicle is positioned and plots the progress along the route. Alternatively, clicking at any point along the route will cause the video to jump instantly to that point. Trainers' comments are added at all the appropriate locations along the route. This sequential audio data is recorded at a separate location and subsequently incorporated into the system. This voice information carries detail on vehicle positioning, hazards, fare stages, road surface, etc. Additional video clips enhance this information for difficult junctions, and the use of road traffic symbols for known road features, such as crossings, etc. Keyboard usage is kept at a minimum, to operate the system drivers may simply press an appropriate icon on a specially adapted keypad.

In this example, several benefits are apparent during driver route training. Drivers can study new and existing bus routes from the safety and comfort of the classroom environment, save on fuel and training costs, reduce the possibility of accidents on the roads, prevent accidents by highlighting problem sections on routes, reducing insurance claims, identify hazard areas on the route, learn points of interest, vehicle positioning and fare stages, improve customer service, plan new routes, investigate accidents. Although the embodiments of the invention described with reference to the drawings comprise computer apparatus and processes performed in computer apparatus, the invention also extends to computer programs, particularly computer programs on or in a carrier, adapted for putting the invention into practice. The program may be in the form of source code, object code, a code of intermediate source and object code such as in partially compiled form suitable for use in the implementation of the processes according to the invention. The carrier may be any entity or device capable of carrying the program.

For example, the carrier may comprise a storage medium, such as ROM, for example a CD ROM or a semiconductor ROM, or a magnetic recording medium, for example, floppy disc or hard disc. Further, the carrier may be a transmissible carrier such as an electrical or optical signal which may be conveyed via electrical or optical cable or by radio or other means.

When the program is embodied in a signal which may be conveyed directly by a cable or other device or means, the carrier may be constituted by such cable or other device or means.

Alternatively, the carrier may be an integrated circuit in which the program is embedded, the integrated circuit being adapted for performing, or for use in the performance of, the relevant processes.

Further modifications and improvements may be added without departing from the scope of the invention herein described.

Claims

1. A system for linking positional references to a plurality of sequential data records, the system comprising:

• a way point modification module adapted to construct a geo-reference record comprising a co- ordinate corresponding to a positional reference and a pointer to a sequential data record; and

2. The system of Claim 1 further comprising a data viewer module adapted to access a sequential data record and an output module adapted to output signals corresponding to said accessed sequential data record.

3. The system of Claim 2 further comprising a map data module for storing map data associated with positional references and a map viewer module adapted to cause said output module to display images corresponding to map data and to display way points corresponding to geo-reference records.

4. The system of Claim 3 further comprising a user input module for providing user input to said map viewer module and the data viewer module.

5. The system of any of Claims 3 to 4 wherein said map viewer module is further adapted to cause said output module to highlight particular way points.

6. The system of any of Claims 3 to 5 wherein said map viewer module is further adapted to cause said output module to display lines connecting way points.

7. The system of any previous Claim wherein said way point modification module is further adapted to access a sequential data record.

8. An authoring method for linking a sequential data record to positional reference data, including the steps of:

9. The method of Claim 8 further including the steps of:

10. The method of Claim 9 further including the step of displaying a line connecting said displayed way point to another way point.

11. A method of linking a sequential data record with positional reference data, including the steps of:

• displaying a map; • displaying a way point corresponding to a stored first geo-reference record, the geo-reference record comprising a first co-ordinate corresponding to said positional reference data and a pointer to a first sequential data record; • outputting a signal corresponding to said first sequential data record; • selecting said displayed way point; • determining a second co-ordinate; • constructing a geo-reference record comprising said second; and • replacing said first geo-reference record with said second geo-reference record.

12. A method of linking a sequential data record with positional reference data, including the steps of:

• displaying a map; • displaying a way point corresponding to a stored first geo-reference record, the geo-reference record comprising a first co-ordinate corresponding to said positional reference data and a pointer to a first sequential data record; • outputting a signal corresponding to said first sequential data record; • selecting said displayed way point; • determining a second pointer to a second sequential data record; • constructing a geo-reference record comprising said second pointer; and • replacing said first geo-reference record with said second geo-reference record.

13. A system for synchronised playback of a first sequential data file comprising first sequential data records and a second sequential data file comprising second sequential data records, the system comprising:

14. The system of Claim 13 further comprising an interpolation module for selecting an intermediate sequential data record between a pair of sequential data records, the pair being referenced by said second geo-reference record.

15. A method of synchronised playback of a first sequential data file comprising first sequential data records and a second sequential data file comprising second sequential data records, the method including:

16. The method of Claim 15 wherein the step of selecting a second sequential data record comprises the steps of selecting a pair of second sequential data records both referenced by said second geo-reference record and selecting an intermediate sequential data record by interpolation between said pair.

17. The method of any of Claims 15 to 16 further including the step of displaying a way point on a map corresponding to a geo-reference record.

18. A method of synchronised playback of a first sequential data file comprising first sequential data records and a second sequential data file comprising second sequential data records, the method including:

19. The method of Claim 18 wherein the step of selecting a second sequential data record comprises the steps of selecting a pair of second sequential data records both referenced by said second geo-reference record and selecting an intermediate sequential data record by interpolation between said pair.

20. The method of any of Claims 18 to 19 further including the step of displaying a way point on a map corresponding to a geo-reference record.

21. Computer software comprising program instructions which, when loaded onto a computer, cause said computer to perform as the system of any of Claims 1 to 7.

22. Computer software comprising program instructions which, when loaded onto a computer, cause said computer to perform as the system of any of Claims 13 to 14.

23. Computer software comprising program instructions which, when loaded onto a computer, cause said computer to carry out the method of any of Claims 8 to 12.

24. Computer software comprising program instructions which, when loaded onto a computer, cause said computer to carry out the method of any of Claims 15 to 20.