US20020064764A1

US20020064764A1 - Multimedia analysis system and method of use therefor

Info

Publication number: US20020064764A1
Application number: US09/725,133
Authority: US
Inventors: Lewis Fishman; Robert Dickinson
Original assignee: Acushnet Co
Current assignee: Acushnet Co
Priority date: 2000-11-29
Filing date: 2000-11-29
Publication date: 2002-05-30
Also published as: JP2002248188A

Abstract

A multimedia analysis system for capturing and comparing sports performances of a player is provided. The system includes at least one video camera for acquiring a plurality of sets of video data of player performances, with the location of the at least one video camera being substantially the same when acquiring each set of video data. The system also includes at least one storage device for storing the plurality of sets of video data, at least one processor for mixing at least two of the sets of video data, and at least one display device for displaying the at least two sets of mixed data. The processor substantially synchronizes and superimposes the at least two sets of video data to form an aggregate set of video data. A launch monitor may also be included for acquiring a plurality of sets of flight data of objects hit by the player.

Description

FIELD OF THE INVENTION

The present invention relates to an apparatus and method for evaluating sports performance, and more particularly relates to an improved multimedia analysis system and method for analyzing performance of a sportsman and the equipment used thereby. In particular, the present invention relates to a multimedia analysis system and method for evaluating the technique of a golf player. The present invention also concerns the creation of a library of data sets related to the performance of a subject in a known set of conditions. More particularly, the present invention permits the creation, archival, and comparison of data sets concerning an athlete's performance during test sessions under a variety of test conditions.

BACKGROUND OF THE INVENTION

Technology is known for capturing images of a motion sequence for use in evaluating athletic performance. Such captured images can serve as instructional tools to teach and demonstrate play characteristics. Athletes and instructors alike may use the captured images to evaluate and compare the technique of the athlete with a preferred technique of an expert, and to indicate points of departure therefrom in order to improve overall performance. Sports that may derive benefit from motion sequence analysis include, but are not limited to, golf, tennis, baseball, football, skiing, and bowling.

An example of the use of visual recording in sport is disclosed in U.S. Pat. No. 5,797,805. A partially prerecorded golf lesson videotape includes gaps in predetermined locations, into which are inserted the full motion video of a player's golf swing and selected still frames. The still frames are selected to match the player's position to a professional golfer's position in corresponding still frames so that a split screen, side-by-side view can be produced showing the player's and professional's positions at various points along a golf swing.

Another sports technique video training device is disclosed in U.S. Pat. No. 5,249,967. The device permits a student athlete to closely pattern his dynamic technique after that of a recognized master. A pair of video cameras transmit live images of the student as seen from those directions that best represent the technique. A video overlay generator receives the live images and combines them, while still in their video signal format, with a corresponding set of self-generated template images that represent, in static outline form, the dynamic technique of the master in the desired sport situation. With respect to golf, an outline may be used to delineate the path of travel of a respective extremity of the master during execution of the technique; for example, the path of travel of the master's hands during the swing may be shown with a continuous or dashed outline. A different outline may be used to represent the path of travel of the golf club head during the backswing, and a further outline may be used to represent positions of the golf club head on the downswing.

U.S. Pat. No. 5,486,001 additionally discloses a personalized instructional aid for enabling and assisting a person to emulate a predetermined movement such as a golf swing. A video camera is used to capture movement of the person undertaking a golf swing, and bio-mechanical information is collected from mechanical devices attached to the person or golf club moved by the person, or from sensing devices such as weight or load sensitive pads, monitors, electronic light beaming devices or the like. A computer database includes selected prerecorded video signals in digital form such as a selection of swings showing the movement of different golf professionals to be emulated. The computer generates an audio-visual presentation in the form of split-screening with images of the preferred technique displayed alongside the present technique, or superimposing of the preferred technique over the present technique such as in the form of stick figures or detailed drawings to show the viewer any variances between his stroke and the preferred stroke.

Despite these developments, there exists a need for an improved multimedia analysis system and method for use by sportsmen. In particular, there is a need for a real-time video-based system for acquiring and archiving an individual's golf swings, along with associated impact data relative to the golf club and golf ball. There further exists a need for a system that allows the individual to compare personal performances from different practice sessions, as well as allows equipment fitting and equipment testing comparisons. Additionally, there is a need for a system that also allows data to be collected regarding various equipment and practice conditions, as well as other personal characteristics that may vary over time.

The present invention provides an apparatus and method that are capable of providing these improvements.

SUMMARY OF THE INVENTION

Broadly, the present invention is directed to a motion sequence analysis system for capturing a set of real-time images of a motion sequence and correlating such images with a set of archived images. Each set of captured, real-time images may be correlated in real-time or time-delay with one or more archived sets of images. Additionally, each set of captured, real-time images may be added to an archive, to permit comparison with other archived sets. Two or more archived data sets may be compared. In a preferred embodiment, video techniques such as ghosting, overlays, and telestrator outlines may be used. Player swing mechanics from swing initiation to ball impact may be captured and analyzed. Data also may be collected on ball launch conditions with regard to specific swing sequences, including ball speed, launch angle, backspin, sidespin rate, rifle spin rate, and carry distances. Other flight conditions and shot patterns may be monitored as well. The use of other non-video data also forms part of the present development. Audio data may be collected for use in analyzing club-ball impacts, and data concerning personal characteristics of a golfer or characteristics of the golfer's equipment may be compiled.

Golf swings of the same player may be compared at different times during the course the player's career, allowing him to see the similarities and differences in his technique as well as equipment performance. Performances may be archived on storage devices, including videotape, CD or DVD, and the captured or recorded images may be used for evaluation purposes by third parties such as instructors or PGA professionals.

Thus, the present invention is related to a method of mixing data concerning sports performances of a player. The method includes: defining a first set of test conditions applied to the player; acquiring a first set of video data on the player's performance at a first time, the first set of video data being acquired during the first set of test conditions; storing the first sets of test conditions and video data; defining a second set of test conditions applied to the player, the first and second sets of test conditions having at least one test condition in common; acquiring a second set of video data on the player's performance at a second time, the second set of video data being acquired during the second set of test conditions; storing the second sets of test conditions and video data; mixing at least a portion of the first and second sets of video data to form a superimposed set; and displaying the superimposed set.

The first and second sets of video data may each have a single video frame, or instead may each have at least two video frames. Preferably, the superimposed sets of video data are substantially synchronized from a start time to a finish time. A video camera with a capture rate of at least about 250 frames per second may be used to acquire the first and second sets of video data. The mixing of video data may occur with a first percentage of the pixels of each frame selected from one of the first and second video data sets and a second percentage of the pixels of each frame selected from the other of the first and second video data sets. Preferably, the sum of the first and second percentages is one hundred, with the first percentage being between about 20 percent and about 40 percent and the second percentage being between about 60 percent and about 80 percent.

The method may further include: acquiring a first set of non-video data on the player's performance at the first time, the first set of non-video data being acquired during the first set of test conditions; storing the first set of non-video data; acquiring a second set of non-video data on the player's performance at the second time, the second set of non-video data being acquired during the second set of test conditions; storing the second set of non-video data; mixing at least a portion of the first and second sets of non-video data to form a non-video data superimposed set; and displaying or playing at least a portion of the non-video data superimposed set. Preferably, the superimposed sets of non-video data are substantially synchronized from the start time to the finish time, and the superimposed sets of non-video data are substantially synchronized from the start time to the finish time with the superimposed sets of video data. The non-video data may include at least one of audio data and launch monitor data.

In the preferred embodiment, the first and second sets of video data are acquired using at least one analog or digital video camera, and the first and second sets of non-video data are acquired using at least one of a microphone and a launch monitor. Each set of test conditions may include at least one of the equipment of the player, the environmental conditions, and the personal characteristics of the player. The mixing of video data may include using at least one key effect, and preferably the video data includes data from at least two video cameras that are disposed about substantially the same axis and/or substantially the same plane.

The present invention is further related to a method of overlaying data sets of sports performances of a player. The method includes: defining a plurality of test conditions applied to the player; acquiring a plurality of sets of video data on the player's performance, each set of video data having at least one test condition in common; storing the plurality of sets of test conditions and video data; mixing at least a portion of at least two of the sets of video data to form a superimposed set; and displaying the superimposed set. The method may further include: acquiring a plurality of sets of non-video data on the player's performance, each set of non-video data having at least one test condition in common; storing the plurality of sets of non-video data; mixing at least a portion of at least two of the sets of non-video data to form a non-video data superimposed set; and displaying or playing the non-video data superimposed set. Also, the method may include synchronizing the plurality of sets of test conditions, video data, and non-video data.

The present invention also is related to a multimedia analysis system for capturing and comparing sports performances of a player. The system includes at least one video camera for acquiring a plurality of sets of video data of player performances, at least one storage device for storing the plurality of sets of video data, at least one processor for mixing at least two of the sets of video data, and at least one display device for displaying the at least two sets of mixed data. The processor substantially synchronizes and superimposes the at least two sets of video data to form an aggregate set of video data. Preferably, the location of the at least one video camera is substantially the same when acquiring each set of video data.

The system may further include a launch monitor for acquiring a plurality of sets of flight data of objects hit by the player, and at least one processor for mixing at least two of the sets of flight data. The at least one processor substantially synchronizes and superimposes the at least two sets of flight data to form an aggregate set of flight data, with the aggregate set of flight data also being synchronized with the aggregate set of video data. The plurality of sets of video data may be acquired by at least two video cameras that are disposed about substantially the same axis and/or substantially the same plane and aimed at the player.

BRIEF DESCRIPTION OF THE DRAWINGS

Preferred features of the present invention are disclosed in the accompanying drawings, wherein similar reference characters denote similar elements throughout the several views, and wherein: [0017]
FIG. 1 is a perspective view of one embodiment of the motion sequence analysis system according to the present invention, showing a golfer addressing a golf ball; [0018]
FIG. 2 is a schematic top view of the system of FIG. 1; [0019]
FIG. 3 is a schematic front view of the system of FIG. 1; [0020]
FIG. 4 is a block diagram showing the overall system operation of one embodiment of the present invention; [0021]
FIG. 5([0022] a) shows an embodiment of the present invention having two views of golfer 110 as captured by cameras at different times, the views being displayed in split-screen format on a display device;
FIG. 5([0023] b) shows the two views displayed in FIG. 5(a) superimposed upon each other;
FIGS. [0024] 5(c)-5(d) show the superimposed views of FIG. 5(b), with the golfer in each view filled with a uniform color;
FIG. 6 shows another embodiment of the present invention with three regions on a display device including two composite views and a region for test conditions to be listed or otherwise displayed; [0025]
FIG. 7 is a flow chart depicting the steps in a preferred embodiment of the present invention; and [0026]
FIG. 8 is a flow chart depicting the steps in another preferred embodiment of the present invention.[0027]

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Referring to FIGS. [0028] 1-3, a multimedia analysis system 100 constructed according the present invention is shown in a preferred configuration with respect to a golfer 110. While the preferred embodiment described herein refers to the sport of golf, the present invention is applicable to analysis of player and equipment performances in other sports or activities for which the study of deviations in technique mechanics and other performance indicators may be useful. Preferably, motion analysis system 100 includes four high speed video cameras 120, 122, 124, 126, optimally disposed at various locations about golfer 110 in a space 105. The golfer 110 is positioned to perform a golf swing, preferably while holding a club 112 and standing on a mat or other surface 108 facing a teed golf ball 114, with the position of the tee being fixed with respect to mat 108. Although club 112 is depicted in FIG. 1 as a wood-type golf club, the present invention may also be used with respect to other clubs such as irons or putters. Motion analysis system 100 is adaptable for use by golfers 110 under various test conditions such as golfers with different physical characteristics, as will be discussed shortly. The location of mat 108 and the space defined thereby, in general, defines the central focal area about which the various components of motion analysis system 100 are positioned. A computer system 130 and display device 163 are included, as also will be described shortly.
Data collection for golf ball trajectory and flight characteristics preferably is facilitated through the use of a golf ball launch monitor [0029] 118, as shown in FIG. 1. Launch monitor 118 is disposed in front of golfer 110 and adjacent to mat 108. The launch monitor measures golf ball flight characteristics and club head swing characteristics. Examples of launch monitors suitable for the use with the present invention are disclosed in U.S. Pat. No. 5,501,463 and U.S. Pat. No. 5,575,719, both issued to Gobush et al. and directed to a “Method and Apparatus to Determine Object Striking Instrument Movement Conditions,” and U.S. patent application No. 09/537,295 directed to “Launch Monitor System with a Calibration Fixture and a Method for Use Thereof” to Gobush et al. All three above-mentioned references are assigned to the Acushnet Company, and the disclosures of these references are hereby incorporated by referenced in their entirety.
Data collection further includes audio data with respect to golf ball impact. Such data may be useful, for example, in synchronizing video data, and also may be an indicator of the quality of ball strike. Audio and video data sets from a test session may be first synchronized with each other, and then may be synchronized with audio and video data sets from other test sessions. The moment in time at which a golf club first impacts a golf ball may be determined, for example, using a particular audio frequency “signature,” permitting multiple sets of data to be synchronized in time. A [0030] microphone 116 is provided proximate teed golf ball 114 for receiving audio. Various audio processing techniques may be used for synchronization of audio data with video data, including those disclosed in U.S. Pat. No. 6,124,895 to Fielder, assigned to Dolby Laboratories Licensing Corporation and directed to “Frame-Based Audio Coding with Video/Audio Data Synchronization by Dynamic Audio Frame Alignment.”
Preferably, [0031] cameras 120, 122, 124, 126 are situated in locations with respect to golfer 110 that facilitate the acquisition of video of the golf swing, ball impact, and ball flight. To this end, a preferred embodiment of motion analysis system 100 is shown, for example, with respect to a right-handed golfer 110, although the present invention is readily adaptable to a left-handed golfer as well. When golfer 110 addresses teed ball 116, front camera 122 has a line of sight to the front of golfer 110, back camera 120 has a line of sight to the back of golfer 110, rear camera 124 has a line of sight to the side of golfer 110, and top camera 126 has a line of sight toward the top of the head of golfer 110. In alternate embodiments, as few as one camera may be provided for video data capture, and preferably, motion analysis system 100 includes at least one camera positioned with a line of sight to the front of golfer 110.
In the preferred embodiment, the positioning of [0032] cameras 120, 122, 124, 126 is selected to permit simultaneous image capture from various body perspectives, thereby allowing more data collection and a concomitantly more detailed analysis. Preferably, cameras 120, 122, 124, 126 are coupled to permanent or semi-permanent mountings to ensure repeatability of data collection conditions from one session to another. More preferably, cameras 120, 122, 124, 126 are coupled to permanent or semi-permanent fixtures disposed toward the perimetral regions of a space that is at least partially enclosed such as space 105. Most preferably, cameras 120, 122, 124, 126 are coupled to permanent or semi-permanent brackets 128 fixed to the walls and ceiling of a space. When golfer 110 hits teed ball 114, the flight of ball 114 may be unrestricted or alternatively constrained as by the use of nets. Advantageously, the preferred configuration of the present development permits golfer 110 to hits balls into a fairway, for example, so that golfer 110 can watch ball flight. In another embodiment, a net is used to permit the positioning of a camera opposite camera 124, although a clean line of sight might not be feasible.
With particular reference to FIGS. [0033] 2-3, in the preferred configuration, cameras 120, 122, 124, 126 are aligned with central point 140. Preferably, cameras 120, 122, 124 are generally coplanar. Moreover, cameras 120, 122 are located along parallel vertical axes 132, 134 respectively, and are coaxial and aimed toward each other along axis 144. In the embodiment shown schematically in FIGS. 2 and 3, cameras 124, 126 are disposed along a center plane that contains the axes 142, 143 and is perpendicular to the plane of the page. Preferably, teed golf ball 114 is placed along an axis 146 that is parallel to axis 142 and closer to camera 122 than camera 120 (for the exemplar right-handed player discussed herein), and is also placed along an axis 148 that is parallel to axis 144 and closer to mat front edge 150 than camera 124. Furthermore, the placement of teed golf ball 114 is preferably offset a distance d₁of between about 0 and about 6 inches from axis 144 and a distance d₂of between about 24 and about 48 inches from axis 142. More preferably, teed golf ball 114 is offset by a distance d₁of between about 2 to about 4 inches from axis 144 and a distance d₂of between about 36 to about 48 inches from axis 142.
[0034] Cameras 120, 122, 124 are preferably set at a vertical height d₃above mat 108 between about 2 feet and about 6 feet, and more preferably above the waist of golfer 110 but not higher than the sternum of golfer 110. Most preferably, cameras 120, 122, 124 are set at a height d₃of about 4 feet. In addition, with cameras 120, 122 set at horizontal distances d₅, d₆respectively away from coplanar axes 142, 143 and also from parallel axes 132, 134, respectively, and with camera 124 set at a horizontal distance d₇away from axis 144, distances d₅, d₆, d₇preferably are each between about 8 feet and about 14 feet and more preferably between about 10 feet and about 12 feet.
[0035] Camera 126 is preferably set at a vertical height that is at least about 8 feet above the head of golfer 110, and more preferably more than about 10 feet above the head of golfer 110. Most preferably, camera 126 is set at a vertical height d₄that is about 16 feet above mat 108.
[0036] Cameras 120, 122, 124, 126 are preferably high speed cameras. In the preferred embodiment, the cameras may capture video data in digital format and in black and white as well as color. Suitable cameras include those commercially available from the Eastman Kodak Company's Motion Analysis Systems Division, and permit data capture meeting a variety of specifications including image type, resolution, shutter time, capture rate, and communications standards. Preferably, black and white image capture is accomplished using a Kodak Model SR-1000 Motion Corder Analyzer digital camera that has a recording rate of 1000 frames per second and a 658 (H)×496 (V) pixel sensor array. Such a CCD camera provides progressive (noninterlaced) scanning, allows electronic zoom, and provides digital data output of 8-bit monochrome TIFF files exported via the SCSI-2 port to a personal computer.
Most preferably, color image capture is achieved with at least one of the four cameras. Suitable equipment for color image capture is a Kodak EKTAPRO Model HG-2000 color digital camera that has a recording rate of 2000 frames per second and a resolution of 512 (H)×384 (V) photo-sensitive pixels in 24-bit color (this model alternatively provides 8-bit monochrome images). The HG-2000 camera permits digital images to be stored during download in a compact (Bayer) or 24-bit color TIFF format onto a PCMCIA hard drive or solid state memory card. In addition to serial communication, the unit permits 100 Mbps ethernet communication with a dedicated 100 Base-T physical link using UDP/IP protocol. [0037]
Videotape recorders (not shown) preferably store output from [0038] cameras 120, 122, 124, 126 as well. A suitable unit is the Sony UVW-1800 editing recorder/player which accepts L-size metal Betacam SP format cassettes, although other recorders/players and formats may be used.
A lighting system (not shown) similar to that used in feature film production may be used to further enhance the image capture of [0039] multimedia analysis system 100. In particular, the choice of lighting can have a pronounced effect on the quality of mixing of data sets. Thus, in accordance with the principles of the present invention, several strong quartz halogen bulbs may be precisely located and aimed at a backdrop or wall so that the background is flooded uniformly with light, while shadows are avoided.
Preferably, [0040] multimedia analysis system 100 further includes at least one computer system 130 for use particularly in data collection, archival, and comparison tasks. Computer 130 typically includes a processing unit such as a central processing unit, or CPU. Parallel processing may be employed, such as with multiple or distributed processors. Storage devices also are included, such as a hard disk, CD ROM, magneto-optical (optical) drive, tape drive or other suitable storage device, and are typically coupled to the processing unit. In addition, a primary memory device is included, and has random access memory (RAM) for storing programming instructions and data for processes operating on the central processing unit. The primary memory device further includes read only memory (ROM) that stores basic operating instructions, data and objects used by the computer to perform its functions. Computer 130 typically also includes one or more input/output sources, which are often chosen as a keyboard, mouse, stylus, and/or other suitable device. Furthermore, a network connection may be provided. Those skilled in the art will also contemplate other configurations of computer 130 and associated technology, which are readily usable with the present development.
With reference to FIG. 4, in one embodiment of the present invention, [0041] cameras 120, 122, 124, 126 send signals 152, 154, 156, 158 respectively to signal/data processing unit 160, which may serve a variety of functions including data storage, data distribution, and data filtering. Processing unit 160 is operatively associated with computer system 130, which governs and/or performs the primary functions of data collection, archival, and comparison, and may additionally be operatively associated with removable media storage devices 162. Preferably, computer system 130 includes one or more display devices 163, and removable media storage devices 162 include one or more videocassette recorder/players. Computer system 130 may further be operatively associated with mass storage systems 164. Each of the aforementioned components or systems may further be operatively associated with a network 166. For example, network 166 can include a server, router or the like, and additional computers that permit data, instructions and/or messages to be passed among the networked components or systems. Additional computer systems 130 may be linked to network 166. Independent network connections also may be provided for one or more of the components or systems. The design, construction, and implementation of a computer network suitable for the present invention may be achieved in a multitude of approaches, as known in the art.
In the preferred embodiment, [0042] multimedia analysis system 100 is configured to permit the creation of a library of data sets related to the performance of golfer 110 in a known set of conditions, and in particular permits the creation, archival, and comparison of data sets concerning the performance of golfer 110 during test sessions under a variety of test conditions. The test conditions may provide the protocols for use in generating comparable data from one session to another, and include the equipment used by golfer 110, environmental conditions, and the personal characteristics of golfer 110 at the time of the session. The focus of each golf session, in part, involves data collection with respect to these test conditions.
A wide variety of data on test conditions may be logged. For example, the equipment used by [0043] golfer 110 is particularly relevant to the analysis of the test session. The range of equipment chosen by such a subject includes the type of club used (i.e., wood or iron), the construction of the club (i.e., material, weighting, grip construction, shaft type, overall dimensions, hosel design), the type of golf ball used (i.e., wound, multilayer, liquid filled, dimple pattern), and even the type of shoes (i.e., spikeless, spikes) worn by golfer 110. Environmental conditions may include temperature, humidity, time of day, season, and lighting. Personal characteristics of golfer 110 may include age, height, weight, posture, eyesight, health, confidence level, and stress level. As some of the test conditions, such as the weight and eyesight of golfer 110 can be expected to vary over time, recordation of these variables permits an instructor or other performance professional to evaluate the many factors that may contribute to a change in the technique or equipment performance of golfer 110.
Advantageously, the archival of data sets representing a golfer's performance during a series of discrete test sessions permits a golfer to compare swing mechanics and resulting launch conditions achieved during different times, and possibly different phases in the golfer's career. This may be useful, for example, in analyzing deviations in performance resulting from changes in swing speed or body positioning. Furthermore, as a golfer's health varies over the course of a career, deviations in technique resulting from medical procedures can be tracked and corrective action prescribed. For example, if [0044] golfer 110 has undergone shoulder or ankle surgery, swing mechanics of golfer 110 may be affected. A before-and-after comparison permits golfer 110 to adapt to the physiological conditions, and potentially adjust swing mechanics to allow a return to pre-surgical form. For the purposes of the present invention, data sets include, but are not limited to, a series of individual video frames or portions of individual video frames, as well as audio data, test conditions data, and data from equipment such as ball launch monitors.
Suitable video capture software and hardware is available from Avid Technology, Inc. of Tewksbury, Massachusetts. In particular, Avid Xpress hardware and software permits real-time video editing. The Avid Xpress system includes Avid Xpress editing software and hardware, Avid Meridien video, audio, effects, display and drive controller hardware and Meridien input/output (I/O) hardware for making connections, analog (component, S-video, composite) and digital (SMPTE-259) video I/O, as well as analog, S/PDIF digital and AES/EBU digital audio I/O. Output options include web video servers, DVD, and CD formats. [0045]
Preferably, [0046] computer system 130 is either a Power Macintosh G3 or G4 with an Avid PCI Extender, or an IBM IntelliStation M Pro running Windows NT. Among the video features offered by the Avid system are the following: ITU R-601 broadcast industry-standard image output (720×486 NTSC or 720×576 PAL), image compression, four video tracks, offline (draft mode) and online image quality, Sony RS-422 serial and VLAN deck control, batch digitizing, waveform monitor and vectorscope, SMPTE timecode support, integrated EDL import/export, 16:9 editing for widescreen and DTV, and a 1:1 uncompressed option. With respect to audio features, Avid Xpress offers the following: eight-track, real-time audio mixing, 48 kHz, 44.1 kHz sampling rate, audio punch-in, real-time rubberband gain adjustments in timeline, real-time audio input/output level adjustments, control volume and pan between channels by clip or track, display of audio waveforms and scan of audio, two or eight channels of audio I/O, downsampling of audio to 22 kHz and 11 kHz for export to CD-ROMs, and other audio mixer features.
With respect to editing capabilities, Avid Xpress is suitable for the present invention, in part, because it offers: tight timeline editing, single-track transitions, snap-to transitions, slip and slide editing, splice, overwrite, extract, replace, extend, lift, and split edits, sync-lock/sync-break detection, a dynamic storyboard, precise timecode editing, pop-up source monitors, matchframe, sift, consolidate, and collapse, built-in logging, a timeline views feature, VTR play emulation, and remote/offline logging. [0047]
The present invention makes use of Avid Xpress, along with modifications that permit enhanced superimposing of data sets and communications with the Titleist Launch Monitor technology. [0048]
As will now be explained, the present invention allows a data-rich archive of a golfer's past performances to be exploited to provide a real-time comparison with a present performance. Such mixing, in general, permits independent sources to be merged together. Simultaneous or near simultaneous mixing of present and past performances is achieved using the present invention. In addition, two or more data sets of a golfer's past performances may be mixed, thereby allowing comparison of archive-only test sessions. [0049]
Preferably, data sets are archived in analog or digital format, using storage media such as hard disk drives, CD-ROM's, laser disks, DVD's, and videotapes as discussed previously. [0050]
Data collection particularly from video, audio, and other ball launch equipment typically requires time spans of less than two seconds per swing of a golf club. Average golfers may have swings that require between about 0.9 to about 1.5 seconds to complete, and most golfers take a full swing in about one second. Thus, image capture of a selection of swings is preferably accomplished using high speed video cameras, such as those filming at speeds of 2000 frames per second. This rate of data collection permits an aggregate data set to be compiled with a sufficient size such that subsequent analysis may be performed on subsets of the aggregate that each have a sufficient number of frames for evaluating particular ranges in time or stages of the golfer's swing. In accordance with the principles of the present invention, a variety of high-speed or relatively high-speed capture rates may be used, such as 500 frames per second. Some embodiments of [0051] multimedia analysis system 100 may include cameras simultaneously operating at different capture speeds, such as one camera operating at 500 frames per second and another operating at 250 frames per second. In some embodiments, a capture rate of at least about 250 frames per second is used.
A session of data capture may be considered one swing, or alternatively a session may be defined as a portion of a swing or multiple swings. Sessions times may occur close to each other (i.e., on the same day), or may be separated by long durations (i.e., in different years). [0052]
In accordance with the principles of the present invention, [0053] multimedia analysis system 100 permits the mixing of multiple sets of video, audio, and other data collected during a golfer's performance sessions, preferably under at least partly repeated test conditions. Mixing is typically achieved using one or more special effects that are known in the art and are incorporated in the present invention. For example, masking permits one or more portions of an image to be isolated for editing. In effect, some portions of the image are protected from changes. Another effect, mattes in the form of videos or images, may be used to create isolated regions through which additional videos or images appear.
With regard to video data sets, several “key” effects (i.e., effects that allow one image to be placed over or within another image) are known in the art, and facilitate mixing of data sets in the present invention. Although the naming of key effects varies from one equipment manufacturer to another, commonly used key effects include chroma key, luminance key, external key, and downstream key. In essence, each of the key effects permits at least two video sources to be combined. A first object may be defined in a first video source, for example, and may be used to define a region within which, or about which, a second video source is placed. [0054]
Most well-recognized is the chroma key effect, typically used for television weather forecasts in which a reporter stands in front of a uniformly blue or green wall that a computer replaces with a weather map for the television viewers. In this method of combining multiple video images, one video is superimposed over another, and it is the particular color of hue that is isolated by the computer. [0055]
Each of the key effects, in general, is characterized by an ability to provide a superimposition of one video upon another. As used herein the terms superimpose, superimposing, superimposition, and the like refer to the range of effects including, but not limited to, those techniques known as layering, overlaying, ghosting, or compositing of sources, along with the range of key effects that have been standardized by industry to permit the combination of multiple image/video sources, preferably in synchronized format. In addition, the terms superimpose, superimposing, superimposition, and the like are applicable to the correlation of non-video data, such as audio data or launch monitor data from multiple data collection sessions. Furthermore, the terms are applicable to frame-by frame analysis, or analysis of portions of several frames. For example, a frame representing a single still from a video stream may be superimposed over another frame in order to discern differences between the frames. [0056]
The present invention may make use of the principle of A/B roll editing, more generally referred to as two-source or multiple-source editing. Through the use of a video mixer, video images may be combined with sophisticated control of transitioning and synchronization. A mixer is commonly used to synchronize multiple sources, for example multiple video sources, so that each video frame of a first source starts at the same instant as each video frame from another source. Thus, when superimposing sets of data in the present invention, synchronization may be used so that the data sets are “in step,” having the same start and finish times. [0057]
Among the graphics, titling, and digital effects that may be used with the present invention are the following, which may also be real-time effects: transitions, dissolves and horizontal, vertical, and box wipes, motion effects, fast-rendered superimpositions, diagonal, matrix, sawtooth, and shape wipes, matte, chroma, and luma key, flips, flops, and resizes, peels, pushes, and spins, conceals, L-conceals, and squeezes, page curls, masking, picture-in-picture (two-dimensional or three-dimensional), compressed keyframed motion graphics and titles with limited or unlimited keyframes per effect, nesting (the composition of multiple video tracks to a single track), two stream keyframeable color effects, fast-rendered masks, titles (three-dimensional), AVX (Avid Visual extension) plug-ins, customizable two- and three-dimensional DVEs, x, y, z position and rotation, and an ability to locate flash frames and black holes. [0058]
Synchronization is particularly important, in order to ensure that each video frame of a first source starts at the same time as each video frame from a second source. For example, a video is composed of many frames that can be rapidly and sequentially shown. Each frame, in turn, includes hundreds of horizontal lines. Two commons standards for a frame are the North American NTSC (National Television System Committee) and the European PAL (Phase Alternation Line). NTSC frames each have 525 lines, and a refresh rate of 30 frames per second, while each frame in PAL is composed of 625 scanned lines, with a refresh rate of 25 frames per second. Furthermore, each frame has two fields, a first field representing every other line beginning with line 1 (i.e., 1, 3, 5, 7 . . . ), and a second field representing every other line beginning with line 2 (i.e., 2, 4, 6, 8 . . . ). [0059]
The difficulty that may be encountered with independent video sources that are to be combined is that a first source may begin, for example, at line [0060] 82 while simultaneously, a second source begins at line 5. Frame synchronizer electronics and/or software permit one of the two sources to be delayed such that the sources are synchronized accordingly, thus avoiding picture shift or waver. Moreover, timebase correction may be undertaken.
Synchronization also is particularly important when multiple sessions are to be viewed together, such that a common start point is chosen. For example, the point of impact of a club with a golf ball may be used as time “zero” for each of two sets of data, in order to synchronize the data. [0061]
[0062] Multimedia analysis system 100 may make use of standard video graphic output types such as BMP, TIF, GIF, PIC, JPG, SGI, and TGA. In addition, multimedia analysis system 100 may use standard motion video formats such as AVI, JPEG, MPEG, JMPEG, as well as standard digital sound formats such as PCM.
With respect to superimposing two video data sets, the superimposition may take a variety of forms. For example, [0063] golfer 110 may take swings on two separate occasions. Advantageously, because the positions of video cameras 120, 122, 124, 126 of motion analysis system 100 are fixed, data collection during the two sessions may be taken in repeatable fashion. In one embodiment of the present invention, a first set of data is collected during a first session of golf swings and stored for later use. A second session is next conducted, during which an exemplar golf swing is chosen for replay from the first set of data. Preferably, computer system 130 is used to store each data set, select a desired data set and if necessary a subset thereof, process, and replay the data. Video data from one or more views may be displayed on one or more display devices 163, such as a computer monitor, television, or projection system, and non-video data also may displayed. The non-video data may include information from a golf ball launch monitor 118, including information on golf ball trajectory as well as estimates of where the golf ball will land (carry), how far the golf ball will roll (total shot distance), and other flight information such as whether the ball will hook or slice and how far off-line the ball will travel. Preferably, one or more of the views include superimposed data.
In one embodiment, shown in FIG. 5([0064] a), two views of golfer 110 as captured by rear camera 124 during separate sessions are shown in split-screen format with screen portions 170, 172 on a display device 163. It should be noted that although display device 163 is shown as a computer monitor, other display devices such as televisions or projection screen systems may be used. A first set of video data 180 of golfer 110 as captured during a first session is shown on the left portion 170 of display device 163, while a second set of video data 190 of golfer 110 acquired during a second session is shown on the right portion 172. Turning to FIG. 5(b), when the first and second sets of video data 180, 190 respectively are superimposed, composite video data 200 is shown on display device 163. Composite video data 200 may represent video data 180, 190 in outline form, which may be accomplished by suitable filtering of the data as with the mixing techniques described herein. Alternatively, as shown in FIGS. 5(c) and 5(d), the first and second sets of video data 180, 190 may each be assigned a fill color, so that each set of data of golfer 110 can be easily discerned from the other. In FIG. 5(c), the first set of video data 180 is shown in the foreground, while in FIG. 5(d) the second set of video data 190 is shown in the foreground. In alternate embodiments, one of the sets of data that is displayed may be a data set showing golfer 110 as seen by a camera without any outlining or other data filtering. Likewise, the display of data sets may be partially or completely shown in color rather than black and white. Furthermore, some or all of the background around golfer 110 may be shown, rather than removing that portion of the data from the data sets.
Superimposing of images from at least two data sets may occur at various percentages. For example, in one embodiment thirty percent of the pixels of each image in a first data set may be superimposed with seventy percent of the pixels of each image in a second data set. In another embodiment, fifty percent of the pixels of each image in each of the first and second data sets may be superimposed. In yet another embodiment, between about 20 percent and about 40 percent of the pixels of each image in a first data set may be superimposed with between about 60 percent and about [0065] 80 percent of the pixels of each image in a second data set.
The present invention may further involve the display of a portion of [0066] golfer 110 from one or more data sets. For example, if the upper body movement of golfer 110 is to be analyzed, the remainder of golfer 110 may be selectively masked or otherwise removed from the images that are captured, thereby permitting analysis to focus on the area of interest. If only foot positioning is to be analyzed, zoomed views of the lower leg and foot position of golfer 110 during the various stages of swings may be selectively displayed. In each example, as before, the superimposing of data sets may occur at various percentages. Thus, the present invention provides a tool for focusing analysis on particular aspects of a golfer's mechanics.
As apparent from FIGS. [0067] 5(a)-(d), golfer 110 may change some swing mechanics from one data collection session to another. For example, as shown in the figures, although golfer 110 placed his feet in the same position in both sets of video data 180, 190, the golfer 110 changed his posture and grip position on the club.
[0068] Composite video data 200, 210, 220 may be viewed in a variety of ways. In one embodiment, first and second sets of video data 180, 190 are acquired during separate sessions, and analyzed following completion of both sessions. In another embodiment, the first and second sets of video data 180, 190 are shown simultaneously with the actual acquisition of the second set of video data. In some embodiments, golfer 110 may not be able to view composite video data 200, 210, 220 until after data collection of the first and second sets of video data 180, 190, while in other embodiments, the first set of video data may be shown on a display device 163 located within the view of golfer 110, such as with a projection screen television located behind a net.
In another embodiment, images captured from multiple cameras may be simultaneously displayed. As shown in FIG. 6, [0069] composite view 220 of golfer 110 as captured by rear camera 124 is shown on screen portion 226 on a display device 163, while an additional composite view 230 of golfer 110 as captured by front camera 122 is shown on screen portion 227. Another screen portion 228 is provided for the display of further data collected on golfer 110, such as the test conditions previously described.
The present invention may also be used to obtain sets of video data that only include two colors, and do not include fine details such as clothing patterns, facial expressions, etc. Software filtering and effects may be used for this purpose, although particular attention to the setup of some equipment included with [0070] multimedia analysis system 100 also may facilitate such data collection. For example, by flooding the area around golfer 110 with light, one or more cameras 120, 122, 124, 126 may capture only a black and white, essentially non-grayscale image. In other words, the individual may appear to be a uniform, solidly filled black body on a white background. Computer 163 and associated software may be used to invert the colors of the image, as desired.
Preferably, data captured by the cameras is received and processed by [0071] computer system 130. The following components may be included with multimedia analysis system 100 to facilitate collection and use of data: a digital capture and overlay board; a video switcher to permit several video signals to be received and selectively routed to display, storage, and processing components; a video contrast enhancer for modifying video signals such as by using shading compensation, brightness compensation, or preselected grey scale levels to introduce picture contrast; a split-screen generator for simultaneously displaying several images from various video sources on the same monitor screen; a video time base corrector for stabilizing the playback of videotape recordings and synchronizing different video sources; a video subtractor for displaying differences between two images; and video peak store memory for generating time-exposure images of physical movements.
In another aspect, non-video data sets may be mixed. For example, flight data sets acquired by a launch monitor may be shown on a [0072] display 163, such as by creating graphs or other charts of the data sets.
Referring now to FIG. 7, in accordance with the present invention, an operator turns the power on for the various components of [0073] multimedia analysis system 100. Next, computer processor 130 boots, resulting in clearing of the computer memory, setting up of devices such as terminals, loading of software information into the computer system's memory, and loading the operating system into memory (STEP 300). After booting of the computer system 130, video capture, audio capture, and other data capture and mixing software is executed so that an operator may subsequently perform data acquisition, manipulation, and analysis (STEP 310). Cameras 120, 122, 124, 126 and lighting associated with multimedia analysis system 100 are then prepared for image capture, with appropriate equipment settings verified and/or established (STEP 320). In addition, launch monitor 118 is calibrated, with appropriate launch monitor settings verified and/or established (STEP 330). Optionally, the settings of a video splitter box may be verified and/or established, permitting one or more images or motion sequences to be displayed in one or more quadrants on a display device such as a monitor 163 (STEP 340).
With the various components of [0074] multimedia analysis system 100 in operational mode, the operator of multimedia analysis system 100 initiates data capture by instructing computer system 130 to begin collecting data, such as by depressing a start key associated with start operations of data capture and mixing software, and a data collection session is initiated (STEP 350). Preferably, the software is Avid Xpress real-time video editing software. The operator instructs golfer 110 to begin a swing sequence (STEP 360), and the golfer then performs a swing sequence (STEP 370). Images captured by cameras 120, 122, 124, 126 are digitized, audio captured by microphone 116 is digitized, and golf ball launch conditions and flight characteristics observed by launch monitor 118 are solved (STEP 380). Data from the session such as video data, audio data, and launch monitor data is next consolidated and correlated by the data capture and mixing software, as necessary (STEP 390).
Historical data such as images or sequences of images may be retrieved from mass storage (STEP [0075] 400), and data capture and mixing software routines may be activated to permit mixing (STEP 410). Software routines are activated to correlate two or more data sets such as a historical aggregated data set and a just-acquired aggregated data set (STEP 420). The term aggregated data set preferably refers to a set of data that includes video, audio, and launch monitor data from one session. The correlated data sets may be processed using mixing software routines so that appropriate effects are achieved, such as superimposition of data sets in video output. The correlated data is displayed in one or more formats, such as multiple correlated video image streams, multiple correlated audio streams, and/or comparisons of test condition data from the data sets (STEP 430).
With reference to FIG. 8, in accordance with the present invention, data sets may be superimposed using the mixing software. The software operator first selects and marks an “in” point of a first data set, which is typically selected to be one frame prior to initiation of a swing sequence (STEP [0076] 500). The first data set, or a portion thereof, is then stored as a timeline-based sequence, such that playing of the first data set sequence occurs over a period of time tracked by a timeline (STEP 510). The software operator next selects and marks an “in” point of a second data set, which also is typically selected to be one frame prior to initiation of a swing sequence (STEP 520). The second data set, or a portion thereof, is then stored as a timeline-based sequence (STEP 530). One or more mixing software tools are next selected by the operator to apply the desired effects to the first and second data sets (STEP 540). The data sets are then displayed in one or more formats, such as multiple correlated video image streams, multiple correlated audio streams, and/or comparisons of test condition data from the data sets (STEP 550).
Preferably, when analyzing a golf swing from start to finish, the selection of “in” points begins at least about 0.75 second before golf ball impact, as this is a typical time span between initiation of a golf swing and contact of a golf club with a golf ball. Also, first and second data sets preferably end no earlier than about 0.25 second after golf ball impact, as this is a typical time span between impact of the golf club with the golf ball and completion of the golf swing. [0077]
While various descriptions of the present invention are described above, it should be understood that the various features can be used singly or in any combination thereof. Therefore, this invention is not to be limited to only the specifically preferred embodiments described and depicted herein. [0078]
Many variations or modifications fall within the scope of the present development. For example, three-dimensional data sets may be compiled using the requisite number of cameras, and visualization techniques, as known in the art, may be used to permit mixing of three-dimensional video data sets so that [0079] golfer 110 may be viewed from any angle. In another embodiment, multimedia analysis system 100 may be configured to be portable. Calibration units may be provided to precisely position cameras, a mat, and a teed golf ball for data collection, and may include devices permitting laser distance measurement. Level sensors also may be provided. With such equipment, along with a portable computer, a multimedia analysis system 100 may be readily transported, so that data capture of a golfer's swing may be commenced at locations such as golf course fairways, pro shops, or trade shows.
While the present invention is particularly suited to sports training, other uses are contemplated especially in areas where an emulation or a precise motion sequence is desired. For example, the present invention may be scaled to smaller dimensions and/or more constrained areas for use in the training of doctors performing particular surgical techniques. Doctors pioneering particular medical procedures may use the apparatus and methodology of the present invention to practice the various steps in a procedure without requiring the presence of a live subject. Alternatively, trade skills may be developed through practice of the present invention, such as welding skills. Law enforcement also may find use of the present invention in firearms training, for example in the practice of sharp shooters. Thus, it should be understood that variations and modifications within the spirit and scope of the invention may occur to those skilled in the art to which the invention pertains. [0080]

Claims

What is claimed is:

1. A method of mixing data concerning sports performances of a player, the method comprising:

defining a first set of test conditions applied to the player;

acquiring a first set of video data on the player's performance at a first time, the first set of video data being acquired during the first set of test conditions;

storing the first sets of test conditions and video data;

defining a second set of test conditions applied to the player, the first and second sets of test conditions having at least one test condition in common;

acquiring a second set of video data on the player's performance at a second time, the second set of video data being acquired during the second set of test conditions;

storing the second sets of test conditions and video data;

mixing at least a portion of the first and second sets of video data to form a superimposed set; and

displaying the superimposed set.

2. The method of claim 1, wherein the first and second sets of video data each comprise a single video frame.

3. The method of claim 1, wherein the first and second sets of video data each comprise at least two video frames.

4. The method of claim 3, wherein the superimposed sets of video data are substantially synchronized from a start time to a finish time.

5. The method of claim 4, wherein the first and second sets of video data are acquired using a video camera with a capture rate of at least about 250 frames per second.

6. The method of claim 4, wherein the mixing of sets of video data occurs with a first percentage of the pixels of each frame selected from one of the first and second video data sets and a second percentage of the pixels of each frame selected from the other of the first and second video data sets.

7. The method of claim 6, wherein the sum of the first and second percentages is one hundred.

8. The method of claim 7, wherein the first percentage is between about 20 percent and about 40 percent and the second percentage is between about 60 percent and about 80 percent.

9. The method of claim 4, further comprising:

acquiring a first set of non-video data on the player's performance at the first time, the first set of non-video data being acquired during the first set of test conditions;

storing the first set of non-video data;

acquiring a second set of non-video data on the player's performance at the second time, the second set of non-video data being acquired during the second set of test conditions;

storing the second set of non-video data;

mixing at least a portion of the first and second sets of non-video data to form a non-video data superimposed set; and

displaying or playing at least a portion of the non-video data superimposed set.

10. The method of claim 9, wherein the superimposed sets of non-video data are substantially synchronized from the start time to the finish time, and the superimposed sets of non-video data are substantially synchronized from the start time to the finish time with the superimposed sets of video data.

11. The method of claim 10, wherein the first and second sets of non-video data include at least one of audio data and launch monitor data.

12. The method of claim 10, wherein the first and second sets of video data are acquired using at least one analog or digital video camera, and the first and second sets of non-video data are acquired using at least one of a microphone and a launch monitor.

13. The method of claim 4, wherein each set of test conditions include at least one of the equipment of the player, the environmental conditions, and the personal characteristics of the player.

14. The method of claim 13, wherein the mixing of video data includes using at least one key effect.

15. The method of claim 4, wherein each of the first and second sets of video data includes data from at least two video cameras.

16. The method of claim 15, wherein at least two video cameras are disposed about substantially the same axis.

17. The method of claim 15, wherein at least two video cameras are disposed about substantially the same plane.

18. A method of overlaying data sets of sports performances of a player, the method comprising:

defining a plurality of test conditions applied to the player;

acquiring a plurality of sets of video data on the player's performance, each set of video data having at least one test condition in common;

storing the plurality of sets of test conditions and video data;

mixing at least a portion of at least two of the sets of video data to form a superimposed set; and

displaying the superimposed set.

19. The method of claim 18, further comprising:

acquiring a plurality of sets of non-video data on the player's performance, each set of non-video data having at least one test condition in common; storing the plurality of sets of non-video data;

mixing at least a portion of at least two of the sets of non-video data to form a non-video data superimposed set; and

displaying or playing the non-video data superimposed set.

20. The method of claim 19, further comprising:

synchronizing the plurality of sets of test conditions, video data, and non-video data.

21. A multimedia analysis system for capturing and comparing sports performances of a player comprising:

at least one video camera for acquiring a plurality of sets of video data of player performances;

at least one storage device for storing the plurality of sets of video data;

at least one processor for mixing at least two of the sets of video data; and

at least one display device for displaying the at least two sets of mixed data, wherein the processor substantially synchronizes and superimposes the at least two sets of video data to form an aggregate set of video data.

22. The multimedia analysis system of claim 21, wherein the location of the at least one video camera is substantially the same when acquiring each set of video data.

23. The multimedia analysis system of claim 22, further comprising:

a launch monitor for acquiring a plurality of sets of flight data of objects hit by the player; and

at least one processor for mixing at least two of the sets of flight data,

wherein the at least one processor substantially synchronizes and superimposes the at least two sets of flight data to form an aggregate set of flight data, the aggregate set of flight data also being synchronized with the aggregate set of video data.

24. The multimedia analysis system of claim 23, wherein the plurality of sets of video data are acquired by at least two video cameras that are disposed about substantially the same axis and aimed at the player.

25. The multimedia analysis system of claim 23, wherein the plurality of sets of video data are acquired by at least two video cameras that are disposed about substantially the same plane and aimed at the player.