EP1702318A4 - Method and apparatus for remote real time collaborative music performance - Google Patents
Method and apparatus for remote real time collaborative music performanceInfo
- Publication number
- EP1702318A4 EP1702318A4 EP03818811A EP03818811A EP1702318A4 EP 1702318 A4 EP1702318 A4 EP 1702318A4 EP 03818811 A EP03818811 A EP 03818811A EP 03818811 A EP03818811 A EP 03818811A EP 1702318 A4 EP1702318 A4 EP 1702318A4
- Authority
- EP
- European Patent Office
- Prior art keywords
- remote
- musical
- local
- event
- delay
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H1/00—Details of electrophonic musical instruments
- G10H1/0033—Recording/reproducing or transmission of music for electrophonic musical instruments
- G10H1/0041—Recording/reproducing or transmission of music for electrophonic musical instruments in coded form
- G10H1/0058—Transmission between separate instruments or between individual components of a musical system
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H1/00—Details of electrophonic musical instruments
- G10H1/36—Accompaniment arrangements
- G10H1/40—Rhythm
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2240/00—Data organisation or data communication aspects, specifically adapted for electrophonic musical tools or instruments
- G10H2240/171—Transmission of musical instrument data, control or status information; Transmission, remote access or control of music data for electrophonic musical instruments
- G10H2240/175—Transmission of musical instrument data, control or status information; Transmission, remote access or control of music data for electrophonic musical instruments for jam sessions or musical collaboration through a network, e.g. for composition, ensemble playing or repeating; Compensation of network or internet delays therefor
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2240/00—Data organisation or data communication aspects, specifically adapted for electrophonic musical tools or instruments
- G10H2240/171—Transmission of musical instrument data, control or status information; Transmission, remote access or control of music data for electrophonic musical instruments
- G10H2240/201—Physical layer or hardware aspects of transmission to or from an electrophonic musical instrument, e.g. voltage levels, bit streams, code words or symbols over a physical link connecting network nodes or instruments
- G10H2240/241—Telephone transmission, i.e. using twisted pair telephone lines or any type of telephone network
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2240/00—Data organisation or data communication aspects, specifically adapted for electrophonic musical tools or instruments
- G10H2240/171—Transmission of musical instrument data, control or status information; Transmission, remote access or control of music data for electrophonic musical instruments
- G10H2240/281—Protocol or standard connector for transmission of analog or digital data to or from an electrophonic musical instrument
- G10H2240/311—MIDI transmission
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2250/00—Aspects of algorithms or signal processing methods without intrinsic musical character, yet specifically adapted for or used in electrophonic musical processing
- G10H2250/041—Delay lines applied to musical processing
Definitions
- the present invention relates generally to a system for electronic music performance. More particularly, the invention relates to a system for permitting participants to collaborate in the performance of music, i.e. to jam, where any performer may be remote from any others .
- An exemplary application may be implemented in Java Studio, by Microsoft Corporation, using calls to SERIALIO, a serial interface class library by Solutions Consulting, Inc., of Santa Barbara, CA and the ActiveX Seer Music Player by Seer Systems of Los Altos, CA.
- An exemplary application may also be implemented in Visual Basic, by Microsoft Corporation, using calls to their DirectX version 8.1 API, specifically the DirectPlay and DirectMusic components.
- Matheson in US Patent 4,570,930 teaches a method for synchronizing two computer games. Applicable only to games having discretely calculated "generations", Matheson provides that each generation is numbered, and each generation calculation uses the users ' inputs gathered during the prior generation. Matheson' s generations may, at best, each be 1/30 of a second long, i.e. at the game's video update rate. However, generations can become arbitrarily long if one or another user's input is not communicated in a timely manner, or needs to be retransmitted. Unfortunately, such a behavior is not conducive to musical performance. Hochstein, et al .
- MIDI commands can be sent over a network to many stations.
- the live performance can be selectively recorded or mixed with other pre-recorded tracks.
- the mechanism is a timestamp that is attached to each musical event (e.g. a MIDI Note-On command) .
- the tracks can be mixed.
- the (almost) live musical performance can be added to the pre-recorded tracks at a remote location.
- a station receiving this performance can play along with the (almost) live performance.
- Moline is limited, however, in that the "play along" performance is not bi-directional. That is, a true jam session is not taking place.
- Tonos Entertainment, Inc of Culver City, CA, (www.tonos.com) provides a similar capability, but is based on MP3 files, rather than MIDI.
- Neumann, et al. in 6,175,872 does not have such a limitation.
- time stamping of MIDI packets the streams of MIDI data generated at remote locations can be sent to the local station, sequenced into the correct musical order, and played aloud.
- the participant playing on the local machine similarly transmits his musical performance, with timestamp, to the other stations.
- communication transport delay shall be less than twenty milliseconds, Neumann provides real time, remote collaboration and jamming.
- the present invention relates to a system and method for playing music with one or more other musicians, that is, jamming, where some of the other people are at remote locations.
- Each musician has a station, typically including a keyboard, computer, synthesizer, and a communication channel.
- the communication channel might be a modem connected to a telephone line, a DSL connection, or other local, wide, or Internet network connection.
- musicians desire a jam session, their respective station computers communicate with each other, or perhaps with a designated host computer, and determine the communication delays to and from each other station in the jam. Subsequently, each musician's performance is immediately transmitted to every other musician's station. However, the performance is delayed before being played locally.
- remote performances are also delayed, with the exception of the performance coming from the station having the greatest associated network delay, which can be played immediately.
- the local performance is played locally after undergoing a delay equal to that of the greatest associated network delay.
- each musician can choose to have his station disregard high delay stations during live jamming, and to allow performance with only low delays .
- a "groove" can be distributed to the stations.
- the groove is a track that provides a framework for the jam session. In its simplest form, it might be a metronome. But it could also be a MIDI sequence, a WAV file (instrumental or lyrical), or an MP3. Regardless of the communication delays, the groove plays in synchrony on all machines, as the command to start the groove is delayed appropriately by each station receiving the play command, and the station issuing the play command . It is the object of this invention to make it possible for a plurality of musicians to perform and collaborate in real time, even at remote locations.
- FIGURE 1 is a detailed block diagram of multiple musical performance stations configured to jam over a communications channel, and including an optional server
- FIGURE 2A is a schematic of multiple musical stations connected over a simplified communications channel topology to illustrate the effect of communications delay
- FIGURE 2B is a schematic like that of FIGURE 2A, but illustrating the added effect of a server
- FIGURE 3 shows the preferred graphical user interface for remote jamming
- FIGURE 4 depicts an instrument picker
- FIGURE 5 is a flow chart for handling musical events. While the invention will be described and disclosed in connection with certain preferred embodiments and procedures, it is not intended to limit the invention to those specific embodiments. Rather it is intended to cover all such alternative embodiments and modifications as fall within the spirit and scope of the invention.
- a plurality of performance stations represented by stations 10, 12, and 14 are interconnected by the communication channel 150.
- the invention is operable with as few as two, or a large number of stations. This allows collaborations as modest as a duet played by a song writing team, up to complete orchestras, or larger. Because of the difficult logistics of managing large numbers of remote players, this invention will be used most frequently by small bands of two to five musicians. Note that while the term "musician” is used throughout, what is meant is simply the user of the invention, though it may be that the user is a skilled musical artist, a talented amateur, or musical student. For some implementations, a fanout server 18 is used.
- Each performance station 10, 12, 14 communicates over communication channel 150 directly with fanout server 18.
- the fanout server is responsible for forwarding all pertinent communications from any of the performance stations to each of the others.
- Communications channel 150 may be a telephone network, a local or wide area Ethernet, the Internet, or any other communications medium.
- each of remote performance stations 12 and 14 mirror the elements of local performance station 10.
- Each of performance stations 12 and 14 have keyboard and controls 100, 100', 100", event interpretation , 110 , 110', 110", event formatting for jam partners 120, 120', 120", transmit module 130, 130', 130", communication channel interface 140, 140', 140", receive module 160, 160', 160", delay 170, 170', 170", instrument synthesizer 180, 180', 180", and audio output 190, 190', 190", respectively.
- Each performance station is preferably comprised of a personal computer having a keyboard and controls 100.
- Other common graphical user interface (GUI) controls such as onscreen menus and buttons operated with a mouse or trackball, are included in keyboard and controls 100, but not specifically illustrated here.
- GUI graphical user interface
- keyboard 100 Certain keys of keyboard 100 are mapped to certain musical notes as explained below in conjunction with FIGURE 3.
- the keys of keyboard 100 when operated, generate events. When a musician presses a key on the keyboard, a “key pressed down” event is generated. When the musician lets go of the key, a “key released” event occurs. Similarly, if the computer's mouse is clicked on an on-screen button, a “button pressed” event is generated.
- a MIDI controller Usually resembling a piano keyboard, though often smaller and covering fewer octaves, a MIDI controller is more intuitive and musically friendly than the computer keyboard. When combined with a MIDI interface for the computer, such as the one provided with well known audio cards such as
- the MIDI controller can generate events in place of or in addition to keyboard and controls 100.
- one or more MIDI controllers are added to the keyboard and controls 100, it becomes possible for more than one musician to perform at a single performance station 10. That is, if a single MIDI controller is added to performance station 10, then one musician could play the MIDI controller, and another musician could play using the computer keyboard.
- Each additional MIDI controller added to keyboard and controls 100 can potentially allow an additional musician to play at the local performance station.
- references to the musician using a performance station will be understood to include the possibility of multiple musicians performing on that single performance station.
- Each of the stations 10, 12, and 14 may be identical, or may have different keyboard and controls 100, 100', 100" as described above.
- the term "keyboard” may be used to refer to the computer keyboard, a MIDI controller, or the GUI or other controls.
- Event interpretation 110 examines the event to determine whether it has significance to the musical performance. An example of a significant event would be "key pressed", where the key has been given an association with a musical note that should be played. A "key released” for the same key would mean that the note, if playing, should be stopped.
- non-significant GUI events would include, for example, mouse actions that take place outside the GUI 300, discussed below in conjunction with FIGURE 3.
- Events determined to be musically significant by Event Interpretation 110 are immediately sent two places: Musical events are formatted for the jam partners at 120, and subsequently the transmit module 130 packages the musical events for the communication channel, possibly merging them with packets from other sources (not shown, discussed below) , and advances them via the communication channel interface 140 to the communication channel 150.
- the musical events are directed to the local instrument synthesizer 180 by way of delay 170, discussed below, to be rendered by audio output 190.
- Distributed multi-player game software is well known in the art. Those skilled in the field will be familiar with a modern personal computer running the Windows operating system by Microsoft Corporation, Redmond, WA, further having Microsoft's
- DirectX real time extensions including DirectPlay - Microsoft's extension for distributed multi-player games.
- the formatting for jam partners 120 preferably consists of a single call to the SendTo method in the DirectPlay API for each musical event. Data representative of the musical event is provided to the method, along with a command code to send the event data to all other stations.
- the transmit module 130 is comprised of the DirectPlay session.
- the DirectPlay session can operate with any of several interconnection technologies, including serial, modem, and TCP/IP, among others.
- Microsoft's DirectPlay API notwithstanding, an implementation of the functionality of the SendTo method is within the capability of an ordinarily skilled programmer, just writing directly to the transmit module 130 as a managed buffer for the communication channel interface 140.
- an implementation of the transmit module 130 without the DirectPlay library is within the ordinarily skilled programmer's capability. While many other alternative implementations of the communications channel 150 can be selected, the following discussion covers the two specific cases where the communications channel 150 is implemented as a telephone modem, and a TCP/IP network. Examples of implementations not discussed in detail include RS-232 serial networks, where a jam fanout server 18 is required for a jam having more than two participating performance stations) ; RS-485 or similar multidrop serial networks, where a jam fanout server 18 is not required; a packet radio network; and other form of LAN or WAN networks, such as token ring, or IPX.
- the transmit module 130 contents are written out to the communication channel interface 140, implemented as a modem, for transmission on communication channel 150, implemented as a telephone line.
- the communication channel will connect performance station 10 to exactly one of the other performance stations 12 or 14, or to a fanout server 18.
- communication channel interfaces (modems) 140 and 140' can connect with each other directly over communication channel (telephone network) 150 without resorting to fanout server 18.
- the communications channel interface 140 of performance station 10 is placed into a waiting- for-call mode, while the other modem, the communications channel interface 140' of performance station 12 dials the former modem's telephone number. Since a telephone modem can connect to only one point at a time, for connecting more than two musicians using a telephone network as the communications channel 150, each performance station 10, 12, and 14 participating in a jam will connect to a common fanout server 18.
- the fanout server 18, when operating with a telephone modem implementation, is comprised of a plurality of modems 40 (only one shown) , each having an associated transmit module 30 and receive module 50. The plurality of modems are all placed into waiting-for-call mode.
- a modem-based bulletin board system having a chat room function.
- the adaptation being that data representative of a musical event is transferred, rather than human-entered text messages which are readably displayed to other chat room participants.
- An advantage of using modems as the communication channel interface 140, 140', 140", and the plurality of 40, communicating over the telephone network as the communication channel 150, and using a BBS implementation of fanout server 18, is that the delay imposed by the telephone network is typically smaller and more stable than that found in switched packet data networks, such as the Internet.
- transmit module 130 includes the TCP/IP stack, and perhaps other software such as the DirectPlay session object when DirectPlay is being used.
- Communication channel interface 140 may be a modem dialed into an Internet Service Provider (ISP) and operating the Point-to-Point Protocol (PPP) to connect with and use the Internet as communication channel 150; a cable modem, DSL, or other communication technology can also be used.
- Interface 140 may be a network interface card (NIC) , connected, for example, using lObaseT to reach a hub or router.
- NIC network interface card
- the invention is operational if musicians at the participating stations 10, 12, and 14 can interconnect over the communications channel 150.
- each performance station 10, 12, and 14 may send musical event messages directly to each of the others.
- a jam fanout server 18 may be used.
- Another alternative is to use a multicast protocol to send each message to the other stations.
- Delay 170 receives musical events generated by the local musician (not shown) at local performance station 10, operating on the keyboard and controls 100 and accepted by event interpretation 110. It also receives musical events generated by remote musicians (not shown) at remote stations 12 and 14, using those keyboards and controls 100' and 100", which were processed similarly and communicated to performance station 10 as described above . By a value that will be specified below, each musical event received by delay 170 is held for a (possibly null) period of time, before being provided to instrument synthesizer 180. Delay 170 can be implemented as a scheduled queue, where each event entered into the queue is given a delay time (to be defined below) .
- delay 170 is to use a sorted queue. Upon receipt of a musical event by delay 170, the musical event is augmented with a future time value, calculated by adding a delay value (selected in a manner described below) to the current time. The musical event with the appended future time is inserted into the sorted queue in order of ascending future time. Delay 170 further operates to ensure that, at the ' time listed as the future time of the first event in the queue, the first musical event is removed from the queue and sent to the instrument synthesizer 180.
- An alternative implementation of the same delay 170 may be partially implemented by the DirectX DirectMusic API.
- the future' time is calculated in the same way, but the future time is then passed as a parameter, along with the musical event data, to the appropriate DirectMusicPerformance method, for example the SendMIDIMSG method, to schedule musical events such as MIDI Note-On or -Off, or the PlaySegmentEx method, to schedule musical events such as the playing of a particular audio file.
- the appropriate DirectMusicPerformance method for example the SendMIDIMSG method, to schedule musical events such as MIDI Note-On or -Off, or the PlaySegmentEx method, to schedule musical events such as the playing of a particular audio file.
- instrument synthesizer 180 can be entirely composed of software, as with the Seer Music Player by Seer Systems of Los Altos, CA.
- a dedicated hardware synthesizer can be used, such as any of the Creative Labs Sound Blaster series, which is a card added to a personal computer. Some computers integral synthesizers.
- the synthesizer can be external to the computer, and receive musical events as a MIDI stream coming from a MIDI output port.
- the term "synthesizer” is not used in a limiting sense. Herein, it is used to indicate any controllable musical device. Examples include systems capable of waveform playback, such as audio samplers and media players, and even automated acoustic instruments such as a MIDI controlled player piano. True synthesizers, such as analog or FM-synthesizers (digital or analog) are also included. The implementation details of any of these alternatives is within the capability of an ordinary skilled programmer.
- FIGURES 2A and 2B illustrate how to obtain the data needed to determine the delay value applied to each music event message by delay 170. The formula for the delay value is given below. " Referring to FIGURE 2A, an idealized connection topology for four performance stations A, B, C, and D (10, 12, 14, and 16 respectively) is shown. No regard is given for the exact nature of the communication channel 150, except that each performance station 10, 12, 14, and 16, can connect directly with any other.
- the communication delay is considered proportional to the distance between each performance station.
- scale 210 shows an exemplary 25 milliseconds between each adjacent pair of performance stations.
- performance stations A and C (10 and 14)
- the resulting communication delays for communication between any two performance stations is shown in table 212.
- a good estimate of a communication delay can be made by measuring how long it takes for a message to make a round trip between two stations. A commonly used measurement is made with the ping protocol, but custom messages implemented from within the application can give a more representative measurement. An estimate is made better by averaging the round trip time for many such messages. Typically, the first few rounds of the message are ignored for the purpose of measurement.
- FIGURE 2B illustrates a different topology.
- each of the four performance stations A, B, C, and D (10, 12, 14, and 16) communicate with each other only through a fanout server S, 18.
- the communication delay between any two performance stations is the sum of the communication delays between each station and the fanout server 18.
- the communication delay from Station A 10 to Server S 18 is 75 milliseconds, as shown by scale 220.
- the fanout server S 18 can undertake to measure the communications delay between itself and each of the performance stations 10, 12, 14, and 16. The results of those measurements can be provided to all of the performance stations. The sums of the delays between each of two stations and the fanout server S 18 can be used to populate the value for each value in the table (with the diagonal elements being held to zero, as above.)
- the values of the communication delay table 212 or 222 are best measured empirically, by the methods known to the art, with the precautions mentioned above.
- the contents of the table can usually be considered fixed, for relatively stable situations.
- the table values can be continuously updated to reflect dynamically changing load placed by unrelated traffic on the communication channel 150.
- Each performance station A, B, C, or D is interested only in the column representing the time it takes for messages sent by other performance stations to reach it. This is contrary to the "fair game" criteria of the prior art, where the whole table had to be considered.
- a musical event message is sent to delay 170, it is associated with a delay value. If the musical event message comes from the local event interpretation (110 for performance station 10, 110' for performance station 12, etc.), then the delay value is maximum value in the table column for that performance station.
- delay 170 For example, local musical events are artificially delayed by delay 170 for the same amount of time that it takes for a message to arrive from the (temporally speaking) furthest participating performance station.
- a musical event is generated by the musician at performance station B 12.
- the delay value set by delay 170' is the maximum delay found in column B of the delay table (212), that is, 50mS: the communication channel delay measured for Station D 16.
- the delay value is calculated as the maximum value in the table column for the receiving performance station, less the value in that column for the transmitting station.
- a remote musical event is artificially delayed by delay 170 for enough additional time to equal the amount of time that it takes for a message to arrive from the (temporally speaking) furthest participating performance station.
- a musical event is generated by the musician at Station A 10.
- the musical event message is sent via communication channel 150 and is received by Station B 12.
- the delay value set by delay 170 ' is the maximum delay found in column B of the delay table 212, again 50mS, but less the time it took the message to travel from Station A to Station B, that is row A of column B, or 25mS.
- the musician of performance station A 10 in FIGURE 2A may elect to set the maximum supported delay to be 60mS.
- the delay value When calculating the delay value, only values less than the maximum supported delay value will be considered. Thus, since (as seen in column A of table 212) only stations B and C are within the maximum supported delay, only those values will be considered when calculating the maximum delay. Thus, the delay applied to local musical events at Station A becomes 50mS (the maximum value in column A less than 60mS) , where otherwise it would have been 75mS (the maximum value in column A) .
- the first is to react to the event as if the calculated delay was zero - play the note now, stop playing the note now, etc.
- policy can be set depending upon the command in the musical event: ignoring the musical event may be suitable for a note on command; responding to the musical event may be suitable for state altering events (e.g., change instrument) .
- Another alternative is to ignore all musical events having a negative delay value, or having a sufficiently negative one. For example, musical events that are received up to 2OmS late (a negative delay value between 0 and -2OmS) may be acceptable, but musical events later than that (having delay value less than -2OmS) would be ignored.
- delay 170 causes local musical events to be delayed before they are sent to the instrument synthesizer 180, is that the instrument takes on an additional quality of prolonged attack. That is, the time from when a musician presses a key to the time the instrument sounds is increased by the delay value. For larger values of the delay value, this can be perceptible to even a novice musician, e.g. a lOOOmS delay would result in the instrument sounding one full second after the key has been pressed. However, for smaller values of the delay, say, less than lOOmS, a novice musician is not notably disturbed by the delay. Experienced musicians can adapt to delay values of 60mS readily.
- delay 170 makes use of the same delay values from delay table 212 or 222.
- each performance station when messages are being sent back and forth between the performance stations to characterize the communication channel delays, each performance station includes in the message the value of its clock, which should have a resolution of about one millisecond, though ten millisecond resolution can suffice.
- each time this clock augmented ping-like message is exchanged not only is the local estimate of the communication channel delay updated, but so is an estimate of the remote station's clock. In this manner, not only is the local performance station able to estimate . that the delay in a message from a particular remote performance station, but it also can determine how far ahead or behind that remote performance station's clock is running.
- a musical event includes the clock time of the transmitting performance station.
- the delay value is added to a computed local time, generated by adding the measured offset of the remote performance station's clock.
- This implementation- is particularly useful for environments where delays in the communication channel are highly volatile. In this way, fluctuations in the actual communications channel delay are removed. Note, however that an uncommonly long communications channel delay may cause the sum of the computed local time and the delay value to fall in the past. In such a case, the musical event can be ignored, or if the lateness does not exceed a threshold, the note can be played immediately, even though it arrived too late .
- An exception to the use of a maximum supported delay is the starting of a groove track.
- the instrument synthesizer 180 includes the ability to play a groove track.
- the groove track is stored as a WAV or MP3 audio file, and thus can include instrumental and lyrical performance.
- the groove track can be stored as a MIDI sequence, which has the advantage of compactness .
- each performance station 10, 12, 14 and 16 possess a local copy of the same groove track file.
- the selection of a groove track file (a non-musical event) at one performance station causes a groove track file selection event message to be propagated to each of the other performance stations .
- the groove track file selection event message arrives at receiver module 160, 160', and 160"
- the message handler determines whether that file is available on the local machine. If so, it is prepared as the current local groove track file, and an acknowledgement is sent.
- the groove track selection event may contain information (such as a URL) for finding the groove track file on the Internet.
- the groove track may be transmitted with the selection event message, though this makes for a heavy message.
- the station making the groove track selection to begin streaming the groove track file to each of the other performance stations, in which case, the performance stations receiving the streaming file begin buffering it.
- the preferred graphical user interface 300 is shown in FIGURE 3.
- the functions of the invention are made available through a menu bar 310 having menu items 312, 314, and 316.
- the middle portion of the display is preferably the local performance station status area 320.
- the lower portion of the display contains the remote performance station status areas 340, 340' .
- the local performance station status area 320 provides a local first instrument indicator 324, and local second instrument indicator 324 ' . Each indicator shows an image of the currently active instrument 322 and 322'.
- Control selection region 326 contains control indicators 328 and 328' for the respective first and second local instruments.
- the control indicators 328 and 328' preferably show which computer keyboard keys correspond to which notes on a chromatic scale.
- the control indicator 328 for the local first instrument an acoustic guitar as shown by local first active instrument 322 (to be synthesized by local instrument synthesizer 180)
- Control indicator 328 suggests a limited range when a computer keyboard is employed as a chromatic keyboard.
- the SHIFT, CONTROL, and ALT modifier keys are preferably used.
- the SHIFT key can be assigned the function of raising the whole keyboard by one octave, and the ALT key by two octaves.
- the CONTROL key commands a drop of one octave. In this way, the musical range of the computer keyboard is expanded to over four octaves.
- Combinations of the modifier keys preferably select the highest modification from those selected. In the alternative, they could be combined to extend the range even further .
- a complication can occur if a note key, say the 'Q', is pressed, followed by the SHIFT key being pressed in anticipation of some future note, after which point the 'Q' is released.
- the respective graphic indicator 328 would indicate the MIDI keyboard and MIDI channel.
- Selection of alternative controls or active instruments can be made through the Instruments menu item 314, or by direct manipulation of elements of the control selection region 326. For example, clicking on the first local instrument control indicator 328 can call up an instrument selection dialog 400, shown in FIGURE 4. The clicking of any instrument choice button 401, 402, or 403 (or any others not called out) will immediately cause that choice to become the active instrument for the local first instrument and the first currently active indicator 322 would be updated to reflect the new choice. Pressing the cancel button 410 causes no change to be made to the active instrument.
- an implementation uses a MIDI representation - at least internally. if so, then the result of an instrument selection is a voice assignment to a MIDI channel.
- the local first instrument can foe implemented as MIDI channel 1, the local second instrument as MIDI channel 2, the first remote musician's instrument selections are on MIDI channels 3 & 4 respectively, and the second remote musician's instruments selections are on MIDI channels 5 & 6.
- each remote musician's name is displayed in remote musician displays 342 and 342'.
- the remote musicians' first instrument indicators 344 and 344', and second instrument indicators 346 and 346' contain images of their currently active instruments 345, 345', 347 and 347' respectively.
- volume controls 350, 350', 352, and '352' are adjusted using the sliders 351, 351', 353, and 353' respectively. Each adjusts the volume with which the nearby instrument or instruments is played.
- volume control 350 sets the overall volume for the local first instrument
- volume control 352 sets the overall volume for the first remote player's instruments.
- the volume setting is used as the "velocity" parameter in MIDI Note-On commands.
- volume settings at the remote performance stations have no effect on the volume at the local performance station.
- the volume settings 350 and 350' at a remote performance station can be combined with the local volume setting 352 for the remote performance station (realizing that the graphical user interface 300 is separately displayed at each performance station 10, 12, 14, and 16, and at each performance station) .
- the velocity parameter of the MIDI Note-On commands resulting can provide a separate volume value for each keypress.
- the velocity values can be combined with the appropriate volume control so that the volume control is able to raise or lower the volume of the instrument relative to the other instruments.
- volume control is an important feature, as it allows a musician to emphasize an instrument of interest, and to de- emphasize instruments (or musicians) that are distracting or otherwise detracting from the local musician's enjoyment. Alternatively, separate volume controls can be provided for each remote instrument .
- the Groove menu item 316 allows a musician to select a previously prepared musical track to be played.
- the groove may be stored as a WAV file containing audio waveform data, or as an MP3 or other compressed sound format. Such audio files are especially well suited to lyrical grooves (i.e. a groove that includes a singer) , or to allow musicians to jam to a prerecorded audio track.
- the groove may be stored as an MID file, i.e. a MIDI sequence file.
- the groove playback control panel 360 becomes active.
- the groove playback control panel 360 contains a play button 362, and a stop button 364.
- play button 362 When play button 362 is pressed, a musical event is generated which, after an appropriate delay introduced by delay 170 and described above, will cause the groove to start playing.
- stop button 364 will cause the groove to stop playing.
- the Jam menu item 312 allows the musician to prepare the local performance station for a jam session, and optionally select remote jam partners. This same functionality is preferably available by clicking jam button 330.
- a first musician at local performance station 10 presses jam button 330 and responds to a dialog that the local performance station 10 is to anticipate incoming remote jam requests.
- dialogs are well known in the art.
- the DplayConnect object has a StartConnectWizard method which provides all necessary dialogs.
- a first dialog requiring selection of a communication channel is preferred.
- the modem is thereby instructed to answer the phone.
- any communications received by the modem are passed to the local performance station software.
- an IP port is prepared and begins listening for connections by remote performance stations.
- the other musicians direct their respective remote performance stations to use the, appropriate communication channel 150, if more than one is available.
- the communication channel selected (or the only one available) must be the same communication channel 150 selected for use by the first musician.
- the remote performance stations 12, 14, and 16 must be provided with an address for the first musician's performance station 10. Preferably, this is done with a dialog asking for a phone number in cases of a modem connection, or for an IP address in cases of an Internet connection.
- a delay display 370 shows the maximum delay detected and used by delay 170 to delay musical events originated at local keyboard and controls 100 before they are provided to local instrument synthesizer 180 to be played.
- delay values list pulldown 372 when pressed, would display the list (not shown) of delay values currently observed for each participating remote station 10, 12, 14, 16. For example, for performance station A 10, this would be the list of values in tables 212 or 222, column A.
- the list of delay values is sorted in ascending order.
- the current selection for maximum delay value would be highlighted.
- the list can include the name of the associated musician next to each remote station delay value.
- the musician can set a local maximum supported delay for the local performance station.
- an additional control can be added to allow the local musician to directly . specify the maximum supported delay. This might be implemented as a dial or slider, and may drive the value displayed in delay display 370 directly. Referring to FIGURE 5, shown is a flowchart of the steps of the present invention.
- the local performance station 10 initializes communication with at least one remote performance station 12, 14, 16, and populates the delay table (212 or 222, as appropriate to the utilization of a fanout server 18) as previously described. Once so initialized, the local performance station awaits the occurrence of a local or remote event.
- a local event occurs 510 from keyboard or controls 100.
- Event interpretation 110 evaluates the event 520 to evaluation whether it is a musical event. If not, the event is passed 522 to other event handlers . If the event is a musical event, the jam status is examined 530. If a jam is in progress, the musical event is sent to the remote performance station (s) by one of two ways: If the system is configured 532 without a fanout server 18, then the musical event is sent 534 to each of the remote performance stations.
- the musical event is sent 536 to the fanout server.
- the delay value for the local performance station is selected 540. If a jam is in progress, the column for the local performance station of the delay table (212 or 222) is pop ⁇ lated, and the largest value in that column is selected. Alternatively, if a maximum supported delay has been set, that delay is used instead. When no jam is in progress, the delay value is zero. Alternatively, if a maximum supported delay has been set, that value is used instead. This permits a musician to practice alone, but to have a delay in the playing of musical events be the same or similar to the delay when a jam is in progress.
- the musical event is played 560.
- a remote musical event occurs 570 when it is received by the local performance station. It is possible for messages to be received that are non-musical or otherwise not intended for the performance station software. Such messages have been directed elsewhere by the receive module 160 to be handled.
- the delay value is selected 580. The delay value depends upon which remote performance station originated the musical event. If a maximum supported delay has been set, then the delay value is compared 582 to the maximum supported delay. If the delay value exceeds the maximum supported delay, the remote musical event is not played 590.
- An alternative step 582 would allow a musical event having a delay value that exceeds a maximum supported delay to be played immediately 560 with no hold step 550.
- the alternative step 582 would allow musical events having an excessive delay value to be played immediately 560, but only if the delay value is within a preset threshold of the maximum supported delay.
- communication channel 150 is the most efficient avenue available for communication between the participating musicians. As such, the ability for the musicians to communicate other than through musical events is highly desirable.
- voice packets are easy.
- a musician's voice is captured by a microphone (not shown) and digitized at remote station 12. Packets of the digitized voice, perhaps 1/10 of a second long, each, are compressed and buffered. When no musical events are pending, the next voice packet is inserted into the message stream at transmit module 130'. The voice packet is received at the local performance station 10. When it is identified by receive module 160, it is passed as a non-musical message to a voice packet buffer (not shown) .
- a process begins the decompression of the remote musician's voice, which is sent to audio output 190.
- the voice capture and transmit process is controlled using a conventional push-to-talk intercom switch.
- a good choice is to assign the spacebar of the keyboard as this intercom switch.
- a talk-to-talk mechanism can be used, where, if the audio level detected by the microphone exceeds some threshold, then voice packets start getting compressed and buffered for sending. If the audio level drops for too long a period of time, no more voice packets are prepared.
- GUI displays While the preferred embodiment is discussed in the context of present day GUI displays, keyboards, MIDI controllers, and communications channels, it is contemplated that other modes of input and communications will be suitable as they are made available .
- the particular features of the user interface and the performance of the application will depend on the architecture used to implement a system of the present invention, the operating system of the computers selected, the communications channel selected, and the software code written.
Abstract
Description
Claims
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/US2003/027063 WO2005031697A1 (en) | 2002-03-01 | 2003-08-29 | Method and apparatus for remote real time collaborative music performance |
Publications (2)
Publication Number | Publication Date |
---|---|
EP1702318A1 EP1702318A1 (en) | 2006-09-20 |
EP1702318A4 true EP1702318A4 (en) | 2007-11-07 |
Family
ID=34392789
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP03818811A Withdrawn EP1702318A4 (en) | 2003-08-29 | 2003-08-29 | Method and apparatus for remote real time collaborative music performance |
Country Status (2)
Country | Link |
---|---|
EP (1) | EP1702318A4 (en) |
AU (1) | AU2003265829A1 (en) |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5820463A (en) * | 1996-02-06 | 1998-10-13 | Bell Atlantic Network Services, Inc. | Method and apparatus for multi-player gaming over a network |
US6009457A (en) * | 1996-04-01 | 1999-12-28 | Rocket Network, Inc. | Distributed real-time communications system |
US6067566A (en) * | 1996-09-20 | 2000-05-23 | Laboratory Technologies Corporation | Methods and apparatus for distributing live performances on MIDI devices via a non-real-time network protocol |
US6175872B1 (en) * | 1997-12-12 | 2001-01-16 | Gte Internetworking Incorporated | Collaborative environment for syncronizing audio from remote devices |
-
2003
- 2003-08-29 EP EP03818811A patent/EP1702318A4/en not_active Withdrawn
- 2003-08-29 AU AU2003265829A patent/AU2003265829A1/en not_active Abandoned
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5820463A (en) * | 1996-02-06 | 1998-10-13 | Bell Atlantic Network Services, Inc. | Method and apparatus for multi-player gaming over a network |
US6009457A (en) * | 1996-04-01 | 1999-12-28 | Rocket Network, Inc. | Distributed real-time communications system |
US6067566A (en) * | 1996-09-20 | 2000-05-23 | Laboratory Technologies Corporation | Methods and apparatus for distributing live performances on MIDI devices via a non-real-time network protocol |
US6175872B1 (en) * | 1997-12-12 | 2001-01-16 | Gte Internetworking Incorporated | Collaborative environment for syncronizing audio from remote devices |
Non-Patent Citations (2)
Title |
---|
CHRIS CHAFE: "Distributed Internet Reverberation for audio collaboration", AES 24TH INTERNATIONAL CONFERENCE ON MULTICHANNEL AUDIO, 26 June 2003 (2003-06-26) - 28 June 2003 (2003-06-28), Banff, Alberta, Canada, XP002451971, Retrieved from the Internet <URL:http://www-ccrma.stanford.edu/~cc/soundwire/dirac.pdf> [retrieved on 20070918] * |
See also references of WO2005031697A1 * |
Also Published As
Publication number | Publication date |
---|---|
AU2003265829A1 (en) | 2005-04-14 |
EP1702318A1 (en) | 2006-09-20 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6653545B2 (en) | Method and apparatus for remote real time collaborative music performance | |
US6975995B2 (en) | Network based music playing/song accompanying service system and method | |
Weinberg | Interconnected musical networks: Toward a theoretical framework | |
US7518051B2 (en) | Method and apparatus for remote real time collaborative music performance and recording thereof | |
US6936758B2 (en) | Player information-providing method, server, program for controlling the server, and storage medium storing the program | |
US7947889B2 (en) | Ensemble system | |
US5631433A (en) | Karaoke monitor excluding unnecessary information from display during play time | |
CN102576524A (en) | System and method of receiving, analyzing, and editing audio to create musical compositions | |
EP1212747A1 (en) | Method and apparatus for playing musical instruments based on a digital music file | |
Weinberg | The aesthetics, history and future challenges of interconnected music networks | |
JP4797523B2 (en) | Ensemble system | |
Freeman | Large audience participation, technology, and orchestral performance | |
JP4700351B2 (en) | Multi-user environment control | |
WO2021246104A1 (en) | Control method and control system | |
Aimi | New expressive percussion instruments | |
EP1702318A1 (en) | Method and apparatus for remote real time collaborative music performance | |
Helmuth | Sound exchange and performance on Internet2 | |
Weinberg et al. | iltur: Connecting novices and experts through collaborative improvisation | |
WO2002080080A1 (en) | Method for providing idol star management service based on music playing/song accompanying service system | |
KR20210026656A (en) | Musical ensemble performance platform system based on user link | |
Freeman | Glimmer: Creating new connections | |
JPH10235015A (en) | System making game proceed simultaneously by sending quiz to numerous terminal by satellite data communication and game terminal | |
Dannenberg et al. | Scaling up live internet performance with the global net orchestra | |
JP3404594B2 (en) | Recording medium and music game apparatus | |
JP2000181468A (en) | Sound image processor |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20060630 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PT RO SE SI SK TR |
|
AX | Request for extension of the european patent |
Extension state: LT LV |
|
RAX | Requested extension states of the european patent have changed |
Extension state: LV Payment date: 20060718 Extension state: LT Payment date: 20060718 |
|
A4 | Supplementary search report drawn up and despatched |
Effective date: 20071010 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G10H 1/40 20060101ALN20071001BHEP Ipc: G10H 1/00 20060101AFI20071001BHEP |
|
17Q | First examination report despatched |
Effective date: 20080122 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN |
|
18D | Application deemed to be withdrawn |
Effective date: 20120313 |