WO2015114216A3 - Audio segments analysis to determine danceability of a music and for video and pictures synchronisaton to the music. - Google Patents

Audio segments analysis to determine danceability of a music and for video and pictures synchronisaton to the music. Download PDF

Info

Publication number
WO2015114216A3
WO2015114216A3 PCT/FI2015/050059 FI2015050059W WO2015114216A3 WO 2015114216 A3 WO2015114216 A3 WO 2015114216A3 FI 2015050059 W FI2015050059 W FI 2015050059W WO 2015114216 A3 WO2015114216 A3 WO 2015114216A3
Authority
WO
WIPO (PCT)
Prior art keywords
music
segment
audio signal
audio
video
Prior art date
Application number
PCT/FI2015/050059
Other languages
French (fr)
Other versions
WO2015114216A2 (en
Inventor
Antti Eronen
Igor Curcio
Juha OJANPERÄ
Mikko ROININEN
Original Assignee
Nokia Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nokia Corporation filed Critical Nokia Corporation
Publication of WO2015114216A2 publication Critical patent/WO2015114216A2/en
Publication of WO2015114216A3 publication Critical patent/WO2015114216A3/en

Links

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/36—Accompaniment arrangements
    • G10H1/40—Rhythm
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • G06F16/43—Querying
    • G06F16/438—Presentation of query results
    • G06F16/4387—Presentation of query results by the use of playlists
    • G06F16/4393—Multimedia presentations, e.g. slide shows, multimedia albums
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/0008—Associated control or indicating means
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H1/00—Details of electrophonic musical instruments
    • G10H1/36—Accompaniment arrangements
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
    • G10H2210/041—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal based on mfcc [mel -frequency spectral coefficients]
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
    • G10H2210/051—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal for extraction or detection of onsets of musical sounds or notes, i.e. note attack timings
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
    • G10H2210/061—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal for extraction of musical phrases, isolation of musically relevant segments, e.g. musical thumbnail generation, or for temporal structure analysis of a musical piece, e.g. determination of the movement sequence of a musical work
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
    • G10H2210/071—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal for rhythm pattern analysis or rhythm style recognition
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
    • G10H2210/076—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal for extraction of timing, tempo; Beat detection
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2240/00—Data organisation or data communication aspects, specifically adapted for electrophonic musical tools or instruments
    • G10H2240/075—Musical metadata derived from musical analysis or for use in electrophonic musical instruments
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2250/00—Aspects of algorithms or signal processing methods without intrinsic musical character, yet specifically adapted for or used in electrophonic musical processing
    • G10H2250/005—Algorithms for electrophonic musical instruments or musical processing, e.g. for automatic composition or resource allocation
    • G10H2250/015—Markov chains, e.g. hidden Markov models [HMM], for musical processing, e.g. musical analysis or musical composition
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2250/00—Aspects of algorithms or signal processing methods without intrinsic musical character, yet specifically adapted for or used in electrophonic musical processing
    • G10H2250/025—Envelope processing of music signals in, e.g. time domain, transform domain or cepstrum domain
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2250/00—Aspects of algorithms or signal processing methods without intrinsic musical character, yet specifically adapted for or used in electrophonic musical processing
    • G10H2250/131—Mathematical functions for musical analysis, processing, synthesis or composition
    • G10H2250/135—Autocorrelation
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2250/00—Aspects of algorithms or signal processing methods without intrinsic musical character, yet specifically adapted for or used in electrophonic musical processing
    • G10H2250/131—Mathematical functions for musical analysis, processing, synthesis or composition
    • G10H2250/215—Transforms, i.e. mathematical transforms into domains appropriate for musical signal processing, coding or compression
    • G10H2250/235—Fourier transform; Discrete Fourier Transform [DFT]; Fast Fourier Transform [FFT]
    • G—PHYSICS
    • G11—INFORMATION STORAGE
    • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
    • G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
    • G11B27/036—Insert-editing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/00127—Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture
    • H04N1/00132—Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture in a digital photofinishing system, i.e. a system where digital photographic images undergo typical photofinishing processing, e.g. printing ordering
    • H04N1/00185—Image output
    • H04N1/00196—Creation of a photo-montage, e.g. photoalbum
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/233—Processing of audio elementary streams

Abstract

A technique for audio processing that comprises obtaining four types of features descriptive of characteristics of a segment of audio signal representing a piece of music and, based on these features, deriving a "club score" that is indicative of at least beat strength associated with said segment of audio signal, thus describing a "danceablility" of music. An application comprises obtaining one or more audio attributes characterizing a segment of audio signal representing the piece of music, calculating a club score, and selecting a switching pattern from a plurality of predetermined switching patterns based on the club score, wherein a switching pattern is arranged to indicate discontinuities, or temporal positions and/or frequency of changes of video sources or image, in a visual content associated with said segment of audio signal, in relation to temporal locations of beats or downbeats identified for the segment of audio signal, for example for generating a visual presentation to accompany remixed music, synchronised to the music.
PCT/FI2015/050059 2014-01-31 2015-01-30 Audio signal analysis WO2015114216A2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB1401626.5 2014-01-31
GB1401626.5A GB2522644A (en) 2014-01-31 2014-01-31 Audio signal analysis

Publications (2)

Publication Number Publication Date
WO2015114216A2 WO2015114216A2 (en) 2015-08-06
WO2015114216A3 true WO2015114216A3 (en) 2015-11-19

Family

ID=50344136

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/FI2015/050059 WO2015114216A2 (en) 2014-01-31 2015-01-30 Audio signal analysis

Country Status (2)

Country Link
GB (1) GB2522644A (en)
WO (1) WO2015114216A2 (en)

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10014841B2 (en) 2016-09-19 2018-07-03 Nokia Technologies Oy Method and apparatus for controlling audio playback based upon the instrument
JP6842558B2 (en) * 2017-09-12 2021-03-17 ＡｌｐｈａＴｈｅｔａ株式会社 Music analysis device and music analysis program
CN111243618B (en) * 2018-11-28 2024-03-19 阿里巴巴集团控股有限公司 Method, device and electronic equipment for determining specific voice fragments in audio
GB2583441A (en) * 2019-01-21 2020-11-04 Musicjelly Ltd Data synchronisation
CN113223487B (en) * 2020-02-05 2023-10-17 字节跳动有限公司 Information identification method and device, electronic equipment and storage medium
CN112435641B (en) * 2020-11-09 2024-01-02 腾讯科技（深圳）有限公司 Audio processing method, device, computer equipment and storage medium
CN115250360A (en) * 2021-04-27 2022-10-28 北京字节跳动网络技术有限公司 Rhythm interaction method and equipment
CN113590076B (en) * 2021-07-12 2024-03-29 杭州网易云音乐科技有限公司 Audio processing method and device
CN113674723A (en) * 2021-08-16 2021-11-19 腾讯音乐娱乐科技（深圳）有限公司 Audio processing method, computer equipment and readable storage medium
CN114268814A (en) * 2021-11-29 2022-04-01 广州繁星互娱信息科技有限公司 Music video acquisition method and device, storage medium and electronic equipment

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040027369A1 (en) * 2000-12-22 2004-02-12 Peter Rowan Kellock System and method for media production
US20050217462A1 (en) * 2004-04-01 2005-10-06 Thomson J Keith Method and apparatus for automatically creating a movie
WO2011051279A1 (en) * 2009-10-30 2011-05-05 Dolby International Ab Complexity scalable perceptual tempo estimation
SG178778A1 (en) * 2007-03-02 2012-03-29 Animoto Llc Automatically generating audiovisual works
WO2013164661A1 (en) * 2012-04-30 2013-11-07 Nokia Corporation Evaluation of beats, chords and downbeats from a musical audio signal
WO2014001849A1 (en) * 2012-06-29 2014-01-03 Nokia Corporation Audio signal analysis

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080300702A1 (en) * 2007-05-29 2008-12-04 Universitat Pompeu Fabra Music similarity systems and methods using descriptors
EP2793223B1 (en) * 2010-12-30 2016-05-25 Dolby International AB Ranking representative segments in media data

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040027369A1 (en) * 2000-12-22 2004-02-12 Peter Rowan Kellock System and method for media production
US20050217462A1 (en) * 2004-04-01 2005-10-06 Thomson J Keith Method and apparatus for automatically creating a movie
SG178778A1 (en) * 2007-03-02 2012-03-29 Animoto Llc Automatically generating audiovisual works
WO2011051279A1 (en) * 2009-10-30 2011-05-05 Dolby International Ab Complexity scalable perceptual tempo estimation
WO2013164661A1 (en) * 2012-04-30 2013-11-07 Nokia Corporation Evaluation of beats, chords and downbeats from a musical audio signal
WO2014001849A1 (en) * 2012-06-29 2014-01-03 Nokia Corporation Audio signal analysis

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
DANIEL P. W. ELLIS: "Beat Tracking by Dynamic Programming", JOURNAL OF NEW MUSIC RESEARCH, vol. 36, no. 1, 16 July 2007 (2007-07-16), pages 51 - 60, XP055177341, ISSN: 0929-8215, DOI: 10.1080/09298210701653344 *
ELIAS PAMPALK: "Computational Models of Music Similarity and their Application in Music Information Retrieval", DOCTOR THESIS, 1 March 2006 (2006-03-01), Wien, XP055177322, Retrieved from the Internet <URL:http://www.ofai.at/~elias.pampalk/publications/pampalk06thesis.pdf> [retrieved on 20150317] *
HERRERA PERFECTO ET AL: "Detrended Fluctuation Analysis of Music Signals: Danceability Estimation and Further Semantic Characterization", AES CONVENTION 118; MAY 2005, AES, 60 EAST 42ND STREET, ROOM 2520 NEW YORK 10165-2520, USA, 1 May 2005 (2005-05-01), XP040507217 *

Also Published As

Publication number Publication date
GB2522644A (en) 2015-08-05
GB201401626D0 (en) 2014-03-19
WO2015114216A2 (en) 2015-08-06

Similar Documents

Publication Publication Date Title
WO2015114216A3 (en) Audio segments analysis to determine danceability of a music and for video and pictures synchronisaton to the music.
USD784392S1 (en) Display screen with an animated graphical user interface
EP4254145A3 (en) Head pose mixing of audio files
MX2018004828A (en) Apparatus and method for generating a filtered audio signal realizing elevation rendering.
WO2015184196A3 (en) Speech summary and action item generation
WO2018013192A3 (en) Extraction of features from physiological signals
EP2846229A3 (en) Systems and methods for generating haptic effects associated with audio signals
MY184715A (en) Apparatus and method for screen related audio object remapping
JP2007248895A5 (en)
MX2015016142A (en) Image display method, image display apparatus, and recording medium.
WO2018085613A3 (en) Intuitive occluded object indicator
WO2015008469A3 (en) Information processing apparatus, information processing method, and information processing system
EP2863339A3 (en) Methods and systems for determing user liveness
MX2015004848A (en) Method relating to presence granularity with augmented reality.
EP4280484A3 (en) Synchronized audio mixing
WO2011041424A4 (en) Providing visual responses to musically synchronized touch input
AU366258S (en) A display screen or portion thereof with an image from a sequence of images forming an animated graphical user interface
MX2016000843A (en) Playback control method and apparatus, and electronic device.
EP2818215A3 (en) Method and system for expressing emotion during game play
WO2013072554A3 (en) Spatial visual effect creation and display such as for a screensaver
EP4239498A3 (en) Image selection suggestions
EP2932889A3 (en) Apparatus for performing multidimensional velocity measurements using amplitude and phase in optical interferometry
PH12016000288A1 (en) Game information analysis system
TWD173517S (en) Video projector
BR112017020011A2 (en) method and apparatus for performing real-time input pass position detection

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15704581

Country of ref document: EP

Kind code of ref document: A2

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15704581

Country of ref document: EP

Kind code of ref document: A2