US9013485B2 - Systems and methods for synthesizing high fidelity stroke data for lower dimension input strokes - Google Patents
Systems and methods for synthesizing high fidelity stroke data for lower dimension input strokes Download PDFInfo
- Publication number
- US9013485B2 US9013485B2 US13/223,139 US201113223139A US9013485B2 US 9013485 B2 US9013485 B2 US 9013485B2 US 201113223139 A US201113223139 A US 201113223139A US 9013485 B2 US9013485 B2 US 9013485B2
- Authority
- US
- United States
- Prior art keywords
- stroke
- data
- query
- samples
- feature vectors
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F3/0416—Control or interface arrangements specially adapted for digitisers
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/0354—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
- G06F3/03545—Pens or stylus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/001—Texturing; Colouring; Generation of texture or colour
Definitions
- Sketching or drawing on paper is a natural way to create and envision shapes.
- a goal of computer-based sketching systems is to leverage this ability in order to create models of real or imaginary objects (including 3D objects or stylized 2D objects) on the computer (e.g., using shading and other stylistic effects).
- humans unlike computers
- Typical sketching applications create only simple 2D input, which may then need to be interpreted as 3D or stylized 2D data in some way. It can be difficult for a computer to perform this interpretation in a manner that is pleasing to people. For example, even when drawings are not precise (or even correct), a person may nevertheless have a strong expectation of how they should be interpreted and/or how the person would prefer that they appear.
- paintbrush simulation technology e.g., Adobe® Illustrator® software from Adobe Systems, Inc.
- Illustrator software computes a dynamic six degree of freedom (6DOF) pose based on heuristics about how an artist typically holds a brush relative to a stroke (e.g., tilted at 45 degrees and oriented tangent to the stroke, with a constant pressure).
- 6DOF six degree of freedom
- Systems and methods for synthesizing paintbrush strokes may use high fidelity pose data of reference strokes to supplement lower dimension stroke data.
- the systems and methods described herein may be used to create a sequence of synthesized poses that follows the trajectory of an input stroke (or query stroke) dependent, at least in part, on a library of collected reference strokes, and to output data representing the sequence of synthesized poses.
- the library may include 6DOF data pre-recorded (or otherwise captured) during the creation of a variety of reference strokes by skilled artists.
- the input stroke data may be compared to data in the library to find the best matches for respective portions of the input stroke (e.g., piece by piece).
- the data representing the input (query) stroke may include data for fewer degrees of freedom than the data representing the reference strokes, and/or fewer degrees of freedom than the data representing the sequence of synthesized poses (i.e. the output stroke).
- the reference strokes and/or the output stroke may include full 6DOF data, while the input stroke may include between two and five DOF data.
- a stroke synthesis engine (which may be a component of an image editing application) may receive data representing a query stroke.
- the data representing the query stroke may include trajectory data, i.e. data that indicates the trajectory of the query stroke with respect to two degrees of freedom (e.g., x, y data in the plane of the stroke).
- the data representing the query stroke may also include pose data (e.g., data in additional degrees of freedom).
- a query stroke made by a less skilled artist/user and/or made using input devices that do not provide 6DOF data may be processed by such a stroke synthesis engine to produce an output stroke that includes 6DOF data.
- a stroke synthesis engine may access data representing one or more reference strokes.
- the data representing each of the reference strokes may include both trajectory data (e.g., data indicating a trajectory of the reference stroke within the plane of the reference stroke) and pose data (e.g., data corresponding to one or more other degrees of freedom, such as tilt, twist, or pressure data).
- the reference strokes may have been created by one or more skilled artists using advanced input devices.
- the collected reference stroke data may be stored in one or more libraries (or databases), grouped by style, content, artist, and/or other criteria.
- synthesizing an output stroke may include constructing feature vectors for samples of the reference strokes and for samples of the query stroke being processed.
- the feature vector for each of the stroke samples may include trajectory data components that are dependent on trajectory data for one or more neighboring samples of the stroke.
- the feature vector for a query stroke or a reference stroke may also include pose data components that are dependent on pose data for one or more neighboring samples.
- feature vectors may be constructed for reference strokes when the reference strokes are created or added to a reference stroke library, or at any time after the reference strokes have been created or added to the library, including in response to the receipt of a query stroke.
- Feature vectors may be constructed for query strokes when the query strokes are created, or in response to the invocation of a stroke synthesis operation on existing query stroke data, in different embodiments.
- a set of nearest neighbor features may be identified from the library of reference strokes by comparing the feature vector for each of the samples of the query stroke to the feature vectors for a plurality of samples of one or more reference strokes.
- the collection of nearest neighbor feature vectors for each sample of the query stroke may include the two or more feature vectors for respective samples of one or more reference strokes that are most similar to the feature vector constructed for the sample of the query stroke.
- a best neighbor feature vector may be identified for each of the samples of the query stroke by selecting one of the nearest neighbor feature vectors from each of the collections of nearest neighbor feature vectors. In some embodiments, selecting the best neighbor feature vectors from each of the collections of nearest neighbor feature vectors includes applying dynamic programming to the collections of nearest neighbor feature vectors to determine the lowest cost sequence of nearest neighbor feature vectors. In some embodiments, selecting the best neighbor feature vectors from each of the collections of nearest neighbor feature vectors may include applying a weighting to the nearest neighbor feature vectors that encourages the selection, for adjacent samples of the query stroke, of nearest neighbor feature vectors for adjacent samples of the same reference stroke. The best neighbor feature vector for each of the samples of the query stroke may include pose data (or aggregated pose data) for a corresponding sample of one of the one or more reference strokes.
- the pose data of the best neighbor feature may be assigned as the pose data for the given sample of the query stroke, and may be included in the sequence of poses in the synthesized output stroke.
- the sequence of poses from the best neighbor features may be smoothed using a blending technique. For example, a smoothing operation may be performed to blend the pose data assigned to one or more pairs of adjacent query samples to produce a smoothed version of the synthesized output stroke.
- 6DOF data representing a synthesized stroke that follows the original x, y trajectory of the input query stroke and includes the smoothed sequence of poses may be output by the stroke synthesis engine. For example, it may be stored for later use and/or displayed by the system, in different embodiments.
- FIG. 1 is a flow diagram illustrating one embodiment of a method for synthesizing high fidelity stroke data for lower dimension input strokes.
- FIG. 2A illustrates an isometric view of the use of a tablet device, according to one embodiment.
- FIG. 2B illustrates a top view of the use of a tablet device, according to one embodiment.
- FIG. 3 is a flow diagram illustrating one embodiment of a method for processing reference stroke data and query stroke data to produce a synthesizing output stroke.
- FIG. 4A illustrates the use of captured and/or estimated stroke data to construct feature vectors at sample points of a query stroke, according to one embodiment.
- FIG. 4B illustrates a collection of triangle filters that may be used in constructing feature vectors for sample points of a query stroke, according to one embodiment.
- FIG. 5 is a flow diagram illustrating one embodiment of a method for constructing feature vectors for a query stroke and for identifying sets of neighbor features.
- FIG. 6 illustrates the association of a set of nearest neighbor feature vectors from a library of reference strokes with each sample in a query stroke, according to one embodiment.
- FIG. 7 is a flow diagram illustrating one embodiment of a method for determining the best neighbor for each of the samples of a query stroke.
- FIG. 8A illustrates a query stroke for which a lowest cost collection of nearest neighbors is to be determined, according to one embodiment.
- FIG. 8B illustrates the determination of a lowest cost collection of nearest neighbors for the samples in a query stroke based on feature vectors of one or more reference curves, according to one embodiment.
- FIG. 8C illustrates two reference curves for which consecutive feature vectors are determined to be in a collection of lowest cost nearest neighbors for the samples in a query stroke, according to one embodiment.
- FIG. 9 illustrates smoothing between the poses of best neighbor features found in two reference curves, according to one embodiment.
- FIG. 10 is a flow diagram illustrating one embodiment of a method for applying the stroke synthesis techniques described herein.
- FIGS. 11A-11E illustrate various reference strokes, query strokes, and output strokes, according to one embodiment.
- FIG. 12 illustrates a stroke synthesis engine that may implement synthesizing high fidelity stroke data for lower dimension input strokes, according to some embodiments.
- FIG. 13 illustrates an example computer system that is configured to implement synthesizing high fidelity stroke data for lower dimension input strokes, according to some embodiments.
- such quantities may take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared or otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to such signals as bits, data, values, elements, symbols, characters, terms, numbers, numerals or the like. It should be understood, however, that all of these or similar terms are to be associated with appropriate physical quantities and are merely convenient labels. Unless specifically stated otherwise, as apparent from the following discussion, it is appreciated that throughout this specification discussions utilizing terms such as “processing,” “computing,” “calculating,” “determining” or the like refer to actions or processes of a specific apparatus, such as a special purpose computer or a similar special purpose electronic computing device.
- a special purpose computer or a similar special purpose electronic computing device is capable of manipulating or transforming signals, typically represented as physical electronic or magnetic quantities within memories, registers, or other information storage devices, transmission devices, or display devices of the special purpose computer or similar special purpose electronic computing device.
- paintbrush simulation technologies may take maximal advantage of available hardware to create a similarly complex set of brush behaviors based on subtle changes in user input. For example, slight twists of a paintbrush during a stroke may be used to add texture and depth. For simulations to be able to reproduce these results, high fidelity input devices such as advanced tablet and/or stylus devices (e.g., an advanced art pen type device) may be used, and these advanced devices may provide six degree of freedom (6DOF) data that can be used to fully reconstruct the virtual brush's pose.
- 6DOF six degree of freedom
- 6DOF input data may not be available.
- many potential users of an image editing application e.g., a painting application
- do not have a tablet device or may have a lower end model.
- These users may only be able to provide stroke data in 2DOF (e.g., using a mouse, touchpad, trackpad, tablet, or other device that provides x, y trajectory data only), 3DOF (e.g., using a tablet that provides both x, y trajectory data and pressure data), or 5DOF (e.g., using a tablet that does not include an art pen, but that provides x, y, pressure, tilt-x, and tilt-y data).
- 2DOF e.g., using a mouse, touchpad, trackpad, tablet, or other device that provides x, y trajectory data only
- 3DOF e.g., using a tablet that provides both x, y trajectory data and pressure data
- 5DOF e.g., using a tablet that does not include an art
- many new classes of devices such as phones and slates, may support only the most rudimentary input technology and thus are frequently only able to provide 2DOF stroke data.
- a user may have created vector line art (using, e.g., a program such as the Illustrator program), which consists of a set of 2D paths, and may wish to apply an expressive stroke style to that vector art.
- a novice user may have access to any of a variety of available input devices (including advanced hardware capable of generating high fidelity stroke data), but may not be able to manipulate them as expertly as a trained artist or to effectively use them to create nicely styled strokes.
- a user of any ability skill level may want to apply an alternate style to their strokes, i.e. one that differs from the style of the input stroke data they provided. In other words, they may wish to “paint with someone else's hands”.
- the techniques described herein may in some embodiments be used to synthesize high quality stroke data (e.g., data in more than two or three degrees of freedom) from lower quality input data, and the high quality output strokes may in turn be used to drive a paintbrush simulation or other advanced image editing application.
- the lower quality input data may be captured using a two degree of freedom (2DOF) mouse, a finger on a touch pad, or another low quality input mechanism, may include a path captured from vector art and/or may include any type of stroke input (e.g., input representing a paint brush stroke) generated by a novice user of various simple or advanced input devices.
- 2DOF two degree of freedom
- the techniques described herein may utilize a database or other repository of high fidelity stroke data recorded (or otherwise captured) when strokes are created by expert artists to synthesize high quality output strokes from the lower quality input data (sometimes referred to herein as “query stroke data” or simply a “query stroke”).
- the systems described herein may utilize a working repository of reference data created by one or more expert artists (e.g., data that was collected from artists of varying skill levels and that represents various artistic styles) as input data.
- This input data may include a collection of stroke data created during the performance of proscribed drawing tasks, including writing English script and making simple line drawings.
- Artists creating such reference data may be told what to draw (e.g., with example images), and/or may be told what brush settings to use (if entering stroke data using an application that simulates the behavior of different paintbrushes through various brush tools and brush tool settings, such as the Photoshop application).
- the movements of the artists' hands and/or brush tools may be recorded (or otherwise captured) while creating the reference stroke data.
- artists may submit data files of their compositions that contain data organized as a set of strokes, where each stroke is a set of input samples that include data in all available degrees of freedom.
- systems and methods for synthesizing paintbrush strokes may use the high fidelity pose data of various reference strokes (or samples and portions thereof) to supplement lower dimension query stroke data.
- 6DOF data representing reference strokes created by skilled artists using advanced tablets and/or stylus devices may be captured and stored in a library of reference strokes.
- the systems described herein may read in a collection of reference stroke files to create a database or repository of expert artists' strokes.
- such a database or repository may be examined to find features that are similar to (i.e. nearest matches to) lower quality user input.
- reference stroke library or “reference curve library”
- reference curve library may be used somewhat interchangeably.
- a query stroke made by a less skilled artist/user and/or made using input devices that do not provide 6DOF data may be processed by a stroke synthesis engine to produce a high quality synthesized stroke that follows the trajectory of the query stroke.
- the stoke synthesis engine may read in files referred to as “queries” that contain stroke data for which poses are to be synthesized, or may receive stroke data directly (e.g., through a user interface of the stroke synthesis engine or an image editing application of which a stroke synthesis engine is a component).
- the system may process the query stroke data to create a sequence of synthesized poses.
- the stroke synthesis engine may construct feature vectors (sometimes referred to herein simply as “features”) from samples of reference strokes and the query stroke, may select the best neighbor feature from a set of nearest neighbor features in the reference stroke library for each query feature based on a lowest cost path through a graph of nearest neighbors, and may smooth the sequence of selected features to produce the synthesized stroke.
- the stroke synthesis engine may then write out a new data file that represents a synthesized output stroke that has the same x, y trajectories as the query stroke, but that also includes these synthesized poses.
- Various examples of systems and methods for synthesizing high quality strokes from lower quality input strokes may include one or more of the following terms:
- FIG. 1 is a flow diagram illustrating one embodiment of a method for synthesizing high fidelity stroke data given lower dimension input strokes.
- the method may include capturing and/or collecting stroke data samples from one or more artists while they perform common drawing tasks.
- data may be captured for reference strokes using an advanced tablet system and/or an advanced input stylus that together are able to provide 6DOF stroke data, as described herein.
- the method may include creating a database of reference strokes and/or corresponding features (as defined by corresponding feature vectors) of those reference strokes, as in 110 .
- reference strokes may be captured, collected, and/or categorized according to the artists who created the reference strokes, the style or content of the reference strokes (or the images from which they are taken), or various other criteria.
- the method may also include receiving query stroke data as input data to be processed and for which a synthesized stroke is to be generated, as in 120 .
- This input data may include at least x, y trajectory information for the query stroke (i.e. data representing the path that the query stroke takes in the x and y dimensions).
- this query stroke data may be lower dimension data than the stroke data that is provided for reference strokes in the library/database.
- the query stroke data may be obtained using less advanced tools than those used to provide the reference stroke data (e.g., less advanced tablets, styluses, or cursor control devices that are not able to provide as much information as the input devices used to provide the reference stroke data).
- the query stroke data may be input using advanced input devices, but by less skilled artists than those who have provided the reference stroke data.
- 6DOF data provided for a query stroke may not contain the amount, type, and/or quality of information that may be needed to create a high quality output stroke for display.
- the method may include processing the query stroke data to create a sequence of synthesized poses that follows the trajectory of the query stroke dependent on reference stroke data found in the database (e.g., dependent on various feature vectors of the stroke data for one or more reference strokes), as in 130 .
- this synthesis may be performed by a stroke synthesis engine executing on a computer system, and that synthesis engine may implement any or all of the techniques described herein for capturing reference stroke information, receiving a query stroke, constructing feature vectors, finding nearest neighbor sets, selecting the best neighbors from those nearest neighbor sets (e.g., using dynamic programming), and/or smoothing the resulting sequence of poses to create a synthesized output stroke.
- the method may then include outputting data representing a resulting synthesized stroke (i.e. an output stroke that includes the sequence of synthesized poses), as in 140 .
- the data may be displayed to a user (e.g., a user who entered the input query stroke or selected the input query stroke for display) as an output stroke (either individually or as part of a larger image) and/or may be stored (either individually or as part of a larger image) for later use or display, in different embodiments.
- the pose of a real paintbrush during a stroke has 6DOF: x and y position (in the plane of the canvas), z position (the height of the brush above the canvas), tilt-x and tilt-y (the orientation of the brush off vertical), and twist (about the brush axis).
- Various combinations of tablet devices and/or stylus devices may be used to provide this same information, in some embodiments.
- an advanced tablet used with an advanced stylus may provide input data in six dimensions (i.e. 6DOF data) to represent a paintbrush stroke.
- they may provide data representing the position of the tip of the stylus on the tablet in the x and y dimensions, the pressure on the stylus, the tilt of the stylus in the x dimension (tilt-x), the tilt of the stylus in the y dimension (tilt-y), and the twist of the stylus.
- the full pose of the tablet stylus may be mapped to the full pose of a virtual brush when creating strokes via a brush simulation engine.
- the output mark may respond to changes in any aspect of the pose. Therefore, the best results may be achieved when all these 6DOF of data are available.
- FIG. 2A illustrates an isometric view of the use of a tablet device 200 , according to one embodiment.
- the tablet device includes a display portion 210 .
- FIG. 2A illustrates the use of a 6DOF stylus ( 220 ) to enter data by drawing various strokes using tablet 200 .
- stylus 220 can provide data corresponding to x and y coordinates, pressure (shown as z), tilt (in two dimensions) and twist (A), as strokes are drawn in the display portion 210 of tablet 200 .
- FIG. 2B illustrates a top view of the use of tablet device 200 , according to one embodiment.
- tablet 200 includes a display portion 210 and data can be input using a 6DOF stylus 220 .
- the two tilt dimensions tilt-x and tilt-y
- the method may include constructing feature vectors for corresponding reference samples and query samples. These feature vectors, which are described in more detail below, may sometimes be referred to herein merely as “features”. For a given query sample (e.g., a given point within a query stroke), the method may include finding the nearest neighbor features in a library of reference strokes (e.g., feature vectors with similar values), as in 310 , and repeating this determination for each of the other query samples of the query stroke (shown as the feedback from 320 to 310 ).
- a library of reference strokes e.g., feature vectors with similar values
- the method may include choosing the best neighbor feature for each of the query samples, as in 330 , and assigning the pose data (e.g., values for any DOFs other than the x, y position values, including pressure, tilt, and/or twist components) of the best neighbor feature to the corresponding query sample.
- the pose data e.g., values for any DOFs other than the x, y position values, including pressure, tilt, and/or twist components
- a dynamic programming technique may be applied to sets of nearest neighbor features to determine the best neighbor for each of the query samples.
- the method may include smoothing between the poses of the sequence of best neighbor features (which may generate modified pose data), as in 340 , and writing out the resulting synthesized stroke.
- the synthesized stroke may follow the trajectory of the query stroke, and may include the pose data of the best neighbor features and/or any modified pose data generated by smoothing between the poses, as in 350 .
- the data captured for an individual sample in a library stroke or a query stroke represents an instantaneous pose.
- the synthesis operations described herein may be applied to small patches of stroke data, rather then on the data captured for these instantaneous samples.
- the system may construct higher dimensional feature vectors based on sequences of stroke sample data that encapsulate the stroke data (e.g., the stroke trajectory) over a short window of time.
- this approach may reduce aliasing effects by combining multiple nearby samples into each feature tap, and may provide importance weighting by increasing the length of the taps (that is, increasing c) for the taps applied to samples that are farther from the target sample s i .
- FIGS. 4A-4B illustrate the use of captured and estimated stroke data to construct feature vectors for a sample of a query stroke, according to one embodiment. More specifically, FIG. 4A illustrates the use of captured (and, in some cases, estimated) stroke data to construct a feature vector at a sample point of a query stroke, and FIG. 4B illustrates a collection of triangle filters that may be used in constructing a feature vector at a sample point of a query stroke, according to one embodiment.
- a feature vector is being constructed for a query stroke sample taken at time 400 .
- 2DOF stroke data which is shown in FIG.
- query stroke data 410 which includes an x component and a y component of the query stroke
- query stroke data may be captured for an entire stroke before any feature vector generation is performed, while in other embodiments, feature vectors may be generated for individual samples and/or portions of a query stroke before the stroke data for the entire stroke is available.
- the captured data may represent the trajectory (or path) of the query stroke in the x, y plane.
- a feature vector for a query sample may be constructed based on the captured and/or estimated stroke data for multiple overlapping sample windows 440 on either side of the query sample currently point being processed.
- FIG. 4A illustrates an example in which five sample windows, labeled as sample windows 440 a - 440 e , are used in constructing a feature vector for the query stroke sample at time 400 .
- the stroke data values (e.g., 2DOF or 6DOF data values) captured and/or estimated in different samples over the width of each sample window may be averaged to determine the contribution from that sample window to the feature vector for the current query sample.
- the x values for each of the sample points in sample window 440 a may be averaged to contribute one of the values in the feature vector for the current query sample point.
- the y values for each of the sample points in sample window 440 a may be averaged to contribute another one of the values in the feature vector for the current query sample point.
- the averages of the x values and the averages of the y values in each of the other sample windows may each contribute another value to the feature vector for the current query sample.
- the feature vector includes the ten values corresponding to the x and y value averages in each of the five sample windows.
- different numbers of sample windows may be processed to determine a feature vector for a given query sample.
- data in more dimensions e.g., up to 6DOF
- the feature vector for the query sample may include a corresponding number of additional component values.
- the sample window that includes the query sample currently being processed may be the smallest sample window, and the stroke data for each individual sample in that window may have a large influence over the stroke data values included in the resulting feature vector.
- the sample windows that are farther from the current query sample in time may be wider than the sample window that include the query sample currently being processed.
- the stroke data for each of the individual samples in these windows may have less influence over the stroke data values included in the resulting feature vector than the stroke data generated from the smallest/nearest sample window.
- different weighting may be applied to the sample data obtained from different ones of the sample windows 440 , in some embodiments.
- the sample data obtained from different ones of the sample windows 440 may be filtered using different and/or more complex filters.
- the sample data obtained from different ones of the sample windows 440 may be filtered using different ones of a collection of triangle filters 450 a - 450 e , as illustrated in FIG. 4B .
- the sample data obtained from sample window 440 a may be filtered using triangle filter 450 a
- the sample data obtained from sample window 440 b may be filtered using triangle filter 450 b , and so on.
- pose data e.g., stroke data for four additional degrees of freedom: twist, tilt-y, tilt-x, and pressure
- pose data may be estimated for previously processed query samples taken prior to the query sample currently being processed, and may be estimated for the current query sample, once processing is complete. This is illustrated in FIG. 4 as estimated stroke data 420 . Embodiments in which such estimated stroke data is employed are described in more detail below.
- each feature vector (which corresponds to a respective reference stroke sample point) may include up to 30 component values, i.e. one component value for each of the six degrees of freedom for which data is available (x, y, pressure, tilt-x, tilt-y, and twist) from each of the five sample windows.
- a variety of mechanisms may be employed in a nearest neighbor searching operation.
- various search algorithms may be applied in high dimensional space with an L2 norm (i.e. a vector norm) to find the reference library feature vectors (which may also be referred to as library features) that most closely resemble the feature vectors constructed for the samples of a query stroke.
- the set of library features may be entered into a data structure to facilitate a nearest neighbor search, and each query feature may be used to find the k nearest neighbors in the data structure.
- k may be a pre-determined value representing the number of nearest neighbors to be identified, and may be a default value or may be configurable by the user.
- the result of the nearest neighbor search may be a corresponding set of k neighbors, n 0 i , . . . , n k ⁇ 1 i .
- the method may include obtaining trajectory data (e.g., raw values of x and y components of the stroke data) and any available pose data (e.g., raw values for tilt, twist, or pressure) for sample points of the query stroke in overlapping sample windows on either side of a given sample point.
- trajectory data e.g., raw values of x and y components of the stroke data
- pose data e.g., raw values for tilt, twist, or pressure
- the method may include obtaining trajectory and pose data captured at and/or associated with the given sample point, trajectory and pose data captured at and/or associated with a pre-determined number of previous sample points (in a pre-determined number of sample windows), and trajectory and pose data captured at and/or associated with a pre-determined number of subsequent sample points (in a pre-determined number of sample windows), in some embodiments.
- the sample windows may be of different widths and the influence of the data obtained in different ones of the sample windows may be weighted differently (e.g., according to the proximity of the sample window to the given sample point).
- the method may include applying a respective triangle filter to the raw data for each component of the stroke data from each of the sample windows, and adding the results to the feature vector for the given sample point, as in 520 .
- the raw data obtained from a collection of sample points may be included in the feature vector, or the sample data may be averaged, pre-processed using other types of filters, or aggregated using other methods to generate component values for the feature vector.
- the resulting feature vector may include an x component value and a y component value from each of the sample windows, and may also include pose component values from each of the sample windows (if any pose data is provided for the input query stroke).
- a query stroke may be input using a tablet device and/or stylus that provides data representing x and y components of the query stroke (i.e. trajectory data) and pressure (which may be mapped to a z component of the query stroke), but does not provide data representing tilt or twist components.
- data representing x and y components of the query stroke i.e. trajectory data
- pressure which may be mapped to a z component of the query stroke
- the method may in some embodiments include obtaining estimated pose data, if any is available, for previously processed sample points, as in 530 , and adding this information to the feature vector.
- pose data e.g., data values representing pressure, tilt and/or twist
- the feature vector may thus include a component value from each of the sample windows for the trajectory data (the x and y components), and for any of the pose components for which query stroke data (whether actual or estimated as described above) is available.
- the method may include comparing the resulting feature vector to feature vectors for sample points of reference strokes, as in 540 .
- only the trajectory data of the feature vector for the query stroke sample point e.g., the ten x and y component values described in previous examples
- the corresponding data of the reference strokes e.g., ten x and y component values for each of the sample points in the reference strokes
- this pose data may not be used to identify the nearest neighbors of the query stroke sample point (e.g., if the user wishes the style of the synthesized stroke to be different from that of the input query stroke).
- any or all available pose data for the query stroke sample point may be compared with the corresponding data of the reference strokes (in addition to the trajectory data) to identify the nearest neighbors of the query stroke sample point.
- the method may include identifying the set of feature vectors for sample points of one or more reference strokes that are closest to the resulting feature vector, and adding those feature vectors to a nearest neighbor set for the given sample point.
- a pre-determined number of nearest neighbors may be identified for each of the sample points.
- this pre-determined number may be a fixed (e.g., a default) number that is applicable in all cases, or may be configurable (e.g., by a user or application) for a given stroke synthesis exercise or drawing session, for the user, for the application, or for synthesizing strokes using a particular stroke synthesis style.
- the operations illustrated in FIG. 5 may be applied to multiple sample points of the query stroke in order to construct a feature vector for each of those sample points and to determine a set of nearest neighbor features for each of those sample points. In some embodiments, these operations may be applied to query sample points as the stroke data necessary to construct the feature vector for each sample point (including, e.g., stroke data from one or more “future” sample points) becomes available. In other embodiments, the operations illustrated in FIG. 5 may not be performed for any samples of the query stroke until all of the stroke data for the query stroke is available. Note also that one or both of the operations illustrated as 510 and 520 may be applied to stroke data for samples of various reference strokes in order to generate feature vectors for those reference strokes.
- FIG. 6 illustrates the association of a set of nearest neighbor feature vectors from a library of reference strokes with each sample in a query stroke, according to one embodiment.
- each of the seven samples of a query stroke 600 e.g., query sample points q 0 -q 6 , shown as 620 a - 620 g
- a respective nearest neighbor set 610 shown as 610 a - 610 g
- a single (best) neighbor may be selected for each of the samples.
- simply selecting the closest neighbor e.g., the neighbor feature that most closely matches the query feature
- the selected poses for neighboring query samples may have very different poses, and may create discontinuities in the output stroke. Therefore, the system may in some embodiments be configured to find an optimal path through the set of neighbors by selecting the best neighbor at each sample while considering the choices made at the samples before and after the current sample.
- the best neighbor selection may be treated as a graph problem in which each of the neighbors in the nearest neighbors sets is a node in the graph and the edges are the paths between each pair of neighbors in adjacent nearest neighbor sets.
- the determination of the best neighbors may include finding the lowest cost path through the graph, i.e. find the sequence of nodes in the graph (one per column) such that the sum of the edge weights for the sequence of nodes has the lowest value (as compared to all other sequences of nodes in other paths across the graph).
- the optimal selection of best neighbor features may be determined by making a graph among the neighbors and finding the minimum cost path through the graph using dynamic programming.
- edges are made to other neighbors n 0 i ⁇ 1 , . . . ,n k ⁇ 1 i ⁇ 1 and n 0 i+1 , . . . ,n k ⁇ 1 i+1 .
- the edge weights may be set to encourage the selection of neighbor vectors that correspond to continuous pieces of library strokes, rather than random samples. In some embodiments, edge weights may be set to encourage the selection of feature vectors corresponding to the beginning and ending samples of library strokes for the starting and ending samples of query strokes.
- the method may include constructing a graph of nearest neighbor sets for at least a portion of the query stroke samples, as in 700 .
- the method may include constructing a graph of the nearest neighbor sets for all of the sample points of a query stroke, and processing the data in that graph to determine the best neighbor for each of the query stroke samples.
- different graphs may be constructed for each of multiple processing windows that include only the data needed for determining the best neighbor for one or a subset of the query stroke samples.
- the processing may take place in a moving processing window in real time, or in a moving processing window that lags real time slightly (e.g., in order to include query stroke data for one or more “future” sample points when determining the best neighbor for a given query stroke sample point).
- the method may include, for each neighbor in each neighbor set, defining an edge to each of the neighbors in the adjacent neighbor sets (e.g., on either side of the neighbor set for a given query stroke sample point).
- edge weights may be assigned that may encourage the selection of neighbors from the beginning and end of reference strokes for the starting and ending samples of the query stroke, respectively. This is illustrated as 720 in FIG. 7 .
- the method may include assigning edge weights that encourage the selection of adjacent neighbors from the same reference stroke as best neighbors for adjacent points in the query stroke, as in 730 . For example, if adjacent features of a single library curve (i.e.
- edges between these features may be assigned a lower cost (or weighting) than edges between features from different library curves.
- edge weighting instead of (or in addition to) a weighting that favors adjacent features from the same library curve and/or a weighting that favors features from beginning and ending reference samples for beginning and ending query stroke samples, in different embodiments.
- the method may include applying dynamic programming techniques to the graph to determine the lowest cost path across the graph, as in 740 .
- the neighbor features on the lowest cost path may then be identified as the best neighbors for the corresponding query samples, as in 750 .
- the pose data for the identified best neighbors may be assigned to the corresponding query stroke sample points in the output stroke, in some embodiments.
- FIGS. 8A-8C illustrate the use of one such graph (shown as graph 800 ) in determining a lowest cost collection of nearest neighbors for the samples in query stroke 600 , according to one embodiment.
- the best neighbors determined from among those in the nearest neighbor sets include feature vectors corresponding to collections of adjacent samples from two reference curves (illustrated in FIG. 8C ).
- each query stroke sample of query stroke 600 in FIG. 8A is associated with a set of five nearest neighbors (i.e.
- each neighbor in a nearest neighbor set may have one of five predecessors and one of five successors, and these predecessors and successors define the edges for a graph problem.
- FIG. 8B illustrates that nearest neighbor n 2 2 (which is one of the nearest neighbors of query sample q 2 , shown at 620 c in FIG. 8A ) may be preceded by one of five best neighbor candidates 810 in the preceding column and may be succeeded by one of five best neighbor candidates 810 in the succeeding column.
- various dynamic programming techniques and various weighting heuristics may be applied to graph 800 and its data to determine the lowest cost path 820 through the graph.
- nearest neighbors n 1 0 (corresponding to query sample g 0 , shown as 620 a in FIG. 8A ), n 1 1 (corresponding to query sample q 1 , shown as 620 b in FIG. 8A ), n 1 2 (corresponding to query sample q 2 , shown as 620 c in FIG. 8A ), and n 1 3 (corresponding to query sample q 3 , shown as 620 d in FIG. 8A ) all represent feature vectors of consecutive samples 831 - 834 in a portion of a reference curve A (shown as 830 in FIG. 8C ).
- nearest neighbors n 1 3 (corresponding to query sample q 3 , shown as 620 d in FIG.
- n 4 4 (corresponding to query sample q 4 , shown as 620 e in FIG. 8A ), n 4 5 (corresponding to query sample q 5 , shown as 620 f in FIG. 8A ), and n 4 6 (corresponding to query sample q 6 , shown as 620 g in FIG. 8A ) all represent feature vectors of consecutive samples 841 - 844 in a portion of a reference curve B (shown as 840 in FIG. 8C ).
- a weighting may be applied to the data in graph 800 that favors the selection of feature vectors corresponding to consecutive samples of the same reference curve.
- these two sequences of feature vectors from reference library curves A and B may have been selected for inclusion in lowest cost path 820 due, in part, to such a weighting.
- the determination of a lowest cost path through a nearest neighbor graph (such as graph 800 illustrated in FIG. 8B ) may be dependent on other rules or heuristics that are applied to the weighting of reference curve samples.
- multiple portions of the same reference stroke may be matched to respective portions of a query stroke.
- reference curve A and reference curve B of FIG. 8C may be different portions of the same reference curve.
- portions of reference strokes that are matched to respective portions of a query stroke may come from different reference strokes in the same reference stroke library or in different reference stroke libraries.
- the high fidelity (e.g., 6DOF) stroke data for each of the reference curve samples whose feature vectors have been selected for inclusion in the lowest cost path (e.g., 820 ) may be included in a synthesized output stroke corresponding to query stroke 600 .
- This output stroke may follow the original x, y trajectory of query stroke 600 , but may include the pose information from these reference curve samples (whether or not the input query stroke data included any pose data).
- discontinuities may be apparent in the sequence of poses selected from best neighbor feature vectors, such as when only small pieces of reference strokes are able to be mapped to portions of the query stroke, with abrupt shifts at the transition points.
- cross-fading of the pose data e.g., using averaging or another blending operation may be performed over small windows between continuations of the reference strokes that contain the best neighbor samples on either side of the transition point.
- FIG. 9 illustrates smoothing between the poses of best neighbor features found in two different reference curves, according to one embodiment.
- four samples shown as 831 - 834 ) from reference curve A (shown as 830 ) and four samples (shown as 841 - 844 ) from reference curve B (shown as 840 ) were determined to be the best matches for different portions of query stroke 600 .
- the thick, gray, vertical lines between 620 c and 833 , and between 620 e and 842 indicate the correspondence between query stroke samples q 2 and q 4 (respectively) and the reference curve samples determined to have the best neighbor feature vectors for those query stroke samples.
- a simple binary assignment of the pose data from these two curves to the output stroke may involve assigning the pose data from curve A to the first half of the output stroke (e.g., for sample points 831 to 834 ) and the pose data from curve B to the second half of the output stroke (e.g., for sample points 842 to 844 ).
- this may result in a discontinuity (shown as discontinuity point 900 , corresponding to query sample q 3 ) between the poses represented by the feature vectors for these two sets of reference samples (i.e. the sets of reference samples from library curves A and B), in this example.
- the stroke synthesis engine may be configured to smooth out the discontinuity by extending the influence of each of the matched curve sections beyond the matched portion (e.g., by averaging the pose data of the two library curves to generate the pose data in the area surrounding the discontinuity.
- the pose data values e.g., pressure, tilt, and/or twist data values
- the pose data values may be averaged to determine the pose data for output stroke 950 .
- an image editing application may include a stroke synthesis engine and one or more libraries of reference strokes.
- reference stroke data created by different artists and/or created in different styles e.g., calligraphy, pointillist, impressionist, etc.
- libraries e.g., separated by artist and/or style
- an image editing application may include a stroke synthesis engine.
- a user may be able to pass lower dimension input data (e.g., 2DOF or 3DOF data for an input stroke that is provided by any of various cursor control devices) through the stroke synthesis engine, and the stroke synthesis engine may synthesize pose data to generate higher dimension and/or higher quality strokes.
- lower dimension input data e.g., 2DOF or 3DOF data for an input stroke that is provided by any of various cursor control devices
- the stroke synthesis engine may synthesize pose data to generate higher dimension and/or higher quality strokes.
- the user may opt to pass those input strokes though the synthesis engine to create strokes styled in the manner of various reference strokes.
- passing input strokes through the synthesis engine may not change the trajectory of an input stroke (e.g., the synthesis engine may not be able to compensate for poor handwriting) but may be used to stylize the stroke so that it appears as if it included additional or different pose data (e.g., the synthesis engine may shade portions of the stroke to match a selected style).
- the method may include a user entering a query stroke (e.g., through a user interface of a stroke synthesis engine or other module of a computer system) for which the user desires to generate a higher fidelity, higher quality, and/or differently styled output stroke.
- the input data for the query stroke may include at least trajectory data (e.g., data representing x and y components at various points in the query stroke).
- the input data for the query stroke may also include pose data (e.g., data representing some or all of pressure, tilt and/or twist components at various points in the query stroke).
- the user may be able to select a library of reference strokes to be used in synthesizing pose data for the query stroke from among multiple reference stroke libraries supported or provided by the system, as in 1020 .
- one or more reference libraries may be used in synthesizing pose data for the query stroke by default or may be selected for use based on other parameters of the stroke synthesis exercise (e.g., based on a selected style, a selected artist, a selected genre, the application through which the synthesis engine is invoke, an identifier of the user, or other criteria).
- the method may include the user invoking the synthesis engine to process the query stroke, and specifying one or more parameter value(s) for the stroke synthesis operation.
- the user may be able to specify input parameter values that enable the use of one or more filters on the raw data provided by the query stroke input, select the filter(s) to be used in constructing a feature vector for the query stroke, configure the number and/or widths of the sample windows to be used in constructing a feature vector for the query stroke, configure the number of neighbors to be included in the neighbor sets for each query stroke sample point, select one or more edge weighting algorithms to be used in nearest neighbor and/or best neighbor identification and selection, or otherwise configure the synthesis engine to perform a desired stroke synthesis operation.
- the values of any or all of such parameters may be selected for use in synthesizing pose data for the query stroke by default or may be selected for use based on other parameters of the stroke synthesis exercise (e.g., based on a selected style, a selected artist, a selected genre, the application through which the synthesis engine is invoke, an identifier of the user, or other criteria).
- the method may include the synthesis engine synthesizing pose data for the query stroke using any or all of the methods described herein and dependent on the specified (or system-selected) reference library (or libraries) and the specified (or system-selected) parameter value(s), as in 1040 .
- the synthesis engine may then output data representing the resulting synthesized stroke for display and/or may add the resulting synthesized stroke to one or more of the reference libraries, as in 1050 .
- the synthesis engine may be configured to process a query stroke only after the entire stroke (e.g., an input representing a paintbrush stroke) has been entered.
- the synthesis engine may be configured to process portions of a query stroke at different times while the stroke is being entered (e.g., before the trajectory and/or pose data for the entire stroke is available) using default references libraries and input parameters or pre-selected reference libraries and input parameters.
- FIGS. 11A-11E illustrate various reference strokes, query strokes, and output strokes, according to one embodiment.
- FIG. 11A illustrates various reference strokes representing English script for which 6DOF stroke data has been captured
- FIG. 11B illustrates query strokes (again representing English script) for which only 2DOF information has been captured.
- FIG. 11C illustrates synthesized output strokes corresponding to the query strokes of FIG. 11B .
- the pose data for the output strokes illustrated in FIG. 11C has been synthesized dependent on the reference stroke data shown in FIG. 11A .
- the output strokes illustrated in FIG. 11C follow the trajectories of the query strokes of FIG.
- FIG. 11B but include pose data that is synthesized from the pose data of one or more samples of the reference strokes of FIG. 11A (e.g., based on their selection as best neighbors according to the trajectory data and/or pose data of the feature vectors for the reference strokes shown in FIG. 11A ).
- FIG. 11D illustrates query strokes for which only 2DOF information was captured
- FIG. 11E illustrates synthesized output strokes corresponding to the query strokes of FIG. 11D .
- the output strokes illustrated in FIG. 11E have been synthesized dependent on the reference stroke data shown in FIG. 11A (e.g., dependent on the trajectory data and/or pose data of the feature vectors for the reference strokes shown in FIG.
- the best neighbor selection process described above may in some embodiments rely on having the entire set of neighbor features for a complete query stroke available before an output stoke can be generated. In such embodiments, this process may not be appropriate for online (e.g., real-time) synthesis of output strokes from query strokes.
- several options for amending the process described above may support online stroke synthesis, and may facilitate the receipt and use of feedback during a stroke (i.e. before all of the query stroke data is available). Two such alternatives are described below.
- a stroke synthesis engine may be configured to apply the dynamic programming approach described herein to a local sub-graph of nearest neighbor features, rather than to a full graph representing nearest neighbors for all of the samples in the entire query stroke.
- some best neighbor features may be selected before the entire stroke is available.
- this approach may result in sub-optimal output, but may still be desirable due to its faster performance.
- dynamic programming may be applied to create poses for each query sample using a moving window (e.g., by lagging the query stroke by the width of a few samples in order to apply stroke data on either side of each query sample before stroke data for the entire stroke is available).
- dynamic programming may be applied to a sub-graph of neighbor features that includes the neighbor feature sets for the previous a samples and the next b samples, but does not necessarily include the full graph.
- a stroke synthesis engine may be configured to build only a portion of graph 800 at a time for query stroke 600 , e.g., a portion of graph that includes only five columns at a time (corresponding to five nearest neighbor sets).
- a sub-graph that includes the data from nearest neighbor sets 610 a - 610 e may be constructed when calculating the best neighbor feature for query sample q 2 (taking into account two nearest neighbor sets on each side of the nearest neighbor set for query sample q 2 ). After solving that portion of the graph (i.e. after determining the best neighbor feature for query sample q 2 ), another sub-graph may be constructed that includes the data from nearest neighbor sets 610 b - 610 f .
- This second sub-graph may be used when calculating the best neighbor feature for query sample q 3 .
- This approach may be applied repeatedly as a five-neighbor-set-wide window is moved one neighbor set to the right as each new query sample point becomes available and a corresponding feature vector and nearest neighbor feature set are determined for the new query sample point.
- a stroke synthesis engine may be configured to apply a greedy best neighbor selection process instead.
- a best neighbor sample may be selected as each new query sample becomes available, by simply choosing the minimum distance neighbor.
- the continuity of the neighbor samples may be improved by amending the feature vector used to determine the minimum distance neighbor to include the full, synthesized 6DOF data for each of the previous samples.
- the pose data assigned to each of the previous query stroke samples e.g., pressure, tilt and/or twist data such as that shown as estimated stroke data 420 in FIG. 4
- the pose data assigned to each of the previous query stroke samples e.g., pressure, tilt and/or twist data such as that shown as estimated stroke data 420 in FIG. 4
- f i s i ⁇ a x ,s i ⁇ a y ,s i ⁇ a p ,s i ⁇ a tx ,s i ⁇ a ty s i ⁇ a ⁇ , . . . ,s i x ,s i y , . . . ,s i+b x ,s i+y b
- the feature vector may include 18 component values: x and y components for each of the five sample points/windows (for a total of 10 values), plus 4 pose data components for each of the two sample points/windows prior to the current sample point/window (for a total of eight additional values).
- the estimated pose data that is included in the feature vector may be raw pose data, or may be pose data that has been averaged, filtered, or otherwise aggregated (similar to manner in which trajectory data may be aggregated).
- the additional pose component values may be included in the distance function to determine the nearest and best neighbors (which may then be dependent on having the most similar trajectory data and the most similar pose histories).
- a feature vector would be constructed for q 3 that includes trajectory data (based on past, present, and future x and y component values) and also pose data (based on the pose component data for one or more of the previous samples, q 0 , q 1 , and q 2 ).
- the nearest neighbors for query sample q 3 may then be selected based on having both similar trajectory data and similar pose histories.
- its pose data may be assumed to be similar to the pose data for any subsequent query sample points, and so on.
- the stroke synthesis techniques described herein may in various embodiments be used to create high fidelity expressive brush strokes using a physical brush engine that supports these techniques, even on devices where full 6DOF input is not available, and/or in cases in which the pose data provided for an input stroke is not desirable to the user due to lack of skill or style considerations.
- a feature vector construction module may receive reference stroke data or query stroke data, and may construct a series of feature vectors for samples of those strokes, as described herein.
- the feature vector construction module may in some embodiments be implemented by a non-transitory, computer-readable storage medium and one or more processors (e.g., CPUs and/or GPUs) of a computing apparatus.
- the computer-readable storage medium may store program instructions executable by the one or more processors to cause the computing apparatus to perform receiving reference stroke data or query stroke data, and constructing a series of feature vectors for samples of those strokes, as described herein.
- Other embodiments of the feature vector construction module may be at least partially implemented by hardware circuitry and/or firmware stored, for example, in a non-volatile memory.
- Some embodiments may include a means for finding the nearest neighbors of a query stroke feature.
- a feature-matching module may compare feature vectors of query stroke samples to those of reference stroke samples to determine the nearest neighbors, as described herein.
- the feature-matching module may in some embodiments be implemented by a non-transitory, computer-readable storage medium and one or more processors (e.g., CPUs and/or GPUs) of a computing apparatus.
- the computer-readable storage medium may store program instructions executable by the one or more processors to cause the computing apparatus to perform comparing feature vectors of query stroke samples to those of reference stroke samples to determine the nearest neighbors, as described herein.
- Other embodiments of the feature-matching module may be at least partially implemented by hardware circuitry and/or firmware stored, for example, in a non-volatile memory.
- Some embodiments may include a means for selecting the best neighbor features for a query sample from among a collection of nearest neighbor features.
- a dynamic programming module may apply various weightings to the nearest neighbor features in a graph of nearest neighbor sets and may determine the lowest cost path across that graph, as described herein.
- the dynamic programming module may in some embodiments be implemented by a non-transitory, computer-readable storage medium and one or more processors (e.g., CPUs and/or GPUs) of a computing apparatus.
- the computer-readable storage medium may store program instructions executable by the one or more processors to cause the computing apparatus to perform applying various weightings to the nearest neighbor features in a graph of nearest neighbor sets and determining the lowest cost path across that graph, as described herein.
- Other embodiments of the dynamic programming module may be at least partially implemented by hardware circuitry and/or firmware stored, for example, in a non-volatile memory.
- a pose-smoothing module may blend feature vector data (e.g., pose data) for selected best neighbors to produce a smooth output stroke, as described herein.
- the pose-smoothing module may in some embodiments be implemented by a non-transitory, computer-readable storage medium and one or more processors (e.g., CPUs and/or GPUs) of a computing apparatus.
- the computer-readable storage medium may store program instructions executable by the one or more processors to cause the computing apparatus to perform blending feature vector data (e.g., pose data) for selected best neighbors to produce a smooth output stroke, as described herein.
- Other embodiments of the pose-smoothing module may be at least partially implemented by hardware circuitry and/or firmware stored, for example, in a non-volatile memory.
- FIG. 12 illustrates a stroke synthesis engine 1220 that may implement synthesizing high fidelity stroke data for lower dimension input strokes, according to various embodiments.
- stroke synthesis engine 1220 may include a feature vector construction module, a feature-matching module, a dynamic programming module, and/or a pose-smoothing module, as described above.
- FIG. 13 illustrates an example computer system on which embodiments of stroke synthesis engine 1220 may be implemented.
- stroke synthesis engine 1220 may receive user input 1210 (e.g., reference stroke data, query stroke data, and/or input parameters for various stroke synthesis operations) via one or more user interfaces 1215 (which may include various tablet devices, stylus devices, and/or other input/output devices suitable for entering such information).
- user input 1210 e.g., reference stroke data, query stroke data, and/or input parameters for various stroke synthesis operations
- user interfaces 1215 which may include various tablet devices, stylus devices, and/or other input/output devices suitable for entering such information.
- the user input may include various types and amounts of stroke information (e.g., trajectory data and/or pose data) in any of multiple dimensions, and may include other input data associated with reference strokes, query strokes, and/or stroke synthesis operations.
- Stroke synthesis engine 1220 may generate as output one or more output images 1240 , which may include one or more output strokes synthesized from query strokes dependent on various combinations of query stroke data and reference stroke data, using any or all of the methods described herein.
- Collected stroke data 1230 (e.g., reference stroke data) and/or output images 1240 may in various embodiments be stored in (or written to) a storage medium 1250 , such as system memory, a disk drive, DVD, CD, etc., and subsequently accessed by stroke synthesis engine 1220 for use in subsequent operations to synthesize high fidelity stroke data for other query strokes.
- output images 1240 may be displayed to users, e.g., via user interface 1215 (not shown).
- Embodiments of a stroke synthesis engine and/or of the various modules thereof, as described herein, may be executed on one or more computer systems, which may interact with various other devices.
- FIG. 13 illustrates an example computer system that is configured to implement synthesizing high fidelity stroke data for lower dimension input strokes, according to some embodiments.
- computer system 1300 may be any of various types of devices, including, but not limited to, a personal computer system, desktop computer, laptop, notebook, tablet device, slate device, netbook computer, mainframe computer system, handheld computer, workstation, network computer, camera, set top box, mobile device, consumer device, video game console, handheld video game device, application server, storage device, peripheral device (such as a switch, modem, or router), or in general any type of computing or electronic device.
- a personal computer system desktop computer, laptop, notebook, tablet device, slate device, netbook computer, mainframe computer system, handheld computer, workstation, network computer, camera, set top box, mobile device, consumer device, video game console, handheld video game device, application server, storage device, peripheral device (such as a switch, modem, or router), or in general any type of computing or electronic device.
- computer system 1300 includes one or more processors 1310 coupled to a system memory 1320 via an input/output (I/O) interface 1330 .
- Computer system 1300 further includes a network interface 1340 coupled to I/O interface 1330 , and one or more input/output devices 1350 , such as cursor control device 1360 , keyboard 1370 , and display(s) 1380 .
- cursor control device 1360 may include one or more simple or advanced stylus devices (e.g., those that provide 2DOF to 6DOF stroke data), mouse devices, touchpads, and/or any other suitable cursor control devices.
- display devices 1380 may include one or more simple or advanced tablet or slate devices (including, but not limited to, those described herein) and/or other types of devices that include a display capability and serve as input and/or output devices.
- embodiments may be implemented using a single instance of computer system 1300 , while in other embodiments multiple such systems, or multiple nodes making up computer system 1300 , may be configured to host different portions or instances of embodiments.
- some elements may be implemented via one or more nodes of computer system 1300 that are distinct from those nodes implementing other elements.
- computer system 1300 may be a uniprocessor system including one processor 1310 , or a multiprocessor system including several processors 1310 (e.g., two, four, eight, or another suitable number).
- Processors 1310 may be any suitable processor capable of executing instructions.
- processors 1310 may be general-purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA.
- ISAs instruction set architectures
- each of processors 1310 may commonly, but not necessarily, implement the same ISA.
- At least one processor 1310 may be a graphics processing unit.
- a graphics processing unit or GPU may be considered a dedicated graphics-rendering device for a personal computer, workstation, game console or other computing or electronic device.
- Modern GPUs may be very efficient at manipulating and displaying computer graphics, and their highly parallel structure may make them more effective than typical CPUs for a range of complex graphical algorithms.
- a graphics processor may implement a number of graphics primitive operations in a way that makes executing them much faster than drawing directly to the screen with a host central processing unit (CPU).
- the data processing methods disclosed herein may, at least in part, be implemented by program instructions configured for execution on one of, or parallel execution on two or more of, such GPUs.
- the GPU(s) may implement one or more application programmer interfaces (APIs) that permit programmers to invoke the functionality of the GPU(s). Suitable GPUs may be commercially available from vendors such as NVIDIA Corporation, ATI Technologies (AMD), and others.
- APIs application programmer interfaces
- System memory 1320 may be configured to store program instructions and/or data accessible by processor 1310 .
- system memory 1320 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or any other type of memory.
- SRAM static random access memory
- SDRAM synchronous dynamic RAM
- program instructions and data implementing desired functions, such as those described above for embodiments of a stroke synthesis engine (and/or modules thereof) are shown stored within system memory 1320 as program instructions 1325 and data storage 1335 , respectively.
- program instructions and/or data may be received, sent or stored upon different types of computer-accessible media or on similar media separate from system memory 1320 or computer system 1300 .
- a computer-accessible medium may include non-transitory storage media or memory media such as magnetic or optical media, e.g., disk or CD/DVD-ROM coupled to computer system 1300 via I/O interface 1330 .
- Program instructions and data stored via a computer-accessible medium may be transmitted by transmission media or signals such as electrical, electromagnetic, or digital signals, which may be conveyed via a communication medium such as a network and/or a wireless link, such as may be implemented via network interface 1340 .
- I/O interface 1330 may be configured to coordinate I/O traffic between processor 1310 , system memory 1320 , and any peripheral devices in the device, including network interface 1340 or other peripheral interfaces, such as input/output devices 1350 .
- I/O interface 1330 may perform any necessary protocol, timing or other data transformations to convert data signals from one component (e.g., system memory 1320 ) into a format suitable for use by another component (e.g., processor 1310 ).
- I/O interface 1330 may include support for devices attached through various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard, for example.
- PCI Peripheral Component Interconnect
- USB Universal Serial Bus
- I/O interface 1330 may be split into two or more separate components, such as a north bridge and a south bridge, for example.
- some or all of the functionality of I/O interface 1330 such as an interface to system memory 1320 , may be incorporated directly into processor 1310 .
- Network interface 1340 may be configured to allow data to be exchanged between computer system 1300 and other devices attached to a network, such as other computer systems, or between nodes of computer system 1300 .
- network interface 1340 may support communication via wired or wireless general data networks, such as any suitable type of Ethernet network, for example; via telecommunications/telephony networks such as analog voice networks or digital fiber communications networks; via storage area networks such as Fibre Channel SANs, or via any other suitable type of network and/or protocol.
- Input/output devices 1350 may, in some embodiments, include one or more display terminals, keyboards, keypads, touchpads, cursor control devices, scanning devices, voice or optical recognition devices, or any other devices suitable for entering or retrieving data by one or more computer system 1300 .
- Multiple input/output devices 1350 may be present in computer system 1300 or may be distributed on various nodes of computer system 1300 .
- similar input/output devices may be separate from computer system 1300 and may interact with one or more nodes of computer system 1300 through a wired or wireless connection, such as over network interface 1340 .
- memory 1320 may include program instructions 1325 , configured to implement embodiments of a stroke synthesis engine as described herein, and data storage 1335 , comprising various data accessible by program instructions 1325 .
- program instructions 1325 may include software elements of embodiments of a stroke synthesis engine as illustrated in the above figures.
- Data storage 1335 may include data that may be used by a stroke synthesis engine, in various embodiments. In other embodiments, other or different software elements and data may be included in memory 1320 .
- computer system 1300 is merely illustrative and is not intended to limit the scope of a stroke synthesis engine as described herein.
- the computer system and devices may include any combination of hardware or software that can perform the indicated functions, including a computer, personal computer system, desktop computer, laptop, notebook, or netbook computer, mainframe computer system, handheld computer, workstation, network computer, a camera, a set top box, a mobile device, network device, internet appliance, PDA, wireless phones, pagers, a consumer device, video game console, handheld video game device, application server, storage device, a peripheral device such as a switch, modem, router, or in general any type of computing or electronic device.
- computer system 1300 may be any of various types of devices, including, but not limited to, a personal computer system, desktop computer, laptop, notebook, tablet device, slate device, netbook computer, mainframe computer system, handheld computer, workstation, network computer, camera, set top box, mobile device, consumer device, video game console, handheld video game device, application server, storage device, peripheral device (such as a switch, modem, or router), or in general any type of computing or electronic device.
- Computer system 1300 may also be connected to other devices that are not illustrated, or instead may operate as a stand-alone system.
- the functionality provided by the illustrated components may in some embodiments be combined in fewer components or distributed in additional components. Similarly, in some embodiments, the functionality of some of the illustrated components may not be provided and/or other additional functionality may be available.
- instructions stored on a computer-accessible medium separate from computer system 1300 may be transmitted to computer system 1300 via transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as a network and/or a wireless link.
- Various embodiments may further include receiving, sending or storing instructions and/or data implemented in accordance with the foregoing description upon a computer-accessible medium. Accordingly, the techniques described herein may be practiced with other computer system configurations.
- a computer-accessible medium may include non-transitory storage media or memory media such as magnetic or optical media, e.g., disk or DVD/CD-ROM, volatile or non-volatile media such as RAM (e.g. SDRAM, DDR, RDRAM, SRAM, etc.), ROM, etc.
- a computer-accessible medium may include transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as network and/or a wireless link.
Abstract
Description
-
- trajectory (or trajectory data), a sequence of 2DOF x, y position values.
- pose (or pose data), 4DOF values that may be synthesized (pressure, tilt-x, tilt-y, and twist).
- library (or reference library), a collection of artist strokes that may have been collected offline.
- query (or query stroke), an input stroke to which the stroke synthesis techniques described herein may be applied.
- output (or output stroke), a high quality stroke that has the same trajectory as the query, but that includes poses created based on data in the library.
-
- 1. Construct feature vectors based on library samples and query samples.
- 2. For each query feature, find the nearest neighbor library features.
- 3. Given a set of nearest neighbor library features for a query feature, choose the best one.
- 4. Smooth the poses of the sequence of best neighbor features.
- 5. Write out a synthesized output stroke that has the same trajectory as the query and includes the selected and/or smoothed pose data.
F(S)={f i|0≦i<n}.
f i = s i−a x ,s i−a y , . . . ,s i+b x ,s i+b x .
F′(S)={f i′|0≦i<n}, where
f i ′= t i−a x ,t i−a y , . . . ,t i+b x ,t i+b y and
t i=Σj=−c c w j s i+j.
n 0 i−1 , . . . ,n k−1 i−1 and
n 0 i+1 , . . . ,n k−1 i+1.
f i = s i−a x ,s i−a y ,s i−a p ,s i−a tx ,s i−a ty s i−a θ , . . . ,s i x ,s i y , . . . ,s i+b x ,s i+y b
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/223,139 US9013485B2 (en) | 2011-06-30 | 2011-08-31 | Systems and methods for synthesizing high fidelity stroke data for lower dimension input strokes |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201161503519P | 2011-06-30 | 2011-06-30 | |
US13/223,139 US9013485B2 (en) | 2011-06-30 | 2011-08-31 | Systems and methods for synthesizing high fidelity stroke data for lower dimension input strokes |
Publications (2)
Publication Number | Publication Date |
---|---|
US20130120237A1 US20130120237A1 (en) | 2013-05-16 |
US9013485B2 true US9013485B2 (en) | 2015-04-21 |
Family
ID=48280086
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/223,139 Active 2034-02-19 US9013485B2 (en) | 2011-06-30 | 2011-08-31 | Systems and methods for synthesizing high fidelity stroke data for lower dimension input strokes |
Country Status (1)
Country | Link |
---|---|
US (1) | US9013485B2 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220122477A1 (en) * | 2020-10-20 | 2022-04-21 | Holistic Language Solutions LLC | Computerized method and apparatus for determining accuracy of written characters and stroke order and compliance with rules and providing visual and audio feedback |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150153898A1 (en) * | 2013-12-03 | 2015-06-04 | Elwha Llc | Latency compensation in a display of a portion of a hand-initiated movement |
WO2015141260A1 (en) * | 2014-03-17 | 2015-09-24 | 株式会社河合楽器製作所 | Handwritten music notation recognition device and program |
JP6624794B2 (en) * | 2015-03-11 | 2019-12-25 | キヤノン株式会社 | Image processing apparatus, image processing method, and program |
US10162435B2 (en) * | 2016-01-04 | 2018-12-25 | Asustek Computer Inc. | Input device and input method using the same |
CN113039511A (en) * | 2018-10-09 | 2021-06-25 | 谷歌有限责任公司 | Writing device with electromagnetic tracking |
US10928946B2 (en) * | 2019-02-15 | 2021-02-23 | Dell Products L.P. | Touchscreen stylus and display module interface |
US11164343B1 (en) * | 2020-10-10 | 2021-11-02 | Adobe Inc. | Synthesizing brush strokes based on user-defined strokes |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5730602A (en) * | 1995-04-28 | 1998-03-24 | Penmanship, Inc. | Computerized method and apparatus for teaching handwriting |
US20020130908A1 (en) | 2001-01-31 | 2002-09-19 | Wilensky Gregg D. | Vector-based painting |
US6801211B2 (en) | 2001-12-21 | 2004-10-05 | Ladd B. Forsline | Computer painting system with passive paint brush stylus |
US6870550B1 (en) | 1999-04-26 | 2005-03-22 | Adobe Systems Incorporated | Digital Painting |
US6919893B2 (en) | 2002-01-07 | 2005-07-19 | Sony Corporation | Image editing apparatus, image editing method, storage medium, and computer program |
US20070216684A1 (en) | 2006-03-17 | 2007-09-20 | Microsoft Corporation | Multiple brush components |
US20110181618A1 (en) | 2008-10-30 | 2011-07-28 | Diverdi Stephen J | Realistic Real-Time Simulation of Brush Behavior |
US20110206277A1 (en) | 2010-02-22 | 2011-08-25 | Casio Computer Co., Ltd. | Image processing apparatus and storage medium having stored therein an image processing apparatus program |
-
2011
- 2011-08-31 US US13/223,139 patent/US9013485B2/en active Active
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5730602A (en) * | 1995-04-28 | 1998-03-24 | Penmanship, Inc. | Computerized method and apparatus for teaching handwriting |
US6870550B1 (en) | 1999-04-26 | 2005-03-22 | Adobe Systems Incorporated | Digital Painting |
US7817159B2 (en) | 1999-04-26 | 2010-10-19 | Adobe Systems Incorporated | Digital painting |
US20020130908A1 (en) | 2001-01-31 | 2002-09-19 | Wilensky Gregg D. | Vector-based painting |
US6801211B2 (en) | 2001-12-21 | 2004-10-05 | Ladd B. Forsline | Computer painting system with passive paint brush stylus |
US6919893B2 (en) | 2002-01-07 | 2005-07-19 | Sony Corporation | Image editing apparatus, image editing method, storage medium, and computer program |
US20070216684A1 (en) | 2006-03-17 | 2007-09-20 | Microsoft Corporation | Multiple brush components |
US20110181618A1 (en) | 2008-10-30 | 2011-07-28 | Diverdi Stephen J | Realistic Real-Time Simulation of Brush Behavior |
US20110206277A1 (en) | 2010-02-22 | 2011-08-25 | Casio Computer Co., Ltd. | Image processing apparatus and storage medium having stored therein an image processing apparatus program |
Non-Patent Citations (4)
Title |
---|
Co-pending U.S. Appl. No. 12/832,960, filed Jul. 8, 2010, Harris et al. |
Marius Muja, et al., "Fast approximate nearest neighbors with automatic algorithm configuration," In International Conference on Computer Vision Theory and Applications, 2009, pp. 1-10. |
Shuto et al. "A Handwritten Character Training System with Haptization of Instructor's Brush-Strokes"; Department of Computer Science and Intelligent Systems, Oita University, Japan; IEEE, 2009, pp. 1030-1034. * |
Stephen DiVerdi, et al., "Industrial-strength painting with a virtual bristle brush," In Symposium on Virtual Reality Software and Technology, 2010, pp. 1-8. |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220122477A1 (en) * | 2020-10-20 | 2022-04-21 | Holistic Language Solutions LLC | Computerized method and apparatus for determining accuracy of written characters and stroke order and compliance with rules and providing visual and audio feedback |
Also Published As
Publication number | Publication date |
---|---|
US20130120237A1 (en) | 2013-05-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9013485B2 (en) | Systems and methods for synthesizing high fidelity stroke data for lower dimension input strokes | |
US8669995B2 (en) | Methods and apparatus for stroke grouping for high-level sketch editing | |
Xing et al. | Autocomplete hand-drawn animations | |
US11741644B2 (en) | Vector graphics based live sketching metods and systems | |
Todi et al. | Sketchplore: Sketch and explore with a layout optimiser | |
Kazi et al. | Vignette: interactive texture design and manipulation with freeform gestures for pen-and-ink illustration | |
US9207858B2 (en) | Method and apparatus for drawing and erasing calligraphic ink objects on a display surface | |
US7483553B2 (en) | Caricature exaggeration | |
Xing et al. | Autocomplete painting repetitions | |
Plimmer et al. | INTERACTING with sketched interface designs: an evaluation study | |
US11087503B2 (en) | Interactive color palette interface for digital painting | |
US20090278848A1 (en) | Drawing familiar graphs while system determines suitable form | |
US11288499B2 (en) | Interactive method for generating strokes with Chinese ink painting style and device thereof | |
Xie et al. | IR2s: interactive real photo to Sumi-e | |
US10970889B2 (en) | Stroke prediction for stylized drawings based on prior strokes and reference image | |
CN111127596B (en) | Incremental Voronoi sequence-based layered oil painting brush drawing method | |
US20210248799A1 (en) | Path-Constrained Drawing With Visual Properties Based On Drawing Tool | |
CN109816770B (en) | Oil painting stroke simulation using neural networks | |
Liu et al. | Name your style: An arbitrary artist-aware image style transfer | |
Gieseke et al. | A survey of control mechanisms for creative pattern generation | |
Baxter et al. | Latent doodle space | |
Yang et al. | Text effects transfer via distribution-aware texture synthesis | |
Eisenacher et al. | Example-based texture synthesis on disney's tangled | |
US10614268B1 (en) | Auto-complete design for content-creating applications | |
Huang et al. | Leveraging the crowd for creating wireframe-based exploration of mobile design pattern gallery |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: ADOBE SYSTEMS INCORPORATED, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:DIVERDI, STEPHEN J.;LU, JINGWAN;FINKELSTEIN, ADAM;SIGNING DATES FROM 20110828 TO 20110831;REEL/FRAME:026844/0796 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
AS | Assignment |
Owner name: ADOBE INC., CALIFORNIA Free format text: CHANGE OF NAME;ASSIGNOR:ADOBE SYSTEMS INCORPORATED;REEL/FRAME:048867/0882 Effective date: 20181008 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |