Variable rate video playback with synchronized audio   

Abstract: A system varies the apparent display rate of an audiovisual display by modifying an original set of audio data in accordance with a target display rate, then modifying a related original set of video data to conform to the modifications made to the audio data set, such that the modified audio and video data sets are synchronized. The apparent display rate can be faster or slower than a normal display rate for the original sets of audio and video data. Preferably, the method for modifying the original audio data set is one that produces a modified audio data set that can be used to generate an audio display having little or no distortion.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No. 12/264,414, entitled “Variable Rate Video Playback With Synchronized Audio,” filed Nov. 4, 2008, now U.S. Pat. No. 8,238,722, which is a continuation of U.S. patent application Ser. No. 10/783,627, entitled “Variable Rate Video Playback With Synchronized Audio,” filed Feb. 20, 2004 (now U.S. Pat. No. 7,480,446), which is a continuation of U.S. patent application Ser. No. 10/041,867 filed Jan. 7, 2002 (now U.S. Pat. No. 6,728,678), which is a continuation of U.S. patent application Ser. No. 09/239,455 filed Jan. 28, 1999 (now U.S. Pat. No. 6,360,202), which is a continuation of U.S. patent application Ser. No. 08/760,769 filed Dec. 5, 1996 (now U.S. Pat. No. 5,893,062), all of which are incorporated by reference herein in their entireties.

TECHNICAL FIELD

This invention relates to the display of audio and video data and, in particular, to variation of the apparent display rate at which the audio and video data is displayed.

It is desirable to be able to vary the apparent display rate (i.e., the rate of change of the display as perceived by an observer, as opposed to the rate at which data is processed to generate the display) of a display generated from audio, video, or related audio and video data. For example, it may be desirable to increase the apparent display rate so that a quick overview of the content of the data can be obtained, or because it is desired to listen to or view the display at a faster than normal rate at which the content of the display can still be adequately digested. Alternatively, it may be desirable to slow the apparent display rate so that the display can be more carefully scrutinized, or because the content of the display can be better digested at a slower rate.

Both audio and video data can be represented in either analog or digital form. The method used to manipulate audio and/or video data to accomplish variation in the apparent display rate of a display generated from that data depends upon the form in which the data is represented. However, conventional devices enable data in one form to be easily converted to the other form (i.e., analog to digital or digital to analog), thus affording wide latitude in the use of methods to accomplish display rate variation, regardless of the form in which the data originally exists.

The apparent display rate of an audio display or a video display can be increased or decreased by deleting specified data from, or adding specified data to (e.g., repeating certain data), respectively, a corresponding set of digital audio data or digital video data that represents the content of the display. Previously, such variation of the apparent display rate of either an audio display or a video display has been accomplished using one of a variety of techniques. For example, the apparent display rate of an audio display represented by a set of digital audio data has been varied by using the synchronized overlap add (SOLA) method (discussed in more detail below) to appropriately modify an original set of digital audio data to produce a modified set of digital audio data from which the audio display is generated.

Often, a set of audio data is related to a particular set of video data and the two are used together to generate an audiovisual display, such as occurs, for example, in television broadcasts, motion pictures or computer multimedia displays. When the apparent display rate of an audiovisual display is varied, the audio display and video display must be synchronized to maintain temporal correspondence between the content of the audio and video displays. (Alternatively, the audio display can be eliminated altogether, thus obviating the need to maintain synchronization; however, the content of the audio display is lost.)

Previously, the apparent display rate of an audiovisual display has been varied by deleting or repeating video data (e.g., video frames) in a uniform manner, as appropriate, and deleting or repeating audio data in a uniform manner that corresponds to the treatment of the video data (e.g., if the apparent display rate of the video display is speeded up to 2 times the original display rate by, for example, eliminating every other video frame, then the audio display is likewise speeded up by eliminating every other audio sample or every other set of a predetermined number of audio samples). While this approach is effective in maintaining synchronization, it can cause distortion in the audio and video displays, particularly at relatively high or low apparent display rates. In particular, the audio display can be distorted so that, as the apparent display rate increases, human voices increasingly begin to manifest a “chipmunk effect,” and, as the apparent display rate decreases, human voices begin to sound as though the speaker is in a stupor. Such distortion of the display is a consequence of the fact that the elimination of audio data from the original set of audio data is done mechanically, without consideration of the content of the audio data being eliminated or retained.

A better way of varying the apparent display rate of an audiovisual display is desirable. In particular, an approach that “intelligently” modifies the audio and/or video data used to generate the display based upon an evaluation of the content of the audio data and/or video data is desirable, since such an approach can reduce or eliminate distortion of the display, and, in particular, the audio display. Good synchronization between the audio and video displays should also be maintained. Additionally, the capability of varying the apparent display rate over a wide range of magnitudes is desirable. Further, preferably the variation of the apparent display rate can be accomplished automatically in a manner that produces an apparent display rate that closely tracks a specified target display rate or rates.

SUMMARY

The invention enables the apparent display rate of an audiovisual display to be varied. The invention can cause an original set of audio data to be modified in accordance with a target display rate (which can be a single target display rate or a sequence of target display rates, as discussed further below) based upon an evaluation of the content of the audio data set, then cause a related original set of video data to be modified to conform to the modifications made to the original audio data set such that the modified audio and video data sets (and, thus, the displays produced therefrom) are synchronized. When the modified audio and video data sets so produced are used to generate an audiovisual display, the audiovisual display has an apparent display rate (or rates) that approximates the target display rate (or rates). Ensuring that the modified audio and video data sets are synchronized minimizes or eliminates the dissonance (e.g., a temporal mismatch between spoken words in the audio display and the corresponding movement of the speaker\'s lips in the video display) that would otherwise be experienced if the audio and video displays were not synchronized. Further, modifying the original audio data set directly, based upon an evaluation of the content of the audio data, to produce variation in the apparent display rate of the audiovisual display is advantageous in that it can enable minimization or elimination of artifacts (e.g., pitch doubling, pops and clicks) in the audio display. Preferably, the original audio data set is modified in a manner that produces a modified audio data set that can be used to generate an audio display having little or no distortion (e.g., there is a reduction or elimination of the tendency of human voices to sound like chipmunks when the apparent display rate is increased above a normal display rate or sound stupefied when the apparent display rate is decreased below a normal display rate). Generally, in accordance with the invention, a target display rate (and, thus, typically, the apparent display rate) can be faster or slower than a normal display rate at which an audiovisual display system generates an audiovisual display from the original sets of audio and video data. In particular, as will be better appreciated from the description below, the methods used to produce the modified audio data set enable a wide range of apparent display rates to be produced without introducing an unacceptable amount of distortion into the audiovisual display (in particular, the audio display).

In one embodiment of the invention, the apparent display rate of an audiovisual display can be varied from a normal display rate at which an audiovisual display system generates the audiovisual display from an original set of audio data and a related original set of video data by: i) defining a correspondence between the original set of audio data and the original set of video data; ii) determining a target display rate (which can be, in fact, a sequence of target display rates) for the audiovisual display; iii) creating a modified set of audio data, based upon the target display rate and an evaluation of the content of the original set of audio data, that corresponds to the original set of audio data; and iv) creating a modified set of video data, based upon the modified set of audio data, the correspondence between the modified set of audio data and the original set of audio data, and the correspondence between the original set of audio data and the original set of video data.

A target display rate can be established “manually” by a user instruction (i.e., by specification of a nominal target display rate by the user). Alternatively, a target display rate can be established automatically, without user input, based upon analysis of the audiovisual data. Or, a target display rate can be established by automatically modifying a user-specified nominal target display rate based upon analysis of the audiovisual data. As indicated above, when a nominal target display rate is specified by a user, a single target display rate can be specified for the entire audiovisual display, or a series of target display rates, each corresponding to a portion of the audiovisual display, can be specified. Likewise, a single target display rate or a series of target display rates can be automatically established (either “from scratch” or based upon an initially specified nominal target display rate or rates) in accordance with the invention. Moreover, as will be better appreciated from the description below, the invention enables a user to vary a nominal target display rate in real time as the audiovisual display is being generated.

Any appropriate method of automatically determining a target display rate, or automatically modifying a nominal target display rate, can be used. Such automatic determination or modification of the target display rate can be accomplished by evaluating the original set of audio data, the original set of video data, or both. Moreover, the target display rate can be established automatically by multiple evaluations of the audio and/or video data sets. The audio data set can be evaluated, for example, to determine the stress with which spoken portions of the audio data are uttered (by, for example, computing an energy term for the spoken portions), and the target display rate based upon the relative stresses of the spoken portions of the audio data. Or, the audio data set can be evaluated to determine the speed with which spoken portions of the audio data are uttered (by, for example, ascertaining spectral changes in the spoken portions), and the target display rate based upon the relative speeds of the spoken portions of the audio data. Or, both the stress and speed with which spoken portions of the audio data set are uttered can be determined and combined to produce audio tension values for the spoken portions, the target display rate being based upon the audio tension values of the spoken portions. The video data set can be evaluated, for example, to determine the relative rate of change of the video data along various population-based dimensions (described in more detail below), and the target display rate based upon that evaluation. Or, the video data set can be evaluated by ascertaining portions of the corresponding video image that change quickly, as well as the frequency with which such quick changes occur, and basing the target display rate on the occurrence and frequency of such quick changes. Or, the video data set can be evaluated by tracking the motion of objects within the corresponding video image, and basing the target display rate on the appearance of new objects in the video image.

The modified set of audio data can be created based upon the magnitude of the target display rate and an analysis of the content of the audio data. For example, the modified set of audio data can be created by: i) dividing the original set of audio data into a plurality of segments, each segment representing a contiguous portion of the set of audio data that occurs during a specified duration of time, each segment being adjacent to one or two other segments such that there are no gaps between segments and adjacent segments do not overlap; ii) overlapping an end portion of a first segment with an adjacent end portion of a second segment that is adjacent to the first segment (the overlap can be negative, as described in more detail below); iii) identifying as part of the modified set of audio data the audio data from the first segment that is not part of the overlapped end portion of the first segment; iv) blending the data of the corresponding overlapped end portions; and v) determining whether there are additional segments in the original set of audio data that have not been overlapped with an adjacent segment, wherein if there are additional segments, the additional segments are processed in accordance with the description above (a new first segment being created from the blended data and the non-overlapped data from the previous second segment), and if there are not additional segments, the blended data and the non-overlapped data from the second segment are included as part of the modified audio data set.

The modified set of video data can be created by: i) establishing a correspondence between the modified audio data set and the original video data set, based upon a correspondence between the modified audio data set and the original audio data set and a correspondence between the original audio data set and the original video data set; ii) grouping the audio data of the modified audio data set into audio segments having the same amount of audio data as found in audio segments of the original audio data set; iii) for each of the audio segments of the modified audio data set, identifying one or more partial or complete subunits of video data from the original video data set that correspond to audio data in the audio segment of the modified audio data set, based upon the correspondence between the modified audio data set and the original video data set; and iv) modifying the video frames in the original video data set as necessary to produce the modified video data set so that there is a one-to-one correspondence between audio segments of the modified audio data set and video frames of the modified video data set. The modified set of video data can be created by eliminating data from the original video data set, adding data to the original video data set, blending data from the original video data set, and/or synthesizing data based on the data in the original video data set.

The modified sets of audio and video data can be stored for later use in generating an audiovisual display, or they can be used immediately to generate an audiovisual display. In particular, in the latter case, the invention can be used to generate an audiovisual display in which the apparent display rate of the display can be varied in real-time. Such real-time variation of the apparent display rate is possible since the method of modifying the audio data set described above does not require knowledge of the audio data of the original audio data set far into the future to enable production of a modified audio data set, but, rather, only the audio data comprising a next segment of the original audio data set. Further, since the calculations for determining modified audio and video data can be done just prior to generating a display from that data, the calculations can be done based on a very recently determined (e.g., specified in real time by a user) target display rate. Moreover, the quantity of calculations required by a method of the invention can be performed by current processing devices sufficiently quickly to enable generation of a real-time display from the modified audio and video data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart of a method in accordance with the invention.

FIG. 2 is a simplified representation of a system with which the invention can be implemented.

FIGS. 3A and 3B together are a flow chart of a method, according to an embodiment of the invention, for modifying a set of audio data in accordance with a target display rate.

FIGS. 4A, 4B and 4C are diagrammatic representations of an audio data set including four segments, and the transformations of the audio data set that can occur during implementation of the method of FIGS. 3A and 3B to modify the audio data set so that the corresponding audio display is speeded up. FIG. 4A illustrates the blending of the first and second segments of the audio data set. FIG. 4B illustrates the blending of the second and third segments of the audio data set. FIG. 4C illustrates the blending of the third and fourth segments of the audio data set.

FIGS. 5A, 5B and 5C illustrate overlap of segments where all of the trial overlaps are positive.

FIGS. 6A, 6B and 6C illustrate overlap of segments where all of the trial overlaps are negative.

FIG. 7 illustrates a range of trial overlaps, including a positive target overlap and some negative overlaps.

FIG. 8A illustrates another range of trial overlaps that require special handling in the method of FIGS. 3A and 3B. FIG. 8B illustrates a way in which the situation shown in FIG. 8A can be addressed.

DETAILED DESCRIPTION

The invention enables the apparent display rate of an audiovisual display to be varied. The audiovisual display can be generated from related sets of audio and video data (the “original” sets of audio and video data) at a normal display rate by an audiovisual display system. The invention can enable the original sets of audio and video data to be modified in accordance with a target display rate or rates (which, in the usual case, are different—either faster or slower—than the normal display rate) so that, when the modified sets of audio and video data are used to generate an audiovisual display with the same audiovisual display system, the apparent display rate or rates of the audiovisual display are approximately equal to the target display rate or rates. (Note that it is possible—particularly when the magnitude of the target display rate is the same as, or similar to, the normal display rate—for a “modified” set of audio data or video data to be the same as the corresponding original set of audio data or video data.)

FIG. 1 is a flow chart of a method 100 in accordance with the invention. The method 100 accomplishes variation in the apparent display rate of the audiovisual display by modifying an original audio data set in accordance with a target display rate to produce a modified audio data set, then modifying a related original video data set to produce a modified video data set such that a correspondence between the audio data and video data of the original data sets is maintained in the modified data sets. Preferably, the original audio data set is modified in a manner that produces a modified audio data set that generates an audio display having little or no distortion.

The sets of audio data and video data can be either digital or analog (in many applications for which the invention is useful, the latter is typically the case). As will be understood by those skilled in the art of computer programming, the method 100 (and other methods of the invention described below) can be implemented, for example, on any appropriately programmed digital computer. Thus, if the audio and/or video data are initially analog, then the audio and/or video data can be digitized, using well-known processes and apparatus, to enable the method 100 to be implemented on a digital computer. The digital audio data can, for example, be represented as a sequence of audio samples that represent the amplitude of the analog audio signal at equally spaced points in time. The digital video data can, for example, be represented as a sequence of frames of pixels of video data (each pixel can further include separate pieces of data that

FIG. 2 is a simplified representation of a system 200 with which the invention can be implemented. The system 200 includes a processing device 201 (e.g., a conventional microprocessor), a data storage device 202 (e.g., a conventional random access memory and/or a conventional hard disk), a display device 203 (e.g., a conventional computer display monitor, television set and/or audio speakers) and a user interface device 204 (e.g., a conventional remote control, computer keyboard, mouse and/or touchscreen). These devices, as well as other conventional devices (not shown), such as those that can be part of a typical digital computer, can interact with each other over a conventional computer bus 205. It is to be understood that the invention can be implemented with other systems having components and/or a configuration other than that shown in FIG. 2. For example, if the audio data and/or video data is initially analog data, and the processing device is implemented as part of a digital computer, a system for implementing the invention can include a conventional A/D converter to convert the analog data to digital data. If such a system also included an analog display device (such as a television), the system could also include a conventional D/A converter to convert the processed digital data to analog form.

Returning to FIG. 1, in step 101 of the method 100, a correspondence between the original audio data set and the original video data set is established. As indicated above, digital audio data can be represented as a sequence of audio samples and digital video data can be represented as a sequence of video frames. As will be apparent from the description below, a convenient basis for establishing a correspondence between such audio and video data is to determine the number of audio samples that have the same duration as a frame of video data, and define that number of samples as an audio segment. (However, more generally, herein, an audio “segment” represents a contiguous portion of a set of audio data that occurs during a specified duration of time.) The determination of the number of audio samples that correspond in duration to a video frame can be made from the video frame rate and audio sample rate at which video data and audio data, respectively, are input to a video display device and audio display device of an audiovisual display system with which the invention is implemented. After establishing the number of samples in an audio segment, a correspondence between audio segments and video frames can then be established. For example, the audio segments can be defined so that each audio segment corresponds to a single particular video frame (such as occurs when the modified SOLA method is used to implement step 103 of the method 100, as described below). Or, overlapping audio segments can be defined so that each segment can correspond to part or all of one or more video frames (such as would occur when the unmodified SOLA method is used to implement step 103 of the method 100, as described below). The former implementation is less computationally expensive (and may, therefore, be preferable), since the matching of audio segments with particular unique video frames is more straightforward than the maintenance throughout the steps of the method 100 of a record of the correspondences between audio segments and video frames of the latter implementation.

In step 102, a target display rate is determined. As described above, generally, the target display rate can be faster or slower than the normal rate at which the audiovisual data (i.e., the audio and/or video data) is displayed. The target display rate can be indicated as a number representing a multiple of the normal display rate, i.e., a target display rate lower than 1 represents a display rate slower than the normal display rate and a target display rate higher than 1 represents a display rate faster than the normal display rate. The choice of a particular method for producing modified audio and video data sets, as well as the content of the audio and video data, can limit the possible display rate changes that can be achieved.

A nominal target display rate can be specified manually by a user of an audiovisual display system with which the invention is implemented. A single target display rate can be specified for the entire audiovisual display, or a series of target display rates, each corresponding to a portion of the audiovisual display, can be specified. The step 102 can ascertain a nominal target display rate value or values specified by a user. If multiple nominal target display rate values are specified, the step 102 can also ascertain a beginning and end time within the audiovisual display for each nominal target display rate. The user can use any appropriate user interface device (such as the exemplary devices described below with respect to FIG. 2) to specify a value or values for the nominal target display rate.

It may be desirable to modify a specified nominal target display rate to increase the likelihood that the audio portion of the audiovisual display will be displayed in an intelligible manner or with acceptable fidelity. According to the invention, this can be done automatically as part of the step 102. The invention can also automatically determine the target display rate “from scratch,” i.e., without specification of a nominal target display rate. However, in such a case, other criteria for governing the determination of the target display rate must be established. Several methods of automatically determining a target display rate, or modifying a nominal target display rate, are discussed in more detail below. However, any other appropriate method could be used.

In step 103, the original audio data set is manipulated, based upon the target display rate, to produce a modified audio data set. The modified audio data set is produced so that, generally, when the modified audio data set is used to generate an audio display, the audio display appears to be speeded up or slowed down by an amount that is approximately equal to the target display rate. Any appropriate method for modifying the original audio data set can be used. Some illustrative particular methods are discussed in more detail below.

In step 104, the original video data set is manipulated to produce a modified video data set. The correspondence between the modified audio data set and the original audio data set (step 103), and the correspondence between the original audio data set and the original video data set (step 101), are used to create a correspondence between the modified audio data set and the original video data set. Using the newly developed correspondence between the modified audio data set and the original video data set, video data can be deleted from or added to, as appropriate, the original video data set to create the modified video data set. Any appropriate method for creating the modified video data set from the modified audio data set can be used. Some illustrative particular methods are discussed in more detail below.

An audiovisual display can be generated from the modified data sets by an audiovisual display system. Any particular audiovisual display system can process audio and video data at a particular rate to produce an audiovisual display at a normal display rate. The audio and video data of the modified audio and video data sets are processed at the same rate by the audiovisual display system. However, since the modified audio and video data sets (in the usual case) have a different amount (either more or less) of data than the original audio and video data sets, the apparent display rate of the audiovisual display generated from the modified audio and video data sets is different than the normal display rate. Further, since, in step 104, the modified video data set is created based upon the content of the modified audio data set and a correspondence between the modified audio data set and the original video data set, the modified video data set is synchronized (at least approximately and, possibly, exactly) with the modified audio data set and produces a display of the same or approximately the same duration.

As will be better appreciated from the description below, the invention can make use of a method for modifying the audio data set that does not require knowledge of the audio data of the original audio data set far into the future to enable production of a modified audio data set, but, rather, only the audio data comprising a next segment of the original audio data set. Thus, the invention can enable the audiovisual display to be generated as the modified audio and video data sets are being produced. In particular, the invention advantageously allows the apparent display rate (via variation in the determined target display rate) to be varied in real time. Alternatively, the modified audio and video data sets can be stored on a conventional data storage device, such as a conventional computer hard disk, and displayed as desired at a later time.

An example will illustrate the operation of the method 100, as well as some minor adjustments that may be necessary in a practical implementation of the method 100. Illustratively, in a typical implementation, the invention can be implemented using a video display device that refreshes the video display with new digital video data at a rate of 30 frames per second, and one or more audio speakers that accepts digital audio data at a rate of 8000 samples per second. Thus, in such an implementation, each frame of video data corresponds in duration to 266.67 audio samples (step 101 of the method 100), i.e., 8000 samples per second divided by 30 frames per second. An audio segment can thus be defined as 266.67 audio samples. In practice, since an audio sample cannot be subdivided, the number of audio samples in each audio segment can be rounded up to the next highest integer or truncated to the next lowest integer, so that, on average, each audio segment includes the calculated number of audio samples per video frame. Thus, in this example, each set of three audio segments includes one segment having 266 audio samples and two segments having 267 audio samples, arranged in any desired order.

Further, let it be assumed in this example that the target display rate is 2.5 times the normal display rate (step 102 of the method 100). The original audio data set is modified in accordance with the target display rate to produce the modified audio data set (step 103 of the method 100). Here, that means creating a modified audio data set that has approximately 60% fewer audio samples than the original audio data set, so that the desired speedup of 2.5 is achieved. The modified audio data set includes subsets of audio data that each correspond to one of the audio segments of the original audio data set. Each of these subsets have, on average, approximately 60% fewer audio samples than were present in the corresponding audio segment of the original audio data set, though particular subsets can vary significantly from this average, as will be better appreciated from the description of FIGS. 3A, 3B, 4A, 4B and 4C below.

The audio speakers and the video display device expect to receive 8000 audio samples per second and 30 video frames per second, respectively. However, each 266.67 audio samples (in practice, 266 or 267 samples) of the modified audio data set correspond to, on average, 2.5 video frames from the original video data set. (This can be verified by noting that there are, on average, 2.5 audio samples in the original audio data set for each audio sample of the modified data set, and that each audio segment—266.67 audio samples—of the original audio data set corresponds to a video frame of the original video data set.) Thus, in order to maintain correspondence between the audio and video displays after modifying the audio data in accordance with the target display rate, it is necessary to reduce the number of video frames, so that each audio segment (i.e., 266.67 audio samples) of the modified audio data set will correspond to a single video frame. In this example, a modified video data set must be created that includes 1 video frame for each 2.5 video frames in the original video data set. Since video frames cannot be split apart temporally, to facilitate the invention, each audio segment of the modified audio data set is deemed to correspond to an integer number of video frames of the original video data set such that, on average, each audio segment of the modified audio data set corresponds to 2.5 video frames. In other words, the first audio segment might correspond to the first three frames of video data of the original video data set, while the second audio segment corresponds to the fourth and fifth (i.e., next two) frames of video data. The original video data set is then modified by retaining only one video frame from each set of video frames that corresponds to an audio segment of the modified audio data set. This can be accomplished, as discussed above, by eliminating or blending video frames.

As mentioned above with respect to step 102 of the method 100 (FIG. 1), a target display rate can be determined, or a specified nominal target display rate can be modified, automatically in accordance with the invention. Generally, a target display rate can be determined or modified automatically by analyzing the original set of audio data, the original set of video data, or both. Several methods that can be used to accomplish such automatic determination or modification of the target display rate are discussed below. It is to be understood, however, that any other appropriate method of determining or modifying a target display rate can be used with the invention.

One method of determining a target display rate is described in detail in the commonly owned, U.S. Pat. No. 5,828,994, entitled “Non-Uniform Time Scale Modification of Recorded Audio,” by Michele Covell and M. Margaret Withgott, issued Oct. 27, 1998 (hereafter referred to as “Covell and Withgott”), the disclosure of which is incorporated by reference herein. In that method, the audio data of a set of audiovisual data is analyzed and a sequence of target display rates are established on that basis. Briefly, in that method, the audio data set is searched for a particular speaking voice or voices (typically, though not necessarily, a voice or voices that are likely to be part of the audio data set). The audio data set is subdivided into sections and each section is evaluated to determine a probability that the section includes the particular speaking voice or voices. The probabilities can be normalized. The target display rate for each section can be established based upon the normalized probability for the section. For example, for sections having a very high probability that the particular speaker is speaking, the target display rate may be established as the normal display rate or even slowed down relative to the normal display rate. As the probability that a particular section includes the particular speaking voice or voices decreases, the target display rate can be made progressively greater. Thus, in sum, this method can produce a sequence of target display rates based upon whether a particular speaker or speakers are speaking. As can be appreciated, while this method has been described above as used to determine a sequence of target display rates from scratch, the method could also be used to modify a nominal target display rate (or sequence of rates) by increasing or decreasing the nominal target display rate based upon the determined probability that the section of audio data corresponding to the nominal target display rate includes the particular speaking voice or voices.

Covell and Withgott describe in detail other methods for using the analysis of an audio data set of a set of audiovisual data to affect the target display rate (i.e., to determine a target display rate or to modify a nominal target display rate). These methods can also be used with the present invention. In one such method, the magnitude of the stress with which the spoken portions of an audio data set are uttered is ascertained. (The magnitude of the stress of unspoken portions of the audio data set is considered to be zero.) Such stress can be ascertained, for example, by computing an energy term for the spoken portions. The target display rate of various portions of the audio data set can then be established (or the nominal target display rate or rates modified) in accordance with the relative magnitudes of stress ascertained for those portions. For example, the magnitude of the target display rate can be varied inversely with respect to the amount of stress with which a spoken portion of the audio data set is uttered. In another such method of Covell and Withgott, the magnitude of the speed with which the spoken portions of an audio data set are uttered is ascertained (unspoken portions have a speed of zero). The speaking speed can be ascertained, for example, from measurement of spectral changes in the speech. The relative magnitudes of speaking speed can be used to establish the target display rate of various portions of the audio data set (or to modify the nominal target display rate or rates). For example, the target display rate of spoken portions that were originally spoken rapidly is made relatively lower than the target display rate of other spoken portions. As further described by Covell and Withgott in yet another method, the ascertained magnitudes of speaking stress and speaking speed can be combined to produce an “audio tension” value, the audio tension value then being used to establish a target display rate or rates (or modify a nominal target display rate or rates) in any desired manner.

As indicated above, the video data of a set of audiovisual data can also be analyzed and a target display rate (or sequence of rates) established (i.e., determined from scratch or obtained by modifying a nominal target display rate) on the basis of that analysis. Several such methods of determining a target display rate are described in detail in the commonly owned, co-pending U.S. patent application entitled “A Method of Compressing a Plurality of Video Images for Efficiently Storing, Displaying and Searching the Plurality of Video Images,” by Subutai Ahmad, Ser. No. 08/528,891, filed on Sep. 15, 1995, (hereafter referred to as “Ahmad”), the disclosure of which is incorporated by reference herein. In one such method, the video data is analyzed to ascertain the relative rate of change of the video data along various population-based dimensions (i.e., dimensions that are selected based upon an analysis of a specified population of the data). Linear (or affine) manifolds are one example of the implementation of such analysis. In a linear manifold, related n-dimensional sets of data (e.g., the pixels of a set of frames of video data) are monitored to determine the variation along each dimension (e.g., pixel) between “adjacent” sets (e.g., successive video frames) of data. A specified number of dimensions (e.g., particular pixels) having the greatest variation can be selected as the population-based dimensions. The change in values of the data (e.g., change in pixel values) along the population-based dimensions are monitored. The target display rate can be established based upon those changes. For example, the target display rate can be made relatively slower when the video data (along the population-based dimensions) changes rapidly and vice versa. In another method described in Ahmad, the video data is analyzed to ascertain portions of a video image that change quickly, as well as the frequency with which such quick changes occur. During periods when numerous such quick changes occur, the target display rate can be made relatively lower; during periods when the video image remains relatively stable, the target display rate can be made relatively higher.

Other methods for establishing a target display rate based upon analysis of the video data set are described in, for example, “A Real-Time System for Automatically Annotating Unstructured Image Sequences,” by Zabih, Woodfill and Withgott, IEEE International Conference on Systems, Man, and Cybernetics, 1993, and “Tracking Non-Rigid Objects in Complex Scenes,” by Huttenlocher, Noh and Rucklidge, International Conference on Computer Vision, 1993, the disclosures of which are incorporated by reference herein. In those methods, the motion of objects in the video image is tracked. The target display rate can be made relatively lower, for example, during periods of time in which new objects appear in the video image.

A target display rate can also be established by combining the results of multiple analyses of the audiovisual data. Generally, the multiple analyses can include analysis of the audio data only, the video data only, or both the audio and the video data. The results of the analyses can be combined in any suitable way. For example, the target display rate (or sequences of rates) established by multiple analyses can be averaged to establish a target display rate. Or, the faster or slower of the target display rates established by multiple analyses can be selected as the target display rate. Other non-linear filtering methods (such as stack filters) or time-variant methods (such as functions which include some temporal hysteresis) can also be used, as understood by those skilled in the art of data analysis, to establish a target display rate from multiple target display rates established by multiple analyses of the audiovisual data.

In step 103 of the method 100 (FIG. 1) described above, a modified set of audio data is created based upon the target display rate. The modified audio data set is created so that, when used to generate an audio display, the apparent display rate of the display approximates or is equal to the target display rate. Below, a particular method for producing such a modified audio data set is described in detail with respect to FIGS. 3A, 3B, 4A, 4B and 4C. The method described is a modification of the previously known synchronized overlap add (SOLA) method. The SOLA method, which can also be used to produce a modified audio data set, is described in more detail in a paper entitled “High quality time scale modification for speech,” by S. Roucos and A. M. Wilgus, published in Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, pages 493-496, IEEE, 1985, the disclosure of which is incorporated by reference herein.

Both the modified SOLA and the SOLA method can be used to produce a modified audio data set that results in a display having an apparent display rate that is either speeded up or slowed down with respect to the normal display rate. Further, both the modified SOLA and SOLA methods can produce a modified audio data set that generates an audio display having little or no distortion of the sound represented by the original audio data set. For example, these methods enable the audio display to be speeded up without producing a “chipmunk effect” when human voices are displayed, or slowed down without making speakers sound as though they are in a stupor. The modified SOLA method described below differs from the SOLA method described in the paper mentioned above principally in that the SOLA method allows segments to initially be overlapping while the modified SOLA method does not; aside from this difference, the two methods are, in principle, the same. Use of the modified SOLA method can be advantageous, as compared to use of the SOLA method, because the modified SOLA method results in a lower computational burden (the SOLA method requires, for example, more “bookkeeping” to keep track of the initially overlapping segments). Additionally, the modified SOLA method can produce less distortion than the SOLA method because the modified SOLA method retains more of the original audio data in an unmodified form. Those skilled in the art of data analysis will readily appreciate how the SOLA method can be used with the invention based upon the description of the modified SOLA method below; primarily, the use of the SOLA method requires that the steps of the modified SOLA method as described below be modified to account for the initial overlap of the segments.

FIGS. 3A and 3B together are a flow chart of a method 300 (a modified SOLA method), according to an embodiment of the invention, for modifying a set of audio data in accordance with a target display rate. The method 300 can be used with any target display rate, either a target display rate intended to speed up the apparent display rate (i.e., when the target display rate is indicated as a number representing a multiple of the normal display rate, a target display rate greater than 1) or a target display rate intended to slow down the apparent display rate (i.e., a target display rate less than 1). FIGS. 4A, 4B and 4C are diagrammatic representations of an audio data set 400 and the transformations of the audio data set 400 that can occur during implementation of the method 300 to modify the audio data set 400 so that the corresponding audio display is speeded up (target display rate greater than 1).

Generally, the modified SOLA method of FIGS. 3A and 3B causes the beginning and end of each segment to be overlapped with the end and beginning, respectively, of adjacent segments. As will be better understood from the description below, the overlapping of segments can result in the addition of data to, elimination of data from and/or blending of data in the audio data set. (Note that, as explained and illustrated in more detail below, the “overlap” can be negative.) For display speedup, the audio data set is decreased in length by the sum of all of the overlap lengths (which will, typically, be predominately or exclusively positive). Correspondingly, for display slowdown, the audio data set is increased in length by the sum of the overlap lengths (which will, typically, be predominately or exclusively negative).

Speech sounds can be divided into two general classes: voiced and unvoiced. Voiced speech is produced as the response of the vocal tract to periodic excitation from vocal cords, which generates pitch pulses. Voiced speech includes the majority of speech sounds, such as the vowels, nasals, semivowels and voiced transients. Unvoiced sounds, which include the fricatives and the unvoiced stops, are created as the result of turbulence produced by a constriction somewhere along the vocal tract. While the method described here is best suited to modifying audio data representing voiced sounds, it performs well in modifying audio data representing both voiced and unvoiced sounds, especially in view of the manner and relative proportion in which such sounds appear in human speech.

The possible lengths of each overlap are constrained in accordance with the amount of display rate variation that is desired. However, within this constraint, the length of each particular overlap is chosen so that the pitch pulses of the overlapped portions closely approximate (i.e., overlie) each other (see FIGS. 5A, 5B and 5C, and FIGS. 6A, 6B and 6C, discussed below). Thus, the addition, elimination and/or blending of the audio data of the overlapped portions does not greatly distort the sound corresponding to the overlapped portions of audio data. As a result, the audio data set can be modified as desired (i.e., the apparent display rate of an audio data set can be varied as desired), while reducing the amount of distortion associated with the modification of the audio data set (i.e., the modified display sounds “normal”). In practice, the method 300 has been found to work well (i.e., produce an audio display with relatively little or no distortion) in varying the apparent display rate of an audio display that includes spoken and/or non-spoken audio.

In step 301, an audio data set is divided into segments of audio data (audio samples, in the discussion below) of equal time duration. Each audio segment preferably corresponds to a specified unit of video data having the same duration. For example, each audio segment can include audio samples spanning a duration of time that corresponds to a single video frame of video data, i.e., each audio segment of the audio data corresponds to a particular video frame of the video data. The segments do not overlap and there are no gaps between segments, i.e., the beginning of each segment (except for a beginning segment of the audio data set) is immediately adjacent to the end of a previous segment and the end of each segment (except for an end segment of the audio data set) is immediately adjacent to the beginning of a subsequent segment. In FIGS. 4A, 4B and 4C, the audio data set 400 is divided into segments 401 through 404. This is shown in the top row of each of FIGS. 4A, 4B and 4C.

In step 302, a first segment is selected. Typically, though not necessarily, the first segment selected in step 302 is a segment at either the beginning or end of the audio data set, e.g., the segment 401 or the segment 404, respectively, of the audio data set 400 of FIGS. 4A, 4B and 4C. (As can be appreciated, when the invention is implemented to vary the apparent display rate of an audiovisual display in real time, the first segment generally must be—given the processing speed capabilities of current devices for processing audiovisual data—the first temporally occurring segment, in order to maintain the real-time display.) In the description of the method 300 below, the first segment selected in step 302 is the segment at the beginning (i.e., temporally, the first occurring segment) of the audio data set (i.e., the segment 401 of the audio data set 400 in FIGS. 4A, 4B and 4C) and segments are processed successively according to the method 300 moving forward in time (i.e., from left to right in FIGS. 4A, 4B and 4C). However, as will be appreciated from the description below, the last segment could have been selected in the step 302 and the method 300 could be implemented moving backward in time (i.e., from right to left in FIGS. 4A, 4B and 4C) in a manner analogous to that described below.

In step 303, a second segment is selected (segment 402 in FIG. 4A). The second segment is temporally adjacent to, but does not overlap, the first segment. The first and second segments are shown in the second row of each of FIGS. 4A, 4B and 4C.

As indicated in the example discussed above, the division of an audio data set into exactly equal audio segments may result in audio segments having a fractional audio sample (e.g., in the example above, 266.67 audio samples). This difficulty can be addressed, as discussed above, by adjusting the number of audio samples in each segment to be an integer value such that, on average, each audio segment includes the exactly calculated number of audio samples. In steps 302 and 303, the integer numbers of audio samples are associated with the segments as the “lengths” of the segments.

In step 304, a target overlap of the first and second segments is determined. The target overlap represents an approximation of the actual overlap between adjacent portions of the first and second segments that is determined in later steps of the method 300. The magnitude of the target overlap is directly related to the target display rate; consequently, as more readily apparent from the description below, the apparent display rate produced by the method 300 closely approximates the target display rate. The target overlap can be determined according to the equation (1):

t=[(s−1)/s]*d   (1)

where t is the target overlap (expressed in time or number of audio samples), s is the target display rate (a dimensionless quantity that, usually, is not equal to 1), and d is the duration of each segment (again, expressed in time or number of audio samples). (Note that, here, d can be a non-integer value if expressed in audio samples.) As can be appreciated from considering equation (1), when the target display rate is greater than 1 (i.e., a speedup of the display is desired), the target overlap is greater than zero. Conversely, when the target display rate is less than 1 (i.e., a slowdown of the display is desired), the target overlap is less than zero.

In step 305, a set of trial overlaps is determined based upon the target overlap. As explained in more detail below, each of the trial overlaps is evaluated to determine which trial overlap works best, e.g., which trial overlap results in modification of the original audio data set with least distortion to the sound produced by the modified audio data set. As can be appreciated, then, the granularity of the trial overlaps (i.e., the amount by which each trial overlap differs from the trial overlap(s) having the next closest magnitude) and the range of trial overlaps can affect the quality of the determination of the best overlap. In general, as the trial overlaps become more finely grained, the quality of the determination of the best overlap increases because more trial overlaps are evaluated for a given range of trial overlaps. Generally, as the range of trial overlaps increases, the quality of the determination of the best overlap increases because more trial overlaps are evaluated for a given granularity. Preferably, the range of trial overlaps is at least large enough to include one pitch pulse of the lowest frequency pitch expected to be encountered in the audio data (e.g., if the lowest frequency pitch has a frequency of 80 hz, such that the pitch pulse is 12.5 milliseconds, and the sampling rate is 8000 samples per second, the range of trial overlaps is preferably at least 100 audio samples in length), since, usually, superposing pitch pulses from overlapping portions of the segments can be found within that range. However, as more trial overlaps are evaluated, the amount of time required to evaluate all of the trial overlaps increases. Additionally, evaluating a wider range of trial overlaps generally increases the chance of determining a best overlap that differs significantly from the trial overlap; if this occurs often enough, the apparent display rate can vary significantly from the target display rate (however, as discussed further below, this problem can be overcome by continually monitoring the actual apparent display rate and modifying the target display rate as necessary to cause the apparent display rate to converge to the target display rate). The granularity and range of the trial overlaps are chosen as a tradeoff between the computational time required and the expected quality of the best overlap produced. Illustratively, for an audio sampling rate of 8000 samples per second, the range of trial overlaps can be 100 audio samples long and the granularity can be one audio sample, i.e., 101 trial overlaps are evaluated.

Preferably, the range of trial overlaps is centered about the target overlap so that, despite the fact that the actual overlap of any two segments can vary from the target overlap, the average actual overlap will tend to equal the target overlap as the number of segments becomes large. The closer the average actual overlap is to the target overlap, the closer the apparent display rate produced by the method 300 is to the target display rate. (However, the produced apparent display rate can also be monitored to enable the target display rate to be varied to cause the produced apparent display rate to remain approximately equal to the target display rate, as mentioned above and described in detail below.)

In step 306, a trial overlap that has not previously been considered is selected. The trial overlaps can be evaluated in any order.

In step 307, the audio data at the end of the first segment that falls within the overlap is identified as first segment overlap data. This is shown, for example, in the third row of each of FIGS. 4A, 4B and 4C, i.e., in FIG. 4A, a set of first segment overlap data 406 is shown.

In step 308, the audio data at the beginning of the second segment that falls within the overlap is identified as second segment overlap data. This is also shown, for example, in the third row of each of FIGS. 4A, 4B and 4C, i.e., in FIG. 4A, a set of second segment overlap data 407 is shown.

When two segments are overlapped, there will be not only segment overlap data (as described above), but also non-overlapped data. Non-overlapped data is also shown in the third row of each of FIGS. 4A, 4B and 4C. For example, in FIG. 4A, first segment 401 includes non-overlapped data 405, while second segment 402 includes non-overlapped data 408.

In step 309, the correlation between the first segment overlap data and second segment overlap data is calculated. Any appropriate method of calculating a correlation between the first segment overlap data and second segment overlap can be used. In one embodiment, the correlation is determined as the mean cross-product of corresponding audio data from the first segment overlap data (after the mean value of the first segment overlap data has been subtracted from each of the first segment overlap data) and the second segment overlap data (after the mean value of the second segment overlap data has been subtracted from each of the second segment overlap data). Let N1 and N2 be the amount of data in the first and second segments, respectively. Let M be the amount of data that is added to the first segment from the second segment to accommodate negative overlaps, if necessary, as described above. (Note that it may also be necessary to add additional data to the second segment if, for example, the range of overlaps is greater than the length of the second segment; however, as discussed below, that eventuality is believed to be unlikely in typical applications of the invention.) Let the amplitude of the audio data in the first segment be represented by a[1], a[2], . . . , a[N1+M] (where M is zero, if no data has been added to the first segment) and the amplitude of the audio data in the second segment be represented by b[1], b[2], . . . , b[N2]. Let k be the amount of audio data in a trial overlap, (2r+1) be the length of the range of trial overlaps (so that there are r pieces of audio data on either side of the trial overlap) and d be the target overlap. The mean values of the first and second segment overlap data for a trial overlap are given by equations (2) and (3), respectively:

a′=(a[N1−k+1]+a[N1−k+2]+ . . . +a[N1+M])/(k+M)   (2)

b′=(b[1]+b[2]+ . . . +b[k+M])/(k+M)   (3)

The correlation measure, C, is then given by equation (4):

C = { ( a  [ N 1 - k + 1 ] - a ′ ) * ( b  [ 1 ] - b ′ ) + ( a  [ N 1 - k + 2 ] - a ′ ) * ( b  [ 2 ] - b ′ ) + … + ( a  [ N 1 + M ] - a ′ ) * ( b  [

Download full PDF for full patent description/claims.




You can also Monitor Keywords and Search for tracking patents relating to this Variable rate video playback with synchronized audio patent application.

Patent Applications in related categories:

20130114939 - Invention title is a digital multi-media keepsake. it is a handheld case with a built in lcd video screen which displays digital photos and plays digital video and audio, with a removable storage container in the center of the case that stores memorabilia - It is a handheld digital keepsake case with a built-in video screen which plays video, photos, and audio, with a removable storage container. The exterior case displays digital photos, plays video and audio automatically when opened and shuts off when closed. It is a keepsake within a case that plays ...


###


Other recent patent applications listed under the agent :