Smartphone-based methods and systems   

Abstract: Arrangements involving portable devices (e.g., smartphones and tablet computers) are disclosed. One arrangement enables a content creator to select software with which that creator's content should be rendered—assuring continuity between artistic intention and delivery. Another utilizes a device camera to identify nearby subjects, and take actions based thereon. Others rely on near field chip (RFID) identification of objects, or on identification of audio streams (e.g., music, voice). Some technologies concern improvements to the user interfaces associated with such devices. Others involve use of these devices in connection with shopping, text entry, sign language interpretation, and vision-based discovery. Still other improvements are architectural in nature, e.g., relating to evidence-based state machines, and blackboard systems. Yet other technologies concern use of linked data in portable devices—some of which exploit GPU capabilities. Still other technologies concern computational photography. A great variety of other features and arrangements are also detailed.

This application is a division of application Ser. No. 13/299,140, filed Nov. 17, 2011, which is a continuation-in-part of international application PCT/US11/59412, filed Nov. 4, 2011 (published as WO2012061760), which claims priority to the following provisional applications: 61/410,217, filed Nov. 4, 2010; 61/449,529, filed Mar. 4, 2011; 61/467,862, filed Mar. 25, 2011; 61/471,651, filed Apr. 4, 2011; 61/479,323, filed Apr. 26, 2011; 61/483,555, filed May 6, 2011; 61/485,888, filed May 13, 2011; 61/501,602, filed Jun. 27, 2011; and which also is a continuation-in-part of each of the following applications: 13/174,258, filed Jun. 30, 2011; 13/207,841, filed Aug. 11, 2011 (published as 20120116559); and 13/278,949, filed Oct. 21, 2011 (published as 20120134548).

The disclosures of these applications are incorporated herein by reference, in their entireties.

TECHNICAL FIELD

The present technology generally primarily concerns sensor-equipped consumer electronic devices, such as smartphones and tablet computers.

The present specification details a diversity of technologies, assembled over an extended period of time, to serve a variety of different objectives. Yet they relate together in various ways, and often can be used in conjunction, and so are presented collectively in this single document.

This varied, interrelated subject matter does not lend itself to a straightforward presentation. Thus, the reader\'s indulgence is solicited as this narrative occasionally proceeds in nonlinear fashion among the assorted topics and technologies.

The detailed technology builds on work detailed in previous U.S. patent filings. These include applications: Ser. No. 13/149,334, filed May 31, 2011; Ser. No. 13/088,259, filed Apr. 15, 2011; Ser. No. 13/079,327, filed Apr. 4, 2011; Ser. No. 13/011,618, filed Jan. 21, 2011 (published as 20110212717); Ser. No. 12/797,503, filed Jun. 9, 2010 (published as 20110161076); Ser. No. 12/774,512, filed May 5, 2010 (published as 20110274310); Ser. No. 12/716,908, filed Mar. 3, 2010 (published as 20100228632); Ser. No. 12/490,980, filed Jun. 24, 2009 (published as 20100205628); Ser. No. 12/271,772, filed Nov. 14, 2008 (published as 20100119208); Ser. No. 11/620,999, filed Jan. 8, 2007 (published as 20070185840); U.S. Pat. No. 7,003,731; and U.S. Pat. No. 6,947,571.

In the few years since their introduction, portable computing devices (e.g., smartphones, music players, and tablet computers) have transitioned from novelties to near-necessities. With their widespread adoption has come an explosion in the number of software programs (“apps”) available for such platforms. Over 300,000 apps are now available from the Apple iTunes store alone.

Many apps concern media content. Some are designed to provide on-demand playback of audio or video content, e.g., television shows. Others serve to complement media content, such as by enabling access to extra content (behind-the-scenes clips, cast biographies and interviews, contests, games, recipes, how-to videos), by allowing social network-based features (communicating with other fans, including by Twitter, Facebook and Foursquare, blogs), etc. In some instances a media-related app may operate in synchrony with the audio or video content, e.g., presenting content and links at time- or event-appropriate points during the content.

Apps are now being specialized to particular broadcast and recorded media content. The ABC television show My Generation, for example, was introduced with a companion iPad app dedicated exclusively to the program—providing polls, quizzes and other information in synchronized fashion. Traditional media companies, such as CNN, ESPN, CBS, etc., are increasingly becoming app companies as well.

It is difficult for apps to gain traction in this crowded marketplace. Searching iTunes, and other app stores, is the most common technique by which users find new apps for their devices. The next most popular technique for app discovery is through recommendations from friends. Both approaches, however, were established when the app market was much smaller, and have not scaled well.

In the case of the My Generation iPad ap, for example, the show\'s producers must reach out to the target audience and entice them to go to the app store, where they must type in the title of the app, download it, install it, and then run it when the television program is playing.

In accordance with certain embodiments of the present technology, a different solution is provided. In one such embodiment, a microphone-equipped user device samples ambient content, and produces content-identifying data from the captured audio. This content-identifying data is then used to look-up an app recommended by the proprietor of the content, which app is then installed and launched—with little or no action required by the user.

By such arrangement, the content effectively selects the app. The user doesn\'t select the software; the user\'s activity selects the software. Over time, each user device becomes app-adapted to the content preferences of the user—thereby becoming optimized to the user\'s particular interests in the content world.

To some degree, this aspect of the present technology is akin to the recommendation features of TiVo, but for apps. The user\'s content consumption habits (and optionally those of the user\'s social network friends) lead the device to recommend apps that serve the user\'s interests.

Desirably, it is artists that are given the privilege of specifying the app(s) to be invoked by their creative works. Many countries have laws that recognize artists\' continuing interest in the integrity with which their works are treated (so-called “moral rights”). Embodiments of the present technology serve this interest—providing artists a continuing role in how their art is presented, enabling them to prescribe the preferred mechanisms by which their works are to be experienced. Continuity is provided between the artist\'s intention and the art\'s delivery.

It is not just stand-alone apps that can be treated in this fashion. More granular software choices can similarly be made, such as the selection of particular rendering codecs to be used by media players (e.g., Windows Media Player). For example, the National Hockey League may prefer that its content be rendered with a codec designed for maximum frame rate. In contrast, the Food Network may prefer that its content be rendered with a codec optimized for truest color fidelity.

Historically, the “channel” was king, and content played a supporting role (i.e., drawing consumers to the channel, and to its advertising). From the consumer\'s standpoint, however, these roles should be reversed: content should be primary. Embodiments of the present technology are based on this premise. The user chooses the content, and the delivery mechanism then follows, as a consequence.

The foregoing and other features and advantages of the present technology will be more readily apparent from the following detailed description, which proceeds with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system that can be used in certain embodiments of the present technology.

FIG. 2 is a representation of a data structure that can be used with the embodiment of FIG. 1.

FIGS. 3-7 detail features of illustrative gaze-tracking embodiments, e.g., for text entry.

FIGS. 8 and 9 detail features of an illustrative user interface.

FIG. 10 shows a block diagram of a system incorporating principles of the present technology.

FIG. 11 shows marker signals in a spatial-frequency domain.

FIG. 12 shows a mixed-domain view of a printed object that includes the marker signals of FIG. 11, according to one aspect of the present technology.

FIG. 13 shows a corner marker that can be used to indicate hidden data.

FIG. 14 shows an alternative to the marker signals of FIG. 11.

FIG. 15 shows a graph representation of data output from a smartphone camera.

FIG. 16 shows a middleware architecture for object recognition.

FIG. 17 is similar to FIG. 16, but is particular to the Digimarc Discover implementation.

FIG. 18 is a bar chart showing impact of reading image watermarks on system tasks.

FIG. 19 further details performance of a watermark recognition agent running on an Apple iPhone 4 device.

FIG. 20 shows locations of salient points in first and second image frames.

FIG. 21 shows histograms associated with geometric alignment of two frames of salient points.

FIG. 22 shows an image memory in a smartphone, including three color bit plane of 8-bit depth each.

FIG. 23 shows a similar smartphone memory, but now utilized to store RDF triples.

FIG. 24 shows some of the hundreds or thousands of RDF triples that may be stored in the memory of FIG. 23.

FIG. 25 shows the memory of FIG. 23, now populated with illustrative RDF information detailing certain relationships among people.

FIG. 26 shows some of the templates that may be applied to the Predicate plane of the FIG. 25 memory, to perform semantic reasoning on the depicted RDF triples.

FIG. 27 names the nine RDF triples within a 3×3 pixel block of memory.

FIG. 28 shows a store of memory in a smartphone.

FIGS. 29A and 29B depict elements of a graphical user interface that uses data from the FIG. 28 memory.

FIG. 30 shows use of a memory storing triples, and associated tables, to generate data used in generate a search query report to a user.

FIG. 31 shows another store of memory in a smartphone, depicting four of more planes of integer (e.g., 8-bit) storage.

FIG. 32 shows a smartphone displaying an image captured from a catalog page, with a distinctive graphical effect that signals presence of a steganographic digital watermark.

FIGS. 33 and 34 show how a smartphone can spawn tags, presented along an edge of the display, associated with different items in the display.

FIG. 35 shows information retrieved from a database relating to a watermark-identified catalog page (i.e., object handles for an object shape).

FIG. 36 shows how detection of different watermarks in different regions of imagery can be signaled to a user.

FIG. 38 shows an LED-based communication system, incorporating both high bandwidth and low bandwidth channels.

DETAILED DESCRIPTION

The present technology, in some respects, expands on technology detailed in the assignee\'s above-detailed patent applications. The reader is presumed to be familiar with such previous work, which can be used in implementations of the present technology (and into which the present technology can be incorporated).

Referring to FIG. 1, an illustrative system 12 includes a device 14 having a processor 16, a memory 18, one or more input peripherals 20, and one or more output peripherals 22. System 12 may also include a network connection 24, and one or more remote computers 26.

An illustrative device 14 is a smartphone or a tablet computer, although any other consumer electronic device can be used. The processor can comprise a microprocessor such as an Atom or A4 device. The processor\'s operation is controlled, in part, by information stored in the memory, such as operating system software, application software (e.g., “apps”), data, etc. The memory may comprise flash memory, a hard drive, etc.

The input peripherals 20 may include a camera and/or a microphone. The peripherals (or device 14 itself) may also comprise an interface system by which analog signals sampled by the camera/microphone are converted into digital data suitable for processing by the system. Other input peripherals can include a touch screen, keyboard, etc. The output peripherals 22 can include a display screen, speaker, etc.

The network connection 24 can be wired (e.g., Ethernet, etc.), wireless (WiFi, 4G, Bluetooth, etc.), or both.

In an exemplary operation, device 14 receives a set of digital content data, such as through a microphone 20 and interface, through the network connection 24, or otherwise. The content data may be of any type; audio is exemplary.

The system 12 processes the digital content data to generate corresponding identification data. This may be done, e.g., by applying a digital watermark decoding process, or a fingerprinting algorithm—desirably to data representing the sonic or visual information itself, rather than to so-called “out-of-band” data (e.g., file names, header data, etc.). The resulting identification data serves to distinguish the received content data from other data of the same type (e.g., other audio or other video).

By reference to this identification data, the system determines corresponding software that should be invoked. One way to do this is by indexing a table, database, or other data structure with the identification data, to thereby obtain information identifying the appropriate software. An illustrative table is shown conceptually in FIG. 2.

In some instances the data structure may return identification of a single software program. In that case, this software is launched—if available. (Availability does not require that the software be resident on the device. Cloud-based apps may be available.) If not available, the software may be downloaded (e.g., from an online repository, such as the iTunes store), installed, and launched. (Or, the device can subscribe to a software-as-service cloud version of the app.) Involvement of the user in such action(s) can depend on the particular implementation: sometimes the user is asked for permission; in other implementations such actions proceed without disturbing the user.

Sometimes the data structure may identify several different software programs. The different programs may be specific to different platforms, in which case, device 12 may simply pick the program corresponding to that platform (e.g., Android G2, iPhone 4, etc.). Or, the data structure may identify several alternative programs that can be used on a given platform. In this circumstance, the device may check to determine which—if any—is already installed and available. If such a program is found, it can be launched. If two such programs are found, the device may choose between them using an algorithm (e.g., most-recently-used; smallest memory footprint; etc.), or the device may prompt the user for a selection. If none of the alternative programs is available to the device, the device can select and download one—again using an algorithm, or based on input from the user. Once downloaded and installed, the application is launched.

(Sometimes the data structure may identify different programs that serve different functions—all related to the content. One, for example, may be an app for discovery of song lyrics. Another may be an app relating to musician biography. Another may be an app for purchase of the content. Again, each different class of software may include several alternatives.)

Note that the device may already have an installed application that is technically suited to work with the received content (e.g., to render an MPEG4 or an MP3 file). For certain types of operations, there may be dozens or more such programs that are technically suitable. However, the content may indicate that only a subset of this universe of possible software programs should be used.

Software in the device 14 may strictly enforce the content-identified software selection. Alternatively, the system may treat such software identification as a preference that the user can override. In some implementations the user may be offered an incentive to use the content-identified software. Or, conversely, the user may be assessed a fee, or other impediment, in order to use software other than that indicated by the content.

Sometimes the system may decline to render certain content on a device (e.g., because of lack of suitable app or hardware capability), but may invite the user to transfer the content to another user device that has the needed capability, and may implement such transfer. (Ansel Adams might have taken a dim view of his large format photography being used as a screen saver on a small format, low resolution, smartphone display. If such display is attempted, the software may invite the user to instead transfer the imagery to a large format HD display at the user\'s home for viewing.)

Instead of absolutely declining to render the content, the system may render it in a limited fashion. For example, a video might be rendered as a series of still key frames (e.g., from scene transitions). Again, the system can transfer the content where it can be more properly enjoyed, or—if hardware considerations permit (e.g., screen display resolution is adequate)—needed software can be downloaded and used.

As shown by the table of FIG. 2 (which data structure may be resident in the memory 18, or in a remote computer system 26), the indication of software may be based on one or more contextual factors—in addition to the content identification data. (Only two context factors are shown; more or less can of course be used.)

One formal definition of “context” is “any information that can be used to characterize the situation of an entity (a person, place or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves.”

Context information can be of many sorts, including computing context (network connectivity, memory availability, processor type, CPU contention, etc.), user context (user profile, location, actions, preferences, nearby friends, social network(s) and situation, etc.), physical context (e.g., lighting, noise level, traffic, etc.), temporal context (time of day, day, month, season, etc.), history of the above, etc.

In the illustrated table, rows 32 and 34 correspond to the same content (i.e., same content ID), but they indicate different software should be used—depending on whether the user\'s context is indoors or outdoors. (The software is indicated by a 5 symbol hex identifier; the content is identified by 6 hex symbols. Identifiers of other forms, and longer or shorter in length, can of course be used.) Row 36 shows a software selection that includes two items of software—both of which are invoked. (One includes a further descriptor—an identifier of a YouTube video that is to be loaded by software “FF245.”) This software is indicated for a user in a daytime context, and for a user in the 20-25 age demographic.

Row 38 shows user location (zip code) and gender as contextual data. The software for this content/context is specified in the alternative (i.e., four identifiers “OR”d together, as contrasted with the “AND” of row 36).

Rows 40 and 42 show that the same content ID can correspond to different codecs—depending on the device processor (Atom or A4).

(By point of comparison, consider the procedure by which codecs are presently chosen. Typically the user isn\'t familiar with technical distinctions between competing codecs, and the artist has no say. Codec selection is thus made by neither party that is most vitally interested in the choice. Instead, default codecs come bundled with certain media rendering software (e.g., Windows Media Player). If the defaults are unable to handle certain content, the rendering software typically downloads a further codec—again with no input from the parties most concerned.)

It will be understood that the software indicated in table 30 by the content can be a stand-alone app, or a software component—such as a codec, driver, etc. The software can render the content, or it can be a content companion—providing other information or functionality related to the content. In some implementations the “software” can comprise a URL, or other data/parameter that is provided to another software program or online service (e.g., a YouTube video identifier).

Desirably, all such software identified in the table is chosen by the proprietor (e.g., artist, creator or copyright-holder) of the content with which it is associated. This affords the proprietor a measure of artistic control that is missing in most other digital content systems. (The proprietor\'s control in such matters should be given more deference than, say, that of a content distributor—such as AOL or iTunes. Likewise, the proprietor\'s choice seems to merit more weight than that of the company providing word processing and spreadsheet software for the device.)

Often the proprietor\'s selection of software will be based on aesthetics and technical merit. Sometimes, however, commercial considerations come into play. (As artist Robert Genn noted, “‘Starving artist’ is acceptable at age 20, suspect at age 40, and problematical at age 60.”)

Thus, for example, if a user\'s device detects ambient audio by the group The Decemberists, artist-specified data in the data structure 30 may indicate that the device should load the Amazon app for purchase of the detected music (or load the corresponding Amazon web page), to induce sales. If the same device detects ambient audio by the Red Hot Chili Peppers, that group may have specified that the device should load the band\'s own web page (or another app), for the same purpose. The proprietor can thus specify the fulfillment service for content objected-oriented commerce.

In some arrangements, the starving artist problem may best be redressed by an auction arrangement. That is, the device 14 (or remote computer system 26) may announce to an online service (akin to Google AdWords) that the iPod of a user—for which certain demographic profile/context information may be available—has detected the soundtrack of the movie Avatar. A mini auction can then ensue—for the privilege of presenting a buying opportunity to the user. The winner (e.g., EBay) then pays the winning bid amount into an account, from which it is shared with the auction service, the artist, etc. The user\'s device responds by launching an EBay app through which the user can buy a copy of the movie, its soundtrack, or related merchandise. Pushing such content detection events, and associated context information, to cloud-based services can enable a richly competitive marketplace of responses.

(Auction technology is also detailed in the assignee\'s previously-cited patent applications, and in Google\'s published patent applications US2010017298 and US2009198607.)

The popularity of content can lead associated software to become similarly popular. This can induce other content proprietors to consider such software for use with their own content, since wide deployment of that software may facilitate consumer exposure to the other proprietor\'s content.

For example, Universal Music Group may digitally watermark all its songs with an identifier that causes the FFmpeg MP3 player to be identified as the preferred rendering software. Dedicated fans of UMG artists soon install the recommended software—leading to deployment of such software on large numbers of consumer devices. When other music proprietors consider what software to designate in table 30, the widespread use of the FFmpeg MP3 software can be one of the factors they weigh in making a choice.

(The software indicated in table 30 may be changed over time, such as through the course of a song\'s release cycle. When a new band issues a song, the table-specified software may include an app intended to introduce the new band to the public (or a YouTube clip can be indicated for this purpose). After the music has become popular and the band has become better known, a different software selection may be indicated.)

Presently, music discovery and other content-related applications are commonly performed by application software. Operating system (OS) software provides a variety of useful services—some of which (e.g., I/O) are commonly used in content-related applications. However, commercial OS software has not previously provided any services specific to content processing or identification.

In accordance with a further aspect of the present technology, operating system software is provided to perform one or more services specific to content processing or identification.

In one particular implementation, an OS application programming interface (API) takes content data as input (or a pointer to a location where the content data is stored), and returns fingerprint data corresponding thereto. Another OS service (either provided using the same API, or another) takes the same input, and returns watermark information decoded from the content data. (An input parameter to the API can specify which of plural fingerprint or watermark processes is to be applied. Alternatively, the service may apply several different watermark and/or fingerprint extraction processes to the input data, and return resultant information to the calling program. In the case of watermark extraction, the resultant information can be checked for apparent validity by reference to error correction data or the like.)

The same API, or another, can further process the extracted fingerprint/watermark data to obtain XML-based content metadata that is associated with the content (e.g., text giving the title of the work, the name of the artist, the copyright holder, etc.). To do this it may consult a remote metadata registry, such as maintained by Gracenote.

Such a content-processing API can establish a message queue (e.g., a “listening/hearing queue) to which results of the fingerprint/watermark extraction process (either literally, or the corresponding metadata) are published. One or more application programs can monitor (hook) the queue—listening for certain identifiers. One app may be alert to music by the Beatles. Another may listen for Disney movie soundtracks. When such content is detected, the monitoring app—or another—can launch into activity—logging the event, acting to complement the media content, offering a buying opportunity, etc.

Alternatively, such functionality can be implemented apart from the operating system. One approach is with a publish/subscribe model, by which some apps publish capabilities (e.g., listening for a particular type of audio), and other subscribe to such functions. By these arrangements, loosely-coupled applications can cooperate to enable a similar ecosystem.

One application of the present technology is to monitor media to which a user is exposed—as a background process. That is, unlike song identification services such as Shazam, the user need not take any action to initiate a discovery operation to learn the identity of a particular song. (Of course, the user—at some point—must turn on the device, and authorize this background functionality.) Instead, the device listens for a prolonged period—much longer than the 10-15 seconds of Shazam-like services, during the course of the user\'s day. As content is encountered, it is processed and recognized. The recognition information is logged in the device, and is used to prime certain software to reflect exposure to such content—available the next time the user\'s attention turns to the device.

For example, the device may process ambient audio for fifteen minutes, for an hour, or for a day. When the user next interacts with the device, it may present a listing of content to which the user has been exposed. The user may be invited to touch listings for content of interest, to engage in a discovery operation. Software associated with this content then launches.

In some implementations the device can prime software applications with information that is based, at least in part, on the content identification data. This priming may cause, e.g., the YouTube app to show a thumbnail corresponding to a music video for a song heard by the user—readying it for selection. Likewise, a 90 second sample audio clip may be downloaded to the iPod music player app—available in a “Recent Encounters” folder. An email from the band might be added to the user\'s email InBox, and a trivia game app may load a series of questions relating to the band. Such data is resident locally (i.e., the user needn\'t direct its retrieval, e.g., from a web site), and the information is prominent to the user when the corresponding app is next used—thereby customizing these apps per the user\'s content experiences.

Social media applications can serve as platforms through which such information is presented, and shared. When the user activates a Facebook app, for example, an avatar may give a greeting, “I noticed that you experienced the following things today . . . ” and then list content to which the user was exposed, e.g., “Billy Liar” by the Decemberists, “Boys Better” by the Dandy Warhols, and the new LeBron James commercial for Nike. The app may remind the user of the context in which each was encountered, e.g., while walking through downtown Portland on Nov. 4, 2010 (as determined, e.g., by GPS and accelerometer sensors in the device). The Facebook app can invite the user to share any of this content with friends. It may further query whether the user would like discographies for any of the bands, or whether it would like full digital copies of the content, is interested in complementary content associated with any, or would like associated app(s) launched, etc.

The app may similarly report on media encounters, and associated activities, of the user\'s friends (with suitable permissions).

From the foregoing, it will be recognized that certain of the foregoing embodiments ease the user\'s dilemma of locating apps associated with certain media content. Instead, the media content serves to locate its own favored apps.

Such embodiments assure continuity between artistic intention and delivery; they optimize the experience that the art is intended to create. No longer must the artistic experience be mediated by a delivery platform over which the artist has no control—a platform that may seek attention for itself, potentially distracting from the art in which the user is interested.

This technology also fosters competition in the app marketplace—giving artists a more prominent voice as to which apps best express their creations. Desirably, a Darwinian effect may emerge, by which app popularity becomes less an expression of branding and marketing budgets, and more a reflection of popularity of the content thereby delivered.

Users are increasingly presented with large volumes of data. Examples include hundreds of channels of television programming, email, and RSS/Twitter/social network/blog feeds. To help users handle such flows of information, technologies have been proposed that filter or highlight the incoming information in accordance with user profile data.

A familiar example is DVR software, such as from Tivo, that presents a subset of the unabridged electronic program guide, based on apparent user interests. The Tivo software notices which television programs have been viewed by the user, invites user feedback in the form of “thumbs-up” or “thumbs-down” rankings, and then suggests future programs of potential interest based on such past behavior and ranking

A more recent example is Google\'s “Priority Inbox” for its Gmail service. Incoming email is analyzed, and ranked in accordance with its potential importance to the user. In making such judgment, Google considers what email the user has previously read, to which email the user has previously responded, and the senders/keywords associated with such mails. Incoming email that scores highly in such assessment is presented at the top of the mail list.

The company My6sense.com offers a similar service for triaging RSS and Twitter feeds. Again, the software monitors the user\'s historical interaction with data feeds, and elevates in priority the incoming items that appear most relevant to the user. (In its processing of Twitter feeds, My6sense considers the links the user has clicked on, the tweets the user has marked as favorites, the tweets that the user has retweeted, and the authors/keywords that characterize such tweets.)

Such principles can be extended to encompass object interactions. For example, if a person visiting a Nordstrom department store uses her smartphone to capture imagery of a pair of Jimmy Choo motorcycle boots, this may be inferred to indicate some interest in fashion, or in motorcycling, or in footwear, or in boots, or in Jimmy Choo merchandise, etc. If the person later uses her smartphone to image River Road motorcycle saddle bags, this suggests the person\'s interest may more accurately be characterized as including motorcycling. As each new image object is discerned, more information about the person\'s interests is gleaned. Some early conclusions may be reinforced (e.g., motorcycling), other hypotheses may be discounted.

In addition to recognizing objects in imagery, the analysis (which can include human review by crowd-sourcing) can also discern activities. Location can also be noted (either inferred from the imagery, or indicated by GPS data or the like).

For example, image analysis applied to a frame of imagery may determine that it includes a person riding a motorcycle, with a tent and a forested setting as a background. Or in a temporal series of images, one image may be found to include a person riding a motorcycle, another image taken a few minutes later may be found to include a person in the same garb as the motorcycle rider of the previous frame—now depicted next to a tent in a forested setting, and another image taken a few minutes later may be found to depict a motorcycle being ridden with a forested background. GPS data may locate all of the images in Yellowstone National Park.

Such historical information—accumulated over time—can reveal recurrent themes and patterns that indicate subjects, activities, people, and places that are of interest to the user. Each such conclusion can be given a confidence metric, based on the system\'s confidence that the attribute accurately characterizes a user interest. (In the examples just given, “motorcycling” would score higher than “Jimmy Choo merchandise.”) Such data can then be used in filtering or highlighting the above-noted feeds of data (and others) with which the user\'s devices are presented.

As a history of device usage is compiled, a comprehensive history of interests and media consumption patterns emerges that can be used to enhance the user\'s interaction with the world. For example, if the user images fashion accessories from Sephora, parameters controlling the user\'s junk email filter may be modified to allow delivery of emails from that company—emails that might have otherwise have been blocked. The user\'s web browsers (e.g., Safari on the smartphone; Firefox on a home PC) may add the Sephora web page to a list of “Suggested Favorites”—similar to what Tivo does with its program suggestions.

A user may elect to establish a Twitter account that is essentially owned by the user\'s object-derived profile. This Twitter account follows tweets relating to objects the user has recently sensed. If the user has imaged a Canon SLR camera, this interest can be reflected in the profile-associated Twitter account, which can follow tweets relating to such subject. This account can then re-tweet such posts into a feed that the user can follow, or check periodically, from the user\'s own Twitter account.

Such object-derived profile information can be used for more than influencing the selection of content delivered to the user via smartphone, television, PC and other content-delivery devices. It can also influence the composition of such content. For example, objects with which the user interacts can be included in media mashups for the user\'s consumption. A central character in a virtual reality gaming world frequented by the user may wear Jimmy Choo motorcycle boots. Treasure captured from an opponent may include a Canon SLR camera.

Every time the user interacts with an object, this interaction can be published via Twitter, Facebook, etc. (subject to user permission and sharing parameters). These communications can also be thought of as “check-ins” in the FourSquare sense, but in this case it is for an object or media type (music, TV, etc.) rather than for a location.

Based on these public communiqués, social frameworks can emerge. People who are interested in hand-built Belgian racing bicycles from the 1980s (as evidenced by their capturing imagery of such bicycles) can coalesce into an affinity social group, e.g., on Twitter or Facebook (or their successors). Object-based communities can thus be defined and explored by interested users.

Social network theorists will recognize that this is a form of social network analysis, but with nodes representing physical objects.

Social network analysis views relationships using network theory, in which the network comprises nodes and ties (sometimes called edges, links, or connections). Nodes are the individual actors within the networks, and ties are the relationships between the actors. The resulting graph-based structures can be complex; there can be many kinds of ties between the nodes. In the case just-given, the relationships ties can include “likes,” “owns,” etc.

A particular 3D graph may place people objects in one plane, and physical objects in a parallel plane. Links between the two planes associate people with objects that they own or like. (The default relationship may be “like.” “Owns” may be inferred from context, or deduced from other data. E.g., a Camaro automobile photographed by a user, and geolocated at the user\'s home residence, may indicate an “owns” relationship. Similarly, a look-up of a Camaro license plate in a public database, which indicates the car is registered to the user, also suggests an “owns” relationship.)

Such a graph will also typically include links between people objects (as is conventional in social network graphs), and may also include links between physical objects. (One such link is the relationship of physical proximity. Two cars parked next to each other in a parking lot may be linked by such a relationship.)

The number of links to a physical object in such a network is an indication of the object\'s relative importance in that network. Degrees of association between two different physical objects can be indicated by the length of the network path(s) linking them—with a shorter path indicating a closer degree of association.

Some objects may be of transitory interest to users, while others may be of long-term interest. If a user images a particular type of object only once, it likely belongs to the former class. If the user captures images of such object type repeatedly over time, it more likely belongs to the latter class. Decisions based on the profile data can take into account the aging of object-indicated interests, so that an object encountered once a year ago is not given the same weight as an object encountered more recently. For example, the system may follow Canon SLR-based tweets only for a day, week or month, and then be followed no longer, unless other objects imaged by the user evidence a continuing interest in Canon equipment or SLR cameras. Each object-interest can be assigned a numeric profile score that is increased, or maintained, by repeated encounters with objects of that type, but which otherwise diminishes over time. This score is then used to weight that object-related interest in treatment of content.

(While the detailed arrangement identified physical objects by analysis of captured image data, it will be recognized that objects with which the user interacts can be identified otherwise, such as by detection of RFID/NFC chips associated with such objects.)

Principles and embodiments analogous to those detailed above can be applied to analysis of the user\'s audio environment, including music and speech recognition. Such information can similarly be applied to selecting and composing streams of data with which the user (e.g., user device) is presented, and/or which may be sent by the user (user device). Still greater utility can be provided by consideration of both visual and auditory stimulus captured by user device sensors.

The front-facing camera on a smartphone can be used to speed text entry, in a gaze-tracking mode.

A basic geometrical reference frame can be first established by having the user look, successively, at two or more known positions on the display screen, while monitoring the gaze of one or both eyes using the smartphone camera 101. In FIG. 3, the user looks successively at points A, B, C and

D. Increased accuracy can be achieved by repeating the cycle. (The reader is presumed to be familiar with the principles of gaze tracking systems, so same are not belabored here. Example systems are detailed, e.g., in patent publications 20110013007, 20100315482, 20100295774, and 20050175218, and references cited therein.)

Once the system has determined the geometrical framework relating the user\'s eye gaze and the device screen, the user can indicate an initial letter by gazing at it on a displayed keyboard 102. (Other keyboard displays that make fuller use of the screen can of course be used.) The user can signify selection of the gazed-at letter by a signal such as a gesture, e.g., an eye blink, or a tap on the smartphone body (or on a desk on which it is lying). As text is selected, it is added to a message area 103

Once an initial letter (e.g., “N”) has been presented, data entry may be speeded (and gaze tracking may be made more accurate) by presenting likely next-letters in an enlarged letter-menu portion 104 of the screen. Each time a letter is entered, a menu of likely next-letters (determined, e.g., by frequency analysis of letter pairs in a representative corpus) is presented. An example is shown in FIG. 4, in which the menu takes the form of a hexagonal array of tiles (although other arrangements can of course be used).

In this example, the user has already entered the text “Now is the time for a_”, and the system is waiting for the user to select a letter to go where the underscore 106 is indicated. The last letter selected was “a.” This letter is now displayed—in greyed-out format—in the center tile, and is surrounded by a variety of options—including “an,” “at,” “al,” and “ar.” These are the four most common letter pairs beginning with “a.” Also displayed, in the indicated hexagonal array, is a “—” selection tile 108 (indicating the next symbol should be a space), and a keyboard selection tile 110.

To enter the next letter “l”, the user simply looks at the “al” display tile 112, and signifies acceptance by a tap or other gesture, as above. The system then updates the screen as shown in FIG. 5.

Here the message has been extended by a letter (“Now is the time for al_”), and the menu 104 has been updated to show the most common letter pairs beginning with the letter “l”. The device solicits a next letter input. To enter another “l” the user gazes at the “ll” tile 114, and gestures.

Initial studies suggest that well over 50% of text entry can be accomplished by the enlarged letter-menu of likely next-letters (plus a space). If a different letter is required, the user simply gazes at the keyboard tile 110 and gestures. A keyboard—like that shown in FIG. 3, or another, appears, and the user makes a selection from it.

Instead of presenting four letter pairs, a space, and a keyboard icon, as shown in FIGS. 2 and 3, an alternative embodiment presents five letter pairs and a space, as shown in FIG. 6. In this arrangement, a keyboard is always displayed on the screen, so the user can select letters from it without the intermediate step of selecting the keyboard tile 110 of FIG. 4.

Instead of the usual keyboard display 102, a variant keyboard display 102a—shown in FIG. 7—can be used. This layout reflects the fact that five characters are not needed on the displayed keyboard, since the five most-likely letters are already presented in the hexagonal menu. In the illustrated example, the five keys are not wholly omitted, but rather are given extra-small keys. The 21 remaining letters are given extra-large keys. Such arrangement speeds user letter selection from the keyboard, and makes gaze tracking of the remaining keys more accurate. (A further variant, in which the five letter keys are omitted entirely from the keyboard, can also be used.)

It will also be noted that the variant keyboard layout 102a of FIG. 7 omits the usual space bar. Since there is an enlarged menu tile 116 for the space symbol, no space bar in the keyboard 102a is required. In the illustrated arrangement, this area has been replaced with common punctuation symbols.

The artisan will recognize that numerous alternatives and extensions can be implemented. There is no need, for example, for the last letter to be displayed in the center of the hexagon. This space can be left vacant, or can be used instead to indicate the next-letter apparently indicated by the user\'s present gaze, so that the user can check the selection before gesturing to confirm. (When updated with the apparently-indicated letter, gazing at the center tile doesn\'t change the earlier gaze-based selection.) A numeric pad can be summoned to the screen by selection of a numeric pad icon—like keyboard tile 110 in FIG. 4. Or a numeric keyboard can be displayed on the screen throughout the message composition operation (like keyboard 102 in FIG. 6). One or more of the hexagonal tiles can present a guess of the complete word the user is entering—again based on analysis of a text corpus.

The corpus used to determine the most common letter pairs, and full word guesses, can be user-customized, e.g., a historical archive of all text and/or email messages authored by the user, or sent from the user\'s device. The indicated display features can naturally be augmented by other graphical indicia and controls associated with the smartphone functionality being used (e.g., a text-messaging application).

In still other embodiments, the user may select from symbols and words presented apart from the smartphone display—such as on a printed page. A large-scale complete keyboard and a complete numeric pad can be presented on such a page, and used independently, or in conjunction with a displayed letter menu, like menu 104. (Again, the smartphone camera can be used to perform the gaze-tracking, and geometrical calibration can be performed by having the user gaze at reference points.)

In a particular embodiment, the smartphone provides additional feedback to the user, e.g., in the form of highlighting that indicates where, on the screen, the user\'s gaze appears to be directed. Similarly, the smartphone can draw trails on the screen corresponding to sensed movements of the user\'s gaze. These trails can “evaporate” a few seconds after being drawn—removing them from the display.

It will be recognized that, in similar fashion, any interface interaction that presently uses fingers can alternatively be implemented with gaze tracking.

Just as a smartphone can watch a user\'s eye, and interpret the movements, it can similarly watch a user\'s hand gestures, and interpret them as well. The result is a sign language interpreter.

Sign languages (American and British sign languages being the most dominant) comprise a variety of elements—all of which can be captured by a camera, and identified by suitable image analysis software. A sign typically includes handform and orientation aspects, and may also be characterized by a location (or place of articulation), and movement. Manual alphabets (fingerspelling) gestures are similar, and are employed mostly for proper names and other specialized vocabulary.

An exemplary sign language analysis module segments the smartphone-captured imagery into regions of interest, by identifying contiguous sets of pixels having chrominances within a gamut associated with most skin tones. The thus-segmented imagery is then applied to a classification engine that seeks to match the hand configuration(s) with a best match within a database library of reference handforms. Likewise, sequences of image frames are processed to discern motion vectors indicating the movement different points within the handforms, and changes to the orientations over time. These discerned movements are likewise applied to a database of reference movements and changes to identify a best match.

When matching signs are found in the database, textual meanings associated with the discerned signs are retrieved from the database records and can be output—as words, phonemes or letters—to an output device, such as the smartphone display screen.

Desirably, the best-match data from the database is not output in raw form. Preferably, the database identifies for each sign a set of candidate matches—each with a confidence metric. The system software then consider what combination of words, phonemes or letters is most likely in that sequential context—giving weight to the different confidences of the possible matches, and referring to a reference database detailing word spellings (e.g., a dictionary), and identifying frequently signed word-pairs and—triples. (The artisan will recognize that similar techniques are used in speech recognition systems—to reduce the likelihood of outputting nonsense phrases.)

The recognition software can also benefit by training. If the user notes an incorrect interpretation has been given by the system to a sign, the user can make a sign indicating that a previous sign will be repeated for re-interpretation. The user then repeats the sign. The system then offers an alternative interpretation—avoiding the previous interpretation (which the system infers was incorrect). The process may be repeated until the system responds with the correct interpretation (which may be acknowledged with a user sign, such as a thumbs-up gesture). The system can then add to its database of reference signs the just-expressed signs—in association with the correct meaning.

Similarly, if the system interprets a sign, and the user does not challenge the interpretation, then data about the captured sign imagery can be added to the reference database—in association with that interpretation. By this arrangement the system learns to recognize the various presentations of certain signs. The same technique allows the system to be trained, over time, to recognize user-specific vernacular and other idiosyncrasies.

To aid in machine-recognition, standard sign language can be augmented to give the image analysis software some calibration or reference information that will aid understanding. For example, when signing to a smartphone, a user may begin with gesture such as extending fingers and thumbs from an outwardly-facing palm (the typical sign for the number ‘5’) and then returning the fingers to a fist.

This allows the smartphone to identify the user\'s fleshtone chrominance, and determine the scale of the user\'s hand and fingers. The same gesture, or another, can be used to separate concepts—like a period at the end of a sentence. (Such punctuation is commonly expressed in American signal language by a pause. An overt hand gesture, rather than the absence of a gesture, is a more reliable parsing element for machine vision-based sign language interpretation.)

As noted, the interpreted sign language can be output as text on the smartphone display. However, other arrangements can also be implemented. For example, the text can be simply stored (e.g., in a ASCII or Word document), or it can be output through a text-to-speech converter, to yield audible speech. Similarly, the text may be input to a translation routine or service (e.g., Google translate) to convert it to another language—in which it may be stored, displayed or spoken.

The smartphone may employ its proximity sensor to detect the approach of a user\'s body part (e.g., hands), and then capture frames of camera imagery and check them for a skin-tone chrominance and long edges (or other attributes that are characteristic of hands and/or fingers). If such analysis concludes that the user has moved hands towards the phone, the phone may activate its sign language translator. Relatedly, Apple\'s FaceTime communications software can be adapted to activate the sign language translator when the user positions hands to be imaged by a phone\'s camera. Thereafter, text counterparts to the user\'s hand gestures can be communicated to the other party(ies) to which the phone is linked, such as by text display, text-to-speech conversion, etc.

In accordance with another aspect of the technology, a smartphone is equipped to rapidly capture identification from plural objects, and to make same available for later review.

FIG. 8 shows an example. The application includes a large view window that is updated with streaming video from the camera (i.e., the usual viewfinder mode). As the user pans the camera, the system analyzes the imagery to discern any identifiable objects. In FIG. 8, there are several objects bearing barcodes within the camera\'s field of view.

In the illustrated system the processor analyzes the image frame starting at the center—looking for identifiable features. (In other arrangements, a top-down, or other image search procedure can be followed.) When the phone finds an identifiable feature (e.g., the barcode 118), it overlays bracketing 120 around the feature, or highlights the feature, to indicate to the user what part of the displayed imagery has caught its attention. A “whoosh” sound is then emitted from the device speaker, and an animated indicia moves from the bracketed part of the screen to a History 122 button at the bottom. (The animation can be a square graphic that collapses to a point down at the History button.) A red-circled counter 124 that is displayed next to the History button indicates the number of items thus-detected and placed in the device History (7, in this case).

After thus-processing barcode 118, the system continues its analysis of the field of view for other recognizable features. Working out from the center it next recognizes barcode 126, and a similar sequence of operations follows. The counter 124 is incremented to “8.” It next notes barcode 128—even though it is partially outside the camera\'s field of view. (Redundant encoding of certain barcodes enables such decoding.) The time elapsed for recognizing and capturing data from the three barcodes into the device history, with the associated user feedback (sound and animation effects) is less than 3 seconds (with 1 or 2 seconds being typical).

By tapping the History button 122, a scrollable display of previously-captured features is presented, as shown in FIG. 9. In this list, each entry includes a graphical indicia indicating the type of feature that was recognized, together with information discerned from the feature, and the time the feature was detected. (The time may be stated in absolute fashion, or relative to the present time; the latter is shown in FIG. 9.)

As shown in FIG. 9, the features detected by the system needn\'t be found in the camera data. They can include features discerned from audio (e.g., identification of a person speaking), or from other sensors. FIG. 9 shows that the phone also sensed data from a near field chip (e.g., an RFID chip)—indicated by the “NFC” indicia.

At a later time, the user can recall this History list, and tap indicia of interest. The phone then responds by launching a response corresponding to that feature (or by presenting a menu of several available features, from which the user can select).

Sometimes the user may wish to turn-off the detailed streaming mode operation, e.g., when the environment is rich with detectable features, and the user does not want multiple captures. A button control 130 on the application UI toggles such functionality on and off. In the indicated state, detection of multiple features is enabled. If the user taps this control, its indicia switches to “Multiple is Off.” When the phone detects a feature in this mode, the system adds it to the History (as before), and immediately launches a corresponding response. For example, it may invoke web browser functionality and load a web page corresponding to the detected feature.

Another aspect of the present technology involves smartphone-based state machines, which vary their operation in response to sensor input.

Application Ser. No. 12/797,503 details how a blackboard data structure is used for passing data between system components. The following discussion provides further information about an illustrative embodiment.

In this illustrative system, there are both physical sensors and logical sensors. A physical sensor monitors a sensor, and feeds data from it to the blackboard. The camera and microphone of a smartphone are particular types of physical sensors, and may be generically termed “media sensors.” Some sensors may output several types of data. For example, an image sensor may output a frame of pixel data, and also an AGC (automatic gain control) signal.

A logical sensor obtains data—typically from the blackboard—and uses it to calculate further data. This further data is also commonly stored back in the blackboard. (The recognition agents discussed in application Ser. No. 12/793,503 are examples of logical sensors. Another is an inference engine.) In some cases the same physical data may pass through multiple stages of logical sensor refinement during processing.

Modules which produce or consume media content may require some special functionality, e.g., to allow format negotiation with other modules. This can include querying a recognition agent for its requested format (e.g., audio or video, together with associated parameters), and then obtaining the corresponding sensor data from the blackboard.

Below follows a table detailing an exemplary six stage physical sensor/logical sensor data flow, for reading a digital watermark from captured imagery (the ReadImageWatermark scenario). The Stored Data column gives the name of the stored data within the blackboard. (MS stands for media sensor; PS stands for physical sensor; LS stands for logical sensor; and RA stands for recognition agent.)

Source Data Sensor Module Stored Data 1 Video frame Camera (MS)

Download full PDF for full patent description/claims.




You can also Monitor Keywords and Search for tracking patents relating to this Smartphone-based methods and systems patent application.
###


Other recent patent applications listed under the agent :