Interactive digital clippings   

The present application claims the right of priority under 35 U.S.C. §119(e) based on U.S. Provisional Patent Application No. 60/974,077, filed Sep. 21, 2007 which is incorporated by reference herein in its entirety as if fully set forth herein.

The present invention relates generally to computing systems and, more particularly, to a method and system for enabling selection and use of objects in a computer system. It has been developed primarily to enable users to receive useful information from a paper-based user interface.

The following applications have been filed by the Applicant simultaneously with the present application:

NPT087US NPT088US NPT089US NPT090US NPT092US NPT093US NPT094US NPT095US NPT096US NPT097US NPS151US NPP090US NPP091US NPP092US NPP093US NPP094US NPP095US NPP096US NPP097US NPS149US NPS150US NPZ032US NPP099US NPP100US NPP101US NPP102US NPP103US NPP104US NPP105US NPP106US NPP107US NPP108US

The disclosures of these co-pending applications are incorporated herein by reference. The above applications have been identified by their filing docket number, which will be substituted with the corresponding application number, once assigned.

The following patents or patent applications filed by the applicant or assignee of the present invention are hereby incorporated by cross-reference.

10/815621 10/815635 10/815647 11/488162 10/815636 11/041652 11/041609 11/041556 10/815609 7204941 7278727 10/913380 7122076 7156289 09/575197 6720985 7295839 09/722174 7068382 7094910 7062651 6644642 6549935 6987573 6727996 6760119 7064851 6290349 6428155 6785016 6831682 6741871 6965439 10/932044 6870966 6474888 6724374 6788982 7263270 6788293 6737591 09/693514 10/778056 10/778061 11/193482 7055739 6830196 7182247 7082562 10/409864 7108192 10/492169 10/492152 10/492168 10/492161 7308148 6957768 7170499 11856061 11/672522 11/672950 11754310 12015507 7148345 12025746 12025762 12025765 10/407212 6902255 6755509

The Applicant has previously described a method of enabling users to access information from a computer system via a printed substrate e.g. paper. The substrate has coded data printed thereon, which is read by an optical sensing device when the user interacts with the substrate using the sensing device. A computer receives interaction data from the sensing device and uses this data to determine what action is being requested by the user. For example, a user may select a printed hyperlink using a sensing device and retrieve a corresponding webpage via a display device or printer.

It would be desirable to enhance the functionality of the above-described system. Enhanced functionality would encourage greater use of the system and, hence, generate increased revenue streams for system providers.

More particularly, it would be desirable to provide users with the ability to complement web-based interactions with paper-based interactions.

In a first aspect the present invention provides a method of generating a clipping from a printed substrate, said substrate having a graphic image and a position-coding pattern disposed thereon, said position-coding pattern identifying an identity and a plurality of locations, said method comprising the steps of: receiving, in a computer system, interaction data indicative of the identity and at least one position of a sensing device relative to the substrate, the sensing device generating the interaction data, when operatively positioned or moved relative to the substrate, by reading at least some of the position-coding pattern; identifying and retrieving at least part of a page description corresponding to the printed substrate using the identity, said page description including a visual description describing the graphic image and an input description describing one or more interactive elements; determining, using the page description and the at least one position of the sensing device, a clipping zone of the substrate indicated by the sensing device, said clipping zone containing a part of the graphic image; generating the clipping, said clipping comprising a clipping identity identifying at least said page description and said clipping zone; and copying the clipping to a clipboard.

Optionally, said clipping identifies a clipping image, said clipping image comprising a clipping visual description describing the part of the graphic image contained in said clipping zone.

Optionally, the method further comprising the step of storing said clipping image on a clipping image store, wherein said stored clipping image is indexed with said clipping identity.

Optionally, the method further comprising displaying said clipping on a display device, said displaying comprising the steps of: retrieving said clipping image from said clipping image store using said clipping identity; and displaying said clipping image on the display device.

Optionally, the method further comprising displaying said clipping on a display device, said displaying comprising the steps of: identifying said page description using said clipping identity; rendering a clipping image from said page description, said clipping image comprising a clipping visual description describing the part of the graphic image contained in said clipping zone; and displaying said clipping image on the display device.

Optionally, said clipping zone is indicated by a path of the sensing device relative to the substrate.

Optionally, said clipping identity identifies a time at which said clipping is generated.

Optionally, said clipping zone contains at least one interactive element, such that said clipping is an interactive clipping.

Optionally, said at least one interactive element is selected from the group comprising: a hyperlink and an input field.

Optionally, said clipping identity identifies a clipping input description describing said at least one interactive element via said page description and said clipping zone.

Optionally, said clipping further identifies a clipping input service configured for interpreting an interaction with said clipping via a graphical user interface.

Optionally, said clipping further identifies citation information associated with the part of the graphic image contained in said clipping zone.

Optionally, the clipping is represented, at least partially, using an HTML fragment containing an image map.

Optionally, the image map is selected from the group comprising: a server-side image map and a client-side image map.

Optionally, said clipping is represented using a script.

Optionally, the method further comprising at least one of: pasting the clipping into an application, e-mailing the clipping, and pasting the clipping into a blog.

Optionally, the method further comprising the steps of: displaying said clipping; and interacting with said at least one interactive element using GUI-based software supported by an operating system of a display device.

Optionally, the interaction with the at least one interactive element identifies a click location and the clipping identity.

Optionally, the clipping identity and the click location are used to identify the corresponding page description and interpret the interaction.

Optionally, the method further comprising identifying a target URI and displaying a corresponding target resource on a display device.

In a second aspect the present invention provides a method of generating a self-contained interactive clipping from a printed substrate, said substrate having a graphic image and a position-coding pattern disposed thereon, said position-coding pattern identifying an identity and a plurality of locations, said method comprising the steps of: receiving, in a computer system, interaction data indicative of the identity and at least one position of a sensing device relative to the substrate, the sensing device generating the interaction data, when operatively positioned or moved relative to the substrate, by reading at least some of the position-coding pattern; identifying and retrieving at least part of a page description corresponding to the printed substrate using the identity, said page description including a visual description describing the graphic image and an input description describing one or more interactive elements; determining, using the page description and the at least one position of the sensing device, a clipping zone of the substrate indicated by the sensing device, said clipping zone containing a part of the graphic image; generating the clipping using said page description and said clipping zone, said clipping comprising a clipping visual description describing the part of the graphic image contained in said clipping zone and a clipping input description describing at least one interactive element contained in said clipping zone; and copying the clipping to a clipboard.

Optionally, the method further comprising displaying said clipping on a display device.

Optionally, said clipping zone is indicated by a path of the sensing device relative to the substrate.

Optionally, said at least one interactive element is selected from the group comprising: a hyperlink and an input field.

Optionally, the clipping is represented, at least partially, using an HTML fragment containing an image map explicitly hyperlinked to at least one target URI.

Optionally, the image map is selected from the group comprising: a server-side image map and a client-side image map.

Optionally, said clipping is represented using a script.

Optionally, the method further comprising at least one of: pasting the clipping into an application, e-mailing the clipping, and pasting the clipping into a blog.

Optionally, the method further comprising the steps of: displaying said clipping on a display device; and interacting with an interactive element using GUI-based software supported by an operating system of the display device.

Optionally, the interaction with the interactive element identifies a click location and a target URI using the clipping input description contained in the clipping.

Optionally, the method further comprising displaying a target resource corresponding to said target URI on said display device.

Preferred and other embodiments of the invention will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which:

FIG. 1 is a schematic of the relationship between a sample printed netpage and its online page description;

FIG. 2 shows an embodiment of basic netpage architecture with various alternatives for the relay device;

FIG. 3 is a schematic view of a high-level structure of a printed netpage and its online page description;

FIG. 4 is a schematic view of a document and page description class diagram;

FIG. 5 is a schematic view of a document and page ownership class diagram;

FIG. 6 is a schematic view of a terminal element specialization class diagram;

FIG. 7 is a schematic view of a static element specialization class diagram;

FIG. 8 illustrates a collection of netpage servers, Web terminals, printers and relays interconnected via a network;

FIG. 9 is a schematic view of a pen class diagram;

FIG. 10 is a perspective view of a netpage pen and its associated tag-sensing field-of-view cone;

FIG. 11 is a perspective exploded view of the netpage pen shown in FIG. 8;

FIG. 12 is a schematic block diagram of a pen controller for the netpage pen shown in FIGS. 10 and 11;

FIG. 13 is a schematic view of a hyperlink element class diagram;

FIG. 14 is a schematic view of a hyperlink element specialization class diagram;

FIG. 15 is a schematic view of a hyperlinked group class diagram;

FIG. 16 is a schematic view of a form class diagram;

FIG. 17 is a schematic view of a digital ink class diagram;

FIG. 18 is a schematic view of a field element specialization class diagram;

FIG. 19 is a schematic view of a checkbox field class diagram;

FIG. 20 is a schematic view of a text field class diagram;

FIG. 21 is a schematic view of a signature field class diagram;

FIG. 22 is a flowchart of an input processing algorithm;

FIG. 22A is a detailed flowchart of one step of the flowchart of FIG. 22;

FIG. 23 is a schematic view of a hyperlink request class diagram;

FIG. 24 is a schematic view of an article group class diagram;

FIG. 25 is a schematic view of a selection hyperlink class diagram;

FIG. 26 is a schematic view of a selection page server command class diagram;

FIG. 27 is a schematic view of a selection retrieval protocol;

FIG. 28 is a schematic view of a styled text object and formatted textflow fragment class diagram;

FIG. 29 is a schematic view of a page fragment bearing some printed text, with two words of the text selected by circumscription (lasso);

FIG. 30 is a schematic view of a page fragment bearing a set of commands that operate on the current selection;

FIG. 31 is a styled paragraph and formatted text line object diagram;

FIG. 32 is a printed netpage with a clipping stroke;

FIG. 33 is a virtual clipping;

FIG. 34 is an interactive clipping class diagram;

FIG. 35 is a clipping display and click-through interaction diagram;

FIG. 36 is an interactive clipping as shown in a graphical HTML client; and

FIG. 37 is an interactive clipping as shown in a non-graphical HTML client.

The present invention is used in connection with the Applicant's netpage system, which has been described in detail in the cross-referenced patent applications identified above.

In brief summary, the preferred form of the netpage system employs a computer interface in the form of a mapped surface, that is, a physical surface which contains references to a map of the surface maintained in a computer system. The map references can be queried by an appropriate sensing device. Depending upon the specific implementation, the map references may be encoded visibly or invisibly, and defined in such a way that a local query on the mapped surface yields an unambiguous map reference both within the map and among different maps. The computer system can contain information about features on the mapped surface, and such information can be retrieved based on map references supplied by a sensing device used with the mapped surface. The information thus retrieved can take the form of actions which are initiated by the computer system on behalf of the operator in response to the operator's interaction with the surface features.

In its preferred form, the netpage system relies on the production of, and human interaction with, netpages. These are pages of text, graphics and images printed on ordinary paper, but which work like interactive webpages. Information is encoded on each page using ink which is substantially invisible to the unaided human eye. The ink, however, and thereby the coded data, can be sensed by an optically imaging sensing device (“reader”) and transmitted to the netpage system. The sensing device may take the form of, for example, a clicker (for clicking on a specific position on a surface), a pointer having a stylus (for pointing or gesturing on a surface using pointer strokes), or a pen having a marking nib (for marking a surface with ink when pointing, gesturing or writing on the surface). Any references herein to “pen” or “netpage pen” are provided by way of example only. It will, of course, be appreciated that the netpage pen may take the form of any suitable optically imaging sensing device or reader.

In one embodiment, active buttons and hyperlinks on each page can be clicked with the sensing device to request information from the network or to signal preferences to a network server. In one embodiment, text written by hand on a netpage is automatically recognized and converted to computer text in the netpage system, allowing forms to be filled in. In other embodiments, signatures recorded on a netpage are automatically verified, allowing e-commerce transactions to be securely authorized. In other embodiments, text on a netpage may be clicked or gestured to initiate a search based on keywords indicated by the user.

As illustrated in FIG. 1, a printed netpage 1 can represent an interactive form which can be filled in by the user both physically, on the printed page, and “electronically”, via communication between the pen and the netpage system. The example shows a “Request” form containing name and address fields and a submit button. The netpage 1 consists of a graphic impression 2, printed using visible ink, and a surface coding pattern 3 superimposed with the graphic impression. The coding pattern 3 is typically printed with an infrared ink and the superimposed graphic impression 2 is printed with colored ink(s) having a complementary infrared window, allowing infrared imaging of the coding pattern 3.

The surface coding pattern 3 typically takes the form of a grid of target dots, which comprises a collection of tags 4. One such tag 4 is shown in the shaded region of FIG. 1, although it will be appreciated that contiguous tags 4, defined by the coding pattern 3, are densely tiled over the whole netpage 1.

A corresponding page description 5, stored on the netpage network, describes the individual elements of the netpage. In particular it has an input description describing the type and spatial extent (zone) of each interactive element (i.e. text field or button in the example), to allow the netpage system to correctly interpret input via the netpage. The submit button 6, for example, has a zone 7 which corresponds to the spatial extent of the corresponding graphic 8.

As illustrated in FIG. 2, a netpage reader 400 (e.g. netpage pen) works in conjunction with a netpage relay device 601, which is an Internet-connected device for home, office or mobile use. The reader 400 is wireless and communicates securely with the netpage relay device 601 via a short-range radio link 9. In an alternative embodiment, the reader 400 utilizes a wired connection, such as a USB or other serial connection, to the relay device 601.

The relay device 601 performs the basic function of relaying interaction data to a page server 10, which interprets the interaction data. As shown in FIG. 2, the relay device 601 may, for example, take the form of a personal computer 601a, a netpage printer 601b or some other relay 601c (e.g. personal computer or mobile phone incorporating a web browser).

The netpage printer 601b is able to deliver, periodically or on demand, personalized newspapers, magazines, catalogs, brochures and other publications, all printed at high quality as interactive netpages. Unlike a personal computer, the netpage printer is an appliance which can be, for example, wall-mounted adjacent to an area where the morning news is first consumed, such as in a user's kitchen, near a breakfast table, or near the household's point of departure for the day. It also comes in tabletop, desktop, portable and miniature versions. Netpages printed on-demand at their point of consumption combine the ease-of-use of paper with the timeliness and interactivity of an interactive medium.

Alternatively, the netpage relay device 601 may be a portable device, such as a mobile phone or PDA, a laptop or desktop computer, or an information appliance connected to a shared display, such as a TV. If the relay device 601 is not a netpage printer 601b which prints netpages digitally and on demand, the netpages may be printed by traditional analog printing presses, using such techniques as offset lithography, flexography, screen printing, relief printing and rotogravure, as well as by digital printing presses, using techniques such as drop-on-demand inkjet, continuous inkjet, dye transfer, and laser printing. As will be explained in more detail below, the present invention provides an alternative means which enables users conveniently to generate netpages, as well as scan or print images.

As shown in FIG. 2, the netpage reader 400 interacts with a portion of the position-coding tag pattern on a printed netpage 1, or other printed substrate such as a label of a product item 251, and communicates, via a short-range radio link 9, the interaction to the relay device 601. The relay 601 sends corresponding interaction data to the relevant netpage page server 10 for interpretation. Raw data received from the netpage reader 400 may be relayed directly to the page server 10 as interaction data. Alternatively, the interaction data may be encoded in the form of an interaction URI and transmitted to the page server 10 via a user's web browser 601c. The web browser 601c may then receive a URI from the page server 10 and access a webpage via a webserver 201. In some circumstances, the page server 10 may access application computer software running on a netpage application server 13.

The netpage relay device 601 can be configured to support any number of readers 400, and a reader can work with any number of netpage relays. In the preferred implementation, each netpage reader 400 has a unique identifier. This allows each user to maintain a distinct profile with respect to a netpage page server 10 or application server 13.

A netpage pen may be registered with a netpage registration server 11 and linked to one or more payment card accounts. This allows e-commerce payments to be securely authorized using the netpage pen. The netpage registration server compares the signature captured by the netpage pen with a previously registered signature, allowing it to authenticate the user's identity to an e-commerce server. Other biometrics can also be used to verify identity. One version of the netpage pen includes fingerprint scanning, verified in a similar way by the netpage registration server.

Netpages are the foundation on which a netpage network is built. They provide a paper-based user interface to published information and interactive services.

As shown in FIG. 1, a netpage consists of a printed page (or other surface region) invisibly tagged with references to an online description 5 of the page. The online page description 5 is maintained persistently by the netpage page server 10. The page description has a visual description describing the visible layout and content of the page, including text, graphics and images. It also has an input description describing the input elements on the page, including buttons, hyperlinks, and input fields. A netpage allows markings made with a netpage pen on its surface to be simultaneously captured and processed by the netpage system.

Multiple netpages (for example, those printed by analog printing presses) can share the same page description. However, to allow input through otherwise identical pages to be distinguished, each netpage may be assigned a unique page identifier in the form of a page ID (or, more generally, an impression ID). The page ID has sufficient precision to distinguish between a very large number of netpages.

Each reference to the page description 5 is repeatedly encoded in the netpage pattern. Each tag (and/or a collection of contiguous tags) identifies the unique page on which it appears, and thereby indirectly identifies the page description 5. Each tag also identifies its own position on the page. Characteristics of the tags are described in more detail below.

Tags are typically printed in infrared-absorptive ink on any substrate which is infrared-reflective, such as ordinary paper, or in infrared fluorescing ink. Near-infrared wavelengths are invisible to the human eye but are easily sensed by a solid-state image sensor with an appropriate filter.

A tag is sensed by a 2D area image sensor in the netpage reader 400, and the tag data is transmitted to the netpage system via the nearest netpage relay device 601. The reader 400 is wireless and communicates with the netpage relay device 601 via a short-range radio link. It is important that the reader recognize the page ID and position on every interaction with the page, since the interaction is stateless. Tags are error-correctably encoded to make them partially tolerant to surface damage.

The netpage page server 10 maintains a unique page instance for each unique printed netpage, allowing it to maintain a distinct set of user-supplied values for input fields in the page description 5 for each printed netpage 1.

Each tag 4, contained in the position-coding pattern 3, identifies an absolute location of that tag within a region of a substrate.

Each interaction with a netpage should also provide a region identity together with the tag location. In a preferred embodiment, the region to which a tag refers coincides with an entire page, and the region ID is therefore synonymous with the page ID of the page on which the tag appears. In other embodiments, the region to which a tag refers can be an arbitrary subregion of a page or other surface. For example, it can coincide with the zone of an interactive element, in which case the region ID can directly identify the interactive element.

As described in some of the Applicant's previous applications (e.g. U.S. Pat. No. 6,832,717), the region identity may be encoded discretely in each tag 4. As described other of the Applicant's applications (e.g. U.S. application Ser. Nos. 12/025,746 & 12/025,765 filed on Feb. 5, 2008), the region identity may be encoded by a plurality of contiguous tags in such a way that every interaction with the substrate still identifies the region identity, even if a whole tag is not in the field of view of the sensing device.

Each tag 4 should preferably identify an orientation of the tag relative to the substrate on which the tag is printed. Strictly speaking, each tag 4 identifies an orientation of tag data relative to a grid containing the tag data. However, since the grid is typically oriented in alignment with the substrate, then orientation data read from a tag enables the rotation (yaw) of the netpage reader 400 relative to the grid, and thereby the substrate, to be determined.

A tag 4 may also encode one or more flags which relate to the region as a whole or to an individual tag. One or more flag bits may, for example, signal a netpage reader 400 to provide feedback indicative of a function associated with the immediate area of the tag, without the reader having to refer to a corresponding page description 5 for the region. A netpage reader may, for example, illuminate an “active area” LED when positioned in the zone of a hyperlink.

A tag 4 may also encode a digital signature or a fragment thereof. Tags encoding digital signatures (or a part thereof) are useful in applications where it is required to verify a product's authenticity. Such applications are described in, for example, US Publication No. 2007/0108285, the contents of which is herein incorporated by reference. The digital signature may be encoded in such a way that it can be retrieved from every interaction with the substrate. Alternatively, the digital signature may be encoded in such a way that it can be assembled from a random or partial scan of the substrate.

It will, of course, be appreciated that other types of information (e.g. tag size etc) may also be encoded into each tag or a plurality of tags.

For a full description of netpage tags 4, reference is made to the Applicant's previously filed patent applications identified above, the contents of which are herein incorporated by reference (see, for example, U.S. Pat. No. 6,832,717 and U.S. application Ser. Nos. 12/025,746 & 12/025,765 filed on Feb. 5, 2008, the contents of which are herein incorporated by reference).

A preferred embodiment of a document and page description class diagram is shown in FIGS. 4 and 5.

In the netpage system a document is described at three levels. At the most abstract level the document 836 has a hierarchical structure whose terminal elements 839 are associated with content objects 840 such as text objects, text style objects, image objects, etc. Once the document is printed on a printer with a particular page size, the document is paginated and otherwise formatted. Formatted terminal elements 835 will in some cases be associated with content objects which are different from those associated with their corresponding terminal elements, particularly where the content objects are style-related. Each printed instance of a document and page is also described separately, to allow input captured through a particular page instance 830 to be recorded separately from input captured through other instances of the same page description.

The presence of the most abstract document description on the page server allows a a copy of a document to be printed without being forced to accept the source document's specific format. The user or a printing press may be requesting a copy for a printer with a different page size, for example. Conversely, the presence of the formatted document description on the page server allows the page server to efficiently interpret user actions on a particular printed page.

A formatted document 834 consists of a set of formatted page descriptions 5, each of which consists of a set of formatted terminal elements 835. Each formatted element has a spatial extent or zone 58 on the page. This defines the active area of input elements such as hyperlinks and input fields.

A document instance 831 corresponds to a formatted document 834. It consists of a set of page instances 830, each of which corresponds to a page description 5 of the formatted document. Each page instance 830 describes a single unique printed netpage 1, and records the page ID 50 of the netpage. A page instance is not part of a document instance if it represents a copy of a page requested in isolation.

A page instance consists of a set of terminal element instances 832. An element instance only exists if it records instance-specific information. Thus, a hyperlink instance exists for a hyperlink element because it records a transaction ID 55 which is specific to the page instance, and a field instance exists for a field element because it records input specific to the page instance. An element instance does not exist, however, for static elements such as textflows.

A terminal element 839 can be a visual element or an input element. A visual element can be a static element 843 or a dyamnic element 846. An input element can be a hyperlink element 844 or a field element 845, as shown in FIG. 6. A static element 843 can be a style element 847 with an associated style object 854, a textflow element 848 with an associated styled text object 855, an image element 849 with an associated image element 856, a graphic element 850 with an associated graphic object 857, a video clip element 851 with an associated video clip object 858, an audio clip element 852 with an associated audio clip object 859, or a script element 853 with an associated script object 860, as shown in FIG. 7.

A page instance has a background field 833 which is used to record any digital ink captured on the page which does not apply to a specific input element.

In the preferred form of the invention, a tag map 811 is associated with each page instance to allow tags on the page to be translated into locations on the page.

A simple netpage network may consist of a distributed set of netpage page servers 10, netpage registration servers 11, netpage ID servers 12, netpage application servers 13, and netpage relay devices 601 connected via a network 19 such as the Internet, as shown in FIG. 8. Of course, other services may be incorporated into the Netpage network, such as those enabling an interactive clipping service as shown in FIG. 35.

The netpage registration server 11 is a server which records relationships between users, pens, printers and/or applications, and thereby authorizes various network activities. It authenticates users and acts as a signing proxy on behalf of authenticated users in application transactions. It also provides handwriting recognition services. As described above, a netpage page server 10 maintains persistent information about page descriptions and page instances. The netpage network includes any number of page servers, each handling a subset of page instances. Since a page server also maintains user input values for each page instance, clients such as netpage relays 601 send netpage input directly to the appropriate page server. The page server interprets any such input relative to the description of the corresponding page.

A netpage ID server 12 allocates document IDs 51 on demand, and provides load-balancing of page servers via its ID allocation scheme.

A netpage relay 601 uses the Internet Distributed Name System (DNS), or similar, to resolve a netpage page ID 50 into the network address of the netpage page server handling the corresponding page instance.

A netpage application server 13 is a server which hosts interactive netpage applications.

Netpage servers can be hosted on a variety of network server platforms from manufacturers such as IBM, Hewlett-Packard, and Sun. Multiple netpage servers can run concurrently on a single host, and a single server can be distributed over a number of hosts. Some or all of the functionality provided by netpage servers, and in particular the functionality provided by the ID server and the page server, can also be provided directly in a netpage appliance such as a netpage printer, in a computer workstation, or on a local network.

The active sensing device (or reader 400) of the netpage system may take the form of, for example, a clicker (for clicking on a specific position on a surface), a pointer having a stylus (for pointing or gesturing on a surface using pointer strokes), or a pen having a marking nib (for marking a surface with ink when pointing, gesturing or writing on the surface). A pen 101 is described herein, although it will be appreciated that clickers and pointers may be of similar construction. The pen 101 uses its embedded controller 134 to capture and decode netpage tags from a page via an image sensor. The image sensor is a solid-state device provided with an appropriate filter to permit sensing at only near-infrared wavelengths. As described in more detail below, the system is able to sense when the nib is in contact with the surface, and the pen is able to sense tags at a sufficient rate to capture human handwriting (i.e. at 200 dpi or greater and 100 Hz or faster). Information captured by the pen may be encrypted and wirelessly transmitted to the printer (or base station), the printer or base station interpreting the data with respect to the (known) page structure.

One embodiment of the netpage pen 101 operates both as a normal marking ink pen and as a non-marking stylus (i.e. as a pointer). The marking aspect, however, is not necessary for using the netpage system as a browsing system, such as when it is used as an Internet interface. Each netpage pen is registered with the netpage system and has a unique pen ID 61. FIG. 11 shows the netpage pen class diagram, reflecting pen-related information maintained by a registration server 11 on the netpage network.

When the nib is in contact with a netpage, the pen determines its position and orientation relative to the page. The nib is attached to a force sensor, and the force on the nib is interpreted relative to a threshold to indicate whether the pen is “up” or “down”. This allows a interactive element on the page to be ‘clicked’ by pressing with the pen nib, in order to request, say, information from a network. Furthermore, the force is captured as a continuous value to allow, say, the full dynamics of a signature to be verified.

The pen determines the position and orientation of its nib on the netpage by imaging, in the infrared spectrum, an area 193 of the page in the vicinity of the nib. It decodes the nearest tag and computes the position of the nib relative to the tag from the observed perspective distortion on the imaged tag and the known geometry of the pen optics i.e. the spatial relationship between the nib and the optical axis. Although the position resolution of the tag may be low, because the tag density on the page is inversely proportional to the tag size, the adjusted position resolution is quite high, exceeding the minimum resolution required for accurate handwriting recognition.

Pen actions relative to a netpage are captured as a series of strokes. A stroke consists of a sequence of time-stamped pen positions on the page, initiated by a pen-down event and completed by the subsequent pen-up event. A stroke is also tagged with the page ID 50 of the netpage whenever the page ID changes, which, under normal circumstances, is at the commencement of the stroke.

Each netpage pen has a current selection 826 associated with it, allowing the user to perform copy and paste operations etc. The selection is timestamped to allow the system to discard it after a defined time period. The current selection describes a region of a page instance. It consists of the most recent digital ink stroke captured through the pen relative to the background area of the page. It is interpreted in an application-specific manner once it is submitted to an application via a selection hyperlink activation.

Each pen has a current nib 824. This is the nib last notified by the pen to the system. In the case of the default netpage pen described above, either the marking black ink nib or the non-marking stylus nib is current. Each pen also has a current nib style 825. This is the nib style last associated with the pen by an application, e.g. in response to the user selecting a color from a palette. The default nib style is the nib style associated with the current nib. Strokes captured through a pen are tagged with the current nib style. When the strokes are subsequently reproduced, they are reproduced in the nib style with which they are tagged.

The pen 101 may have one or more buttons 209, which are pressed by the user to select a mode of the pen. The button(s) are used to determine a behavior of the pen, which, in turn, determines how a stroke is interpreted by the page server 10. For example, a ‘clipping mode’ may be selected using a button 209 (see Section 7.3).

Whenever the pen is within range of a relay device 601 with which it can communicate, the pen slowly flashes its “online” LED. When the pen fails to decode a stroke relative to the page, it momentarily activates its “error” LED. When the pen succeeds in decoding a stroke relative to the page, it momentarily activates its “ok” LED.

A sequence of captured strokes is referred to as digital ink. Digital ink forms the basis for the digital exchange of drawings and handwriting, for online recognition of handwriting, and for online verification of signatures.

The pen is wireless and transmits digital ink to the relay device 601 via a short-range radio link. The transmitted digital ink is encrypted for privacy and security and packetized for efficient transmission, but is always flushed on a pen-up event to ensure timely handling in the printer.

When the pen is out-of-range of a relay device 601 it buffers digital ink in internal memory, which has a capacity of over ten minutes of continuous handwriting. When the pen is once again within range of a relay device, it transfers any buffered digital ink.

A pen can be registered with any number of relay devices, but because all state data resides in netpages both on paper and on the network, it is largely immaterial which relay device a pen is communicating with at any particular time.

A more detailed description of the Netpage pen may be found in the Applicant's earlier cross-referenced patents and patent applications (see, for example, U.S. Pat. No. 7,105,753; U.S. Pat. No. 7,015,901; U.S. Pat. No. 7,091,960; and US Publication No. 2006/0028459, the contents of which are herein incorporated by reference).

Referring to FIGS. 10 and 11, the pen, generally designated by reference numeral 101, includes a housing 102 in the form of a plastics moulding having walls 103 defining an interior space 104 for mounting the pen components. Mode selector buttons 209 are provided on the housing 102. The pen top 105 is in operation rotatably mounted at one end 106 of the housing 102. A semi-transparent cover 107 is secured to the opposite end 108 of the housing 102. The cover 107 is also of moulded plastics, and is formed from semi-transparent material in order to enable the user to view the status of the LED mounted within the housing 102. The cover 107 includes a main part 109 which substantially surrounds the end 108 of the housing 102 and a projecting portion 110 which projects back from the main part 109 and fits within a corresponding slot 111 formed in the walls 103 of the housing 102. A radio antenna 112 is mounted behind the projecting portion 110, within the housing 102. Screw threads 113 surrounding an aperture 113A on the cover 107 are arranged to receive a metal end piece 114, including corresponding screw threads 115. The metal end piece 114 is removable to enable ink cartridge replacement.

Also mounted within the cover 107 is a tri-color status LED 116 on a flex PCB 117. The antenna 112 is also mounted on the flex PCB 117. The status LED 116 is mounted at the top of the pen 101 for good all-around visibility.

The pen described herein can operate both as a normal marking ink pen and as a non-marking stylus (‘pointer’). An ink pen cartridge 118 with nib 119 and a stylus 120 with stylus nib 121 are mounted side by side within the housing 102. Either the ink cartridge nib 119 or the stylus nib 121 can be brought forward through open end 122 of the metal end piece 114, by rotation of the pen top 105. Respective slider blocks 123 and 124 are mounted to the ink cartridge 118 and stylus 120, respectively. A rotatable cam barrel 125 is secured to the pen top 105 in operation and arranged to rotate therewith. The cam barrel 125 includes a cam 126 in the form of a slot within the walls 181 of the cam barrel. Cam followers 127 and 128 projecting from slider blocks 123 and 124 fit within the cam slot 126. On rotation of the cam barrel 125, the slider blocks 123 or 124 move relative to each other to project either the pen nib 119 or stylus nib 121 out through the hole 122 in the metal end piece 114. The pen 101 has three states of operation. By turning the top 105 through 90° steps, the three states are: Stylus 120 nib 121 out; Ink cartridge 118 nib 119 out; and Neither ink cartridge 118 nib 119 out nor stylus 120 nib 121 out.

A second flex PCB 129, is mounted on an electronics chassis 130 which sits within the housing 102. The second flex PCB 129 mounts an infrared LED 131 for providing infrared radiation for projection onto the surface. An image sensor 132 is provided mounted on the second flex PCB 129 for receiving reflected radiation from the surface. The second flex PCB 129 also mounts a radio frequency chip 133, which includes an RF transmitter and RF receiver, and a controller chip 134 for controlling operation of the pen 101. An optics block 135 (formed from moulded clear plastics) sits within the cover 107 and projects an infrared beam onto the surface and receives images onto the image sensor 132. Power supply wires 136 connect the components on the second flex PCB 129 to battery contacts 137 which are mounted within the cam barrel 125. A terminal 138 connects to the battery contacts 137 and the cam barrel 125. A three volt rechargeable battery 139 sits within the cam barrel 125 in contact with the battery contacts. An contacts. An induction charging coil 140 is mounted about the second flex PCB 129 to enable recharging of the battery 139 via induction. The second flex PCB 129 also mounts an infrared LED 143 and infrared photodiode 144 for detecting displacement in the cam barrel 125 when either the stylus 120 or the ink cartridge 118 is used for writing, in order to enable a determination of the force being applied to the surface by the pen nib 119 or stylus nib 121. The IR photodiode 144 detects light from the IR LED 143 via reflectors (not shown) mounted on the slider blocks 123 and 124.

Rubber grip pads 141 and 142 are provided towards the end 108 of the housing 102 to assist gripping the pen 101, and top 105 also includes a clip 142 for clipping the pen 101 to a pocket.

The pen 101 is arranged to determine the position of its nib (stylus nib 121 or ink cartridge nib 119) by imaging, in the infrared spectrum, an area of the surface in the vicinity of the nib. It records the location data from the nearest location tag, and is arranged to calculate the distance of the nib 121 or 119 from the location tab utilising optics 135 and controller chip 134. The controller chip 134 calculates the orientation of the pen and the nib-to-tag distance from the perspective distortion observed on the imaged tag.

Utilising the RF chip 133 and antenna 112 the pen 101 can transmit the digital ink data (which is encrypted for security and packaged for efficient transmission) to the computing system.

When the pen is in range of a relay device 601, the digital ink data is transmitted as it is formed. When the pen 101 moves out of range, digital ink data is buffered within the pen 101 (the pen 101 circuitry includes a buffer arranged to store digital ink data for approximately 12 minutes of the pen motion on the surface) and can be transmitted later.

The nib ID allows the computing system to identify which nib (stylus nib 121 or ink cartridge nib 119) is presently being used. The computing system can vary its operation depending upon which nib is being used. For example, if the ink cartridge nib 119 is being used the computing system may defer producing feedback output because immediate feedback is provided by the ink markings made on the surface. Where the stylus nib 121 is being used, the computing system may produce immediate feedback output.

Since a user may change the nib 119, 121 between one stroke and the next, the pen 101 optionally records a nib ID for a stroke 175. This becomes the nib ID implicitly associated with later strokes.

Cartridges having particular nib characteristics may be interchangeable in the pen. The pen controller 134 may interrogate a cartridge to obtain the nib ID 175 of the cartridge. The nib ID 175 may be stored in a ROM or a barcode on the cartridge. The controller 134 notifies the system of the nib ID whenever it changes. The system is thereby able to determine the characteristics of the nib used to produce a stroke 175, and is thereby subsequently able to reproduce the characteristics of the stroke itself.

The controller chip 134 is mounted on the second flex PCB 129 in the pen 101. FIG. 10 is a block diagram illustrating in more detail the architecture of the controller chip 134. FIG. 12 also shows representations of the RF chip 133, the image sensor 132, the tri-color status LED 116, the IR illumination LED 131, the IR force sensor LED 143, and the force sensor photodiode 144.

Referring to FIG. 12, the pen controller chip 134 includes a controlling processor 145. Bus 146 enables the exchange of data between components of the controller chip 134. Flash memory 147 and a 512 KB DRAM 148 are also included. An analog-to-digital converter 149 is arranged to convert the analog signal from the force sensor photodiode 144 to a digital signal.

An image sensor interface 152 interfaces with the image sensor 132. A transceiver controller 153 and base band circuit 154 are also included to interface with the RF chip 133 which includes an RF circuit 155 and RF resonators and inductors 156 connected to the antenna 112.

The controlling processor 145 captures and decodes location data from tags from the surface via the image sensor 132, monitors the force sensor photodiode 144, controls the LEDs 116, 131 and 143, and handles short-range radio communication via the radio transceiver 153. It is a medium-performance (˜40 MHz) general-purpose RISC processor. The controlling processor 145 also receives input via mode selection button(s) 209, which can be used to control interpretation of pen-substrate interactions.

The processor 145, digital transceiver components (transceiver controller 153 and baseband circuit 154), image sensor interface 152, flash memory 147 and 512 KB DRAM 148 are integrated in a single controller ASIC. Analog RF components (RF circuit 155 and RF resonators and inductors 156) are provided in the separate RF chip.

The image sensor 132 may be a 215×215 pixel CCD (such a sensor is produced by Matsushita Electronic Corporation, and is described in a paper by Itakura, K T Nobusada, N Okusenya, R Nagayoshi, and M Ozaki, “A 1 mm 50 k-Pixel IT CCD Image Sensor for Miniature Camera System”, IEEE Transactions on Electronic Devices, Volt 47, number 1, January 2000, which is incorporated herein by reference) with an IR filter.

The controller ASIC 134 enters a quiescent state after a period of inactivity when the pen 101 is not in contact with a surface. It incorporates a dedicated circuit 150 which monitors the force sensor photodiode 144 and wakes up the controller 134 via the power manager 151 on a pen-down event.

The radio transceiver communicates in the unlicensed 900 MHz band normally used by cordless telephones, or alternatively in the unlicensed 2.4 GHz industrial, scientific and medical (ISM) band, and uses frequency hopping and collision detection to provide interference-free communication.

In an alternative embodiment, the pen incorporates an Infrared Data Association (IrDA) interface for short-range communication with a base station or netpage printer.

In a further embodiment, the pen 101 includes a pair of orthogonal accelerometers mounted in the normal plane of the pen 101 axis. The accelerometers 190 are shown in FIGS. 11 and 12 in ghost outline.

The provision of the accelerometers enables this embodiment of the pen 101 to sense motion without reference to surface location tags, allowing the location tags to be sampled at a lower rate. Each location tag ID can then identify an object of interest rather than a position on the surface. For example, if the object is a user interface input element (e.g. a command button), then the tag ID of each location tag within the area of the input element can directly identify the input element.

The acceleration measured by the accelerometers in each of the x and y directions is integrated with respect to time to produce an instantaneous velocity and position.

Since the starting position of the stroke is not known, only relative positions within a stroke are calculated. Although position integration accumulates errors in the sensed acceleration, accelerometers typically have high resolution, and the time duration of a stroke, over which errors accumulate, is short.

The netpage relay device 601 receives data relating to a stroke from the pen 101 when the pen is used to interact with a netpage 1. The coded data 3 of the tags 4 is read by the pen when it is used to execute a movement, such as a stroke. The data allows the identity of the particular page to be determined and an indication of the positioning of the pen relative to the page to be obtained. Interaction data comprising the page ID 50 and at least one position of the pen, is transmitted to the relay device 601, where it resolves, via the DNS, the page ID 50 of the stroke into the network address of the netpage page server 10 which maintains the corresponding page instance 830. In one form of the netpage system, the relay device 601 then transmits the stroke to the page server. If the page was recently identified in an earlier stroke, then the relay device may already have the address of the relevant page server in its cache. Each netpage consists of a compact page layout maintained persistently by a netpage page server (see below). The page layout refers to objects such as images, fonts and pieces of text, typically stored elsewhere on the netpage network.

When the page server 10 receives the stroke from the pen, it retrieves the page description to which the stroke applies, and determines which element of the page description the stroke intersects. It is then able to interpret the stroke in the context of the type of the relevant element.

A “click” is a stroke where the distance and time between the pen down position and the subsequent pen up position are both less than some small maximum. An object which is activated by a click typically requires a click to be activated, and accordingly, a longer stroke is ignored. The failure of a pen action, such as a “sloppy” click, to register may be indicated by the lack of response from the pen's “ok” LED.

There are two kinds of input elements in a netpage page description: hyperlinks and form fields. Input through a form field can also trigger the activation of an associated hyperlink.

A hyperlink is a means of sending a message to a remote application, and typically elicits a displayed or printed response in the netpage system.

A hyperlink element 844 identifies the application 71 which handles activation of the hyperlink, a link ID 54 which identifies the hyperlink to the application, an “alias required” flag which asks the system to include the user's application alias ID 65 in the hyperlink activation, and a description which is used when the hyperlink is recorded as a favorite or appears in the user's history. The hyperlink element class diagram is shown in FIG. 13.

When a hyperlink is activated, the page server sends a request to an application somewhere on the network. The application is identified by an application ID 64, and the application ID is resolved in the normal way via the DNS. There are three types of hyperlinks: general hyperlinks 863, form hyperlinks 865, and selection hyperlinks 864, as shown in FIG. 14. A general hyperlink can implement a request for a linked document, or may simply signal a preference to a server. A form hyperlink submits the corresponding form to the application. A selection hyperlink submits the current selection to the application. If the current selection contains a single-word piece of text, for example, the application may return a single-page document giving the word's meaning within the context in which it appears, or a translation into a different language. Each hyperlink type is characterized by what information is submitted to the application.

The corresponding hyperlink instance 862 records a transaction ID 55 which can be specific to the page instance on which the hyperlink instance appears. The transaction ID can identify user-specific data to the application, for example a “shopping cart” of pending purchases maintained by a purchasing application on behalf of the user.

The system includes the pen's current selection 826 in a selection hyperlink activation. The system includes the content of the associated form instance 868 in a form hyperlink activation, although if the hyperlink has its “submit delta” attribute set, only input since the last form submission is included. The system includes an effective return path in all hyperlink activations.

A hyperlinked group 866 is a group element 838 which has an associated hyperlink, as shown in FIG. 15. When input occurs through any field element in the group, the hyperlink 844 associated with the group is activated. A hyperlinked group can be used to associate hyperlink behavior with a field such as a checkbox. It can also be used, in conjunction with the “submit delta” attribute of a form hyperlink, to provide continuous input to an application. It can therefore be used to support a “blackboard” interaction model, i.e. where input is captured and therefore shared as soon as it occurs.

A form defines a collection of related input fields used to capture a related set of inputs through a printed netpage. A form allows a user to submit one or more parameters to an application software program running on a server.

A form 867 is a group element 838 in the document hierarchy. It ultimately contains a set of terminal field elements 839. A form instance 868 represents a printed instance of a form. It consists of a set of field instances 870 which correspond to the field elements 845 of the form. Each field instance has an associated value 871, whose type depends on the type of the corresponding field element. Each field value records input through a particular printed form instance, i.e. through one or more printed netpages. The form class diagram is shown in FIG. 16.

Each form instance has a status 872 which indicates whether the form is active, frozen, submitted, void or expired. A form is active when first printed. A form becomes frozen once it is signed or once its freeze time is reached. A form becomes submitted once one of its submission hyperlinks has been activated, unless the hyperlink has its “submit delta” attribute set. A form becomes void when the user invokes a void form, reset form or duplicate form page command. A form expires when its specified expiry time is reached, i.e. when the time the form has been active exceeds the form's specified lifetime. While the form is active, form input is allowed. Input through a form which is not active is instead captured in the background field 833 of the relevant page instance. When the form is active or frozen, form submission is allowed. Any attempt to submit a form when the form is not active or frozen is rejected, and instead elicits an form status report.

Each form instance is associated (at 59) with any form instances derived from it, thus providing a version history. This allows all but the latest version of a form in a particular time period to be excluded from a search.

All input is captured as digital ink. Digital ink 873 consists of a set of timestamped stroke groups 874, each of which consists of a set of styled strokes 875. Each stroke consists of a set of timestamped pen positions 876, each of which also includes pen orientation and nib force. The digital ink class diagram is shown in FIG. 17.

A field element 845 can be a checkbox field 877, a text field 878, a drawing field 879, or a signature field 880. The field element class diagram is shown in FIG. 18. Any digital ink captured in a field's zone 58 is assigned to the field.

A checkbox field has an associated boolean value 881, as shown in FIG. 19. Any mark (a tick, a cross, a stroke, a fill zigzag, etc.) captured in a checkbox field's zone causes a true value to be assigned to the field's value.

A text field has an associated text value 882, as shown in FIG. 20. Any digital ink captured in a text field's zone is automatically converted to text via online handwriting recognition, and the text is assigned to the field's value. Online handwriting recognition is well-understood (see, for example, Tappert, C., C. Y. Suen and T. Wakahara, “The State of the Art in On-Line Handwriting Recognition”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 12, No. 8, August 1990, the contents of which are herein incorporated by cross-reference).

A signature field has an associated digital signature value 883, as shown in FIG. 21. Any digital ink captured in a signature field's zone is automatically verified with respect to the identity of the owner of the pen, and a digital signature of the content of the form of which the field is part is generated and assigned to the field's value. The digital signature is generated using the pen user's private signature key specific to the application which owns the form. Online signature verification is well-understood (see, for example, Plamondon, R. and G. Lorette, “Automatic Signature Verification and Writer Identification—The State of the Art”, Pattern Recognition, Vol. 22, No. 2, 1989, the contents of which are herein incorporated by cross-reference).

A field element is hidden if its “hidden” attribute is set. A hidden field element does not have an input zone on a page and does not accept input. It can have an associated field value which is included in the form data when the form containing the field is submitted.

“Editing” commands, such as strike-throughs indicating deletion, can also be recognized in form fields.

Because the handwriting recognition algorithm works “online” (i.e. with access to the dynamics of the pen movement), rather than “offline” (i.e. with access only to a bitmap of pen markings), it can recognize run-on discretely-written characters with relatively high accuracy, without a writer-dependent training phase. A writer-dependent model of handwriting is automatically generated over time, however, and can be generated up-front if necessary,

Digital ink, as already stated, consists of a sequence of strokes. Any stroke which starts in a particular element's zone is appended to that element's digital ink stream, ready for interpretation. Any stroke not appended to an object's digital ink stream is appended to the background field's digital ink stream.

Digital ink captured in the background field is interpreted as a selection gesture. Circumscription of one or more objects is generally interpreted as a selection of the circumscribed objects, although the actual interpretation is application-specific.

Table 1 summarises some typical pen interactions with a netpage.

TABLE 1 Summary of some typical pen interactions with a netpage Object Type Pen input Action Hyperlink General Click Submit action to application Form Click Submit form to application Selection Click Submit selection to application Form field Checkbox Any mark Assign true to field Text Handwriting Convert digital ink to text; assign text to field Drawing Digital ink Assign digital ink to field Signature Signature Verify digital ink signature; generate digital signature of form; assign digital signature to field None — Circumscription Assign digital ink to current selection

The system maintains a current selection for each pen. The selection consists simply of the most recent stroke captured in the background field. The selection is cleared after an inactivity timeout to ensure predictable behavior.

The raw digital ink captured in every field is retained on the netpage page server and is optionally transmitted with the form data when the form is submitted to the application. This allows the application to interrogate the raw digital ink should it suspect the original conversion, such as the conversion of handwritten text. This can, for example, involve human intervention at the application level for forms which fail certain application-specific consistency checks. As an extension to this, the entire background area of a form can be designated as a drawing field. The application can then decide, on the basis of the presence of digital ink outside the explicit fields of the form, to route the form to a human operator, on the assumption that the user may have indicated amendments to the filled-in fields outside of those fields.

FIG. 22 shows a flowchart of the process of handling pen input relative to a netpage. The process consists of receiving (at 884) a stroke from the pen; identifying (at 885) the page instance 830 to which the page ID 50 in the stroke refers; retrieving (at 886) the page description 5; identifying (at 887) a formatted element 839 whose zone 58 the stroke intersects; determining (at 888) whether the formatted element corresponds to a field element, and if so appending (at 892) the received stroke to the digital ink of the field value 871, interpreting (at 893) the accumulated digital ink of the field, and determining (at 894) whether the field is part of a hyperlinked group 866 and if so activating (at 895) the associated hyperlink; alternatively determining (at 889) whether the formatted element corresponds to a hyperlink element and if so activating (at 895) the corresponding hyperlink; alternatively, in the absence of an input field or hyperlink, appending (at 890) the received stroke to the digital ink of the background field 833; and copying (at 891) the received stroke to the current selection 826 of the current pen, as maintained by the registration server.

FIG. 22A shows a detailed flowchart of step 893 in the process shown in FIG. 22, where the accumulated digital ink of a field is interpreted according to the type of the field. The process consists of determining (at 896) whether the field is a checkbox and (at 897) whether the digital ink represents a checkmark, and if so assigning (at 898) a true value to the field value; alternatively determining (at 899) whether the field is a text field and if so converting (at 900) the digital ink to computer text, with the help of the appropriate registration server, and assigning (at 901) the converted computer text to the field value; alternatively determining (at 902) whether the field is a signature field and if so verifying (at 903) the digital ink as the signature of the pen's owner, with the help of the appropriate registration server, creating (at 904) a digital signature of the contents of the corresponding form, also with the help of the registration server and using the pen owner's private signature key relating to the corresponding application, and assigning (at 905) the digital signature to the field value.

The netpage system provides a mechanism to allow users to select an object on a printed netpage and submit it to an application, e.g. to associate the selected object with another object in the netpage system.

In one preferred embodiment, the selection mechanism is effected by circumscribing the graphical representation of an object using the netpage pen. The user may then submit the selected object to an application by activating a selection hyperlink via the same or another printed netpage.

The registration server maintains a current selection for each pen, describing a region of a page instance from which the selection has been made. This description includes the most recent digital ink stroke captured by way of transmitted signals from the pen relative to the background area of the page. Pen strokes are interpreted in an application-specific manner once they are submitted to an application via a selection hyperlink activation. When the user ‘clicks’ on a selection hyperlink, the page server obtains the pen's current selection from the registration server and transmits it to the corresponding application as part of the selection hyperlink activation, thus associating the two objects.

When the application receives a selection hyperlink activation, it retrieves the content of the selection from the page server which manages the page from which the selection was made. The application may then retrieve the selection as formatted data, allowing it to interpret the object in the form of the selected region in an application-specific manner in relation to the formatted data, or as unformatted data, allowing it to rely on the page server to interpret the selection region in a meaningful way.

When requesting unformatted data, the application may specify a desired scope to assist the interpretation of the selected region by the page server. Possible scopes include letter, word, phrase, line, paragraph and article. If the page server is unable to interpret the selection region region according to the desired scope, it may reject the application's request for the content of the selection.

The selection content returned by the page server to the application may include field values. Typically however, only field values which have been submitted as part of a form submission are included.

An author of a document can assist selection of articles by grouping all the elements of an article into an article group 507, as shown in FIG. 31. If the application specifies an article as the scope in its selection request, the page server attempts to find an article group related to the selection region.

The protocol for selection hyperlink activation and subsequent selection content retrieval, illustrated diagrammatically in FIG. 27, operates as follows.

When page server A receives a selection hyperlink ‘click’, it retrieves 510 the current selection associated with the pen from the registration server. The selection is described by a page ID 511 and a region 512. The page server then constructs a selection hyperlink request 508 (i.e. a specialization of the hyperlink request 934, shown in FIG. 23). The selection hyperlink request contains the pen's current selection as shown in FIG. 25. The page server then transmits its own server ID, the hyperlink request ID, and the link ID to the application in the usual way. It also sends the pen's current selection to the application. The application uses the page ID in the current selection to identify the page server managing the page on which the selection was made (ie. page server A). It then requests the content of the selection from this page server in the desired format 514 and according to the desired scope 513. Page server B uses the server ID and hyperlink request ID supplied by the application to obtain the selection from original page server A. Page server B obtains the selection from page server A rather than from the application in order to ensure that the application does not modify the selection to obtain information not intended by the user.

As an alternative, page server A could sign the selection sent to the application, allowing page server B to easily verify that the selection supplied by the application has not been modified. Once page server B has the selection, it retrieves the selected page from its database and determines the content of the selection according to the application's desired format and scope. Finally the page server returns the selection content 515 to the application for application-specific processing.

If the page on which the selection was made was generated by the same application as that handling the selection hyperlink activation, then the application has direct access to the page (i.e. the application can retrieve the entire page from the relevant page server by way of the page ID of the page) and may already have done so before the selection hyperlink is activated. In this case, the application preferably interprets the selection region without reference to the selection content retrieval mechanism.

Selection of an object may of course alternatively be performed using a draggable command. This has an advantage in that a selection stroke can be distinguished from a normal input stroke wherever the stroke is made. The user may therefore select an object without inadvertently entering an input stroke into a field. As such, a user is also able to select an object that resides entirely within a field relatively easily.

The application may define the selection command in the form of a selection page server command 509, as shown in FIG. 26. This may be placed in a standard location on all netpages, to provide consistent support for selection in much the same way that there is consistent support for page duplication.

When a selection page server command is activated by a user, the page server forwards the corresponding stroke to the registration server to be recorded as the current selection for the pen. Apart from this difference, the selection mechanism operates in the same way as previously described.

As an example, in a netpage e-mail application, clicking on an <attach> button at the bottom of every e-mail composition page effects attachment of the current selection at the current end of the body of the e-mail. The attachment can consist of any object or objects, which are capable of selection, on any netpage page. The user may have made this selection from, say, selecting text (eg. word/sentence/paragraph/article) from another netpage page, or selecting a photograph (eg. by circumscription) from a photo collection page. The entire e-mail may then be reprinted with the attachment included, additional pages being automatically added to the e-mail to accommodate the attachment.

FIG. 28 shows the styled text object 855 of FIG. 7 decomposed into a set of styled paragraphs 1012, styled words 1014 and styled characters 1016. Styling may be applied at any level. In the absence of styling at a particular level styling is inherited from the parent level. Each styled character 1016 has an associated character code. This may utilize the Unicode encoding scheme or another encoding scheme.

A formatted element 835 associated with a textflow element 848 is a formatted textflow fragment 1018. The formatted textflow fragment consists of a set of formatted text lines 1020. Each formatted text line has a spatial extent or zone on the page. Each formatted text line consists of a set of formatted word fragments 1022, each of which has its own zone. The zone of formatted text line is the union of the zones of its word fragments. Each formatted word fragment is associated with a styled word 1014. Where a styled word is broken across multiple lines it has multiple formatted word fragments. Where the entire styled word lies within a single line it has a single formatted word fragment. Each formatted word fragment consists of a set of formatted characters 1024, each of which has a zone and is associated with a styled character. The zone of a formatted word fragment is the union of the zones of its formatted characters.

FIG. 29 shows a fragment 1030 of a printed page bearing a single two-line paragraph of text. The words (“in Spain”) are shown circumscribed by a stroke 1032 of a netpage pen.

FIG. 30 shows a fragment 1034 of a printed page including a set of command buttons 1036, 1038, 1040, 1042, 1044 and 1046 designed to operate on the current selection. It also includes a text field 1048 for entering an arbitrary text string.

FIG. 31 shows part of the object diagram corresponding to the styled text of FIG. 29. The styled text consists of a single styled paragraph 1012a, the first two styled words 1014a and 1014b of which are shown. Because the words are unbroken, they consist of one formatted word fragment each, 1022a and 1022b respectively. The two styled words each consists of a set of styled characters 1016a-c and 1016d-g respectively. The two formatted word fragments each consists of a set of formatted characters 1022a-c and 1022d-g respectively. As shown in the Figure, formatted character 1022d corresponds to the initial letter “r” of “rain”.

The user can utilize a number of different pen gestures to effect text selection. A click on a word can be used to select that word. The position of the click is compared with the zone of each word, and the word whose zone it lies within is selected. Circumscription (or lassoing) can be used to select of one or more words. The degree of overlap between the region enclosed by the circumscribing stroke and the zone of each word is used to determine whether the word is selected. The required overlap can be configured by the user. Underlining can be used to select one or more words. A line-like stroke which doesn't otherwise select any words is interpreted as an underline. Since the orientation of the text is known, the text which lies above the underlining stroke can be identified.

As described previously, the current selection is available to an application which is the target of a selection hyperlink 864. When an application receives a selection hyperlink activation, it can request the current selection.

A number of useful applications can utilize text selections. The selected text can be copied to the clipboard of the user's graphical user interface (GUI) operating system (e.g. Microsoft Windows, Apple Macintosh OS X, or Linux) for use by other GUI applications. FIG. 30 shows a <copy text> command button 1038 which causes the currently selected text to be copied to the clipboard. Variants of this function can be provided, such as <copy & paste text> which copies and then pastes the selected text into the currently active GUI application. Under Microsoft Windows this can usually be effected by inserting a control-V keystroke into the keystroke input queue. It can also be effected using Windows Automation, by identifying and invoking the current application's Paste method.

In addition to utilizing an explicit <copy text> command, the user can operate in a mode where all strokes, if interpretable as selection strokes, cause the selected text to be automatically copied to the clipboard of the GUI operating system. The user can also cause such copying to occur by utilizing one or more specific gestures, such as tapping on a word twice, or circumscribing a set of words twice, or writing the letter “C” and then circling it.

FIG. 30 shows a <search> command button 1040 which causes the currently selected text to be used in a search, such as a search of the World-wide Web using a search engine such as Google or Yahoo!. Invoking a Web search engine consists of constructing a URI which identifies the search engine and includes the selected text as a parameter, and then issuing an HTTP GET or POST using that URI. For example, in the case of Google it's a matter of appending the selected text to “http://www.google.com/search?q=”, separating multiple words by “+”, and optionally enclosing multiple words in quotes (“% 22”) to cause them to be interpreted as a single phrase. The result of the search is either displayed on the user's desktop or mobile computer, on the user's mobile phone or PDA, or is printed on a netpage printer, all according to the user's configured preference. The search command may also be linked to a search of the user's personal file system.

FIG. 30 shows a <dictionary> command button 1042 which causes the currently selected text to be used in a dictionary lookup, such as a lookup using a Web-based dictionary. The result is either displayed or printed according to the user's configured preference.

FIG. 30 shows an <encyclopedia> command button 1044 which causes the currently selected text to be used in an encyclopedia lookup, such as a lookup using a Web-based encyclopedia such as wikipedia. The result is either displayed or printed according to the user's configured preference.

FIG. 30 shows a <translate> command button 1046 which causes the currently selected text to be translated into the user's preferred (or previously specified) language. The language of selected text is often known from the resource description 842 associated with the styled text object 855. Where the document 836 is derived from a Webpage, the language is commonly known. The result is either displayed, printed or output as sampled or synthesized audio according to the user's configured preference.

Each command button 1038 through 1046 is implemented as a selection hyperlink.

When user invokes a selection hyperlink which causes the target application to request the currently selected text, it is possible for the most recent strokes entered by the user to lie in a text field. As a configurable option (and as a per-request option) the system delivers the corresponding recognized text in place of any previously selected text.

FIG. 30 shows a text field 1048. This is provided for the convenience of the user to allow the user to manually enter text to be processed in the same way as a text selection, as described in the previous paragraph.

The entire collection of command buttons shown in FIG. 30, as well as the text field 1048, can be provided for the user's convenience on a separate “command card”, or can be optionally printed as a “command bar” at the bottom of every page.

As described earlier in relation to the <copy text> command, the user can also invoke these commands by utilizing a specific gestures, such as writing a corresponding letter and then circling it, e.g. “C” for <copy>, “S” for <search>, “D” for <dictionary>, “E” for <encyclopedia>, and “T” for <translate>.

An application which is a target of a selection hyperlink can query the content of the most recent selection, and can specify the format that the selection should be returned in. For example, the application can use the selection to retrieve text (a word, multiple words, a line, multiple lines, a paragraph, an article, etc.), an object (such as any of the object types listed in FIG. 15), a page, or the entire document. In the latter case the Netpage system can export the document as a self-contained file, or provide a portable reference to it (with the unique document ID or document instance ID encoded in a URI).

The Netpage system also allows a user to take a virtual clipping from a printed page. A virtual clipping appears visually as a fragment of the printed page, bounded by the hand-drawn stroke used to circumscribe the clipping. The clipping is copied to the clipboard of the operating system of the user's display device, e.g. the user's PC, PDA or mobile phone. The user can subsequently paste the clipping into another application or document, such as a word processing document, an e-mail, or a blog. A clipping may include, by default, any digital ink strokes on the original page.

FIG. 32 shows a printed page with a clipping stroke. FIG. 33 shows the virtual clipping corresponding to the clipping stroke.

The application may also request a “clipping”, i.e. graphic page content clipped to the most recent selection stroke. A clipping optionally remains active, i.e. hyperlinks and/or fields embedded in the page become part of the clipping, e.g. encoded in HTML (see Section 7.4). A clipping includes, by default, any digital ink strokes on the original page.

Using a non-marking Netpage pointer the user can take a clipping using a single clipping stroke. Using a marking Netpage pen the user can identify a stroke as a clipping stroke via a subsequent action, such as clicking on a ‘copy’ command on command card or selecting a ‘copy’ command from the option menu. FIG. 30 shows a <copy image> command button 1036. It causes the currently selected image or clipping to be copied to the clipboard of the GUI operating system for use by other GUI applications. The user may, for example, select an image by tapping it, or the user may select a clipping by circumscribing it.

Many other commands which copy objects to the clipboard of the GUI operating system are possible. It is also possible to provide a generic <copy> command which causes the current selection to be advertised on the clipboard in multiple formats (e.g. plain text, rich text, HTML, image, document, or document reference). The GUI application which retrieves the selection from the clipboard can then allow the user to select the desired format. Most GUI operating systems allow a form of “lazy” copying to the clipboard, where the copying application initially only copies an object reference to the clipboard, and only copies the object to the clipboard in its final format once it is notified that an application has attempted to retrieve the object from the clipboard.

An interactive clipping retains the interactivity of the original printed page. Any user interaction with the clipping is interpreted in much the same way as a corresponding user interaction with the original printed page. At a minimum an interactive clipping supports explicit and implicit hyperlinks. Depending on implementation, it may also support gestural commands and the context-sensitive option menu.

An interactive clipping provides the user with greater utility than a passive clipping, and allows the publisher of the original page to obtain value from user interactions with the clipping. A printed Netpage publication is an extension to the publisher's web presence, and each interactive clipping further extends this web presence.

As shown in FIG. 34, an interactive clipping 310 has an associated clipping image 312, a clipping input service 314, clipping information 316, and an optional citation 318. The clipping image 312 provides the visual appearance of the clipping. The clipping input service 314 interprets user interactions with the clipping. The clipping information 316 identifies the printed page instance (i.e. impression) that the clipping was taken from, and the offset between the clipping and the page. The clipping information 316 allows the clipping input service to interpret user input received via the clipping against the page description 5 associated with the impression. The citation 318 identifies the publication, issue and page number that the clipping was taken from, if known, and provides a link to the publication's web presence.

In order to minimize the size of the clipping 310, the clipping image 312 is stored separately and may be shared by a number of clippings. The clipping image 312 may be stored explicitly, or it may be generated from the original impression on demand. For the purpose of generating the clipping image 312 from the impression, the clipping information needs to record the entire clipping path rather than just the offset between the clipping and the impression, as well as the time the clipping took place to allow digital ink (etc.) added to the impression after the clipping took place to be ignored.

The interactive clipping is represented using an HTML fragment containing a server-side image map (see W3C, Server-Side Image Maps, HTML 4.01 Specification, W3C Recommendation 24 Dec. 1999, http://www.w3.org/TR/html1401/struct/objects.html#h-13.6.2):

<A href=“http://www.example.com/cgi-bin/inputService?id=1234”> <IMG src=“http://www.example.com/clippings/id1234.gif” ismap></A>

Using HTML allows the interactive clipping to be supported by popular word processors, e-mail programs and blogs, as well as the clipboards of popular operating systems.

For illustrative purposes in the example the clipping ID is 1234. The anchor element identifies the clipping input service, and the id parameter in the input service URI identifies the clipping. The image element identifies the clipping image and the fact that it's an image map.

A click on the clipping image generates a URI of the following form: http://www.example.com/cgi-bin/inputService?id=1234?25,37

For illustrative purposes in the example the click coordinates are (25,37). Note that the syntax of the parameters in the URI is not strictly conformant, since HTML does not define a way to combine image-map ‘query’ parameters with other query parameters. However, the indicated syntax is generally supported.

FIG. 35 shows the interaction between an HTML/HTTP client 330 such as a web browser, the Netpage clipping image web service 332, the Netpage clipping input web service 334, the Netpage document service 336 (e.g. the page server 10), and a target web service 338. The HTML/HTTP client 330 is typically the relay device 601 which receives interaction data from the reader 400, and may take the form of a web-enabled laptop computer, mobile phone, PDA etc.

When displaying a clipping, the client uses a clipping image URI to retrieve the corresponding clipping image 312 from the Netpage clipping image web service 332. The image service 332 may in turn rely on the Netpage document service 336 to render the clipping image from the original impression. In this case the image service 332 extracts the clipping ID from the clipping image URI and passes it to the Netpage document service 336.

When handling a click on the clipping image 312, the client 330 generates a URI for the Netpage clipping input web service 334, including the clipping ID and click location. The input service 334 in turn relies on the Netpage document service 336 to identify the impression from the clipping ID, and interpret the click location relative to the impression (suitably adjusted according to the offset between the impression origin and clipping origin). The Netpage document service 336 returns the URI of the final target of the click, and the input service 334 returns this to the client as a redirect. The client uses the target URI to retrieve and display the target web page (or other web resource) from the target web service 338.

Although the Netpage image web service 332, input web service 334 and document service 336 are shown here as separate services, they may be implemented as multiple interfaces to a single service (e.g. the page server 10).

Server-side image maps provide maximum flexibility, since they allow the exact click location to be interpreted against the original impression's page description. However, some users object to server-side image maps because they don't reveal a click's target URI. Client-side image maps may be used in place of server-side image maps with the added complication that implicit hyperlinks, e.g. associated with images and text, must be made explicit in the HTML of the interactive clipping.

An impression may bear digital ink and other attached objects (e.g. photos). When the user takes a clipping it can be automatically placed on the clipboard in multiple formats, including variants that include and exclude digital ink (and other objects). Variants can also be included that are passive rather than interactive. The user can then use the ‘paste special’ function provided in most applications to select an appropriate variant of the clipping.

An image-map only supports clicks. To support more complex gestures, such as swipe to search or access to a contextual option menu, an interactive clipping can be represented using a script (i.e. widget-style). However, this makes it less compatible with extant HTML clients, either due to their implementation or their security policies.

Where the citation for the clipping is known, this should be displayed below the clipping in the form of an anchor linking to the publication's web presence, as shown in FIG. 36.

For non-graphical HTML clients it is important for the user to have a way of viewing and interacting with the clipping. This can be provided via a separate anchor, with a distinct URI, included after the image map anchor. This is preferable to using the alternate text attribute of the image-map image element, since that generates an image-map click with a location of (0,0).

The non-graphical anchor should point to a web page that displays the clipping and allows interactions with it. Ideally this web page will also support script-based access to the contextual option menu.

FIG. 37 shows the appearance of the interactive clipping in a non-graphical HTML client.

The present invention has been described with reference to a preferred embodiment and number of specific alternative embodiments. However, it will be appreciated by those skilled in the relevant fields that a number of other embodiments, differing from those specifically described, will also fall within the spirit and scope of the present invention. Accordingly, it will be understood that the invention is not intended to be limited to the specific embodiments described in the present specification, including documents incorporated by cross-reference as appropriate. The scope of the invention is only limited by the attached claims.