CROSS REFERENCE TO RELATED APPLICATIONS
This application is related to the following commonly owned and co-pending patent applications, all filed concurrently herewith, and identified below by their titles and attorney docket numbers:
“System for Annotating Documents Served by a Document System without Functional Dependence on the Document System,” Attorney Docket Number G0006-1001;
“System for Creating and Editing Temporal Annotations of Documents,” Attorney Docket Number G0006-1002; and
“System for Programmatically Accessing Document Annotations,” Attorney Docket Number G0006-1003.
Massive computer networks, ready access to such communications by consumers and businesses alike, and the prevalence of personal computing devices have made access to multimedia documents widespread. Computers supporting graphical user interfaces and audio have made web pages incorporating multi-media content a common form of document and web pages represent a common interface for computer programs. Gigantic numbers of online multimedia documents constitute a massive amount of content accessible to hundreds of millions of users. Currently however, as is well understood by those skilled in the art, whilst the user may view such online documents, and thereby passively consume such content, the user's ability to actively engage with such content is extremely limited. There has been a trend for site publishers to encourage user engagement by such means as blogs, forums and bulletin boards and such means have indeed resulting in a massive explosion of user generated content but such attempts to go beyond the passive consumption of packaged content do not represent a powerful general model for interaction with content because the user's ability to comment on content in such cases is limited, typically constrained to features provided by the web site implementation and authorised by the website publisher. Such limitations dictate which content can be commented on at what time and in what way. The general definition of ‘annotation’ is extra information associated with a particular point in a document and hence we note that such limited user interaction with online content constitutes a weak form of annotation—the forms of extra information are limited, the points that may be annotated are limited, and the documents that may be annotated are limited.
- Top of Page
In one embodiment, an annotation system includes first annotation data for annotating a manifestation of a first instance of a first XML document. The first instance of the first XML document is served by a document system. The first annotation data includes a first XML document identifier for the first XML document. The first annotation data is uniquely identified by a first annotation data identifier. The document system is configured to respond to a request containing the first XML document identifier with a manifestation of a second instance of the first XML document. A method for use with the annotation system includes: (A) receiving a request containing the first annotation data identifier; (B) issuing a request to the document system, wherein the request contains the first XML document identifier; and (C) in response to the request containing the first annotation data identifier, manifesting the first annotation data in connection with a manifestation of a second instance of the XML document.
The first data may include first annotation data representing a first association between: (a) a first manifestation of first annotation content in connection with a manifestation of the first instance of the first XML document and (b) a manifestation of a first instance of a first target in the manifestation of the first instance of a first XML document. The first UALI may include a first Unique Annotation Identifier (UAI) that uniquely identifies the first annotation data. Operation (A) may include receiving a request containing the first UAI, and operation (C) may include, in response to the request containing the first UAI, manifesting the first annotation data in connection with a manifestation of a second instance of the first XML document. The first annotation data may include first annotation content data representing the first annotation content, first target locating data for locating the second instance of the first target, and first association data representing the first association.
The first data may include first annotation set data representing a first annotation set. The first UALI may include a first Unique Set Identifier (USI) that uniquely identifies the first annotation set. Operation (A) may include receiving a request containing the first USI, and operation (C) may include, in response to the request containing the first USI, manifesting the first annotation set data in connection with a manifestation of a second instance of the first XML document. The first annotation set may further contain second annotation data for annotating the first instance of the first XML document; and operation (C) may further include, in response to the request containing the first USI, manifesting the second annotation data in connection with the manifestation of the second instance of the first XML document. The annotation set data may further include second annotation data for annotating the first instance of the first XML document; and operation (A) may further include, in response to activation of the hyperlink, manifesting the second annotation data in connection with the manifestation of the second instance of the first XML document.
Operation (A) may include: (A)(1) at a client device: (A)(1)(a) receiving input containing the first UALI; (A)(1)(b) retrieving the first data from storage; (A)(1)(c) requesting the second instance of the first XML document from the document server; and (A)(1)(d) manifesting the first data in connection with the manifestation of the second instance of the first XML document. Alternatively, operation (A) may include: (A)(1) at a client device: (A)(1)(a) receiving input containing the first UALI; (A)(1)(b) transmitting a request containing the first UALI to an annotation server; (A)(2) at the annotation server (A)(2)(a) requesting the second instance of the first XML document from the document server; (A)(2)(b) creating a composite of the first data and the second instance of the first XML document; (A)(2)(c) transmitting the composite to the client device; and (A)(3) at the client device: (A)(3)(a) manifesting the composite. Alternatively, operation (A) may include: (A)(1) at a client device: (A)(1)(a) receiving input containing the first UALI; (A)(1)(b) transmitting a request containing the first UALI to an annotation server; (A)(1)(c) requesting the second instance of the first XML document from the document server; (A)(1)(d) transmitting the first data and the second instance of the first XML document to the annotation server; (A)(2) at the annotation server: (A)(2)(a) creating a composite of the first data and the first XML document; (A)(2)(b) transmitting the composite to the client device; and (A)(3) at the client device: (A)(3)(a) manifesting the composite.
The first annotation data may represent a first association between: (a) a first manifestation of first annotation content in connection with a manifestation of the first instance of the first XML document and (b) a manifestation of a first instance of a first target in the manifestation of the first instance of a first XML document. The first annotation content may include first manifestation data, and the first manifestation of the first annotation content may be a manifestation of the first manifestation data.
The first annotation data may represent a first association between: (a) a first manifestation of first annotation content in connection with a manifestation of the first instance of the first XML document and (b) a manifestation of a first instance of a first target in the manifestation of the first instance of a first XML document, wherein the first association comprises a spatial association between the first manifestation of the first annotation content and the manifestation of the first instance of the first target. The first association may be a temporal association and/or a spatial association between the first manifestation of the first annotation content and the manifestation of the first instance of the first target.
The first annotation data may be stored: in a distinct file from the first instance of the first XML document; on an annotation server that is distinct from the document server; by the annotation system without modifying the document server; by the annotation system without modifying any software on the document server; by the annotation system without modifying any content on the document server.
Other features and advantages of various aspects and embodiments of the present invention will become apparent from the following description and from the claims.
BRIEF DESCRIPTION OF THE DRAWINGS
- Top of Page
FIG. 1 is an exemplary high level block diagram of a system enabling the storage, retrieval, transmission, processing and display of documents according to one embodiment of the present invention.
FIG. 2 is a high level block diagram of a document system and a functionally distinct annotation system, wherein the annotation system may annotate documents managed by the document system according to one embodiment of the present invention.
FIG. 3 illustrates the use of manifestation control data to influence the timing of the manifestation of annotations according to one embodiment of the present invention.
FIG. 4 shows how the position of manifested annotation content on the page is determined by its spatial offset from the target according to one embodiment of the present invention.
FIG. 5 shows how the time when the annotation content is manifested on the page is determined by its target on the annotated page object timeline and any temporal offset from this target according to one embodiment of the present invention.
FIG. 6 is a diagrammatic representation of the annotation data structure according to one embodiment of the present invention.
FIG. 7 shows a multi-component annotation on web page according to one embodiment of the present invention.
FIG. 8 illustrates the use of both the ‘temporal offset’ and ‘MCD delay’ for controlling the temporal behavior of an annotation according to one embodiment of the present invention.
FIG. 9 is a flowchart depicting a high level view of the program's restore process according to one embodiment of the present invention.
FIG. 10 is a diagram depicting the annotation save process according to one embodiment of the present invention.
FIG. 11 is a high level overview of how an annotation set is restored according to one embodiment of the present invention.
FIG. 12 is a diagram showing a simple example of how annotation programming and communication between cooperating annotations can be used to implement a dynamic manifestation according to one embodiment of the present invention.
FIG. 13 is a high level diagram of the system architecture of contemporary social plugins that may be used in connection with one embodiment of the present invention.
FIG. 14 is a high level diagram of a novel system architecture for the incorporation of social data into third party websites by means of novel annotations implemented according to embodiments of the present invention.
FIG. 15 is a depiction of the effects of programmatically editing elements of the annotation triple according to one embodiment of the present invention.
FIG. 16 is a depiction of the relationship between a USI hyperlink, the annotation set data, and the underlying document data according to one embodiment of the present invention.
FIG. 17 is an illustration of semantic integrity according to one embodiment of the present invention.
FIG. 18 is a depiction of the relationship between a UAI hyperlink, the annotation data, and the underlying document data according to one embodiment of the present invention.
- Top of Page
A user should be able to annotate any web page content, of any media type, on any web page, on any website, at any time, with any number of annotations of any media type; to be able to save, retrieve and share such annotations, and to do all this without requiring the support, or even the permission, of the site publisher, without taking copies of the content, and without changing the content, the site itself (including any software being used to support the site) or the browser software used to view the content. As will be evident to those skilled in the art this is a much stronger model of annotation for web page content than currently exists.
The strong functional independence of the annotation system from the website and system that serves content to the website is an extremely important characteristic of certain embodiments of the invention; it means that such embodiments of the invention are able to provide valuable functionality to users without demanding changes to existing web infrastructure and may therefore readily made available. Furthermore, only with such independence may an annotation system enable users to interact with web page content without requiring the cooperation and permission of the site publisher.
In the case of embodiments of the present invention a document to be annotated may, for example, be a web page and the type of extra information which may be used to annotate web pages should be interpreted very broadly. Annotations may include not just text commentary on original page content, but much richer annotation such as multi-media augmentation of (or complete changes to) this content, linking the content to other web pages, and enabling additional interaction with content (e.g., by the introduction of new hyperlinks.).
Some systems exist that allow annotating web pages and saving annotations. For example, the most popular web page annotation modes supported by existing annotation systems have been (a) post-it style notes, (b) drawing, and (c) text highlighting. Post-it style notes enable the user to drop a notelet onto a web page, to edit that notelet and save and retrieve it. An example of such notelet technology may be found at www.mystickies.com. Drawing onto a web page is less common but may be found at www.drawhere.com. Perhaps the most common annotation mode is text highlighting which may be found, for example, at www.diigo.com or www.sharedcopy.com. Diigo further enables the association of notelets with highlighted text. The site www.shiftspace.com provides software to enable users to attach notes, highlight text, change photographs and edit underlying HTML.
There are also existing systems that recognise the utility of persistent annotations. Typically such annotations are saved to and restored from a server in a way obvious to one skilled in the art. An example would be U.S. Pat. No. 6,697,838 Method and System for Annotating Information Resources in connection with browsing, in both connected and disconnected States which enables someone browsing the web to create notes, link them to the displayed web page and automatically retrieve such notes on returning to the web page in question. Such persistence is also demonstrated in the implementations, for example, of www.shiftspace.org or www.mystickies.com. In such cases the annotation content is stored separately from the content being annotated and the annotation system is functionally independent of the web server system.
Despite such existing systems there remains a major issue which until now has had no general solution for such functionally independent annotation systems: the problem of semantic integrity. For annotation to have value its meaning must be preserved and in many cases this meaning lies not just in the content of the annotation, but in its relationship to the underlying content. In some cases that relationship is very general as when the annotation relates to the page as a whole. An example of this might be a sticky notelet with a text comment about the page. But in other cases the user may wish to annotate not the page in general but a specific element, or set of elements, on the page. An intuitively obvious example of this is text highlighting. The yellow highlighting annotation has meaning in relation not to the page but to a specific block of text. This is one instance of a much more general issue. We can envisage countless more styles of annotations a user might wish to make in which the relationship between the annotation content and the underlying page content is an integral part of the meaning. In such cases users will place annotation at a specific location for good reason. An arrow points to a specific word or image not to the page in general. A specific part of an image is circled. A user paints a picture of a tail onto the rear end of a dog. In such cases the meaning of the annotation is very sensitive to its location on the page. It is the challenge of ensuring that this relationship between annotation and content is preserved despite transformation of the page so that the true meaning of the annotation is preserved that is the problem of semantic integrity.
It might seem that the provision of semantic integrity is a trivial problem which may be readily solved by saving the annotation\'s position on the page. If a user were annotating a book this would work because book pages do not change. Web pages do change however and such changes can easily render annotations meaningless. For this reason the obvious solution does not work. Even in the early days of the World Wide Web, when most pages were static and their content did not change a great deal, problems of semantic integrity could easily arise. For example, if a user annotated a web page and that page was long enough to require scrolling to display page content, then the annotations become spatially misplaced. If the page employed fluid rather than fixed display so that when the browser display was resized the page layout changed, then the annotations become misplaced. Fluid display sites are not exceptional. An example of a popular such website is www.wikipedia.org. Furthermore, the same web page may be rendered differently by different platforms, and by different web browsers on the same platform, resulting in changing the rendered location of elements on the page. And of course, sometimes the publisher would change the page content.
With today\'s web, often referred to as Web 2.0, the problem is compounded because today\'s web is characterized by many technologies that result in pages changing continually. A large and growing percentage of today\'s web pages are implemented with a database driven design whereby updated content is pushed onto webpages, composed then rendered in real time. The web page retains its URL but its content changes regularly. Personalization may mean the page accessed by the same URL being displayed differently for different users. Pages may not only change from one instance of a user accessing the page to another but may even change whilst the same user is viewing the page. A high profile example is the newsfeed on www.facebook.com where a sub-section of the page, a nested viewport, scrolls continually. In this case any annotation that comments on an element of the newsfeed will quickly become meaningless as that item drops down the list of updates and then disappears from the screen. Another example is an annotated element of a scrollable region which is initially off-screen and then moved on-screen by page embedded action code; such an annotation will not be displayed as the element appears on the screen. Such dynamic behavior of web pages means that it is impossible to view today\'s web pages as static resources, their content must be assumed to be subject to continual change.
The problem has become acute. Changes in page content and media with a temporal dimension are now so prevalent that without the ability to maintain semantic integrity the provision of web page annotation software is a license to create annotations that quickly appear broken or meaningless.
One approach to this problem is to restrict what users can do in the way of annotation. Prevent page layout problems due to resizing the browser window by enforcing a standard window size for annotation. Only allow annotation of pages with static design. Address content change by only allowing annotation styles less likely to quickly become meaningless, for example stickies which in some sense comment on the whole page rather than a page element. Don\'t allow annotation on media types with tricky timing issues. Analyse the structure of the underlying document and use this structure as a proxy for the on screen spatial relationships that the user cares about. Take a copy of the page to be annotated and annotate that copy. But such approaches are not foolproof and severely limit the ways in which users may annotate and copying raises potential copyright concerns. Hence such attempts fall far short of the ideal.
Due to this unresolved problem of semantic integrity today\'s annotation tools are weak. Annotations such as the notelets of www.mystickies.com that float above the whole page aren\'t specific enough for many applications. Users should be able to annotate specific page elements, and regions or points on such a page element, not just the page at large. Text highlighting is a step in the right direction but it is a small step indeed, as is associating notelets with such highlighted text blocks. Web page content is increasingly media rich and it is natural that user should wish to be able to annotate all page media types with content of any media type and be assured of spatial and temporal semantic integrity. The ideal is for users to have access to tools which can generate annotation content in any media and for those annotations to deliver semantic integrity where possible and graceful degradation if changes to the page are so radical that it is not possible; to be able to collaborate on such annotation creation both synchronously and asynchronously; and to be able to generate such content in a framework which can be extended to address unforeseen annotation requirements and media types.
Finally, existing systems do not comprehensively support annotations as programmable objects which may be accessed and modified not just by interaction with human users, but by other computer programs, through a well defined API. The provision of such an API is important because it enables programmers to develop applications that can create and modify multimedia annotations with semantic integrity.
As the description above makes clear, existing annotation systems have a variety of shortcomings As the description below will make clear, various embodiments of the present invention overcoming these shortcomings and provide a variety of other advantages.
A prominent aspect of today\'s computing environment is networked computing. Client computing devices are connected over the Internet or an intranet to each other and to server computers. Such client devices include PCs running Microsoft Windows or Macs running MacOS, mobile phones running the Android operating system or the Apple iOS operating system, tablet computers such as Apple Computer\'s iPad running the iOS operating system, tablets running Android, or netbooks running Google\'s Chrome OS. Such server devices include computers running web server software such the Apache Foundation\'s Apache or Microsoft\'s Internet Information Server; database software such as Oracle\'s MySql or Microsoft\'s SQL Server; and application specific applications server software such a programs written in Ruby or Java.
In such an environment, computers issue requests for resources and servers respond to such requests and forward the requested resource. A common such resource is documents stored on servers and encoded in a scheme appropriate for consumption by a document client program. Commonly the client will translate such encoded documents into a format which may be rendered and then displayed.
One important instance of such network computing is the World Wide Web, commonly abbreviated to WWW or W3 and commonly known as “the web.” In this case the document is a web page. Users use a web browser to view web pages which may well incorporate rich media and which typically include hyperlinks by means of which users may navigate between pages. The web is therefore a system of interlinked web pages.
Web pages are documents encoded in a markup language—a language with a syntax which adds structural information to a document by means of special character sequences, hereinafter the ‘markup’. The markup languages typically used to structure documents (web pages) on the web are the HyperText Markup Language (hereinafter ‘HTML’) and the similar eXtended HyperText Markup Language (hereinafter ‘XHTML’). XHTML is a stricter form of HTML with consistent rules about the format of markup and the nesting of document elements and both are derived from SGML (Standards General Markup Language) and the SGML derived XML (eXtensible Markup Language) which is widely used as a portable method for encoding document structure and encapsulating content. Web page structure (hereinafter ‘structure’) is specified by the markup which identifies the start and end of individual components of the document. These components form HTML/XHTML elements (hereinafter ‘elements’) and the page content is contained within the elements.
XML is a text based language, thus the content of an XML element is always text. However, the text content may be encoded to represent any media type, including, but not limited to images, audio, video. Furthermore, the text may contain an identifier used to access the actual content stored elsewhere; for example by using the Uniform Resource Locator (URL) to specify the location of the media data and the method for retrieving it.
For the sake of simplicity hereinafter we shall refer to documents that may be XML, HTML or XHTML in the context as ‘XML documents’ and similarly to document elements that may be XML, HTML or XHTML as ‘XML elements.’ Furthermore, we shall refer to ‘HTML or XHTML’ as ‘HTML’ where the two are interchangeable for practical purposes.
Web browsers can issue requests for web pages. Servers respond with a web page either taken directly from a local file specified by the request or encapsulates the output of an application component referenced by the request. Such requests and responses typically use the HyperText Transfer Protocol (hereinafter ‘HTTP’), a client/server based protocol for requesting whole or partial documents.
A widely used technique for the implementation of applications that process XML documents is to first parse the XML structure into an in-memory representation using the Document Object Model (hereinafter ‘DOM’). The DOM is well known to those skilled in the art as a cross-platform and language independent means of representing and interacting with objects in XML documents. The DOM has an application programming interface (hereinafter ‘DOM API’) by means of which programs may interact with and process well formed and hence valid such documents. The DOM is a tree structure which mirrors the XML document structure; nodes on the DOM tree represent XML document elements, their attributes and content. Thus, objects within the DOM representation correspond to XML elements and their content—hence, the XML element is often described, by those skilled in the art, as an XML object.
With the DOM API, programmers can build documents, navigate their structure, and add, modify, or delete elements and content. Anything found in an XML document can be accessed, changed, deleted, or added using the DOM API. Those skilled in the art will recognize the almost universal use of the DOM and DOM API for manipulating XML documents.
Those skilled in the art will recognize that a web browser is a specific implementation of an application which uses the DOM API to parse XML document structure in order to access and transform its content. Furthermore, web browsers are open platforms which support the extension of their functionality via third party components. The interface and protocol specifications to provide access to the inner operation of the browser, (which access is necessary to develop such extensions) is published and freely available to developers. Examples of extensions are: support for new media types; customised web page display behaviour; additional menu options and web page development tools.
FIG. 1 is an exemplary high level block diagram of a system 100 enabling the storage, retrieval, transmission, processing and display of documents The components of the system are enclosed in the dotted line 105. A high level description of a typical operation (document retrieval and display) using this system 100 follows. A user on a client computer 110 interacts with a web browser 120 and by interaction with that program generates a request 130 for an XML document 180 stored in a document store 160 on a document server 150. The request 130 is transmitted over the communication channel 140 to the document server 150 storing the document. Those skilled in the art will appreciate that the communication channel 140 may for example be the interne, an intranet or in the case wherein the client computer 110 and the document server computer 150 are the same, an internal system bus. The document server 150 locates the requested document 180 in its document store 160 and transmits it over the communication channel 140 as a response 170 to the client computer 110 and thence to the web browser 120. The web browser 120 parses the encoded document 180, processes it into a format suitable for representation in the DOM and inserts this formatted data into the DOM by means of the DOM API. The rendering engine module of the web browser 120 operates on the resultant DOM data structure(s) to create a rendered version of the document 190. This version of the document 190 is manifested.
Those skilled in the art will recognise that whilst this is only one use case and it is simplified for the purposes of exposition. It is nonetheless representative of the key components and processes of the document system.
The embodiment of the ‘annotation system’ (hereinafter ‘program’) comprises two components a) an annotation server, and b) an annotation client. These components are distinct from those of the document system. As the annotation system exists to annotate documents it presumes the prior existence of a document system.
FIG. 2 is a high level block diagram of a document system 100, whose components are enclosed in the dotted line 105 and an annotation system 200, whose components are enclosed in the solid line 205. The annotation system 200 and the document system 100 share some resources: the client computer 110, the communication channel 140 and the web browser 120. The annotation system 200 has resources the document system 100 does not: the annotation server 250 with its annotation store 260 and the browser extension 285. The document system 100 has resources the annotation system 200 does not: the document server 150 with its document store 160.
The annotation system 200 assumes the existence of a document system 100 and enables the storage, retrieval, transmission, processing and display of annotations on documents served by the document server 150. A high level description of a typical operation (annotation retrieval and display of an annotation 280 and its display on its associated document 180 served by the annotation system 100) using this system 200 follows.
A user on a client computer 110 interacts with a web browser 120 and with the browser extension 285 and by that interaction generates a request 230 for an annotation 280 stored in an annotation store 260 on an annotation server 250. The request 230 is transmitted over the communication channel 140 to the annotation server 250 storing the annotation. The annotation server 250 locates the requested annotation 280 in its annotation store 260 and transmits it over the communication channel 140 as a response 270 to the client computer 110 and thence to the browser extension 285. The browser extension 285 parses the retrieved annotation 280 and processes it into a format suitable for representation in the DOM. As will be outlined in detail below, the browser extension 285 then inserts this processed data into the DOM by means of the DOM API thereby creating a composite of the document to be annotated 180 and the annotation 280 which is annotating document 180. The rendering engine module of the web browser 120 operates on the resultant composite DOM structure to create a rendered version of the composite, document 290. This version of the document 290 is displayed.
It will be recognized by those skilled in the art that notwithstanding shared components the document system 100 and the annotation system 200 are functionally independent. This point will be discussed in greater detail below.
Whilst the exemplary implementation describes a browser extension module and a particular division of labor between the extension module and the server modules it will be recognised by those skilled in the art that the functionality of browsers themselves continues to evolve as does technology to support browser extensions and server software. The invention may, for example, be implemented using any mechanism which supports extension of browser functionality such as Google\'s “Native Client” (NaCL). Furthermore, in an alternative implementation, the functionality of the extension may be incorporated into the core browser in its entirety. Indeed, as is obvious to one skilled in the art, in the light of such ongoing technical change, the division of labor between the browser, the browser extension and the server is itself subject to change and the program may therefore take the form of many embodiments. The functionality of the program may, for example, be heavily weighted to the server with a thin client or alternatively may, for example, be heavily weighted to the client with the server simply storing annotations created and edited by interaction with the client.
The invention enables powerful and novel annotations on web pages. Web page content may be as simple as text or it may be complex media such as video. XML elements specify how and where visible content is rendered on the browser\'s display area and how non-visible content is actioned e.g. how and when audio is played. These XML elements with their content form a rich set of objects. The content of XML elements are of various types. Hereinafter we refer to such types simply as ‘media types’.
Web pages are structured compound objects that include objects of many media types. Many such objects are directly included, for example text and images with a data URI, but others may be included by reference and all are subject to dynamic modification.
Page objects may have spatial and or temporal dimension. Visual manifestations of page objects are spatially related to each other on the page. Those skilled in the art will recognise that page layout may change for many reasons and that as it does, for whatever reason, such spatial relations may also change. Similarly, the manifestation of page objects are temporally related on the page and such temporal relations may also change.
In an intuitive sense, annotations may be said to be placed “onto” a web page, thereby annotating the page. The location on the page to which the annotation is attached is the ‘annotation target’. In placing the annotation, the user specifies a relationship between the annotation content and the annotation target. This relationship has meaning to the user and hence is a ‘semantic relationship.’ Recognizing that the importance of this semantic dimension of annotation, an important purpose of the described embodiment is to capture the semantic relationship and ensure that annotations are re-manifested with the semantic relationship intact on subsequent instances of the web page. Components of the embodiment undertake analysis of page content and structure to capture this semantic relationship. This analysis is hereinafter ‘semantic analysis.’ The property of preservation of this semantic relationship derived from semantic analysis (despite changes in the underlying document from one instance of its manifestation to another) is hereinafter termed ‘semantic integrity.’
FIG. 17 illustrates the property of semantic integrity. The FIG. 17 shows two instances of the same web page 2306 and 2316. Although they differ in layout and specific content, they are the same web page as they result from a retrieval of the same URL. The first instance of the web page 2306 shows the page comprising of four page objects, 2304, 2307, 2308 and 2309. Examples of such page objects are text blocks or images. Page object 2304 has been annotated. The annotation content 2303 has been placed onto the web page 2306 in a specific relationship 2302 to a point 2301 on the annotated page object 2304. This relationship 2302 is a semantic relationship, determined by the user, as described above. The second instance of the same web page 2316 shows the page has changed significantly: it now comprises three page objects, 2304, 2310 and 2312. Only the page object 2304 remains from the earlier instance of the web page 2306 and this page object is now shown to be a different size and in a different location on the page 2316. In this second instance the same annotation content 2303 has been placed onto the web page 2316 in the same specific relationship 2302 to the point 2301 on the annotated page object 2304. Hence the semantic relationship, determined by the user, between the annotation content 2303 and the location on the page 2301 is maintained. This example of the preservation of the semantic relationship created by the user, despite significant change in the web page being annotated is an example of the property of semantic integrity.
Content placed ‘on’ the page is what is typically naively thought of as the annotation. Familiar examples of annotations are notelets and text highlighting but the program enables annotation content of any media type or combination of such types. Whilst some annotations have a visual dimension e.g. notelets which are of type text or composite objects such as widgets, other annotations may not have such a visualization e.g. audio with no visualized player. Recognizing such differences, we say that different annotation media types are “manifested” differently. For example, visualizations such as images are manifested by being rendered and displayed, audio attached to a page with no visible player is manifested by being played, video is manifested by being rendered and played, and links are manifested by being followed. The act of manifesting annotation content creates a ‘manifestation’ of the annotation content (e.g., a rendered and displayed image in the case of image annotation content). It is this manifestation which the user experiences and it is such manifestations that the user manipulates to place the annotation in relation to a target manifestation to establish a semantic relationship. In the case of visual manifestations this is a direct and evident manipulation. In the case of non-visual manifestations it remains logically true, but the relationship is manipulated by means of a user interface representing this relationship.
The program enables the creation of annotation content in ways that range from the familiar and simple, for example typing text into a notelet, to the rich, for example ‘painting’. The program allows for all such styles of annotation. Whilst some content is of very general applicability e.g. a notelet may be used to annotate practically any type of page object, other content is much more specific and only makes sense when applied to certain types of page objects e.g. a video annotation tool may only be applied to video.
The data which the program generates to describe the annotation content itself comprises one or two components (a) data required to manifest the content on the annotated web page, hereinafter the ‘manifestation data’ and optionally (b) data which controls elements of how the manifestation behaves; hereinafter ‘manifestation control data’ or ‘MCD’. An annotation always has ‘manifestation data’ but may not have ‘manifestation control data.’
The invention supports manifestation data of any type supported by the DOM. Annotation need not therefore be simple text or images but may be, for example, multimedia such as video, hyperlinks or form elements such as text fields, checkboxes, buttons and selection menus. Script and embedded object elements are also valid content media types and, as will be discussed in more detail below. Such dynamic elements within the annotation content may be used to implement ‘annotation behavior’ by interacting with the user, responding to external events and generating events to influence the behavior of other annotations. The manifestation may also be of combinations of such media types within an HTML fragment, e.g. a speech balloon image into which the user may input text or other data. In another example, the manifestation may not be visual at all, but may rather be an audio clip that is played. It is the manifestation of this manifestation data which may be naively considered to be the annotation.
The MCD is a specification of how the program manifests the annotation: it comprises