WO2004021166A1

WO2004021166A1 - Guidance method and device

Info

Publication number: WO2004021166A1
Application number: PCT/FI2003/000598
Authority: WO
Inventors: Jukka-Pekka METSÄVAINIO; Manne Hannula
Original assignee: Myorigo Oy
Priority date: 2002-09-02
Filing date: 2003-08-12
Publication date: 2004-03-11
Also published as: FI20021560A0; FI115255B; FI20021560A; EP1535142A1; AU2003249135A1

Abstract

This invention relates to the scrolling and reading of a larger document on the display of small hand-held electronic equipment, such as mobile phones or personal digital assistance devices, where the view on the display screen is based on a tilt angle of the equipment. An objective is to provide a guidance method that enables reading text lines in a document whose actual size is larger than the screen of the display, without un-wanted effects such as jitter or bouncing. Thus the guidance method is adapted to follow accurately the reading direction of the text lines from the beginning to the end of the text lines.

Description

Guidance method and device

Field of the invention

The present invention generally relates to the display of electronic equipment such as mobile phones and more particularly to the browsing and reading of a large document on a display screen.

Background of the invention

Today portable electronic equipment, such as mobile phones and personal digital assistance devices PDA, are getting ever-smaller. However, when the size of the portable equipment is reduced also the size of the equipment's display is reduced. On the other hand, there is an increasing need to provide services with text and pictures to subscribers. Thus the screen of the display is usually smaller in size than the actual document, i.e. only a portion of the document is visible at a time. It is clear that reading a large document from a small display screen is rather difficult.

Different kinds of mechanisms have been developed to ease reading from a small display screen. Some of them are considered briefly in the following. In some small-sized, hand-held devices which can be used while being carried, scrolling the view on the display screen is based on the location (at least one of x, y, or z coordinates) and/or orientation (at least one of azimuth, elevation, roll) of the device. In such a motion-control environment, the portion displayed on the display screen of the hand-held device depends constantly on the movements of the hand. Due to the fact that the hand is inaccurate, some unwanted effects arise, such as jitter or bouncing, which makes reading even more difficult. The same kind of unwanted effects may arise also when the hand-held device is used in a moving vehicle.

In order to avoid the above drawbacks, a method has been de- vised whereby the direction of the scrolling the data can be locked. However, one drawback with this method is that it allows browsing of the data in one direction only, i.e. it does not take into account the direction of the text lines in the data.

Methods with scroll bars permitting movements of the text either horizontally or vertically do not have jitter. That is, scroll bars can be used to permit the user to change the portion of the view on the display screen hori- zontally by sliding the horizontal scroll bar and vertically by sliding the vertical scroll bar. However, a drawback with such methods is that scrolling is always either horizontal or vertical, i.e. no other direction is possible.

At the moment there is no method available that reduces the er- rors arising from the bouncing movements around the text lines when reading a document on the display screen of a portable device assisted by a motion-control method.

Summary of the invention An objective of the present invention is to provide a guidance method for scrolling the display on a screen of a hand-held device having motion- controlled scrolling and a hand-held device performing the method. The view shown on the display is a portion of larger content comprising text lines that are too long to be displayed entirely on the small display. The guidance method should be adapted to follow accurately the reading direction of the text lines from the beginning of the text lines to the end of the text lines. The reading direction should remain as uniform as possible during the entire reading event, i.e. unwanted effects such as jitter or bouncing of the text lines should not arise during the reading event. A typical situation may be when a user is browsing a large full-size image or a document by scrolling the view shown on a mobile phone's display using motion-control scrolling.

The guidance method discovers when the user stops browsing and starts to read text lines. Then the need for guidance is examined. If the result of examination indicates that the user is reading the text lines, the scrolling direction is guided to follow the reading direction of the text lines. The guidance is invisible to the user.

The objective is achieved in the manner described in the independent claims. The guidance method comprises three main steps: 1) the direction of the text lines shown on the display screen is periodically examined; 2) the movements of the mobile phone are periodically examined and, based on the movements, it is concluded whether the user is reading the text lines; 3) if the result of examination in step 2 indicates that the user is reading the text lines, the scrolling direction is reinforced in the direction of the text line so long as the difference between the current scrolling direction and the text line direction is less than a predetermined threshold value.

Brief description of the drawings The invention is described more closely with reference to the accompanying drawings, in which

FIG. 1a-b illustrates a page of the document to be read on a display of a small portable device, FIG. 2 is a flowchart illustrating definitions of the direction of text lines on the display, FIG. 3 is a flowchart illustrating the method according to the invention, FIG. 4 illustrates the inventive idea with the help of a full-size image to be read on a display of a small portable device, FIG. 5a-b show an example of the determination of coordinates on the full- size image, and FIG. 6a-b demonstrate how filtering is carried out.

Detailed description of the invention The invention can be applied to a terminal having a small display, at least one sensor such as a magnetic pulse sensor, a acceleration transducer or gyroscope which is mounted in the back of the display, and a character recognition method or, alternatively, some other method for recognizing the direction of the text lines. The principle of the invention and the embodiments are described using a mobile phone as an example, but the same principle can, of course, be applied to any kind of terminal.

FIG. 1a shows a typical situation where a user is browsing a large full-size image 10 on the display of a mobile phone. In this context the full- size image refers to a whole document or page, of which only a part can be displayed on the actual screen of the terminal. The actual display of the mobile phone is illustrated by a dotted square 16 on the document page for visualizing the proportions of the display to the document. Thus the user is reading a document whose actual size is larger than the physical size of the display. The document page has been divided into several columns consisting of text sections 11 - 13 and non-text sections 14 - 15, such as pictures. Lines in the text column 13 are in a different direction than in the other two text columns 11 and 12.

FIG. 1b is an enlargement of the circled area in FIG. 1a. As seen, only a portion of the data is seen on the display at a time, i.e. a line of the document does not fit from beginning to end on the display. Scrolling directions are shown by double-ended arrows. Other scrolling directions are also possible, e.g. in the diagonal direction.

It is assumed that motion-controlled scrolling is used. In order to browse the document, the data is displayed proportionally to the orientation or location of the mobile phone. That is, data scrolling in the vertical direction when the mobile phone is tilted vertically, and correspondingly, the data scrolls horizontally when the mobile phone is tilted horizontally.

When a piece of interesting text 17 appears on the display and the user wants to go into the matter in greater detail, i.e. to read through the whole document or a chapter concerning the matter, the portion of the view displayed is moved to the beginning of the document or appropriate chapter by tilting the mobile phone in a suitable way. For example, a movable cursor is used to indicate a position of interest on the display surface by tilting the mobile phone. Thus the cursor moves horizontally when the mobile phone is tilted towards the vertical axis of the display, and correspondingly, the cursor moves vertically when the mobile phone is tilted towards the horizontal axis of the display.

Thus, the display shown on the screen depends on the motion of the hand. However, as stated above, hand movements are inaccurate. Vv th- out accurate guidance, unwanted jitter arises, making the reading uncomfortable and slow. This drawback is eliminated by the guidance method described in more detail in the following.

The guidance method comprises three main steps: 1) the direction of the text lines shown on the display screen is periodically examined; 2) the movements of the mobile phone are periodically examined and, based on the movements, it is concluded whether the user is reading the text lines; 3) if the result of examination in step 2 indicates that the user is reading the text lines, the scrolling direction is reinforced in the direction of the text line so long as the difference between the current scrolling direction and the text line direction is less than a predetermined threshold value. FIG. 2 shows as a flowchart an example of the implementation of examination of the direction of text lines.

It is assumed here that a user scrolls documents on the mobile phone display screen in a way that is described in association with FIG. 1a and 1b.

At stage 20, the first task is to analyze roughly whether there is text in the current portion on display. If no text is detected, free scrolling of the data is allowed at stage 21 , and the control is returned to stage 20.

There are numerous implementation alternatives for analysis. One alternative is that the portion of the data that is seen on the display screen is first analyzed by defining the major components, i.e. whether the type of component represents the text or non-text. Some character recognition method such as an optical character recognition method OCR, can be used. In some cases information about the type and size of the document, and the type and size of the font may already be included in the document, in which case no character recognition is needed.

At the next stage 22, the direction of the text lines is defined. In

FIG. 1a most of the page is covered with text. When this page is analyzed by the character recognition method the direction of text lines is defined to be horizontal. However, if the column 13 had been shown on the display, the text lines would have been defined to be vertical.

One alternative for the direction definition of the text lines can be created on the basis of the middle point of the current portion of view displayed. This method will be considered later in greater detail. Another alternative utilizes image analysis and character recognition. The direction is defined on the basis of features of the analyzed data.

The definition results concerning the direction of the text lines are saved at stage 23. The information also includes current time information. The above-described analysis and definition are came out periodically at certain intervals.

A flowchart in FIG. 3 is used to demonstrate how the need for guidance is examined and carried out while one is browsing a large document on the display screen of a terminal by scrolling the display.

Coordinates of the current partial view of the full-size image are read at the fixed reading point (xi, yi) of the display. In this example, the fixed reading point is the middle point of the display. The coordinates of the current view at the middle point of the display are then saved at certain time instances e.g. every second, at stage 30. With the aid of the set of coordinates, a set of motion vectors is obtained. The motion distribution as a time function is calculated at stage 31 by using the motion vector set.

At stage 32 the need for guidance is examined. If the distribution is narrow, it means that movements have been found, which do not deflect much from the main movement direction or scrolling direction. In other words, if the scrolling direction is in the direction of the x-axis, the coordinates of the page, which have been read at the middle point of the display, seem to follow the general direction of the x-axis (the text line), i.e. the scrolling direction is not exactly rectilinear. If such movement as described above is not detected or the distribution is relatively broad, free scrolling is still allowed, at stage 35. Otherwise, the next stage 33 is to examine more accurately whether there is a need to guide the reading direction. In this case the distribution is compared with the saved information (see FIG. 2 stage 23) about the direction of the text lines. At stage 34 it is analyzed whether the result of the comparison indicates, in a given time, successive movements which are roughly in the same direction. If the result does not indicate such movements, free scrolling of the data is still allowed at stage 35. Otherwise, the data is filtered in such a way that the direction of the text lines are taken into account at stage 36. Regardless of the result of the comparison, the steps above are repeated from stage 30.

FIG. 4 demonstrates how the defining and filtering are carried out mathematically. The assumption in this example is that the direction of the text lines on the display is defined to be horizontal by the character recognition method. A vector 42 represents that direction. The additional assumption is that coordinates of the partial view 40 of the full-size image are read at the middle point 43 of the display. Based on the successive coordinates, a set of motion vectors are obtained, wherein if the directions of said motion vectors are roughly in the direction of the text line, this indicates that the user is reading the text lines. The curve 41 represents the set of motion vectors.

The similarity of directions between the vector 42 and curve 41 is examined. One way of examining the similarity is described in the following.

Coordinates of the partial view of the full-size image are controlled by a cer- tain time window. In other words while the document is scrolled the position of the middle point 43 on the full-size image 10 varies as time function.

The curve 41 is sampled by a calculating unit using a sample rate FS (e.g. FS=1 s). Each sample consists of the coordinates (xi, yi) of a point Pi on the curve at a given time. For example, a current coordinate pair (xi,yi) is saved every second within a certain time interval, such as 10 s. These samples formulate a temporal matrix T, where the first row of the matrix represents the x-coordinates and the second row the y-coordinates, so that each column presents one coordinate pair (xi,yi). The parameter D represents the direction (angle) of the text line under examination. The definition value for D is 0 radians when the text line is in the vertical direction and π/2 radians when the text line is in the horizontal direction. Of course, parameter D can have any angle value between 0 - n2π radians. The current direction (angle) between each consecutive point in temporal matrix T is calculated. For example, if at time t1 the coordinates at point P1 are (x1 , y1) and at time t2 at point P2 (x2, y1), the vector between the two points P1 and P2 defines the direction (angle) at time t1 , where Δt is (t2-t1 , t2 > t1). In general the result of calculation is N-1 directions or vectors when the number of samples is N. The calculated N-1 directions formulate a vector U. Here within 10 seconds the result is nine vectors, each of which has a direction (angle) between 0-n2π radians. That is the result is the distribution of directions within 10 seconds. When the shape of distribution of the directions is narrow, a lot of movements have been found in the same direction. Correspondingly, when the shape of distribution is broad, only some movements are in the same direction.

The similarity of curve 41 and vector 42 is examined by comparing the above vectors in the following way. First a comparison is made as to whether the absolute value of the difference between the mean of vector U and parameter D is smaller than the predetermined threshold parameter V and whether the standard deviation of vector U is smaller than the predetermined parameter X using the following equations Abs(Mean(U) - D) < V (1) Std(U) < X (2)

If the results are smaller in both comparisons, curve 41 and vector 42 are in the same direction. Otherwise curve 41 and vector 42 are in different directions. In the first case mentioned above, i.e. when curve 41 and vector

42 are in the same direction, the scroll direction is guided to follow accurately the reading direction of the text lines by filtering from the scrolling motion those components which differ from the direction of the text lines.

The effect of filtering is seen in FIG. 4 with the help of curve 44 and vector 45. Curve 44 demonstrates movement of the page before filtering and vector 45 represents movement of the page after filtering. Thus, filtering provides for movement of the portion of view in the full-size image accurately in the direction of the text lines. The result is that the user is able to read the text lines with ease because unwanted effects are prevented that might make reading uncomfortable.

One way to carry out filtering is described with reference to FIG. 5a - 5b and 6a - 6b.

First, it is assumed that filtering is not in action in FIG. 5a and 5b. FIG. 5a illustrates how the view shown on the display looks at time t1. It is seen that the middle point of the current view 500 is in point P1 at time t1. Within time (t1+ Δt) - 11 the display on the screen has been moved or the middle point of the previous display has been moved on the full-size image from point P1 to point P2, when the current view 500 is now the view 501 as shown in FIG. 5b. FIG. 6a and 6b illustrate a situation similar to what was explained above with reference to FIG. 5a-5b, but here the filtering is in action.

In FIG. 6a the vector a from the initial point P1 to the terminal point P2 represents the displacement of the full-size image within a given time. The vector b is a unit vector (i.e. | b | =1) in the direction of the x-axis. The initial point of vector b is at point P1.

In this case it is enough to determine only the component of a which is parallel with x-axis, because the text lines are in the direction of the x-axis.

FIG. 6b illustrates said determination. First the projection of a on b is determined by a scalar product: p = a • b.= |a||b|cosθ (3) where p is the product of the magnitudes of a and b and the cosine angle θ between them. In this example θ is π/4. Then the vector a_x represents the component vector of a in the x direction, as denoted by a_x = pb (4)

The initial point of the vector a_x is at point P1 and the terminal point at point PS. The vector a_x represents the displacement of the full-size image in the x-direction within the time Δt.

The calculation described above is repeated correspondingly for displacements at intervals as long as the filtering is activated. The time period is a sample period 1/FS.

The above-described method can be used in a hand-held terminal having a motion controlled scrolling operation by adapting the terminal to: read coordinates of the scrolling display at a fixed reading point of the screen at predetermined time intervals, wherein a set of motion vectors is obtained; determine the scrolling direction based on the set of motion vectors; examine whether the scrolling direction is roughly the same as the direction of a text line; and if so, reinforce the displayed scrolling in the direction of the text line as long as the difference between the current scrolling direction and the text line direction is less than a predetermined threshold value.

The implementation and embodiment of the present invention has been explained above with some examples. However, it is to be understood that the invention is not restricted by the details of the above embodiments and that numerous changes and modifications can be made by those skilled in the art without departing from the characteristic features of the invention. The described embodiment is to be considered illustrative but not restrictive. Therefore, the invention should be limited only by the attached claims. Thus alternative implementations defined by the claims, as well as equivalent implementations, are included in the scope of the invention. For example, guidance can be implemented in any scrolling direction, also diagonally. Parameters such as the sample rate and the number of samples, as well as the time intervals, depend on the application used. In some cases it is also possible that a user adjusts some of the above parameters, wherein the guidance method reacts differently to a slow user and a quick user.

Claims

1. A guidance method for scrolling data shown on the display of a terminal, where the scrolling direction depends on the tilt angle of the terminal, characterized by the steps of: reading the coordinates of the scrolling display at a fixed reading point of the screen at predetermined time intervals, wherein a set of motion vectors are obtained, determining the scrolling direction based on the set of motion vec- tors, examining whether the scrolling direction is roughly the same as the direction of a text line, and if so, reinforcing the displayed scrolling in the direction of the text line as long as the difference between the current scrolling direction and the text line direction is less than a predetermined threshold value.

2. The method according to 1, characterized further by comprising the steps of: detecting periodically whether text is shown on the display, defining the direction of the text lines in response to the detection and saving the result together with the detection time, calculating the motion distribution as a time function, by using the motion vector set, and comparing the motion distribution with the saved result.

3. The method according to 1, characterized in that the di- rection of text lines is defined by character recognition.

4. The method according to 1, characterized in that the direction of text lines is defined on the basis of the coordinates.

5. A hand-held terminal having a motion-controlled scrolling operation, characterized in that the terminal is adapted to: read coordinates of the scrolling display at a fixed reading point of the screen at predetermined time intervals, wherein a set of motion vectors is obtained, determine the scrolling direction based on the set of motion vec- tors, examine whether the scrolling direction is roughly the same as the direction of a text line, and if so, reinforce the displayed scrolling in the direction of the text line as long as the difference between the current scrolling direction and the text line direction is less than a predetermined threshold value.

6. The hand-held terminal as in claim 5, c h a ra cte r i z e d in that the user terminal is further adapted to detect periodically whether text is shown on the display, define the direction of the text lines in response to the detection and save the result together with the detection time, calculate the motion distribution as a time function, by using the motion vector set, and compare the motion distribution with the saved result.