Information processing device

Background of the invention

1. Field of the invention

One embodiment of the present invention relates to a program and an information processing device.

Note that one embodiment of the present invention is not limited to the above technical field. One embodiment of the invention disclosed in this specification and the like relates to an object, a method, or a manufacturing method. One embodiment of the present invention relates to a machine, a process, manufacture, or a composition of matter. In particular, the present invention relates to, for example, a semiconductor device, a storage device, a processor, a display device, a light-emitting device, an input device, an input/output device, a sensing device, a lighting device, a power storage device, a manufacturing method thereof, or a driving method thereof.

2. Description of the related art

In rapid development of mobile electronic devices, input technique of data by a touch panel or the like plays an important role.

In particular, optical touch panels using photo sensor units serving as touch panels have been attracting attentions. For example, a semiconductor device in which a photo sensor unit and a display unit are provided in different regions can control the display state of the display unit based on a data signal generated by the photo sensor unit. For example, when a user touches the display unit with his/her finger, the photo sensor unit senses the position of the finger based on a data signal generated by the photo sensor unit, so that the photo sensor unit can serve as a touch panel.

Non-patent document 1 discloses an example of information processing flow by which a touch position is sensed in an optical touch panel.

Ambient light (light in an ambience where the photo sensor unit is located) enters the photo sensor unit. Thus, ambient light might influence a data signal produced by the photo sensor unit, thereby causing difficulties in the sensing of a touch position.

Patent Document 1 discloses a method of suppressing adverse effects of ambient light in a touch panel, in which the state of a light unit included in a semiconductor device is switched between a lighting state and a non-lighting state, and in each period of the lighting state and non-lighting state, data is produced by a photo sensor unit and a data signal is produced by obtaining a difference between the two produced data signals. REFERENCE Patent Document

[Patent Document 1] Japanese Published Patent Application No. 2012-008541 Non-Patent Document

[Non-patent document 1] Adi Abileah and Patrick Green, “Optical Sensors Embedded within AMLCD Panel: Design and Applications,” ACM SIGGRAPH 2007 SUMMARY OF THE INVENTION

Optical touch panels, which are easily affected by ambient light as described above, are also affected by a display image. Accordingly, a program that determines a touch position more stably in various ambient light and during display of various images, or an information processing device having the program are demanded.

One embodiment of the present invention has at least one object of the following: to provide a program that can extract input data more stably; to provide a program that can extract input data with higher accuracy; to provide a more convenient information processing device; and a provide a novel information processing device.

Note that the description of these objects do not disturb the existence of other objects. Note that in one embodiment of the present invention, there is no need to achieve all the objects. Other objects will be apparent from and can be derived from the description of the specification, the drawings, and the claims.

One embodiment of the present invention is a program including first to fourth steps. The first step includes a step of fetching a first sensor image at the time when a light unit is lighting and a second sensor image at the time when the light unit is not lighting. The second step includes a step of calculating a third image by calibrating the first sensor image using a first calibration image; and a step of calculating a fourth image by calibrating the second sensor image using a second calibration image. The third step includes a step of calculating fifth to seventh images using the third image and the fourth image; and a step of calculating an eighth image by obtaining a weighted average of the fifth to the seventh images in accordance with an illumination and a display image. The fourth step includes a step of calculating touch data using the eighth image.

Another embodiment of the present invention is a program including first to fourth steps. The first step includes a step of fetching a first sensor image at the time when a light unit is lighting and a second sensor image at the time when the light unit is not lighting. The second step includes a step of calculating a third image by calibrating the first sensor image using a first calibration image; and a step of calculating a fourth image by calibrating the second sensor image using a second calibration image. The third step includes a step of calculating fifth to seventh images using the third image and the fourth image; a step of calculating an eighth image by obtaining a weighted average of the sixth and the seventh images in accordance with an illumination; a step of fetching a display image; a step of calculating a ninth image by converting the display image into a gray scale; and a step of calculating a tenth image by obtaining a weighted average of the fifth and the eighth images in accordance with the ninth image. The fourth step includes a step of calculating touch data using the tenth image.

Another embodiment of the present invention is a program including first to fourth steps. The first step includes a step of fetching a first sensor image at the time when a light unit is lighting and a second sensor image at the time when the light unit is not lighting. The second step includes a step of calculating a third image by calibrating the first sensor image using a first calibration image; and a step of calculating a fourth image by calibrating the second sensor image using a second calibration image. The third step includes a step of calculating a fifth image by normalizing a difference between the third and the fourth images; a step of calculating a sixth image by normalizing the third image; a step of calculating a seventh image by normalizing the fourth image; a step of calculating an eighth image by obtaining a weighted average of the sixth and the seventh images in accordance with an illumination; a step of fetching a display image; a step of calculating a ninth image by converting the display image into a gray scale; and a step of calculating a tenth image by obtaining a weighted average of the fifth and the eighth images in accordance with the ninth image. The fourth step includes a step of calculating touch data using the tenth image.

Another embodiment of the present invention is a program including first to fourth steps. The first step includes a step of fetching a first sensor image at the time when a light unit is lighting and a second sensor image at the time when the light unit is not lighting. The second step includes a step of calculating a third image by calibrating the first sensor image using a first calibration image; and a step of calculating a fourth image by calibrating the second sensor image using a second calibration image. The third step includes a step of calculating a fifth image by normalizing a difference between the third and the fourth images; a step of calculating a sixth image by normalizing the third image; a step of calculating a seventh image by normalizing the fourth image; a step of calculating an eighth image by obtaining a weighted average of the sixth and the seventh images assuming that the weights of the sixth and the seventh images are 1 and 0, respectively, in the case where an illumination is lower than the threshold illumination and are 0 and 1, respectively, in the other cases; a step of fetching a display image; a step of calculating a ninth image by converting the display image into a gray scale; and a step of calculating a tenth image by obtaining a weighted average of the fifth and the eighth images in accordance with the value of the ninth image such that the weights of the fifth and the eight images are 1 and 0, respectively, in a white display portion and the weights of the fifth and the eighth images are 0 and 1, respectively, in a black display portion. The fourth step includes a step of calculating touch data using the tenth image.

Another embodiment of the present invention is a program including first to third steps. The first step includes a step of fetching a sensor image at the time when a light unit is not lighting. The second step includes a step of calculating a first image by deducting a calibration image from the sensor image. The third step includes a step of calculating a second image by smoothing the first image; a step of finding the brightest component in the second image; a step of calculating estimated first and second illuminations using the value of the brightest component; a step of fetching a display image; a step of calculating a third image by converting the display image into a gray scale; and a step of calculating an estimated third illumination by obtaining a weighted average of the estimated first and second illuminations in accordance with the coordinates of the brightest component and the third image.

Another embodiment of the present invention is a program including the program of any one of the embodiments

to

as a first program and the program of the embodiment

as a second program. In the first program, the estimated illumination calculated in the second program is used.

Another embodiment of the present invention is the program of the embodiment

including fifth to seventh steps. The fifth step includes a step of fetching a third sensor image at the time when the light unit is not lighting. The sixth step includes a step of calculating a ninth image by deducting a third calibration image from the third sensor image. The seventh step includes a step of calculating a tenth image by smoothing the ninth image; a step of finding the brightest component in the tenth image; a step of calculating estimated first and second illuminations using the value of the brightest component; a step of calculating an eleventh image by converting the display image into a gray scale; and a step of calculating an estimated third illumination by obtaining a weighted average of the estimated first and second illuminations in accordance with the coordinates of the brightest component and the eleventh image. The estimated third illumination is used as the illumination.

Another embodiment of the present invention is the program of the embodiment

including fifth to seventh steps. The fifth step includes a step of fetching a third sensor image at the time when the light unit is not lighting. The sixth step includes a step of calculating an eleventh image by deducting a third calibration image from the third sensor image. The seventh step includes a step of calculating a twelfth image by smoothing the eleventh image; a step of finding the brightest component in the twelfth image; a step of calculating estimated first and second illuminations using the value of the brightest component; and a step of calculating an estimated third illumination by obtaining a weighted average of the estimated first and second illuminations in accordance with the coordinates of the brightest component and the ninth image. The estimated third illumination is used as the illumination.

Another embodiment of the present invention is an information processing device including a memory and an execution unit. The memory includes a program, and the execution unit conducts the program. The program includes first to fourth steps. The first step includes a step of fetching a first sensor image at the time when a light unit is lighting and a second sensor image at the time when the light unit is not lighting. The second step includes a step of calculating a third image by calibrating the first sensor image using a first calibration image; and a step of calculating a fourth image by calibrating the second sensor image using a second calibration image. The third step includes a step of calculating fifth to seventh images using the third image and the fourth image; and a step of calculating an eighth image by obtaining a weighted average of the fifth to the seventh images in accordance with an illumination and a display image. The fourth step includes a step of calculating touch data using the eighth image.

Another embodiment is an information processing device including an optical touch panel and a storage portion in which the program of any one of the embodiments

to

is stored.

A program that can extract input data more stably can be provided. Furthermore, a program that can extract input data with higher accuracy can be provided. Furthermore, an information processing device with improved convenience can be provided. Furthermore, a novel information processing device can be provided.

Note that the description of these effects does not disturb the existence of other effects. One embodiment of the present invention does not necessarily have all the effects listed above. Other effects will be apparent from and can be derived from the description of the specification, the drawings, the claims, and the like.

Brief description of the drawings

In the accompanying drawings:

FIGS. 1A and 1B are block diagrams of information processing devices of embodiments of the present invention;

FIG. 2 is a block diagram of an information processing device of one embodiment of the present invention;

FIGS. 3A and 3B show flow charts of programs of embodiments of the present invention;

FIG. 4 is a flow chart of a program of one embodiment of the present invention;

FIG. 5 is a flow chart of a program of one embodiment of the present invention;

FIG. 6 is a flow chart of a program of one embodiment of the present invention;

FIG. 7 is a flow chart of a program of one embodiment of the present invention;

FIG. 8 is a flow chart of a program of one embodiment of the present invention;

FIG. 9 is a block diagram of an information processing device of one embodiment of the present invention;

FIGS. 10A , 10 B 1 , 10 B 2 , 10 C, and 10 D illustrate information processing devices of embodiments of the present invention;

FIG. 11 is a block diagram of a module including a display unit and a photo sensor unit;

FIG. 12 is a circuit diagram illustrating a cell including a pixel and a photodiode;

FIG. 13 is a timing chart showing the operation of a cell including a photodiode;

FIG. 14 is a block diagram of an information processing device of one embodiment of the present invention;

FIG. 15 is a flow chart of a program of one embodiment of the present invention;

FIG. 16 is a flow chart of a program of one embodiment of the present invention;

FIGS. 17A, 17B, 17C, 17D, and 17E are images containing touch levels each obtained by information processing;

FIG. 18 is a flow chart of a program of one embodiment of the present invention;

FIG. 19 is a schematic view of a module including a display unit and a photo sensor unit;

FIG. 20 is a graph showing estimated illuminations and measured illuminations

FIG. 21 is a flow chart of a comparative example; and

FIG. 22 is a graph showing sensor values when a backlight is on and when a backlight is off and differences thereof.

Detailed description of the invention

Embodiments and an example of the present invention will be described in detail with the reference to the drawings. However, the present invention is not limited to the description below, and it is easily understood by those skilled in the art that modes and details disclosed herein can be modified in various ways. Furthermore, the present invention is not construed as being limited to the description of the embodiments and example. In describing structures of the present invention with reference to the drawings, common reference numerals are used for the same portions in different drawings. Note that the same hatched pattern is applied to similar parts, and the similar parts are not especially denoted by reference numerals in some cases.

Note that the size, the thickness of films (layers), or regions in drawings is sometimes exaggerated for simplicity.

Note that the ordinal numbers such as “first” and “second” are used for the sake of convenience and do not denote the order of steps or the stacking order of layers. Therefore, for example, description can be given even when “first” is replaced with “second” or “third”, as appropriate. In addition, the ordinal numbers in this specification and the like are not necessarily the same as those which specify one embodiment of the present invention.

In this specification, an element X is abbreviated as X in some cases. A quantity Y is abbreviated as Y in some cases. Examples of the element include a wiring, a signal line, a power supply line, a circuit, an element, a conductor, an insulator, and a film. Examples of the quantity include a variable, a function, and a parameter. For example, an illumination “illum” is simply referred to as “illum” in some cases. A function “norm” is simply referred to as “norm” in some cases. A signal line SL is simply referred to as SL in some cases. Embodiment 1

In this embodiment, a program and an information processing device of one embodiment of the present invention will be described with reference to drawings.

Structural examples of information processing devices of embodiments of the present invention will be described with reference to FIGS. 1A and 1B and FIG. 2 .

FIG. 1A is a block diagram illustrating the structure of an information processing device 180 A.

The information processing device 180 A illustrated in FIG. 1A includes an execution unit 101 and a memory 102 . The following two sensor images that are output from a photo sensor unit outside the device are input to the information processing device 180 A: a sensor image when a light unit outside the device is lighting (sensor image (light unit=on)) and a sensor image at the time when the light unit is not lighting (sensor image (light unit=off)). The information processing device 180 A outputs display images to be input to a display unit outside the device. A program of one embodiment of the present invention for processing the sensor images is stored in the memory 102 . The execution unit 101 is configured to execute the above program.

The execution unit 101 is, for example, an arithmetic logic unit (ALU). The memory 102 is, for example, a cache memory. In another example, the execution unit 101 may be a central processing unit (CPU). The memory 102 may be a main memory unit.

The information processing device 180 A is connected to an optical touch panel, and is capable of calculating touch data such as a touch position by using the program stored in the memory 102 , for example.

FIG. 1B is a block diagram illustrating the structure of an information processing device 180 B. The information processing device 180 B in FIG. 1B differs in only input/output of display images from the information processing device 180 A in FIG. 1A . Accordingly, for the other structure, the description of that of the information processing device 180 A in FIG. 1A can be referred to as appropriate. Display images are output from the information processing device 180 A in FIG. 1A , whereas display images are input to the information processing device 180 B shown in FIG. 1B . This means that the information processing device 180 A in FIG. 1A produces the display images, whereas the information processing device 180 B in FIG. 1 B does not. Since the program of one embodiment of the present invention is configured to refer to display images, when the information processing device 180 B does not produce the display images, the display images are configured to be externally input.

FIG. 2 is a block diagram illustrating the structure of an information processing device 190 A.

The information processing device 190 A in FIG. 2 includes a display and photo sensor unit (display & PS unit) 120 A and an information processing unit 110 . The display and photo sensor unit 120 A includes a photo sensor unit 121 , a display 122 and a light unit 123 . The information processing unit 110 includes an execution unit 101 and a memory 102 . The following two sensor images which are output from the photo sensor unit 121 are input to the information processing unit 110 : a sensor image at the time when the light unit 123 is lighting and a sensor image at the time when the light unit is not lighting. The information processing unit 110 is configured to output display images to be input to the display 122 . A program of one embodiment of the present invention for processing the sensor images is stored in the memory 102 . The execution unit 101 is configured to execute the above program.

The photo sensor unit 121 includes cells two-dimensionally arranged. Each cell, for example, includes a photodiode and has a function of producing a voltage in accordance with the illumination of incident light. A display 122 includes pixels two-dimensionally arranged. Each pixel includes, for example, sub-pixels displaying respective colors of red, green, and blue. Each sub-pixel includes, for example, a liquid crystal element. The light unit 123 , for example, functions as a backlight of a liquid crystal display unit.

The display 122 may include, for example, a light-emitting element instead of a liquid crystal element. The light-emitting element refers to an element whose luminance is controlled with a current or a voltage. As the light-emitting element, an electroluminescent element (also referred to as an EL element) or the like can be used.

The term “light unit” in this specification refers to a light emission unit provided with a light source and having a function of lighting when the light source emits light. For example, a backlight used in the case where a display unit includes a liquid crystal element is a light unit. A white light-emitting diode or a light-emitting diode that emit more than one color can be used as a light source of the light unit. For example, a display unit that includes a light-emitting element is a light unit, for the display unit itself is provided with the light source. In the case where a display unit includes a light-emitting element, a light unit is not necessarily provided separately from the display unit.

The information processing device 190 A, for example, is provided with an optical touch panel, and can calculate touch data such as a touch position by using the program stored in the memory 102 .

Next, examples of programs of embodiments of the present invention will be described with reference to FIG. 3A to FIG. 8 and FIG. 18 .

FIGS. 3A and 3B are examples of flow charts for explaining programs of embodiments of the present invention. FIG. 3A is an example of a flow chart for explaining Program P 1 with which touch data such as a touch position can be calculated. FIG. 3B is an example of a flow chart for explaining Program P 2 with which illumination under ambient light can be estimated. When touch data such as a touch position is calculated by using Program P 1 , the illumination under the ambient light is used. The illumination calculated by using Program P 2 can be used as the illumination under the ambient light.

The flow chart shown in FIG. 3A includes Steps S 1 , S 2 , S 3 and S 4 .

In Step S 1 , a sensor image at the time when the light unit is lighting and a sensor image at the time when the light unit is not lighting are fetched.

In Step S 2 , the fetched two sensor images are calibrated. The calibration is performed on each of the two sensor images. The calibration is executed by using a sensor image for the calibration. The calibration is executed by using an independent standard value for each cell.

In Step S 3 , an image containing a touch level is calculated by using the two calibrated sensor images. The above calculation is executed by using the illumination and a display image. As the illumination, the illumination under the ambient light or its estimated value is used.

In Step S 4 , the touch data such as a touch position is calculated by using the images containing touch levels.

The information processing performed according to the above steps enables calculation of the touch data such as a touch position. In particular, a stable calculation of the touch data can be executed by using the illumination under the ambient light and/or the display image as well as the sensor image at the time when the light unit is lighting and the sensor image at the time when the light unit is not lighting.

The flow chart shown in FIG. 3B includes Steps T 1 , T 2 , and T 3 .

In Step T 1 , a sensor image at the time when the light unit is not lighting is obtained.

In Step T 2 , a background sensor image is deducted from the obtained sensor image.

The background sensor image can be obtained under the darkest ambient light when the display image is entirely black and the light unit is not lighting. The processing of Step T 2 can reduce an influence caused by a defect and/or a variation of the photo sensor unit.

In Step T 3 , an estimated illumination under ambient light is calculated by using the sensor image from which the background sensor image has been deducted.

The information processing performed according to the above steps enables estimation of the illumination under ambient light.

The flow charts shown in FIG. 3A and FIG. 3B will be described in more detail.

FIG. 4 is an example of a flow chart for explaining Step S 2 (calibration). The flow chart in FIG. 4 includes Steps S 2 - 1 A and S 2 - 1 B.

At the start of Step S 2 , the sensor image at the time when the light unit is lighting and the sensor image at the time when the light unit is not lighting are obtained. The sensor image at the time when the light unit is lighting is expressed as a set of sensor input values si_on(c) for each cell. The sensor image at the time when the light unit is not lighting is expressed as a set of sensor input values si_off(c) for each cell. Note that sensor input means data read from a cell. A sensor input value means its value. The parameter c means a coordinate of the cell.

In Step S 2 - 1 A, the sensor images at the time when the light unit is not lighting are normalized by using a sensor image for calibration.

Examples of the sensor image for calibration are sensor images obtained under two kinds of ambient light when the lighting unit is not lighting and the display image is entirely white. The two kinds of ambient light may be, for example, the brightest ambient light supposed as the user's circumstance (hereinafter simply referred to as the brightest ambient light), and the darkest ambient light supposed as the user's circumstance (hereinafter simply referred to as the darkest ambient light). Assume that the sensor input value obtained under the brightest ambient light is si_off_l(c) and the sensor input value obtained under the darkest ambient light is si_off_d(c). In the following description, it is supposed that si_off_d(c)<si_off_l(c) at any normal cell.

An example of a function for normalization will be described. The function for normalization returns a value that is greater than or equal to s0 and less than or equal to s1 with the function si_off(c) as a parameter (s0 and s1 are real numbers). For example, the function is an increasing function when si_off(c) is a value between si_off_d(c) and si_off_l(c). It returns s0 when si_off(c) is equal to si_off_d(c), and returns s1 when si_off(c) is equal to si_off_l(c).

If a function fn(x) is an increasing function, f(x1)≤fn(x2) is satisfied when x1<x2. If a function fn(x) is a decreasing function, fn(x1)≤fn(x2) is satisfied when x1<x2.

When si_off(c) is smaller than si_off_d(c) in a specified cell, the cell may be labeled as a defective cell. In that case, for example, the defective cell may be distinguished or excluded at the calculation of the touch data. When si_off(c) is smaller than si_off_d(c), the function may return s0.

When si_off(c) is larger than si_off_l(c) in a specified cell, the cell may be labeled as a defective cell. In that case, for example, the defective cell may be distinguished or excluded at the calculation of the touch data. When si_off(c) is larger than si_off_l(c), the function may return s1.

A specific example of a function for normalization is shown. The function for normalization can be a function norm1 shown in Mathematical Formula 1 where si_off(c), si_off_d(c), and si_off_l(c) are substituted for x, x0, and x1, respectively.

[ Mathematical ⁢ ⁢ Formula ⁢ ⁢ 1 ] ⁢ norm ⁢ ⁢ 1 ⁢ ( x ) = { s ⁢ ⁢ 0 for ⁢ ⁢ x < x ⁢ ⁢ 0 s ⁢ ⁢ 1 for ⁢ ⁢ x ⁢ ⁢ 1 < x s ⁢ ⁢ 0 + ( ⁢ s ⁢ ⁢ 1 - s ⁢ ⁢ 0 ) .Math. f ⁡ ( x - x ⁢ ⁢ 0 x ⁢ ⁢ 1 - x ⁢ ⁢ 0 ) for ⁢ ⁢ x ⁢ ⁢ 0 ≤ x ≤ x ⁢ ⁢ 1 ( 1 )

Here, a function f is, for example, an identity function. The function f may be a nonlinear increasing function that converts a value of 0 or more and 1 or less into a value of 0 or more and 1 or less.

Another specific example of a function for normalization is shown. The function for normalization can be a function norm2 shown in Mathematical Formula 2 where si_off(c), si_off_d(c), and si_off_l(c) are substituted for x, x0, and x1, respectively.

[ Mathematical ⁢ ⁢ Formula ⁢ ⁢ 2 ] ⁢ norm ⁢ ⁢ 2 ⁢ ( x ) = { s ⁢ ⁢ 1 for ⁢ ⁢ x ⁢ ⁢ 1 ≤ x s ⁢ ⁢ 0 + ( ⁢ s ⁢ ⁢ 1 - s ⁢ ⁢ 0 ) .Math. f ⁡ ( .Math. x - x ⁢ ⁢ 0 .Math. x ⁢ ⁢ 1 - x ⁢ ⁢ 0 ) for ⁢ ⁢ x < x ⁢ ⁢ 1 ( 2 )

Here, a function f is, for example, an identity function. The function f may be a nonlinear increasing function that converts a value of 0 or more and 1 or less into a value of 0 or more and 1 or less.

For example, s0 may be 0, and s1 may be 1.

In Step S 2 - 1 B, the sensor images at the time when the light unit is lighting are used as the sensor images for calibration. Thus, processing similar to that in the steps S 2 - 1 A may be executed in Step S 2 - 1 B except for this difference.

In this way, the calibrated sensor images are calculated.

Furthermore, in Step S 2 - 1 A, sensor images subjected to two kinds of calibration can be obtained by using as sensor images for calibration four sensor images obtained under two kinds of ambient light when a black image is displayed and when a white image is displayed. The two kinds of ambient light may be, for example, the brightest ambient light and the darkest ambient light.

For example, a sensor input value obtained under the brightest ambient light when a black image is displayed is si_off_lb(c), and a sensor input value obtained under the brightest ambient light when a white image is displayed is si_off_lw(c). A sensor input value obtained under the darkest ambient light when a black image is displayed is si_off_db(c), and a sensor input value obtained under the darkest ambient light when a white image is displayed is si_off_dw(c).

In a normal cell, the above calculation is performed supposing that si_off_dw(c)<si_off_lw(c) and si_off_db(c)<si_off_lb(c) are satisfied. A calibrated sensor image calib_off_black is generated by using si_off_lb(c) and si_off_db(c). In a similar manner, a calibrated sensor image calib_off_white is generated by using si_off_lw(c) and si_off_dw(c).

Furthermore, Step S 2 - 1 B is similar to Step S 1 - 1 A. Two calibrated sensor images can be generated by using as sensor images for calibration four sensor images obtained under the brightest light and the darkest light when a black image or a white image is displayed.

In Step S 2 - 1 B, a sensor input value obtained under the brightest ambient light when a black image is displayed is si_on_lb(c), and a sensor input value obtained under the brightest ambient light when a white image is displayed is si_on_lw(c). A sensor input value obtained under the darkest ambient light when a black image is displayed is si_on_db(c), and a sensor input value obtained under the darkest ambient light when a white image is displayed is si_on_dw(c).

In a normal cell, the above calculation is performed supposing that si_on_dw(c)<si_on_lw(c) and si_on_db(c)<si_on_lb(c) are satisfied. A calibrated sensor image calib_on_black is generated by using si_on_lb(c) and si_on_db(c). A calibrated sensor image calib_on_white is generated by using si_on_lw(c) and si_on_dw(c).

The above description is given supposing that si_off_d(c)<si_off_l(c). This relation depends on a method for reading out sensor images. It is needless to say that calibration can be conducted in a similar way even when the inequality sign is reversed.

An example will be described in which si_off_d(c)>si_off_l(c) is satisfied in a normal cell. In this case, the function for normalization can be, for example, a function norm3 shown in Mathematical Formula 3 where si_off(c), si_off_d(c), and si_off_l(c) are substituted for x, x0, and x1, respectively.

[ Mathematical ⁢ ⁢ Formula ⁢ ⁢ 3 ] ⁢ norm ⁢ ⁢ 3 ⁢ ( x ) = { s ⁢ ⁢ 0 for ⁢ ⁢ x ⁢ ⁢ 0 < x s ⁢ ⁢ 1 for ⁢ ⁢ x < x ⁢ ⁢ 1 s ⁢ ⁢ 0 + ( s ⁢ ⁢ 1 - s ⁢ ⁢ 0 ) .Math. f ⁡ ( x - x ⁢ ⁢ 0 x ⁢ ⁢ 1 - x ⁢ ⁢ 0 ) for ⁢ ⁢ x ⁢ ⁢ 1 ≤ x ≤ x ⁢ ⁢ 0 ( 3 )

Here, a function f is, for example, an identity function. The function f may be a nonlinear increasing function that converts a value of 0 or more and 1 or less into a value of 0 or more and 1 or less.

The above processing in Step S 2 is performed, whereby calibration using different reference values for respective cells can be performed by using the sensor images for calibration. As a result, a defective cell can be labelled to prevent any influence in the subsequent processing even if the defective cell is included in the photo sensor unit. In addition, the influence of variations in characteristics among the cells included in the photo sensor unit can be reduced.

The above processing in Step S 2 enables the calibration depending on the use environment. As a result, the program enables more precise and/or more stable calculation of the touch data.

In Step S 2 described above, the calibration of the sensor images at the time when the light unit is not lighting and the calibration of the sensor images at the time when the light unit is lighting are independently conducted. As a result, the program enables more precise and/or more stable calculation of the touch data.

Note that, in some cases, Step S 2 can be divided into a step of determining the maximum sensor input value and/or the minimum sensor input value for each cell on the basis of the sensor images for calibration and a step of normalizing the sensor input value of each cell by using the maximum sensor input value and/or the minimum sensor input value.

Note that, in some cases, Step S 2 can be divided into a step of labeling a defective cell on the basis of the sensor images for calibration, a step of processing the sensor input value of the defective cell, and a step of processing the sensor input values of cells (also called normal cells) other than the defective cell.

FIG. 18 is an example of a flow chart for explaining Step S 3 (calculation of images containing touch levels). The flow chart in FIG. 18 includes Steps S 3 C- 1 A, S 3 C- 1 B, S 3 C- 1 C, S 3 C- 3 , S 3 C- 4 , and S 3 C- 5 .

At the start of Step S 3 , the calibrated sensor images at the time when the light unit is lighting and the calibrated sensor images at the time when the light unit is not lighting are produced. When the display image is entirely white, whatever the illumination of the ambient light is, a touch level (a quantity expressing the degree of touching) can be obtained by the prescribed calculation using the above two calibrated sensor images. When the display image is entirely black, light from the light unit is blocked, so that less light from the light unit is reflected by a user's finger. In other words, information on touch levels contained in the sensor images at the time when the light unit is lighting decreases. The sensor image at the time when the light unit is not lighting contains more information on a touch level when ambient light is brighter. In contrast, when ambient light is darker, the sensor image at the time when the light unit is not lighting contains less information on a touch level. This is because the degree of the light blocking by the existence of a finger (whether a touch panel is touched) is higher when ambient light is brighter. Therefore, it is effective to take the illumination into account to extract the information on the touch level. In Step S 3 , the information on the touch level is extracted as above, taking the illumination of the ambient light and the display images into account.

In Step S 3 C- 1 A, an image containing a touch level is calculated by a calculating method for calculating touch levels when the display image is entirely white, using the whole or part of the two calibrated sensor images.

The foregoing image containing a touch level is expressed as a set of touch levels tl_w(c) for each cell. The touch level is a quantity expressing a degree of touching. The parameter c means a coordinate of the cell. For example, it can be 0 in the case of a standard touching state (full touching state) and 1 in the case of a standard not-touching state (no touching state). The full touching state may be, for example, a state of being actually touched under a real environment. The no touching state may be, for example, a state of being actually not touched under a real environment. Either of them may be a virtual state, which is decided by adjusting parameters adequately.

In Step S 3 C- 1 B, an image containing a touch level is calculated by a calculating method for calculating touch levels when the display image is entirely black under the brightest ambient light, using the whole or part of the two calibrated sensor images. The foregoing image containing a touch level is expressed as a set of touch levels tl_bl(c) for each cell.

In Step S 3 C- 1 C, an image containing a touch level is calculated by a calculating method for calculating touch levels when the display image is entirely black under the darkest ambient light, using the whole or part of the two calibrated sensor images. The foregoing image containing a touch level is expressed as a set of touch levels tl_bd(c) for each cell.

In Step S 3 C- 3 , a display image is obtained.

In Step S 3 C- 4 , the obtained display image is converted into a gray scale.

The display image is expressed by a set of display output values to be output to pixels. In the case where each pixel includes sub-pixels of red, green, and blue, the display output values can be expressed as do_red(p), do_green(p), and do_blue(p), which are the display output values of the sub-pixels of red, green, and blue, respectively. A parameter p represents a coordinate of the pixel. In the case of an 8-bit grayscale, do_red(p), do_green(p), and do_blue(p) are each an integer of 0 to 255 inclusive.

The conversion into a grayscale may be performed using appreciate weight for each color. For example, a value gdo(p) obtained by converting the display output value of the pixel into a grayscale can be expressed as follows: gdo(p) w_red.Math.do_red(p)+w_green.Math.do_green(p)+w_blue.Math.do_blue(p). The values w_red, w_green, and w_blue are weights of red, green, and blue, respectively, in conversion into a grayscale.

The description continues in the full USPTO document.