Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Biological Inspired Deformable Image Sensor
Blekinge Institute of Technology, Faculty of Computing, Department of Technology and Aesthetics.ORCID iD: 0000-0003-3887-5972
2019 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Nowadays, cameras are everywhere thanks to the tremendous progress on sensor technology. However, their performance is far away from what we experience by our eyes. The study from evolution process shows how the sensor arrangement of retina in human vision has differentiated from other species and is formed into a specific combination of sub-arrangements from hexagonal to elliptical ones. There are three major key differences between our visual cell arrangement and current camera sensors which are: the sub-arrangements, the pixel form and the pixel density.

Despite the advances in sensor technology we face limitations in their further development; i.e. to make the cameras close to the visual system. This is due to the optical diffraction limit which prevents us to increase the sensor resolution, and rigidity of hardware implementation which prevent us to change the image sensor after manufacturing. In the thesis the possibilities to overcome such limitations are investigated where the intention is to find a closer sensory solution to the visual system in comparison to the current ones.

Breaking the diffraction barrier and solving the rigidity problem are simultaneously achieved by introducing and estimating virtual subpixels. A statistical framework consisting of local learning model and Bayesian inference for predicting the incident photons captured on each such a subpixel is used to resample the captured image by any current camera sensor. By investigating the virtual variation of pixel size and fill factor the validity of the proposed idea is proven by which the results show significant changes of dynamic range and tonal levels in relation to the variation. As an example, for both monochrome and color images the results show that by virtual increase of fill factor to 100%, the dynamic range of the images are widened and the tonal levels are enriched significantly over 256 levels for each channel.

The results of virtual variation of the fill factor and pixel size indicates that it is feasible to change the rigidity of the image sensor using the software-based method. Inspired by the mosaic in the fovea, the center of human retina, the hexagonal sub-arrangement and pixel form are proposed to generate images based on the estimated virtual subpixels. Compared to the original square images, not only the dynamic range and tonal levels are improved, but also the hexagonal images are superior in detection of edges, i.e. more edge points on the contour of the objects are detected in hexagonal images.

The evaluation of different sub-arrangements or pixel forms of the image sensor is a challenging task and should be directed to a more specific task. Since the curvature contours contain most of the information related to object perception and human vision is highly evolved to detect curvature object, the task is focused to investigate the impact of the curviness on the different pixel forms and sub-arrangements, by comparing two categories of images; having curved versus linear edges of the objects in a pair of images which have exact similar contents but different contours. The detectability of each of the different sensor structures for curviness is estimated and the results show that the image on hexagonal grid with hexagonal pixel form is the best image type for distinguishing the curvature contours in the images.

According to the pattern of pixels tiling, there are two types of pixel sub-arrangements, i.e. periodic (e.g. square or hexagonal), and aperiodic (e.g. Penrose). Each type of sub-arrangements is investigated where the pixel forms and density are variable. By having at least two generated images of one configuration (i.e. specific sub-arrangement, pixel form and density), the result of histogram of gradient orientation of the certain sensor arrangement shows a stable and specific distribution which we called it the ANgular CHaracteristic of a sensOR structure (ANCHOR). Each ANCHOR has a robust pattern which is changed by the change of the sensor sub-arrangement. This makes it feasible to plan a sensor sub-arrangement in the relation to a specific application and its requirements, and more alike the biological vision sensory. To generate such a flexible sensor, a general framework is proposed for virtual deforming the sensor with a certain configuration of the sensor sub-arrangement, pixel form and pixel density.

Assessing the quality difference between the images generated by different sensor configuration or addressing from on configuration to another one generally needs the conversion of one to another. To overcome this problem, a common space is proposed by implementing a continuous extension of square or hexagonal images based on the orbit function, for quality evaluating the images with different arrangements and addressing from one type of image to another one. The evaluation results show that the creation of such space is feasible which facilitates a usage friendly tool to address an arrangement and assess the changes between different spatial arrangements, for example, it shows richer intensity variation, nonlinear behavior, and larger dynamic range in the hexagonal images compared to the rectangular images.

Place, publisher, year, edition, pages
Karlskrona: Blekinge Tekniska Högskola, 2019. , p. 207
Series
Blekinge Institute of Technology Doctoral Dissertation Series, ISSN 1653-2090 ; 4
Keywords [en]
image sensor, pixel form, sub-arrangements, fill factor, square image, hexagonal image, deformable sensor, quality assessment.
National Category
Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
URN: urn:nbn:se:bth-17149ISBN: 978-91-7295-366-6 (print)OAI: oai:DiVA.org:bth-17149DiVA, id: diva2:1272821
Public defence
2019-03-14, J1650, Campus Gräsvik, Karlskrona, 13:15 (English)
Opponent
Supervisors
Available from: 2018-12-20 Created: 2018-12-20 Last updated: 2019-03-05Bibliographically approved
List of papers
1. Novel Software-based Method to Widen Dynamic Range of CCD Sensor Images
Open this publication in new window or tab >>Novel Software-based Method to Widen Dynamic Range of CCD Sensor Images
2015 (English)In: / [ed] Yu-Jin Zhang, Springer, 2015, Vol. 9218, p. 572-583Conference paper, Published paper (Refereed)
Abstract [en]

In the past twenty years, CCD sensor has made huge progress in improving resolution and low-light performance by hardware. However due to physical limits of the sensor design and fabrication, fill factor has become the bottle neck for improving quantum efficiency of CCD sensor to widen dynamic range of images. In this paper we propose a novel software-based method to widen dynamic range, by virtual increase of fill factor achieved by a resampling process. The CCD images are rearranged to a new grid of virtual pixels com-posed by subpixels. A statistical framework consisting of local learning model and Bayesian inference is used to estimate new subpixel intensity. By knowing the different fill factors, CCD images were obtained. Then new resampled images were computed, and compared to the respective CCD and optical image. The results show that the proposed method is possible to widen significantly the recordable dynamic range of CCD images and increase fill factor to 100 % virtually.

Place, publisher, year, edition, pages
Springer, 2015
Series
Lecture Notes in Computer Science, ISSN 0302-9743
Keywords
Dynamic range, Fill factor, CCD sensors, Sensitive area, Quantum efficiency
National Category
Signal Processing
Identifiers
urn:nbn:se:bth-11169 (URN)10.1007/978-3-319-21963-9_53 (DOI)978-3-319-21963-9 (ISBN)
Conference
International Conference on Image and Graphics 2015, Tianjin, China
Available from: 2015-12-11 Created: 2015-12-11 Last updated: 2018-12-20Bibliographically approved
2. A Software Method to Extend Tonal Levels and Widen Tonal Range of CCD Sensor Images
Open this publication in new window or tab >>A Software Method to Extend Tonal Levels and Widen Tonal Range of CCD Sensor Images
2015 (English)In: 2015 9TH INTERNATIONAL CONFERENCE ON SIGNAL PROCESSING AND COMMUNICATION SYSTEMS (ICSPCS), IEEE Communications Society, 2015Conference paper, Published paper (Refereed)
Abstract [en]

As one of important outcomes of the past decades of researches on sensor arrays for digital cameras, the manufacturers of sensor array technology have responded to the necessity and importance of obtaining an optimal fill factor, which has great impact on collection of incident photons on the sensor, with hardware solution e.g. by introducing microlenses. However it is still impossible to make a fill factor of 100% due to the physical limitations in practical development and manufacturing of digital camera. This has been a bottle neck problem for improving dynamic range and tonal levels for digital cameras e.g. CCD cameras. In this paper we propose a software method to not only widen the recordable dynamic range of a captured image by a CCD camera but also extend its tonal levels. In the method we estimate the fill factor and by a resampling process a virtual fill factor of 100% is achieved where a CCD image is rearranged to a new grid of virtual subpixels. A statistical framework including local learning model and Bayesian inference is used for estimating new sub-pixel intensity values. The highest probability of sub-pixels intensity values in each resampled pixel area is used to estimate the pixel intensity values of the new image. The results show that in comparison to the methods of histogram equalization and image contrast enhancement, which are generally used for improving the displayable dynamic range on only one image, the tonal levels and dynamic range of the image is extended and widen significantly and respectively.

Place, publisher, year, edition, pages
IEEE Communications Society, 2015
Keywords
CCD sensor, tonal range, fill factor, quantum efficiency
National Category
Communication Systems
Identifiers
urn:nbn:se:bth-12967 (URN)000380405700050 ()978-1-4673-8118-5 (ISBN)
External cooperation:
Conference
9th International Conference on Signal Processing and Communication Systems (ICSPCS), DEC 14-16, 2015, Cairns, AUSTRALIA
Available from: 2016-08-31 Created: 2016-08-30 Last updated: 2018-12-20Bibliographically approved
3. Estimation of Image Sensor Fill Factor Using a Single Arbitrary Image
Open this publication in new window or tab >>Estimation of Image Sensor Fill Factor Using a Single Arbitrary Image
2017 (English)In: Sensors, E-ISSN 1424-8220, Vol. 17, no 3, p. 620-Article in journal (Refereed) Published
Abstract [en]

Achieving a high fill factor is a bottleneck problem for capturing high-quality images. There are hardware and software solutions to overcome this problem. In the solutions, the fill factor is known. However, this is an industrial secrecy by most image sensor manufacturers due to its direct effect on the assessment of the sensor quality. In this paper, we propose a method to estimate the fill factor of a camera sensor from an arbitrary single image. The virtual response function of the imaging process and sensor irradiance are estimated from the generation of virtual images. Then the global intensity values of the virtual images are obtained, which are the result of fusing the virtual images into a single, high dynamic range radiance map. A non-linear function is inferred from the original and global intensity values of the virtual images. The fill factor is estimated by the conditional minimum of the inferred function. The method is verified using images of two datasets. The results show that our method estimates the fill factor correctly with significant stability and accuracy from one single arbitrary image according to the low standard deviation of the estimated fill factors from each of images and for each camera.

Place, publisher, year, edition, pages
MDPI, 2017
Keywords
fill factor; virtual image; image sensor; pipeline; virtual response function; sensor irradiance
National Category
Computer Vision and Robotics (Autonomous Systems)
Identifiers
urn:nbn:se:bth-14046 (URN)10.3390/s17030620 (DOI)000398818700193 ()
Note

open access

Available from: 2017-03-24 Created: 2017-03-24 Last updated: 2022-02-10Bibliographically approved
4. Image Quality Assessment of Enriched Tonal Levels Images
Open this publication in new window or tab >>Image Quality Assessment of Enriched Tonal Levels Images
2017 (English)In: Image and Graphics 9th International Conference, ICIG 2017, Shanghai, China, September 13-15, 2017, Revised Selected Papers, Part II / [ed] Yao Zhao, Xiangwei Kong, David Taubman, Springer, 2017, Vol. 10668, p. 134-146Conference paper, Published paper (Refereed)
Abstract [en]

The quality assessment of a high dynamic image is a challenging task. The few available no reference image quality methods for high dynamic range images are generally in evaluation stage. The most available image quality assessment methods are designed to assess low dynamic range images. In the paper, we show the assessment of high dynamic range images which are generated by utilizing a virtually flexible fill factor on the sensor images. We present a new method in the assessment process and evaluate the amount of improvement of the generated high dynamic images in comparison to original ones. The results show that the generated images not only have more number of tonal levels in comparison to original ones but also the dynamic range of images have significantly increased due to the measurable improvement values.

Place, publisher, year, edition, pages
Springer, 2017
Keywords
Image quality assessment, High dynamic range image, Tonal levels, Tone mapping, Fill factor
National Category
Signal Processing
Identifiers
urn:nbn:se:bth-15709 (URN)10.1007/978-3-319-71598-8_13 (DOI)000612986800013 ()978-3-319-71597-1 (ISBN)
Conference
9th International Conference on Image and Graphics, Shanghai
Available from: 2018-01-02 Created: 2018-01-02 Last updated: 2021-12-22Bibliographically approved
5. Back to basics: Towards novel computation and arrangement of spatial sensory in images
Open this publication in new window or tab >>Back to basics: Towards novel computation and arrangement of spatial sensory in images
2016 (English)In: Acta Polytechnica, ISSN 1210-2709, Vol. 56, no 5, p. 409-416Article in journal (Refereed) Published
Abstract [en]

The current camera has made a huge progress in the sensor resolution and the low-luminance performance. However, we are still far from having an optimal camera as powerful as our eye is. The study of the evolution process of our visual system indicates attention to two major issues: the form and the density of the sensor. High contrast and optimal sampling properties of our visual spatial arrangement are related directly to the densely hexagonal form. In this paper, we propose a novel software-based method to create images on a compact dense hexagonal grid, derived from a simulated square sensor array by a virtual increase of the fill factor and a half a pixel shifting. After that, the orbit functions are proposed for a hexagonal image processing. The results show it is possible to achieve image processing operations in the orbit domain and the generated hexagonal images are superior, in detection of curvature edges, to the square images. We believe that the orbit domain image processing has a great potential to be the standard processing for hexagonal images.

Place, publisher, year, edition, pages
CZECH TECHNICAL UNIV PRAGUE, 2016
Keywords
Convolution, Fill factor, Hexagonal pixel, Hexagonal processing, Hexagonal sensor array, Orbit functions, Orbit transform, Square pixel
National Category
Other Computer and Information Science
Identifiers
urn:nbn:se:bth-13798 (URN)10.14311/AP.2016.56.0409 (DOI)000411584300010 ()2-s2.0-85008440583 (Scopus ID)
Note

open access

Available from: 2017-01-20 Created: 2017-01-20 Last updated: 2018-12-20Bibliographically approved
6. The impact of curviness on four different image sensor forms and structures
Open this publication in new window or tab >>The impact of curviness on four different image sensor forms and structures
2018 (English)In: Sensors, E-ISSN 1424-8220, Vol. 18, no 2, article id 429Article in journal (Refereed) Published
Abstract [en]

The arrangement and form of the image sensor have a fundamental effect on any further image processing operation and image visualization. In this paper, we present a software-based method to change the arrangement and form of pixel sensors that generate hexagonal pixel forms on a hexagonal grid. We evaluate four different image sensor forms and structures, including the proposed method. A set of 23 pairs of images; randomly chosen, from a database of 280 pairs of images are used in the evaluation. Each pair of images have the same semantic meaning and general appearance, the major difference between them being the sharp transitions in their contours. The curviness variation is estimated by effect of the first and second order gradient operations, Hessian matrix and critical points detection on the generated images; having different grid structures, different pixel forms and virtual increased of fill factor as three major properties of sensor characteristics. The results show that the grid structure and pixel form are the first and second most important properties. Several dissimilarity parameters are presented for curviness quantification in which using extremum point showed to achieve distinctive results. The results also show that the hexagonal image is the best image type for distinguishing the contours in the images. © 2018 by the authors. Licensee MDPI, Basel, Switzerland.

Place, publisher, year, edition, pages
MDPI AG, 2018
Keywords
Critical points, Curviness quantification, Fill factor, Grid structure, Hessian matrix, Hexagonal image, Pixel form, Software-based, Virtual, Image processing, Image sensors, Semantics, Grid structures, Hessian matrices, Pixels
National Category
Other Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
urn:nbn:se:bth-15920 (URN)10.3390/s18020429 (DOI)000427544000112 ()2-s2.0-85041511195 (Scopus ID)
Note

open access

Available from: 2018-02-22 Created: 2018-02-22 Last updated: 2022-02-10Bibliographically approved
7. Virtual deformable image sensors: Towards to a general framework for image sensors with flexible grids and forms
Open this publication in new window or tab >>Virtual deformable image sensors: Towards to a general framework for image sensors with flexible grids and forms
2018 (English)In: Sensors, E-ISSN 1424-8220, Vol. 18, no 6, article id 1856Article in journal (Refereed) Published
Abstract [en]

Our vision system has a combination of different sensor arrangements from hexagonal to elliptical ones. Inspired from this variation in type of arrangements we propose a general framework by which it becomes feasible to create virtual deformable sensor arrangements. In the framework for a certain sensor arrangement a configuration of three optional variables are used which includes the structure of arrangement, the pixel form and the gap factor. We show that the histogram of gradient orientations of a certain sensor arrangement has a specific distribution (called ANCHOR) which is obtained by using at least two generated images of the configuration. The results showed that ANCHORs change their patterns by the change of arrangement structure. In this relation pixel size changes have 10-fold more impact on ANCHORs than gap factor changes. A set of 23 images; randomly chosen from a database of 1805 images, are used in the evaluation where each image generates twenty-five different images based on the sensor configuration. The robustness of ANCHORs properties is verified by computing ANCHORs for totally 575 images with different sensor configurations. We believe by using the framework and ANCHOR it becomes feasible to plan a sensor arrangement in the relation to a specific application and its requirements where the sensor arrangement can be planed even as combination of different ANCHORs. © 2018 by the authors. Licensee MDPI, Basel, Switzerland.

Place, publisher, year, edition, pages
MDPI AG, 2018
Keywords
Deformable sensor, Framework, Hexagonal, HoG, Penrose, Pixel form, Sensor grid, Deformation, Image sensors, Pixels, Histogram of gradients, Sensor arrangements, Sensor configurations, Sensor grids, Specific distribution, Anchors
National Category
Other Electrical Engineering, Electronic Engineering, Information Engineering Other Computer and Information Science
Identifiers
urn:nbn:se:bth-16631 (URN)10.3390/s18061856 (DOI)000436774300190 ()2-s2.0-85048303219 (Scopus ID)
Note

open access

Available from: 2018-06-27 Created: 2018-06-27 Last updated: 2022-02-10Bibliographically approved

Open Access in DiVA

fulltext(59692 kB)340 downloads
File information
File name FULLTEXT02.pdfFile size 59692 kBChecksum SHA-512
afcb0f16458c41c574e8aa521b0f52937d8dfb6656cf89f667a5f7867928e7aa9b156e8538bdc6c1c1524332d704e28226aa16a190fac1a4fadc0f2ddb6a062f
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Wen, Wei
By organisation
Department of Technology and Aesthetics
Electrical Engineering, Electronic Engineering, Information Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 368 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 1321 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf