This is a rough plan before any code is written, it is a plan and we all know in life plans can change.
The objective of our project is to Isolate (by masking) the cell identified by a user's mouse click in the sample shown on the previous blog entry.This can be broken down into 4 key steps.
- Load image
- Identifying cells
- Watershed Algorithm
- Cell selection
1. Load Image
The first step is to load the image that we are wish to work with. for this project we will use cervical.jpg. But going forward we will design the code in a way that will allow other similar images to be used. Loading the image is probably the easiest step of this project as it is required in every image processing task that myself and other members of my team have done to date. This is simply done by using the line:
I = cv2.imread("cervical.jpg")
The openCV function imread loads an image from the specified file and stores it as an array I . To display I we use the function:
cv2.imshow("image", I)
This will take the array I and display it in a window.
2. Identifying cells
When the image is displayed in the window the next step is to identify what is a cell and what isn't and show the individual cells. This can be done automatically when the image is displayed or add an extra piece of code so that the user has to prompt the identification of cells. This can be achieved by displaying the image with a prompt that could say "Press I to locate cells" the program waits for the user to enter the letter I and then the cells will be identified with a colored line around them. But how will the cells be identified?
When something needs to be identified from an image the easiest way is looking at the characteristics of what is being identified with respect to everything else in that image. This could mean a specific color range/shape, then isolating that from everything else in the image. Then the next step is to separate the larger clumps of cells to identify individual cells.
The cells can be identified by a process called isolation by way of masking. The first step is to identify what is a cell and what isn't. This is done by creating a mask. A mask is a filter that can be used to separate a region of interest from a region of insignificance. Masks are used for edge detection but are also used for a wide range of other applications.
The mask is generated by applying a threshold to the image there are two main thresholding techniques , Simple thresholding and Adaptive thresholding. other thresholding techniques do exist and may be looked into later which include Otsu thresholding. First the image is converted to grey-scale then the threshold is applied.
Simple thresholding:"If pixel value is greater than a threshold value, it is assigned one value (may be white), else it is assigned another value (may be black) [1]".
Simple thresholding:"If pixel value is greater than a threshold value, it is assigned one value (may be white), else it is assigned another value (may be black) [1]".
retval, dst=cv.threshold(src, thresh, maxval, type[, dst])
The different types of of simple thresholding are:
Adaptive thresholding: "In this, the algorithm calculate the threshold for a small regions of the image. So we get different thresholds for different regions of the same image and it gives us better results for images with varying illumination [1]".
dst = cv.adaptiveThreshold( src, maxValue, adaptiveMethod, thresholdType, blockSize, C[, dst]
The different types of of adaptive thresholding are:
In an ideal world once the appropriate thresholds are set (which could be either simple/adaptive or a mixture of both or by other methods) the mask is a perfect representation of the region of interest. This however is hardly ever the case with real world images. therefore it is likely that the mask will contain some amount of noise. This can be resolved in a couple of different ways by smoothing the image before applying the threshold or using morphological transformations .
Smoothing images can be achieved with convolution and image blurring (average, Gaussian, Median or bilateral filtering) the image. Whereas morphological transformations include such techniques as Erosion, Dilation, Opening, Closing etc of the mask. I feel that the morphological technique of Opening is the most appropriate for this application. "Opening is just another name of erosion followed by dilation. It is useful in removing noise [2]". By Eroding the image which will reduce the boundaries of foreground object removing small amounts of noise completely and then applying dilation which will increase the boundaries of foreground again in theory should remove a large amount of noise. This can be used in conjunction with some of the convolution techniques could result in a noise free mask.
The mask may then be applied to the image which will show only the region of interest (ie. the cells)
The idea is to fill isolated valleys with different colored water. The valleys in this case are known as local minima which is the lowest intensity areas of the grey-scale image. The different color water areas are known as labels. applying the watershed starts the flood and as the water rises different color waters will begin to merge. To avoid this from happening barriers are added where the water tries to merge. barriers are continuously put up as the water tries to merge in different areas until the last peak is under water.
Smoothing images can be achieved with convolution and image blurring (average, Gaussian, Median or bilateral filtering) the image. Whereas morphological transformations include such techniques as Erosion, Dilation, Opening, Closing etc of the mask. I feel that the morphological technique of Opening is the most appropriate for this application. "Opening is just another name of erosion followed by dilation. It is useful in removing noise [2]". By Eroding the image which will reduce the boundaries of foreground object removing small amounts of noise completely and then applying dilation which will increase the boundaries of foreground again in theory should remove a large amount of noise. This can be used in conjunction with some of the convolution techniques could result in a noise free mask.
The mask may then be applied to the image which will show only the region of interest (ie. the cells)
3. Watershed Algorithm [3]
Next we need to identify individual cells this can be done by applying a watershed algorithm. Any grey-scale image can be seen as a topographic surface. for those who haven't taken geography in school a topographic surface is a surface with peaks, troughs, hills and valleys. The peaks and hills denotes the high intensity parts of the grey scale image and the troughs and valleys denote the low intensity parts. The watershed algorithm applies a flood to the topographic surface which will denote the cell boundries.
The idea is to fill isolated valleys with different colored water. The valleys in this case are known as local minima which is the lowest intensity areas of the grey-scale image. The different color water areas are known as labels. applying the watershed starts the flood and as the water rises different color waters will begin to merge. To avoid this from happening barriers are added where the water tries to merge. barriers are continuously put up as the water tries to merge in different areas until the last peak is under water.
The resulting barriers give the segmentation result. This however tends to give a over-segmented result due to noise and other regularities. This should be minimal in the case of our program as we will have performed a lot of the noise removal in the last step. However as an added step to remove any additional noise OpenCV have implemented an interactive marker-based watershed algorithm. different labels are given for our object. The following labels are given:
- Region which we are sure of being the foreground or object with one color (or intensity)
- Region which we are sure of being the background or or non-object with another colour
- Region which we are not sure of anything is labeled with 0.
The watershed algorithm is then applied. The marker will be updated with the lables that have been given and the boundaries of the objects will have a value of -1.
The following code is an example of a watershed algorithm identifying the borders of coins
import numpy as np
import cv2
from matplotlib import pyplot as plt
img = cv2.imread('coins.png')
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
ret, thresh = cv2.threshold(gray,0,255,cv2.THRESH_BINARY_INV+cv2.THRESH_OTSU)
# noise removal
kernel = np.ones((3,3),np.uint8)
opening = cv2.morphologyEx(thresh,cv2.MORPH_OPEN,kernel, iterations = 2)
# sure background area
sure_bg = cv2.dilate(opening,kernel,iterations=3)
# Finding sure foreground area
dist_transform = cv2.distanceTransform(opening,cv2.DIST_L2,5)
ret, sure_fg = cv2.threshold(dist_transform,0.7*dist_transform.max(),255,0)
# Finding unknown region
sure_fg = np.uint8(sure_fg)
unknown = cv2.subtract(sure_bg,sure_fg)
# Marker labelling
ret, markers = cv2.connectedComponents(sure_fg)
# Add one to all labels so that sure background is not 0, but 1
markers = markers+1
# Now, mark the region of unknown with zero
markers[unknown==255] = 0
markers = cv2.watershed(img,markers)
img[markers == -1] = [255,0,0]
 |
| Original Image |
 |
| Marker Image after segmentation and Image after watershed algorithm |
4. Cell selection
Once each of the cells are clear with their borders defined the next step is to give the user the power to be able to select a cell of interest. When the user selects a cell they wish to examine everything else bar that cell is removed from the window leaving the cell isolated. The cell is then cropped removing all the unnecessary blank space left behind from the cells that were not worthy of being selected.
When the user selects a cell, the pixel location is taken and the pixels RGB value of that location is taken from the marker image after segmentation from the watershed algorithm. This can be done with the line:
if event == cv2.EVENT_LBUTTONDOWN:
refPt = [(x, y)]
When the user selects a cell, the pixel location is taken and the pixels RGB value of that location is taken from the marker image after segmentation from the watershed algorithm. This can be done with the line:
if event == cv2.EVENT_LBUTTONDOWN:
refPt = [(x, y)]
The first line detects whether the left mouse button has gone down and if so
Then a thresholding is applied to the marker image leaving everything bar that color. This is then turned into a binary mask that can used on the original image resulting in an image with only the cell that was selected and everything . The cell can then be cropped using the following function that I had used in a previous assignment.
# Crop Image
p = np.argwhere(thresh1==0)
p = np.fliplr(p)
x, y, w, h = cv2.boundingRect(p)
x = x-10
y = y-10
w = w+20
h = h+20
c = ROI[y:y+h, x:x+w]
np.argwhere() - Finds the indices of array elements that are non-zero, grouped by element.
np.fliplr() -Flips array in the left/right direction. Flipa the entries in each row in the left/right direction. Columns are preserved, but appear in a different order than before.
cv2.boundingRect(p) - Returns x, y, w, h. Lets (x,y) be the top-left coordinate of the rectangle and (w,h) be its width and height.
The cropped image is then to be saved to a file location using the line:
Then a thresholding is applied to the marker image leaving everything bar that color. This is then turned into a binary mask that can used on the original image resulting in an image with only the cell that was selected and everything . The cell can then be cropped using the following function that I had used in a previous assignment.
# Crop Image
p = np.argwhere(thresh1==0)
p = np.fliplr(p)
x, y, w, h = cv2.boundingRect(p)
x = x-10
y = y-10
w = w+20
h = h+20
c = ROI[y:y+h, x:x+w]
np.argwhere() - Finds the indices of array elements that are non-zero, grouped by element.
np.fliplr() -Flips array in the left/right direction. Flipa the entries in each row in the left/right direction. Columns are preserved, but appear in a different order than before.
cv2.boundingRect(p) - Returns x, y, w, h. Lets (x,y) be the top-left coordinate of the rectangle and (w,h) be its width and height.
The cropped image is then to be saved to a file location using the line:
cv2.imwrite('image.png',img)
References
[1 ]Anon, (2018). [online] Available at: https://docs.opencv.org/3.4.0/d7/d4d/tutorial_py_thresholding.htmlaa669a943a [Accessed 12 Apr. 2018].
[2] Docs.opencv.org. (2018). Morphological Transformations — OpenCV 3.0.0-dev documentation. [online] Available at: https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_imgproc/py_morphological_ops/py_morphological_ops.html [Accessed 12 Apr. 2018].
[3] Docs.opencv.org. (2018). OpenCV: Image Segmentation with Watershed Algorithm. [online] Available at: https://docs.opencv.org/3.1.0/d3/db4/tutorial_py_watershed.html [Accessed 12 Apr. 2018].
[4] N. Li, M. Liu and Y. Li, "Image Segmentation Algorithm using Watershed Transform and Level Set Method," 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07, Honolulu, HI, 2007, pp. I-613-I-616.
[5] Hai Gao, Ping Xue and Weisi Lin, "A new marker-based watershed algorithm," 2004 IEEE International Symposium on Circuits and Systems (IEEE Cat. No.04CH37512), 2004, pp. II-81-4 Vol.2.
[4] N. Li, M. Liu and Y. Li, "Image Segmentation Algorithm using Watershed Transform and Level Set Method," 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07, Honolulu, HI, 2007, pp. I-613-I-616.
[5] Hai Gao, Ping Xue and Weisi Lin, "A new marker-based watershed algorithm," 2004 IEEE International Symposium on Circuits and Systems (IEEE Cat. No.04CH37512), 2004, pp. II-81-4 Vol.2.
No comments:
Post a Comment