Tutorial 2: Interacting with images in phenopype¶

In this tutorial we learn how to open and close images in phenopype, and how to use the interactive featues of the program. Phenopype uses OpenCV’s HighGUI module to display images and to allow users to interact with images. HighGUI has a few pros and cons:

+ native OpenCV GUI (no extra GUI libraries required)
+ the module is extremely fast in displaying images 
+ it can display very large image-arrays (> 10000x10000 pixels)
+ it can display multiple images side by side
+ interactions (drawing and measuring) are possible 

- sometimes unstable (e.g. windows are not closed but freeze)
- issues with cross plattform stability (e.g. on macOS)
- displaying instructions is hacky (text is "painted" onto a displayed image) 
- user input (key strokes and mouse clicks) sometimes isn't captured properly

Currently Phenopype uses the standard HighGUI libraries that ship with the most recent precompiled opencv-contrib-python package, which is Qt for Linux and macOS, and Win32 UI on Windows. The Qt GUI is a bit more userfriendly with builtin buttons, scrollbars, RGB info and zoom, but you don’t actually need those things for basic Phenopype GUI interactions.

GUI control¶

IMPORTANT - read before continuing:

Open

In a very simple case, where you want to just inspect an image file from within phenopype, you use the load_image function to load an image file into Python as an array, and then display it with show_image. A HighGUI window will pop up and display the array. While the window is open, the Python kernel is “busy”, and you cannot interact with the console, i.e. run any code - you first have to close the image again.

Close

Although the HighGUI window has the “red crossed” close button in the upper right corner, DO NOT USE IT! For practical reasons, phenopype relies on key strokes to control the windows. Make sure that the window is selected / highlighted, and use the following key combinations to close it:

Enter - close window or finish an interactive function in pype-mode
Ctrl+Enter - close and finish a window in pype-mode
Esc - close a window and quit the Phenoype process that invoked it. This may also work when the process is frozen.

Issues

Unfortunately the HighGUI isn’t exactly stable, and may sometimes behave unexpectedly. Here are solutions to the most comment issues:

If a keystroke doen’t do anything the first time, try a few times more (Phenopype “listens” to your keystroke while refreshing the presented image, and sometimes a refreshing operation overlaps with user input and it is not recognized).
If your keystore still isn’t recognized, make sure the window is highlighted (i.e. click on it) and try again.
If you killed a process but the window still open (i.e. the kernel is not busy anymore), type import cv2 and cv2.destroyAllWindows() into the console to close the window.
If a window and the Python kernel is frozen permanently, you need to restart it - sorry!

Opening images in phenopype¶

To open an image, it first has to be loaded as an array using load_image. The array can then be passed on to show_image, which simply displays an image (no interactions, except zooming in using the mousewheel). The window is closed by keystroke (Enter, or Esc which both closes the window and ends ongoing processes).

import phenopype as pp
import os

img = pp.load_image(os.path.join("data",'stickle1.jpg')) 
pp.show_image(img)

show_image can also handle multiple images. Here we loop through the images folder, attach all images to a list, and pass that list of arrays to the functions - it will give a warning if more than 10 images are being opened at the same time.

import os
images = [] # square-brakets make an empty list
names = os.listdir("data") # making a list of all the files names inside a directory

for i in names: # looping along our list of names
    filepath = os.path.join("data", i) # joining name and path strings 
    images.append(pp.load_image(filepath)) # load images and store them in list

pp.show_image(images, position_offset=100) ## show all images in the image folder in the tutorial directory

Invalid file extension ".mp4" - could not load image:

skipped showing list item of type NoneType
phenopype - 2
phenopype - 3
phenopype - 4
phenopype - 5
phenopype - 6

The function has a few more options to arrange and separte images across the screen. Future version of phenopype will do this in more meaningful manner (e.g., arrange small images side by side until the screen is filled).

pp.show_image(images, 
              window_max_dim=100,     # maximum dimension (in either direction) for the windows
              position_offset=100,    # window offset if multiple windows are displayed
              position_reset=True)    # reset window positions (i.e. window position will not be remembered from previous call)

skipped showing list item of type NoneType
phenopype - 2
phenopype - 3
phenopype - 4
phenopype - 5
phenopype - 6

Creating masks¶

Masking, i.e. removing unwanted parts of an image that contain noise by including or excluding certain parts of the image, is an important preprocessing step in any computer vision workflow. Phenopype’s create_mask tool provides flexibility when drawing masks.

Create masks

Fig. 1: Phenopype’s mask tool in action. You can include or exclude certain parts of the image; the resulting coordinates are recognized in subsequent computer vision steps (e.g. thresholding)

Using create_mask results in a DataFrame object that contains coordinates of the created mask. You can add multiple “submasks” that belong to the same mask layer that don’t have to be connected, and right click takes you one step back. Single polygons are finished with Ctrl, finish with Enter - the last open polyon will automatically be completed. Finish with Enter.

img = pp.load_image(os.path.join("data",'stickle1.jpg')) 
mask_plates = pp.preprocessing.create_mask(img, tool="polygon", label="plates")
mask_plates

{'mask': {'a': {'info': {'annotation_type': 'mask',
    'phenopype_function': 'create_mask',
    'phenopype_version': '3.2.3'},
   'settings': {'tool': 'polygon',
    'line_width': 5,
    'line_colour': (0, 255, 0),
    'label_size': 1,
    'label_width': 1,
    'label_colour': (0, 255, 0)},
   'data': {'label': 'plates',
    'include': True,
    'n': 1,
    'mask': [[(1373, 271),
      (1376, 443),
      (1859, 429),
      (1878, 378),
      (1702, 361),
      (1390, 268),
      (1373, 271)]]}}}}

Creating masks like this corresponds to the low throughput workflow, in which we first use create_mask to retrieve the mask coordinates, and then draw_masks to visualize the mask that was just created. A “canvas” is created with the original image and the mask coordinates, which can then be shown using show_image:

canvas = pp.visualization.draw_mask(img, 
                                    annotations=mask_plates, 
                                    label=True, 
                                    label_size=3, 
                                    line_width=2,
                                    label_width=4)    
pp.show_image(canvas)

In this step, select the reference card to and set include=False to exclude it. We use the previously created canvas to know which areas have been masked already. Then we create and visualize a new canvas, which will draw both masks onto the original image.

mask_scale = pp.preprocessing.create_mask(
    canvas, 
    label="scale", 
    line_colour="red",
    include=False,
)
canvas = pp.visualization.draw_mask(
    canvas, 
    annotations=mask_scale, 
    line_colour="red",
    label=True, 
    label_colour="red",
    label_size=3, 
    line_width=2,
    label_width=4,
)    
pp.show_image(canvas)

With this tutorial we have learned how to handle images and some basic interactions - proceed with Tutorial 3 to get started with image anaysis and learn about the different workflows that phenopype offers.