Combine multiple video frames into one using OpenCV

I’m playing a lot with OpenCV and object recognition recently (I started from Python, but now I’ve moved to C++) and what I wanted to do to make my life easier was putting 4 different video frames (original, thresholded, grayscaled & thresholded and original with marked face). Even if it’s not very difficult task, it took me a while to do it, because – definitely – I’m not an OpenCV expert. This article shows how I did it using the new 2.3 (or – actually – 2.X) OpenCV API.

The result I was expecting was like this:

Combined frames

Of course without this pixelization blur ;-)

I’ll start from a brief description on what we have in the code and which variables we will use:

1
2
3
4
5
6
7
Mat frame,                     // original frame captured from webcam
    working_frame,             // resized frame where I detect face
    detected_frame,            // here I mark the face(s)
    threshold_grayscale_frame, // frame -> grayscale -> threshold
    threshold_color_frame,     // frame -> threshold
    merged_frame,              // big frame for combining four frames
    roi;                       // Region Of Interest for merged_frame

Original frame is 640×480, all other Mats except merged_frame are 320×240. Merged frame will be later defined as:

1
merged_frame = Mat(Size(640, 480), CV_8UC3);

Secondly – some processing I did so you could know what objects do these variables refer to:

1
2
3
4
5
6
7
/* 
 * For better performance I work on a frame which has half of the
 * width and height of the original frame and - for my comfort -
 * flipped horizontally 
 */
resize(frame, working_frame, Size(), 0.5, 0.5);
flip(working_frame, working_frame, 1);
1
2
3
4
5
6
7
8
/* 
 * I clone the working frame, detect objects and mark them on the
 * cloned frame. Note: "detect" and "mark_objects" functions are
 * not a part of OpenCV library
 */
detected_frame = working_frame.clone();
detected_objects = detect(working_frame, cascade);
mark_objects(detected_frame, detected_objects, 1, 1);
1
2
3
4
5
/* 
 * Simple binary threshold
 */
threshold(working_frame, threshold_color_frame,
            100.0, 255.0, CV_THRESH_BINARY);
1
2
3
4
5
6
7
8
9
10
11
/* 
 * Convert frame to grayscale and threshold. Here's a little trick!
 * We cannot use a grayscale image in merged_frame, but we want it
 * to be grayscale for threshold - so we have to convert the frame
 * two times - first, to grayscale and then - back to RGB 
 */
cvtColor(working_frame, threshold_grayscale_frame, CV_RGB2GRAY);
threshold(threshold_grayscale_frame, threshold_grayscale_frame,
            100.0, 255.0, CV_THRESH_BINARY);
cvtColor(threshold_grayscale_frame, threshold_grayscale_frame,
            CV_GRAY2RGB);

Having all the frames we need: working_frame, detected_frame, threshold_color_frame, threshold_grayscale_frame and the merged_frame to combine them, we can do the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
/* 
 * I think it's self-descriptive. We pick 4 different ROIs and copy
 * the frames we need into them. Piece of cake!
 */
merged_frame = Mat(Size(640, 480), CV_8UC3);
roi = Mat(merged_frame, Rect(0, 0, 320, 240));
working_frame.copyTo(roi);
roi = Mat(merged_frame, Rect(320, 0, 320, 240));
detected_frame.copyTo(roi);
roi = Mat(merged_frame, Rect(0, 240, 320, 240));
threshold_color_frame.copyTo(roi);
roi = Mat(merged_frame, Rect(320, 240, 320, 240));
threshold_grayscale_frame.copyTo(roi);

Of course I could (should?) write a nice-looking function for this, but it’s not something you can’t do on your own.

Now you only have to display merged_frame using – for example – imshow(), like you do with any other Mat or IplImage – that’s all!

Comments are closed.