I designed an video surveillance system (demo for evaluation), you can download them in follow cloud storage :

Download the demo
Shared video clips [1], [2], [3]

—————————————————————————–

Notes :

1.you need connect the internet, my server will authorize the trial demo to run.

2. do not delete the ‘configure.xml’ file in the folder, you can find some settings in it.

3. if you can’t play the video file, you need install the codec.

—————————————————————————–

Snap 1: Multi-people tracking + Auto camera calibration ( create 3D scene and mapping the people on it , the background event-thread will do nonliear fit and RANSAC to estimate the camera KRT matrics by peoples)

Snap 2. Abandon object detection. In left-bottom window, the abandon object is hint by the red bounding box and print the stop time (minutes:seconds)

Snap 3. The new stopped car is also ‘abandon object’ because the original background model does not contain it, it can applied in the high-speed way

to detect the stopped car.

This article is taken from MathWorks ( Thanks MathWorks, its very nice article ):

－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－

filename = 'shaky_car.avi';
hVideoSrc = vision.VideoFileReader(filename, ...
                                      'ImageColorSpace', 'Intensity',...
                                      'VideoOutputDataType', 'single');

imgA = step(hVideoSrc); % Read first frame into imgA
imgB = step(hVideoSrc); % Read second frame into imgB

cvexShowImagePair(imgA, imgB, 'Frame A', 'Frame B');

figure; imshow(cat(3,imgA,imgB,imgB));
title('Color composite (frame A = red, frame B = cyan)');

maxPts = 150;
ptThresh = 1e-3;
hCD = vision.CornerDetector( ...
    'Method','Local intensity comparison (Rosen & Drummond)', ...
    'MaximumCornerCount', maxPts, ...
    'CornerThreshold', ptThresh, ...
    'NeighborhoodSize', [9 9]);
pointsA = step(hCD, imgA);
pointsB = step(hCD, imgB);

cvexShowImagePair(imgA, imgB, 'Corners in A', 'Corners in B', ...
    'SingleColor', pointsA, pointsB);

% Extract features for the corners
blockSize = 9; % Block size.
[featuresA, pointsA] = extractFeatures(imgA, pointsA, ...
    'BlockSize', blockSize);
[featuresB, pointsB] = extractFeatures(imgB, pointsB, ...
    'BlockSize', blockSize);

% Match features which were found in the current and the previous frames
indexPairs = matchFeatures(featuresA, featuresB, 'Metric', 'SSD');
pointsA = pointsA(indexPairs(:, 1), : );
pointsB = pointsB(indexPairs(:, 2), : );

cvexShowMatches(imgA, imgB, pointsA, pointsB, 'A', 'B', 'RC');

[a_1 a_3 t_r;
 a_2 a_4 t_c;
   0   0   1]

hGTE = vision.GeometricTransformEstimator;
hGT = vision.GeometricTransformer;
hGTPrj = vision.GeometricTransformer;

% Run multiple RANSAC trials
nRansacTrials = 1;
Ts = cell(1,nRansacTrials);
costs = zeros(1,nRansacTrials);
nPts = int32(size(pointsA,2));
inliers = cell(1,nRansacTrials);

for j=1:nRansacTrials
    % Estimate affine transform
    [Ts{j},inliers{j}] = step(hGTE, pointsB, pointsA);

    % Warp image and compute error metric.
    imgBp = step(hGT, imgB, Ts{j});
    costs(j) = sum(sum(imabsdiff(imgBp, imgA)));
end
% Take best result.
[~,ix] = min(costs);
imgBp = step(hGT, imgB, Ts{ix});
pointsBp = [single(pointsB), ones(size(pointsB,1), 1)] * Ts{ix};
H = [Ts{ix} [0 0 1]'];

cvexShowMatches(imgA, imgBp, pointsA(inliers{ix}, : ), ...
    pointsBp(inliers{ix}, : ), 'A', 'B');

[s*cos(ang) s*-sin(ang) t_x;
 s*sin(ang)  s*cos(ang) t_y;
          0           0   1]

% Extract scale and rotation part sub-matrix.
R = H(1:2,1:2);
% Compute theta from mean of two possible arctangents
theta = mean([atan2(R(2),R(1)) atan2(-R(3),R(4))]);
% Compute scale from mean of two stable mean calculations
scale = mean(R([1 4])/cos(theta));
% Translation remains the same:
translation = H(3, 1:2);
% Reconstitute new s-R-t transform:
HsRt = [[scale*[cos(theta) -sin(theta); sin(theta) cos(theta)]; ...
  translation], [0 0 1]'];

imgBold = step(hGTPrj, imgB, H);
imgBsRt = step(hGTPrj, imgB, HsRt);
figure(2), clf;
imshow(cat(3,imgBold,imgBsRt,imgBsRt)), axis image;
title('Color composite of affine and s-R-t transform outputs');

% Reset the video source to the beginning of the file.
reset(hVideoSrc);
hGTE = vision.GeometricTransformEstimator;
hGT = vision.GeometricTransformer;
hGTPrj = vision.GeometricTransformer;

hVPlayer = vision.VideoPlayer; % Create video viewer

hCD = vision.CornerDetector( ...
    'Method','Local intensity comparison (Rosen & Drummond)', ...
    'MaximumCornerCount', maxPts, ...
    'CornerThreshold', ptThresh, ...
    'NeighborhoodSize', [9 9]);

% Process all frames in the video
movMean = step(hVideoSrc);
imgB = movMean;
imgBp = imgB;
correctedMean = imgBp;
ii = 2;
Hcumulative = eye(3);
while ~isDone(hVideoSrc) && ii < 10
    % Read in new frame
    imgA = imgB; % z^-1
    imgAp = imgBp; % z^-1
    imgB = step(hVideoSrc);
    movMean = movMean + imgB;

    % Estimate transform from frame A to frame B, and fit as an s-R-t
    H = cvexEstStabilizationTform(imgA,imgB,hCD,hGT,hGTE);
    HsRt = cvexTformToSRT(H);
    Hcumulative = HsRt * Hcumulative;
    imgBp = step(hGTPrj, imgB, Hcumulative);

    % Display as color composite with last corrected frame
    step(hVPlayer, cat(3,imgAp,imgBp,imgBp));
    correctedMean = correctedMean + imgBp;

    ii = ii+1;
end
correctedMean = correctedMean/(ii-2);
movMean = movMean/(ii-2);

% Here you call the release method on the objects to close any open files
% and release memory.
release(hVideoSrc);
release(hVPlayer);

cvexShowImagePair(movMean, correctedMean, ...
    'Raw input mean', 'Corrected sequence mean');

Recently I found a nice single object tracker named Predator.

Here’s the Introduce :

—————————————————————————————————————————————————————————————————————————————-

The Predator contains :

• Tracking : use Backward-forward LK tracker, the highlight of this tracker is it can correct mismatching in optical flow. ( for example, the point A in image t-1 is matching to A’  in image t by LK,  we call this forward, and then backward from A’  in image t to A” in image t-1 , if  A” is not matching  orginal point A, we call this point is error matching, remove this point’s optical flow )
• Learning : 1. When initialize, normalize the target to a small N*N patch, generate a lots of features ( random blocks in patch ), encode them by 2bitLP (haar-like features), then we get feature vector of this target to be positive examples. and we use feature vector generated from background (far from target) to be negative examples. we use those to initialize a random forest. 2. When tracking, we use P-N constraint to distinguish whether is unseen object appearence or background, to update the classifier.
• Detection : use sliding window + integral image + 2bitLP (haar-like features)

—————————————————————————————————————————————————————————————————————————————-

Here’s examples :

This video is reference in OpenCV jp.

Data set used in the program can be downloaded from here.

This is a demo program to project the 2D image pixel to 3D space parallax image.

————————————————————————————————————————————

And you can find how to operate in here

————————————————————————————————————————————

[1] Makoto Ota, Norishige Fukushima, Tomohiro Yendo, Masayuki Tanimoto, Toshiaki Fujii, “Rectification of Pure Translation 2D Camera Array,” Proc. Of IWAIT2009, 0044, Jan. 2009.

Why human eye can tracking object ? What’s algorithm the human brain run? Let’s talk about this.

1 . Segmentation . Human can easily distinguish and tracking target different with background but hardly tracking target similar to background, for example,  a polar bear on snow vs a a polar bear  on grass, people wear camouflage uniform lying in the grass vs lying in the snow.

2. Motion detection. Do you feel dizzy when you staring at a waving stick ?  People look a static background scene are more easily than watching a moving background scene, such as rotating car at playground can make people dizzy, the moving background scene need run complex motion detection algorithm  on brain.

If you wanna search data in a disorder dataset, you can divide dataset to a lots of blocks, and use each thread to seach each block, here is a example :

Definition :

Input :       void **ppDataSet : A huge dataset which you wanna seach target data in it

Input:        int nLen : Dataset length

Input:        void *pTargetData :  Target data (You wanna search it in dataset and return it location)

Input:        CompareFunc Comp : Compare data (Matching datas to obtain similarity)

Output:     vector<int> VLoc : Target data locatoin in dataset

Sample Code:

vector<int>   ParallelSearchData( void** ppDataSet, int nLen, void* pTargetData, CompareFunc Comp )

{

.        int i, k;

.        int nCore   = omp_get_num_procs();

.        int nStep   =    nLen / Core;

.        vector<int>  VLoc;

#pragma omp parallel for

.        for ( k = 0 ; k < nCore ; k++ )

.        {

.                  int iBegin  = k * nStep;           // Each Processor Search  Start Loc

.                  int iEnd     =   (k+1) * nStep;    // Each Processor Search End Loc

.                 // Divide step cumulative may not accurate equal nLen (int type)

.                  if ( k = kCore – 1 )         iEnd =  nLen;        

.                  for ( i = iBegin ; i < iEnd ; i++ )

.                  {

.                          // We assume confidence = 1.0 is target

.                            if (  Comp( ppDataSet[i], pTargetData ) == 1.0 )  

.                                       VLoc.insert( i );

.                  }

.        }

.        return VLoc;

}

Nowadays, I encountered a problem when I designing a new real-time tracking algorithm in company, inevitable for loop and time-consuming operation make me upset, so I borrow a book from Lib and search multi-thread to solve this problem, now I find answer:

here is example :

int i;

int iLoopTimes    =    1000;

void Job()
{
for ( i = 0 ; i < iLoopTimes ; i++ )
{
IplImage*    pImage    =    cvCreateImage( cvSize(100,100), 8, 3 );
cvCvtColor( pImage, pImage, CV_BGR2HSV );
SAFE_FREE_CV_IMAGE( pImage );
}
}
void ParallelSection()
{
//////////////////////////////////////////////////////////////////////////
CLOCK_START
#pragma omp parallel
{
#pragma omp sections
{
#pragma omp section
Job();
#pragma omp section
Job();
}
#pragma omp sections
{
#pragma omp section
Job();
#pragma omp section
Job();
}
}
CLOCK_STOP
//////////////////////////////////////////////////////////////////////////

CLOCK_START
#pragma omp parallel sections
{
#pragma omp section
Job();
#pragma omp section
Job();
#pragma omp section
Job();
#pragma omp section
Job();
}
CLOCK_STOP

//////////////////////////////////////////////////////////////////////////
CLOCK_START
Job();
Job();
Job();
Job();
CLOCK_STOP
//////////////////////////////////////////////////////////////////////////
}

}

Time Usage Compare:

Test1  time = 628.708 ms (Double Double sections)
Test2  time = 831.781 ms (Four sections)
Test3  time = 2741.43 ms (Do not use OpenMP)

See ? My friend.

Here is the C++ Code Solve the Nonlinear function below

 Use the Binary Divisive Procedure

#include <math.h>
#include <stdio.h>
#include <iostream>
using namespace std;

double TestEquation( double x )
{
 double y;
 y = (((((x-5.0)*x+3.0)*x+1.0)*x-7.0)*x+7.0)*x-20.0;
 return y;
}
// Function : BinDivSearchRoot 
// Parameters:
//  LowerBound  Lower Bound of Root
//  UpperBound  Upper Bound of Root
//      So, This Function will Find Root in [a,b]
//  StepWidth  Progressive Step
//  Eps    Error Rande
//  Root[]   the Value of roots
//  RootCount  Expected number of Roots
//  Email :   Author Email : novertina@aol.com
int BinDivSearchRoot( double LowerBound, double UpperBound, double StepWidth, double Eps,double Root[], int RootCount, char* mail )
{
 int SearchedRoot, rootStatus;
 double x, y, x1, y1, x0, y0;
 SearchedRoot = 0; x = LowerBound;  y = TestEquation( x );

 while ( ( x <= UpperBound + StepWidth/2.0 )&&( SearchedRoot!=RootCount ) ) // Search root in Sub-interval
 {
  if ( fabs(y) < Eps ) // If this is a ROOT !
  {
   SearchedRoot++;
   Root[SearchedRoot-1] = x;
   x += StepWidth/2.0;
   y = TestEquation( x );
  }
  else // Or Not a ROOT
  {
   x1 = x + StepWidth;
   y1 = TestEquation( x1 );
   if ( fabs(y1) < Eps ) // If next Step is a ROOT !
   {
    SearchedRoot++;
    Root[SearchedRoot-1] = x1;
    x = x1 + StepWidth/2.0;
    y = TestEquation( x );
   }
   else if ( y * y1 >0.0 ) // NULL ROOT ;-(
   {
    y = y1;
    x = x1;
   }
   else // A ROOT in this Sub-interval !
   {
    rootStatus = 0 ;// Status : 1. Still Not Found
        //   2. Have Found

    while ( rootStatus == 0 )
    {
     if ( fabs( x1-x ) < Eps ) // Sub-Interval length is less than Given Eps, It can be used as a ROOT !
     {
      SearchedRoot ++;
      Root[SearchedRoot-1] = (x1+x)/2.0; // Consider the ROOT is Median value of this Sub-Interval
      x = x1 + StepWidth/2.0; // In order to Find Root in next Sub-Interval, this Interval is clear !
      y = TestEquation( x );
      rootStatus = 1; // Don’t forget Set the Status OK !
     }
     else // Sub-Interval length is Bigger than Given Eps, Binary Divide it
     {
      x0 = (x1+x)/2.0;// Binary Divide it
      y0 = TestEquation( x0 );

      if ( fabs(y0) < Eps ) // Here is a ROOT !
      {
       Root[SearchedRoot] = x0;
       SearchedRoot ++;
       x = x0 + StepWidth/2.0;
       y = TestEquation( x );
       rootStatus = 1;
      }
      else if ( (y*y0) < 0.0 )// a ROOT exist in [y,y0]
      {
       x1 = x0;
       y1 = y0;
      }
      else
      {
       x = x0;
       y = y0;
      }
     }
    }
   }
  }
 }
 if ( strcmp(mail,”novertina@aol.com“) )
 {
  // Here is a Joke
  // Purpose is to prevent lazy students who just copy the code do not understand the meaning
  // And let them get Null result
  // Sorry
  return 0;
 }
 return(SearchedRoot);  // Return Root Count
}

void SolveFunctionMain()
{

 int    rootCount;
 static int  m = 6;
 static double x[6];
 double   LowerBound = -2.0;
 double   UpperBound =  5.0;
 double   StepWidth =  0.2;
 double   Eps   =  0.000001;
 char   mail[100] = ”novertina@aol.com“;

 cout<<”———   This is a program to solve Nonlinear function”<<endl;
 cout<<”———   By Binary Divisive Procedure”<<endl;
 cout<<endl;

 cout<<”———   The Nonlinear function is:”<<endl;
 cout<<”y = (((((x-5.0)*x+3.0)*x+1.0)*x-7.0)*x+7.0)*x-20.0″<<endl;
 cout<<endl;

 rootCount = BinDivSearchRoot( LowerBound, UpperBound, StepWidth, Eps, x, m, mail );

 cout<<”Solve Successfully!”<<endl;
 cout<<endl;

 cout<<”The function has “<<rootCount<<” roots,they are:”<<endl;
 for (int i = 0 ; i < rootCount ; i++ )
 {
  cout<<”x(“<<i<<”) = “<<x[i]<<endl;
 }
 
 system(“PAUSE”);
}

The Symbol on my head is my Family name;-)

Just like the Symbol on the head of Tiger

Happy New Year:-)

If you’re new to OpenCV, you need to know exactly how to manage all the huge amounts of memory you’re using. C/C++ isn’t a garbage collected language (like Java), so you need to manually release memory as soon as its use is over. If you don’t, your program could use up hundreds of MBs of highly valuable RAM… and often even crash (out-of-memory errors?)

It can be a daunting task to hunt exactly where memory needs to be released. So I’ve compiled this short list of places where you should look out for memory leaks.

Create it, then Release it

If you create something, make sure you release it before “returning”. This is probably the very first thing you should check when fixing memory leak problems with OpenCV. For example, if you do a cvCreateImage, make sure you do a cvReleaseImage. There are many things you can create. Here are some functions that “create” and their corresponding “release” functions

cvCreateImage cvReleaseImage
cvCreateImageHeader cvReleaseImageHeader
cvCreateMat cvReleaseMat
cvCreateMatND cvReleaseMatND
cvCreateData cvReleaseData
cvCreateSparseMat cvReleaseSparseMat
cvCreateMemStorage cvReleaseMemStorage
cvCreateGraphScanner cvReleaseGraphScanner
cvOpenFileStorage cvReleaseFileStorage
cvAlloc cvFree

One warning though: If you create something and want to return it, don’t release it. Lets say a function that creates a checkerboard image and returns it. If you release the image before returning it, you’re freeing all memory that stores the image data. And when you try accessing memory that isn’t yours, you get a crash.

Release returned structures

This is the second thing you should check for. Often, once you return a structure (say, an image).. you forget about it.

Multiple Memory Allocations

This is the third thing you should check for: Allocating memory, and then changing the pointer itself. Here’s some example code:

view plaincopy to clipboardprint?
IplImage* image = cvCreateImage(whatever);
image = CreateCheckerBoard(whatever);
…
cvReleaseImage(ℑimage);
IplImage* image = cvCreateImage(whatever);
image = CreateCheckerBoard(whatever);
…
cvReleaseImage(ℑimage);

This function creates a memory leak. First, you allocate some memory for image . Then, you call the function CreateCheckerBoard. This function itself creates new memory. And image now points to this new memory. The memory created in the first step is lost forever. No variable points to it. A memory leak. To fix this, you need to modify the code like this:

view plaincopy to clipboardprint?
IplImage* image = NULL;
image = CreateCheckerBoard(whatever);
…
cvReleaseImage(ℑimage);
IplImage* image = NULL;
image = CreateCheckerBoard(whatever);
…
cvReleaseImage(ℑimage);

If you return a sequence, release its storage

There are many instances where you use the CvSeq data structure. And often you might want to return this structure for further use. If you release its storage (a CvMemStorage structure) within the function itself, you’d free the memory where the sequence is stored. And then you’d try and access it in the calling function. Again, crash.

A temporary fix would be to just erasing the cvReleaseMemStorage statement… but that would mean lots of memory.
To fix this, you don’t release the memory in the function itself. You release it in the calling function like this:

view plaincopy to clipboardprint?
cvReleaseMemStorage(&thesequence->storage);
cvReleaseMemStorage(&thesequence->storage);

storage is a member of the CvMemStorage structure that always points to the memory where its stored.

Again, this is just an example. There are more structures where a similar situation could arise.

Dependence on other structures

I quite recently discovered this memory leak. To explain this, I’ll use an example: Lets say you find out the contours of an image. OpenCV would return a “linked list” type structure calledCvSeq . You decide to access the third element of this linked list. OpenCV returns a pointer to the third element. All going great till this moment.

Now you decide to save all the points of this contour (the third element) in a data structure of your own. Since this is an array of points, you do something like:

view plaincopy to clipboardprint?
mystructure->points = thirdcontour->points;
mystructure->points = thirdcontour->points;

You set the pointer to equal to the thirdcontour . This is the bug. If you release the storage of the sequence (which you should), mystructure has a bad pointer. To fix this, allocate new memory to mystructure->points and then copy contents of thirdcontour->points … something like this:

view plaincopy to clipboardprint?
mystructure->points = (CvPoint*)malloc(sizeof(CvPoint) * thirdcontour->total);
memcpy(mystructure->points,thirdcontour->points,sizeof(CvPoint)*thirdcontour->total);
mystructure->points = (CvPoint*)malloc(sizeof(CvPoint) * thirdcontour->total);
memcpy(mystructure->points,thirdcontour->points,sizeof(CvPoint)*thirdcontour->total);

This creates new memory for your structure and then copies each element there. Once you’ve done this, you can release the storage of the sequence without fear.

[Original source from: LiquidMetal , all rights reserved by original authors]