1 Pedagogical objectives
Experience with practical implementations of sensor-based path planning al
gorithms. Simple feedback controllers. Empirical analysis of runtimes and
robustness. Understand the Kalman filter. Understand the role of the vari
ances. Learn to develop vision-based sensing to close the Kalman filter loop.
1.1 Setup Guideline
We currently support two versions, Python 2.7 and Python 3.+. We recom
mend using the python 3.+ version as it is easier to install. If for some reason
your python 3.+ version is not working, the python 2.7 version has a docker
image which should work but is more tricky to use. Required libraries are as
pip install NumPy
pip install pygame
pip install vispy
pip install opencv-python
pip install imutils
Furthermore, you will also need a graphic library from one of the following,
[’PyQt4’, ’PyQt5’, ’PyQt6’, ’PySide’, ’PySide2’, ’PySide6’, ’Pyglet’, ’Glfw’,
’SDL2’, ’wx’, ’EGL’, ’osmesa’, ’tkinter’]
In this assignment, the environment consists of a red ball whose position can
be controlled by fans. The ball moves up and down depending on the speed
of the fans. The state of the environment is provided as an RGB image. Note
that the ball can fly outside the line of sight of the camera if the fan rpm is
set high enough.
There are 3 modes, validation, validation noisy, and validation save; each
lasts 15 seconds. Validation mode is the standard mode; the target position
slider will bob up and down. The noisy mode adds significant Gaussian noise
to the RGB image provided, making it significantly more difficult to detect
the ball. Finally, validation save is the standard validation mode but saves
all RGB images in a NumPy matrix in the ../output/ directory.
Your goal in this assignment is to first detect the ball’s location. Then,
using the ball’s location, compute the proper fan rpm to move it closer
to the target position. For this assignment, you only need to modify the
opencv ball tracker.py and kalman ball tracker.py files. More specifically,
the functions detect ball, get fan rpm, and for the Kalman version, get kalman filter estimation.
Note there are several variables you can modify; PID variables, bgr color,
thresh which can improve performance.
2.3 Problem 1 – Blob Tracker
Implement a blob tracker and controller using OpenCV to extract the posi
tion of the red ball in the simulated environment. Plot the position of the
blob as a function of time with the validation mode. Adjust the parameters
of detect image for good results. Note that depending on the randomness of
the validation mode’s target positions, it can be difficult to obtain perfect
Now plot the position in 1D as a function of time, simply taking the horizontal
position component of the blob position as returned by the sensor.
You can use any programming language you want, but your code must func
tion on a Linux-based computer.
Your code should be runnable using a command of the form:
python opencv ball tracker.py
Your solution may use any logic that you like for this component as long as
you manage to follow the ball successfully.