Vision Target Identification System
Robot Casserole developed a vision processing system for the 2017 season. This page describes that vision processing system's setup and theory of operation.
The System's purpose is to capture camera images, and identify collections of pixels which may correspond to parts of the various targets on the field (high goal, gear spring, etc).
Many many many thanks to FRC2481 for their simple instructions on getting this started.
Their awesome work can be found here.
##Basic Setup
The system presently consists of a beaglebone black and Axis M1011 network camera, all on the same ethernet network.
We will use the default angstrom linux distro.
For first setup, the BBB must be connected to a network with internet access.
On a brand new BBB, ssh into it using your favorite ssh utility. I use putty because of reasons. You will likely have to look up the IP address by looking at the router setup page, and determining what DHCP address was assigned to the BBB.
When prompted, login with:
user: root
password: <leave blank>
Yes it's bad to do stuff as root. But also easier. Just don't do silly things. Never run "rm -rf /", no matter how cool you think it will make you look in front of your friends.
Update the time so SSL certificate validation occurs ok:
ntpdate -b -s -u pool.ntp.org
Now install all the python utilities needed:
opkg update
opkg install python-pip
opkg install python-setuptools
opkg install python-opencv
opkg install python-misc
opkg install python-modules
pip install http://pypi.python.org/packages/source/p/pynetworktables/pynetworktables-2015.3.1.tar.gz
pip install psutil
Now that all the dependencies are installed, the unit should be moved and placed on the robot network (probably no internet access). Once this is done, log back in.
Set a static IP address of 10.17.36.20, with the default gateway pointing to the router (usually 10.17.36.1)
ls -la /var/lib/connman/
cd /var/lib/connman/ethernet_[xx]
more settings
cd /usr/lib/connman/test
./set-ipv4-method ethernet_[xx]_cable manual 10.17.36.20 255.255.255.0 10.17.36.1
reboot
Once the BBB boots again, you should be able to connect via the set up static IP address.
DHCP can be restored using the commands:
cd /usr/lib/connman/test
./set-ipv4-method ethernet_[xx]_cable dhcp
reboot
It's highly recommended to set up a static IP allocation on the router for the BBB to ensure it's ip address is always available.
There are a number of things the BBB does at boot time. Lots of these should be turned off to prevent vision lag from loading things which don't need to be loaded. Here's the service disable commands we ran:
systemctl disable bonescript-autorun.service
systemctl disable cloud9.service
systemctl disable bonescript.socket
systemctl disable gdm.service
Note some of these just disable things you used to be able to do on the BBB's hosted website (nothing we ever attempted). Note that gdm is the Gnome Desktop Manager, which you will need to re-enable if you want to remote connect or plug in an hdmi cord/mouse/keyboard and expect anything other than a black screen. SSH should be preserved, though.
In general, all vision processing code changes are stored in this repo, along with all the rest of the 2017 code - this is to keep the whole set of content in sync.
At present we have a fairly annoying development setup - figure out how to change the code, make the changes, deploy the changes to the BBB, see if it breaks or works, repeat till functional. The need to deploy to test code changes is frustrating, and hopefully we will find a solution.
If more rapid developmetn is needed, one could theoretically SSH into the BBB, stop the service, and start manually editing code files/running the script (VIM is my favorite command line editor). A remote connect graphical interface method is also described below. Editing files directly on the BBB is faster, but be sure to move any changes made there back to this repo! Any subsequent deploy will overwrite your changes if you're not careful.
We are having issues with this at the moment. For now, although GRIP is great for developing algorithms, the pipeline design must be manually ported to python (with the cv2 libraries).
Under the deploy folder, there is a deploy.bat file. Running this will attempt to contact the BBB, copy the appropriate files, and reboot it. When it comes back up, in theory, the changed files should be up and running.
There are a few other batch files for making sure the service is enabled, or checking the system logs for script errors or crashes.
After the very first deploy to the BBB, you will have to manually SSH into it, and start the service:
systemctl start CasseroleVision
systemctl enable CasseroleVision
reboot
Every time the BBB boots, it should automatically start running the vision processing algorithm
As is, the script is set up to maintain a robust connection with the IP camera, process frames as they come in, and post the processing results to NetworkTables (or just print them out, as it does now)
The psutil library is used to monitor CPU and memory loading.
The exact data processed out of the image will change year to year. Some possibiilities include:
- Total time from reciving the image to posting the processing results
- For each target:
- X/Y position within the image
- Information for acessing the validity of the target
- Bounding Box Size/Area
- Infill percentage
- Skew
The read and connect routines have fairly short timeouts, but will attempt to reconnect indefinitely if the connection is ever lost. There's been some basic desktop testing to prove out that there are no race conditions between starting the camera and starting the BBB.
To indicate status, second LED is turned on when the BBB is sending info to the RIO. If it can't connect to the network, read the camera, or has any other runtime error, the light will stay off. If the light is on, I believe we can be certan that data is being streamed to the RIO.
Additionally, the RIO should be monitering the right port for packets. After a pre-defined timeout of not getting packets, the vision system is declared inoperative and unavailable.
The deploy script should set this up for you. However, if you ever want to do it manually or want to know about the process, this section may help.
A "service" is a linux-OS mechanism for running background tasks, automatically. By creating and installing a service for our vision processing algorithm, we can have the OS start it at boot, and restart it if it crashes.
Copy the service definition file to its proper home on the in the OS system files.
cp CasseroleVision.service /lib/systemd/system/
Instruct the OS to run this at Boot:
systemctl enable CasseroleVision
Then reboot the system to test:
reboot
Once the next boot happens, the vision processing script should start up right away. Anything which was printed to console may be viewed in the system logs (grepped for relevant lines, since many many many things in the OS get dumped here):
journalctl | grep python
If you've changed the python script and want the system to reload your updates, you can restart the service
systemctl restart CasseroleVision
If you want to just manually run the script and not have it going in the background for some reason, you can temporarily stop it or start it back up:
systemctl stop CasseroleVision
systemctl start CasseroleVision
Finally, if you need it to not run at boot for debugging reasons, you can also uninstall the service:
systemctl disable CasseroleVision
Remember to re-run enable before you go to competition!
To get good images for processing, it's usually best to try to get a dark image. The only lit-up pixels should be the target in question.
This usually means
- Fixed (non-automatic) white balance
- Exposure turned down to minimum
- Brightness turned down to minimum
- All auto-adjustement disabled
Note that I've been testing with an axis M1013, which is a horrible camera choice since it is impossible to disable all brightness compensation. M1011 is better for vision processing, I think... But, basicallly, we wouldn't ever want to use the M1013.
This is a good resource on this sort of thing.
Framerate is another consideration. Our BBB system could process frames at a max of around 15 frames per second. Due to this, we set up our axis camera to cap all streams at 13FPS - close to the maximum, but not putting more data onto the network than was needed.
#Remote Desktop Connections
x11vnc is installed and runable on the BBB. It's not recommended to have it always on since it can soak up processor cycles, which are very valuable. However, for some vision debugging, it's nice to have.
To remote desktop, first ssh into the BBB as described above. Then run the following magic incantation:
x11vnc -bg -o %HOME/.x11vnc.log.%VNCDISPLAY -auth /var/run/gdm/auth-for-gdm*/database -display :0 -forever
Then connect to it using any VNC utility. For windows, I like this one.
To adjust the screen resolution, sadly this all has to be done on the server (BBB) side, and only after the connection has been started. To do this, open the Terminal application under Applications->System, and run the following command:
xrandr --fb 1600x900
Of course, resolution may be adjusted to suit.
All these things can be put into .sh scripts if desired to speed setup
