We developed a visual tracking method which is robust, simple, and operates at a maximum rate of 17Hz for stereo laparoscopes. The use of a stereo laparoscope enables the robot to track the instrument in both lateral and depth motions. Due to the multiplicity of problems coming up with shape analysis, we do not check the presence of any particular shape or structure. Instead, we use color information alone for instrument segmentation. Since the instrument color is not unique, we decided to use an artificial color mark to distinguish the instrument. We analyzed the color distribution of typical laparoscopic images and chose a color which does not usually appear to mark the instrument. Thus, even if only a very small part of the instrument is visible, reliable data can still be obtained for robot control. With color image segmentation, the mark can be correctly located in the image and used to control the robot motion. The image processing part was initially implemented on a commercially available image processing system, MaxVideo MV200, from Datacube, Inc., Danvers, MA. The robot we use is the AESOP 1000 (Automated Endoscope System for Optimal Positioning), from Computer Motion, Inc., Goleta, CA. The system was tested in a dummy abdomen used for surgery training. In December 1995, the system was also tested on a pig at the Klinikum Rechts der Isar, Technical University of Munich. After a major redesign, the system now runs without any special image-processing hardware on a general-purpose PC. It can be controlled by voice commands such as "faster", "slower", "track", and "stop".