Mainpage
  Stereovision
  Segmentation
  Publications
  Online-Processing

Links
Contact
Spinking Neurons
Prev: Deconstruction Up: ^ Stereo Mainpage ^ Next: Fusion and Vergence

Coherence-Detection by Integrate-And-Fire Neurons


The neural hardware of cognitive systems does an amazing job: stable perceptions are created out of unreliable, inconsistent data, transmitted and analyzed by unprecise and noisy hardware.

The crucial postulate of coherence-based stereo is that one can, and has to, detect the coherence between independently operating disparity units in a fast and reliable way, by neural hardware.

Coherence-Detection is a highly non-linear, selecting operation, since only the data available within the coherent cluster is fed forward to later processing stages. The operation is therefore extremely robust: most of the available data is thrown away, and only the tiny coherent portion of the neural information streams are transmitted for further analysis.

There are several possible neural realizations of this process, but the simplest one uses the synchronization properties of pools of neural oscillators.

As an example, Fig. 1 displays spiketrains of integrate-and-fire neurons, driven by actual data from a single disparity-stack. In the simulation of this figure, the neurons operated independently from each other, and as expected, no obvious structure shows up in the spiketrains of these neurons. Due to the high amount of noise signals, it would be very hard for subsequent processing layers to make sense of this data.

 - Spiketrains of uncoupled I&F-Neurons  -
Fig. 1: Spiketrains of integrate-and-fire neurons, driven by data from a single disparity stack. Simulation time runs horizontal, neuron index vertically. No prominent structure is visible in the outputs of these neurons, even so the spiketrains were sorted according to mean firing rate.

However, there is of course structure in this data, simply because the data is created by the outside world. The object at which the disparity stack is looking at has a certain disparity, and the whole stack is split by this into two classes: one which is coding coherent values within each other, and the rest of the neurons.

Now: by simply introducing a fixed, but weak global coupling between the neurons, the dynamical behaviour of the pool of neurons changes dramatically, as displayed in Fig. 2. The coherently coding cluster is synchronizing fast; it becomes clearly marked by synchrony within the spiketrains of the neurons belonging to the coherent cluster!

 - Spiketrains of synchronizing neurons  -
Fig 2: With appropriate coupling, the neurons in the disparity stack quickly synchronize - but only within a small part of the network, coherent with each other. This is due to the weak-coupling paradigm. The coherent cluster becomes clearly visible in the spiketrains as vertical blocks of spikes. Note that the coherence cluster makes up less than half of the whole neural population in this pool.

Of course, the synchronized pool of neurons is easily detectable by further processing layers, and in doing so, one obtains Fig. 3, a disparity map calculated out of real image data by a network of simple spiking integrate-and-fire neurons:

 - Right stereo image -  - Left stereo image -
Right stereo image Left stereo image
 ->  - Disparity Map calculated by spiking neurons -

Disparity Map
Fig 3: A disparity map, calculated by a network of spiking neurons (integrate-and-fire neurons; white means "near", black = "far"). The original stereopair used as input to the network is displayed also.
(A click on the disparity map loads an animated .gif-file, displaying the virtual view created by the coherence detection process within the neural network. This gives an impression of the original 3d structure of the scene.)

The neural synchronization process described here is actually computing the disparity map, by performing a dynamical coherence detection between signals in a very noisy environment.

A simple average or weigthed linear combination of the data would not lead to a useable disparity map, due to the high noise level within the disparity pools. Click here for a comparision, with and without the synchronizing links.

In Fig. 4, the output currents of two identical pool of neurons are displayed. The pool without weak-coupling (brown trace) shows only small modulations of its total output current. The pool where the weak-coupling is switched on at time=20 (blue trace) develops very fast an oscillatory component indicating the development of a coherence cluster.

 - Output Currents of synced & non-synced neuron pool -
Fig 4: Traces of output currents of two identical neural pools. The blue pool has weak-coupling switched on at Time=20.

Actually, the modulation depth of this output current reflects the number of coherent neural units, and is thus a measure of the validity of the signal. This can be seen in Fig. 5, which shows the modulation depth for the example image of Fig. 3. The dark areas in the coherence map clearly mark image areas where disparity estimates are in doubt, for example around the borders of the dragon head, where occlusion effects prevent correct estimation.

 - Disparity Map -  - Coherence Map -
Fig 5: The disparity map from above again (left), which actually displays the firing rate of read-out neurons, plus the corresponding coherence map (right), which shows the modulation depth of the output current. Dark areas indicate a low modulation depth, and they are found in image areas where it is difficult to estimate disparity correctly.

More information about the topic covered in this page can be found in technical reports, available at this page. You can run a neural network simulation using coherence detection on your own stereo images, by clicking here.

Check also the "Mus silicium" by John Hopfield and Carlos Brody, which uses the coherence detection scheme described here as computational tool in word recognition experiments.


© 1994-2003 - all rights reserved.