Visual Cortex v2

In the past two weeks, I’ve focused on refining the visual cortex implementation. In the previous iteration, the visual ERUs lacked biological fidelity; so to mirror the audio cortex, I implemented each visual cortex level as its own micro environment, with feedback mechanisms and nonlinear controls.

Cortices Development Retrospective

After reviewing my past notes, here are the design elements that I aimed for and how well the implementation fits to that design -

Algorithms / Development

Building the Environmental Response System (ERS)

1. Define a center-surround mechanism (similar to that in horizontal cells) and temporal kernels for magnocellular/parvocellular channels from midget/parasol cells
2. If activity < threshold:
   • Use burst mode for increased LGN gain
3. Else:
   • Use tonic mode for moderate LGN gain
4. Apply second-level center-surround filtering (in addition to horizontal cells in the retina), to enhance edges

1. Note: It works similar to a 3D STRF with spatial and temporal components
2. Define a spatial map with Gaussian-cosine gabor
3. Define a temporal map with Gaussian-cosine gabor
4. Convolute the incoming (stream of) spikes onto the STRF

1. For each L4 unit:
   • It focuses on a patch in the receptive field and applies the visual STRF
   • The STRF output is converted into synapses, which have their own dynamics (fatigue and facilitations)
   • The neuron then integrates those currents and fires if needed. This step is equivalent to pooling / the activation layer in CNN-like models
2. For each L2/3 unit:
   • Pool nearby L4 units and connect collinear segments
   • Coincidental (and oriented) segments are assumed to be part of a longer segment, so horizontal facilitation improves synapse drive in the neighborhood
3. For each 5/6 unit:
   • It holistically observes the drive of L2/3 units and via neuromodulators, feeds back into L4 / LGN gain

1. Note: these cells primarily detect edges and corners
2. Multiply synapses to detect coincidences. Only when both synapses are strong, the V2 neurons fire
3. Note: the pre-neurons are connected in different orientations to detect T-junctions and L curves
4. Use neural dynamics to spike when threshold (of -30 mV) is crossed

1. Similar in operation to V2, the V4 cells detect curves
2. Normalize luminance to reduce sensitivity to lighting (color invariance)
3. Linearly pool synapses to detect curves. Only when both synapses are strong, the V4 neurons fire

1. IT cells establish “routes” i.e., chains of neurons when fired in a particular sequence, which would result in Glutamate (Glu) release. These routes and releases are how thoughts occur in your brain
2. When a route is traced, fire Glu spikes in the target neuron

Development Activity - https://github.com/akhil-reddy/beads/graphs/commit-activity
Building the visual cortex components - https://github.com/akhil-reddy/beads/blob/main/beads/core/eru/hub/visual/v_cortex.py

Please note that some code (class templates, function comments, etc) is AI generated, so that I spend more of my productive time thinking and designing. However, I cross-verify each block of generated code with its corresponding design choice before moving ahead.

Next Steps

Deployment

1. Overlaying video frames onto the retina, including code optimization for channel processing
2. Post processing in the visual cortex
3. Overlaying audio clips onto the cochlea, including optimization for wave segment processing
4. Post processing in the auditory cortex
5. Parallelization / streaming of cellular events via Flink or equivalent

Building the Environmental Response System (ERS)

1. Building the ERUs
2. Neurotransmitters - Fed by vision’s bipolar and amacrine cells, for example, to act on contrasting and/or temporal stimulus
3. Focus - Building focus and its supporting mechanisms (of which acetylcholine is one)