Year: 2023
Type: peer-review
Abstract: Article Figures and data Abstract Editor's evaluation Introduction Results Discussion Materials and methods Data availability References Decision letter Author response Article and author information Metrics Abstract Epifluorescence miniature microscopes ('miniscopes') are widely used for in vivo calcium imaging of neural population activity. Imaging data are typically collected during a behavioral task and stored for later offline analysis, but emerging techniques for online imaging can support novel closed-loop experiments in which neural population activity is decoded in real time to trigger neurostimulation or sensory feedback. To achieve short feedback latencies, online imaging systems must be optimally designed to maximize computational speed and efficiency while minimizing errors in population decoding. Here we introduce DeCalciOn, an open-source device for real-time imaging and population decoding of in vivo calcium signals that is hardware compatible with all miniscopes that use the UCLA Data Acquisition (DAQ) interface. DeCalciOn performs online motion stabilization, neural enhancement, calcium trace extraction, and decoding of up to 1024 traces per frame at latencies of <50 ms after fluorescence photons arrive at the miniscope image sensor. We show that DeCalciOn can accurately decode the position of rats (n = 12) running on a linear track from calcium fluorescence in the hippocampal CA1 layer, and can categorically classify behaviors performed by rats (n = 2) during an instrumental task from calcium fluorescence in orbitofrontal cortex. DeCalciOn achieves high decoding accuracy at short latencies using innovations such as field-programmable gate array hardware for real-time image processing and contour-free methods to efficiently extract calcium traces from sensor images. In summary, our system offers an affordable plug-and-play solution for real-time calcium imaging experiments in behaving animals. Editor's evaluation This article presents a novel realtime processing pipeline for miniscope imaging which enables accurate decoding of behavioral variables and generation of feedback commands within less than 50ms. The efficiency of this important tool for experiments requiring close-loop interaction with brain activity is demonstrated based on compelling measurements in two experimental contexts. This pipeline will be useful for a wide range of questions in system neuroscience. https://doi.org/10.7554/eLife.78344.sa0 Decision letter Reviews on Sciety eLife's review process Introduction Miniature epifluorescence microscopes ('miniscopes') can be worn on the head of an unrestrained animal to perform in vivo calcium imaging of neural population activity during free behavior (Ghosh et al., 2011; Ziv et al., 2013; Cai et al., 2016; Aharoni et al., 2019; Hart et al., 2020). Imaging data are usually collected while subjects are engaged in a task and stored for later offline analysis. Popular offline analysis packages such as CaImAn (Giovannucci et al., 2019) and MIN1PIPE (Deng et al., 2015) employ algorithms (Pnevmatikakis et al., 2016) for demixing crossover fluorescence between multiple sources to extract calcium traces from single neurons, but these algorithms cannot be implemented in real time because they incur significant computing delays and rely on acausal computations. Emerging techniques for online trace extraction (Friedrich et al., 2017; Mitani and Komiyama, 2018; Chen et al., 2019; Chen et al., 2020; Chen et al., 2022a; Friedrich et al., 2021; Taniguchi et al., 2021) offer potential for carrying out real-time imaging experiments in which closed-loop neurostimulation or sensory feedback are triggered at short latencies in response to neural activity decoded from calcium fluorescence (Aharoni and Hoogland, 2019; Zhang et al., 2018; Liu et al., 2021). Such experiments could open new avenues for investigating the neural basis of behavior, developing brain–machine interface devices, and preclinical testing of neurofeedback-based therapies for neurological disorders (Grosenick et al., 2015). To advance these novel lines of research, it is necessary to develop and disseminate accessible tools for online calcium imaging and neural population decoding. Here we introduce DeCalciOn, a plug-and-play hardware device for Decoding Calcium Images Online that is compatible with existing miniscope devices which utilize the UCLA Miniscope DAQ interface board (Cai et al., 2016; Scott et al., 2018; de Groot et al., 2020; Blair et al., 2021). We show that this system can decode hippocampal CA1 calcium activity in unrestrained rats (n = 12) running on a linear track, and can also categorically classify behaviors performed by rats (n = 2) during an instrumental touchscreen task from orbitofrontal cortex (OFC) calcium activity. Performance tests show that the system can accurately decode up to 1024 calcium traces to trigger TTL outputs within <50 ms of fluorescence detection by the miniscope's image sensor. To achieve these short real-time decoding latencies, DeCalciOn relies upon customized hardware for online image processing (Chen et al., 2019; Chen et al., 2022a) running in the fabric of a field-programmable gate array (FPGA); the system also utilizes efficient software for trace decoding (Chen et al., 2022b) that achieves microsecond variability in decoding latencies by running on an ARM core under the FreeRTOS real-time operating system. We show that the time required for training the real-time decoder from example data (prior to the initiation of online imaging) can be minimized by employing contour-free methods for defining regions of interest (ROIs) from which calcium traces are extracted. In summary, DeCalciOn provides the research community with a low-cost, open-source, easy-to-use hardware platform for short latency real-time decoding of calcium trace population activity. All hardware, software, and firmware are openly available through miniscope.org. Results The DeCalciOn hardware system is implemented on an Avnet Ultra96 development board featuring a Xilinx Zynq Ultrascale + multiprocessor system-on-a-chip (MPSoC) with 2-GB DRAM (Figure 1A). A custom interface board mated to the Ultra96 receives real-time image data from a modified version of the UCLA Miniscope Data Acquisition (DAQ) interface, which has 14 flywires soldered to the printed circuit board (PCB) for transmitting deserialized video to the Ultra96 (pre-modified DAQ boards can be obtained through miniscope.org). Throughout online imaging sessions, the DeCalciOn system is controlled from a host PC running standard Miniscope DAQ software (to focus the lens and adjust the LED light source) alongside newly developed real-time interface (RTI) software which communicates with our custom ACTEV (Accelerator for Calcium Trace Extraction from Video) hardware accelerator, which runs in the fabric of the MPSoC's FPGA. Raw video data (including real-time motion correction vectors) and calcium trace values are transmitted via Ethernet from the Ultra96 to a host PC for storage, so that additional analyses of calcium data can be performed offline. ACTEV accelerator for the two versions of the UCLA Miniscope used here (MiniLFOV and V4, both sampling at a frame rate of ~20 Hz), as well as RTI software resources for the host PC, can be found at https://github.com/zhe-ch/ACTEV, (Chen, 2023 copy archived at swh:1:rev:aa6393d3bd2dd490aa5369e1f2677e85e8a64a82). Figure 1 Download asset Open asset Real-time imaging protocol and system hardware. (A) Sequence of steps for a real-time imaging and decoding session. (B) Miniscope connects to DAQ via coax cable, DAQ connects to Ultra96 via flywires, host PC connects to Ultra96 via Ethernet and to DAQ via USB 3.0; TTL pinouts from Ultra96 can drive closed-loop feedback stimulation from external devices. Steps of a real-time imaging session DeCalcion is designed for conducting real-time calcium imaging experiments in freely behaving animals. Although the animal species, targeted brain regions, and behavioral tasks will necessarily differ depending on the goals of a given experiment, real-time imaging sessions conducted with DeCalciOn will generally be carried out by performing a sequence of five common steps (Figure 1B). Step 1 is to mount the miniscope on the subject's head and collect 1000 frames of data to derive a reference alignment image for online motion correction. It is preferable for the subject to be in a quiescent state during this initial data collection period, to minimize brain motion while the reference image data are being stored. Imaging and behavioral data are transmitted via Ethernet from the Ultra96 for storage on the host PC. After 1000 frames of data have been stored to the host PC (~50 s after the start of acquisition at the ~20 Hz frame rates used here), imaging acquisition automatically pauses and the user is prompted to select a 128 × 128 pixel subregion of the image to use for motion correction. The host PC derives a reference image for motion correction by computing the mean image within the selected area over the 1000 frames of stored data. The reference image is then uploaded to the Ultra96 and data collection resumes with online motion correction enabled. Collecting the alignment data and uploading the reference frame takes ~3 min. Step 2 is to collect an initial set of motion-corrected data for training the decoder. Online motion correction vectors are derived in real time for each frame and uploaded to the host PC for storage along with imaging and behavioral data. The time required to collect initial training data depends upon how much data is needed to train the decoder, which in turn depends upon the quality of calcium signals and what behaviors the decoder will be trained to predict. In performance tests presented here (see below), asymptotic accuracy for decoding a rat's position on a linear track was achieved with as little as ~3 min of training data whereas asymptotic accuracy for classifying categorical behaviors during an instrumental task required 20–40 min of training data (~100 experimental trials). The size of the training dataset required for a given experiment can be determined in advance (prior to real-time imaging sessions) by storing offline imaging data from the brain ROI while subjects are engaged in the experimental task, then carrying out offline simulations of real-time decoder performance with training sets of varying size to determine the minimum amount of training data needed for accurate decoding. Step 3 is to pause data acquisition so that the host PC can perform analyses to extract calcium traces from each frame of the initial training dataset. Calcium traces can be extracted by two methods: a contour-based method that requires 30–60 min of processing time on the host PC during Step 3 (for reasons explained below), or a contour-free method that requires no processing time during Step 3 because this method allows calcium traces to be extracted online and stored to the host PC concurrently with collection of the initial training dataset, thereby obviating the need to devote additional time or effort to calcium trace extraction during Step 3. It is preferable for the miniscope to remain attached to the animal's head throughout Step 3 (and Step 4), because removing and remounting the miniscope can introduce shifts in the focus or alignment of the brain image and thereby degrade the accuracy of subsequent real-time decoding. Since the contour-based method of calcium trace extraction consumes considerable processing time to perform offline identification of cell contours, it also incurs the inconvenience of monitoring the subject during this extended time period while the miniscope remains attached to the head. Step 4 is to use extracted calcium traces for training a decoder to predict behavior from vectors of calcium trace values. The time required for this depends upon the complexity of the user-defined decoder that is being trained (the decoder is trained on the host PC and runs in real time on the Ultra96 ARM core; see 'Stage 4: population decoding'). Thus, a multilayer deep network decoder would take longer to train than a single-layer linear classifier of the kind used here for performance testing. Training consumed <60 s for the linear classifiers used in performance tests presented below. Trained decoder weights (along with derived cell contour pixel masks if contour-based decoding was used) are uploaded from the host PC to the Ultra96 (a process which takes a maximum of 5 min to complete). Once the decoder weights have been uploaded, DeCalciOn is fully configured for real-time decoding of calcium traces. Step 5 is to resume the behavior experiment while DeCalciOn performs real-time imaging and decoding of calcium traces to generate feedback output at TTL pins. Real-time image processing pipeline Figure 2 shows a schematic diagram of DeCalciOn's real-time image processing pipeline, which performs four sequential stages of image processing and calcium trace decoding: (1) online motion stabilization, (2) image enhancement via background removal, (3) calcium trace extraction, and (4) fluorescence vector decoding. As explained below, Stages 1–3 are performed by our custom ACTEV hardware accelerator (Chen et al., 2019; Chen et al., 2020; Chen et al., 2022a) running in the programmable logic fabric of the FPGA, whereas Stage 4 is performed by a C++ program running under the FreeRTOS operating system on the MPSoC's embedded ARM core. Figure 2 Download asset Open asset Online imaging and control pipelines. Serialized video data from the MiniLFOV are transmitted through a coaxial tether cable to the DAQ, where it is deserialized and transmitted via the flywire bus to ACTEV (Accelerator for Calcium Trace Extraction from Video) firmware programmed on the field-programmable gate array (FPGA) of the Ultra96. ACTEV crops the incoming image from its original size (1296 × 972 for the LFOV sensor shown here) down to a 512 × 512 subwindow (manually selected to contain the richest area of fluorescing neurons in each animal) before storing video frames to a BRAM buffer on the FPGA. Subsequent motion correction and calcium trace extraction steps are performed by the FPGA fabric as described in the main text. Extracted calcium traces are stored to DRAM from which they can be read as inputs to a decoder algorithm running on the Ultra96 ARM core. Decoder output is fed to a logic mapper that can trigger TTL output signals from the Ultra96, which in turn can control external devices for generating closed-loop feedback. Stage 1: motion stabilization Incoming frames arriving to the Ultra96 from the Miniscope DAQ are cropped to a 512 × 512 subregion, manually selected to contain the richest area of fluorescing neurons in each animal. Pixel data from the cropped subregion are stored to a BRAM buffer on the Ultra96 where it can be accessed by ACTEV accelerator running in the FPGA logic fabric. To correct for translational movement of brain tissue, a 128 × 128 pixel area with distinct anatomical features is manually selected during Stage 1 (Figure 1B) from within the 512 × 512 imaging subwindow to serve as a motion stabilization window. ACTEV's image stabilization algorithm (Chen et al., 2019) rigidly corrects for translational movement of brain tissue by convolving the 128 × 128 stabilization window in each frame with a 17 × 17 contrast filter kernel, and then applying a fast 2D FFT/IFFT-based algorithm to correlate the stabilization window contents with a stored reference template (derived by averaging 1000 contrast-filtered frames from the beginning of each experimental session) to obtain a 2D motion displacement vector for the current frame. Video 1 demonstrates online performance of ACTEV's real-time motion stabilization algorithm. Video 1 Download asset This video cannot be played in place because your browser does support HTML5 video. You may still download the video for offline viewing. Download as MPEG-4 Download as WebM Download as Ogg Real-time motion correction. The left and right windows show sensor video data before and after motion correction, respectively. The line graphs at bottom show x (yellow) and y (green) components of the image displacement between frames before (left) and after (right) motion correction. Stage 2: background removal After motion stabilization, ACTEV removes background fluorescence from the 512 × 512 image by performing a sequence of three enhancement operations (Chen et al., 2020): smoothing via convolution with a 3 × 3 mean filtering kernel, estimating the frame background via erosion and dilation with a 19 × 19 structuring element (Lu et al., 2018), and subtracting the estimated background from the smoothed image. These operations produce an enhanced image in which fluorescing neurons stand out in contrast against the background (see Video 2; 'Enhanced image' in Figure 2). Calcium traces are then extracted (by methods described under 'Stage 3') from the motion-corrected (Stage 1) and background-removed (Stage 2) image. Video 2 Download asset This video cannot be played in place because your browser does support HTML5 video. You may still download the video for offline viewing. Download as MPEG-4 Download as WebM Download as Ogg Real-time interface (RTI) view of contour-based calcium trace extraction. The online image display (left window) shows the motion corrected and enhanced (i.e., background subtracted) sensor image data as it arrives in real time from the Ultra96 board. The right window shows a scrolling display of 63 selected calcium traces from regions outlined by colored borders in the image display window. These traces are derived on the Ultra96 by summing fluorescence within their respective contour regions, and the resulting trace values are transmitted (along with sensor image data) via ethernet to the host PC for display in the RTI window. For demonstration purposes, traces shown in the window on the right are normalized within the ranges of their own individual minimum and maximum values. Stage 3: trace extraction To derive calcium trace values, the enhanced image is filtered through a library of up to 1024 binary pixel masks, each of which defines a unique ROI for extracting a calcium trace. Each mask is defined as a selected subset of pixels within a 25 × 25 square region that can be centered anywhere in the 512 × 512 imaging subwindow (Figure 2, upper left). In performance test results presented below, decoding accuracy was compared for two different methods of selecting ROI pixels: contour-based versus contour-free. For contour-based ROI selection, the CaImAn (Giovannucci et al., 2019) algorithm was used during the intermission period to identify contours of individual neurons in the training dataset, and pixel mask ROIs were then aligned to the identified regions where neurons were located. For contour-free ROI selection, the 512 × 512 imaging subwindow pixel was uniformly covered with a 32 × 32 array of square tiles, each measuring 16 × 16 pixels, which served as ROIs for calcium trace extraction. This contour-free approach did not align ROIs with individual neurons, so each extracted calcium trace could detect fluorescence originating from more than one neuron. It is shown below that this method of contour-free ROI selection can improve the computational efficiency of online decoding at little or no cost to prediction accuracy. After ROIs had been defined by the contour-based or contour-free method, calcium traces were derived by summing pixel intensities within each ROI. Demixing was not performed during real-time calcium trace extraction; it is shown below that similar prediction accuracy can be achieved with offline (demixed) versus online (non-demixed) decoding using identical contour-based ROIs. Stage 4: population vector decoding The vector of calcium trace values extracted from each frame is stored to the Ultra96 DRAM, from which trace values are then read into the MPSoC's ARM core. The ARM core executes user-defined decoding algorithms running as a C++ program under the FreeRTOS operating system (which affords highly consistent execution times and therefore minimizes frame-to-frame variability in decoding latency). In performance tests reported below, we used simple linear classifier algorithms to decode calcium population vector activity of neurons imaged in CA1 and OFC of freely behaving rats. However, the C++ programmable ARM core is capable of implementing a wide range of different decoder architectures, including deep convolutional neural networks and long short-term memory networks (Chen et al., 2022b). Performance testing Results presented below were obtained using two different versions of the UCLA Miniscope: CA1 position decoding tests were conducted using the Large Field-of-View (Blair et al., 2021) (MiniLFOV) version featuring 1296 × 972 pixel resolution sampled at 22.8 fps, whereas OFC instrumental behavior decoding tests were conducted using the V4 version featuring 600 × 600 pixel resolution sampled at For performance we used a sensor (see that was capable of stored image data to the ACTEV accelerator at the frame rate for V4, 22.8 for as video data arriving in real This different real-time image processing algorithm contour-based contour-free to be compared and on the stored Calcium traces obtained with the sensor were to be identical with obtained during real-time imaging ACTEV for sensor testing and for real-time imaging with both versions of the UCLA Miniscope (MiniLFOV and can be found at https://github.com/zhe-ch/ACTEV, (Chen, 2023 copy archived at swh:1:rev:aa6393d3bd2dd490aa5369e1f2677e85e8a64a82). decoding from CA1 For the set of performance the MiniLFOV was used to image neurons in the hippocampal CA1 region while rats (n = 12) and on a linear track (Figure Each linear track session was min in frames of calcium trace data per CA1 neurons are to as that an animal in and so a position can be decoded from CA1 population activity et al., 2013; Cai et al., 2016; and et al., 2018; for As of the neurons we imaged in CA1 were on the linear track (see example in Figure data in of to Figure For real-time we did not a for classifying CA1 traces as from versus calcium traces were decoded from all ROIs of their Figure 3 with 1 see all Download asset Open asset decoding from CA1 (A) The CA1 of the was imaged while rats on a linear for position the rat's was and into position each ~20 (B) linear inputs per calcium were to outputs using a the track show with outputs of at each of interest (ROIs) from which contour-based contour-free and contour-based calcium traces were extracted in ROI shows the for traces extracted from the ROI. from training and testing for and traces extracted from are by of activity in the testing Decoding performance did not differ for versus offline traces (left but was for sessions with of calcium traces Decoding performance over track within each session (left and over sessions at each track position Decoding performance as a of training set significant with the of more frames to the training For all hippocampal activity is or decoded offline than it is common to perform speed filtering that time the is This is because during the a state during which place cell activity is less for the animal's current 2015). speed filtering was not implemented during online decoding of CA1 fluorescence because the linear classifier real-time calcium trace data and not position data that would be needed for speed As shown below, DeCalciOn achieved accurate decoding of the rat's position from calcium traces speed ACTEV can utilize a real-time to each calcium trace into an per frame. However, decoding information from from of neural population it is standard to perform a by in time of a In calcium imaging the as a of activity. The used in our CA1 imaging experiments has a of ms et al., which is similar to the of time typically used for decoding a rat's position from of place et al., 2013; Cai et al., 2016; and et al., 2018; for In with of position decoding from CA1 calcium fluorescence et al., we found (see A of to Figure that the linear classifier was more accurate at decoding the rat's position from calcium trace (which on a time of the to the than from (which on a much time of to the miniscope's frame we decoded the rat's position from calcium traces than in performance tests reported below. The linear classifier was trained on calcium trace and position data from the of each then on data from the of the session. The output vector of that used a (Figure to position ~20 along a of the After the linear classifier had been trained on data from the of the weights were uploaded from the host PC to the Ultra96 and image data from the of the session data) was fed through the sensor to real-time decoding. To that position with than performance was compared for trained on calcium traces that were aligned versus with position data to Figure decoding performance was compared for different methods of extracting online fluorescence traces including contour-based versus contour-free methods as explained below. trace decoding trace extraction derived calcium traces by summing fluorescence within pixel masks defining ROIs with identified neurons (Figure Video 2). were identified during the intermission period by using the CaImAn pipeline (Giovannucci et al., 2019) to perform (Giovannucci et al., 2019; et al., 2016) on the training dataset and extract ROIs for calcium traces. CaImAn on frames of training data required 30–60 min of computing time on the host PC, this the performed during the intermission period (see Figure 1B). After contour ROIs had been an additional min of computing time was required to all of the training data frames through a to online calcium traces that would have been extracted in real time during the of the session using the extracted contour The online traces were then aligned with position data and used to train a linear classifier for decoding the rat's position on the linear track from calcium training the decoder required a maximum s of computing position decoding requires that calcium trace inputs to the decoder must the training and testing of the session. To that this was the were derived for calcium traces during the versus of each session (Figure left). of and in to Figure that of traces in
Source: eLife
Institution: University of California, Los Angeles
Language: English
Subfield: Biophysics
Domain: Life Sciences
Open Access status: gold
Estimated APC paid: $2,500
Funder: National Science Foundation
OpenAlex ID: W4319783129