Flash communication pattern analysis of fireflies based on computer vision

a stopwatch


Introduction
Fireflies' bioluminescence behavior is an interesting phenomenon. The wonderful light of adult fireflies plays a role in reproductive species-specific isolation according to the pattern of emitted light [1]. Fireflies have various kinds of communication systems, especially nocturnal fireflies [2]. Different species emit light in different patterns. The characteristics of the flash, for instance, light intensity, lantern size, and pulse duration are used for species-specific reproductive separation [3]. Several species of female Photinus mimic the flash response of the other female species to attract and devour their males [4]. In addition, their bioluminescence is used for illumination during landing and walking, which protects fireflies from the spider's webs and flooded areas [5]. Therefore, the study of bioluminescence behavior can lead to an understanding of the biology of fireflies.
Since the firefly flash is a sophisticated behavior, a variety of methods were used to study flashing behavior. One method was direct human observations using a stopwatch [1][6] [7]. The stopwatch technique is limited in that it is prone to inaccuracies because the stopwatch operator has a significant delay in switching the watch on and off.
The photomultiplier tube (light sensor) detects and records using a data acquisition system is another technique used for firefly behavior study [8]- [10]. However, the photomultiplier tube is not appropriate for recording several fireflies simultaneously, because it senses all of the light sources at the same time and leads to interference of the signal.
In addition, the video recording method is analyzed based on the frame by frame analysis [3] [11]. Flashing behavior study by video recording can decrease the limitation of multiple object recording. Normally, one second of video length consists of 25-30 frames. There is a significant time needed for data interpreting, especially during the process of capturing the pictures and analyzing them frame by frame. This method also limited the analyzing capability due to the manual tracking of individual fireflies in each frame during the frame by frame analysis [11].
Computer-assisted techniques are used extensively to improve the performance of many processes in studies such as robotics, automated agriculture, digital devices, as well as automation monitoring. Dankert and colleagues [12] used computer vision to track fruit fly behavior, which gave highthroughput and accurate results. Computer vision was also applied to a variety of biological studies such as taxonomy (automated identification), plant phenotyping, and cell culture [13]- [15]. Computerized image processing is more accurate and takes less time to investigate and analyze data [16].
Due to the advantages of computer vision, the goal of this study is to develop a program to assist in analyzing firefly flashing behavior based on computer vision. The program tracks the firefly, records the flash amplitude and the time of the flash during frame by frame analysis. The developed program also assists flash interpretation by calculating the pulse duration and flash interval. The developed program can enhance the capability of routine tasks of biologists and entomologists for studying insects and animal behavior.

Development of the program: flash data extraction from the video
The recorded video is converted from ".MOV" to ".mp4" format before the image-processing process by Movie Maker, Microsoft, 2012. Image processing, all frames of the video file are processed reclusively based upon three steps. In the first step, each frame is converted from RGB (red-green-blue) color space to HSV (hue-saturation-value) color space because it is more suitable for image segmentation than the RGB model [17]. The conversion follows the study of Pekel [18]. Secondly, the firefly flash is extracted from an image of the value channel (an array image of HSV color space) by the multiple-thresholding technique [19]. Then the flash area (pixel 2 ) is calculated to represent the flash illuminance. The coordinates of the position and time of each flash area were also collected.

Development of the program: firefly tracking process
Global object tracking can be classified into two types: probabilistic and deterministic methods [20]. The probabilistic method solves the tracking problem based on Bayesian inference or uncertainty modeling, such as Monte Carlo, Particle Filtering framework [21] [22]. Deterministic methods solve the tracking problem by comparison to the region of interest in the present and previous frames [23]. However, the use of an algorithm to track the movement of fireflies has not been reported in firefly flashing behavior studies.
Accordingly, an algorithm was developed that was motivated by nature [24]- [26]. The tracking process in this study was designed using aspects of probability and mimicry of female firefly behavior. The fireflies are tracked by using the coordinate position of the occurring flash and predicting the area where the next flash will occur because the flash continuously appears and disappears as it flashes on and off. The problem is complicated as the observer needs to consider whether the flash is made by the same insect or not. The algorithm operations are based on two concepts.
The first, these females are represented by a set A = {f1, f2, f3, …, fn}, where each member of this set is the female firefly. When a flash occurs, the female firefly is simulated by the computer (f1) and tracks this flash. The computer (f1) as an observer that observes the flash position, and time of the flash occurrence to predict where the next flash will occur (area of interest, I). When the next flash occurs, if this flash is in the area of interest of f1, the computer is going to track and calculate the area of interest for f1. However, if the flash is not in the area of interest of any member in A. The computer will begin to observe this flash as a second female firefly (fn+1). The fireflies tracking algorithm is represented by algorithm 1 (Fig. 1).  The Second, the area of interest (I) represents the prediction area calculated for each female (member in set A), which follows by the highest-flying velocity of the firefly: Vmax (approximate) and the flying directions (which are random). If the time of the first flash is tn and the next flash is tn+1, the velocity of fireflies between tn and tn+1 can be calculated as shown in (1). In addition, 's' = the distance between the first and the next flash. The distance between the two flashes is two dimensions, the 's' is calculated on Euclidean space as shown in (2). Assuming that the flying directions are random, the flash will happen in the circumferential (c). The radius between each flash at tn and tn+1 is the Euclidean distance (3).
In reality, the velocity of the firefly movement is variable, which ranges from 0 to V max . Therefore, the area of interest (I) is calculated by equation 4.
Where 's' is the distance of 'v' at a time between tn and tn+1. 0 and smax are the distances of v = 0 and v = Vmax, respectively. The setting of the area of interest (I) in the developed program is defined by distance (s, pixel).

Development of the program: flash data interpretation
The firefly has two types of bioluminescence, glow (continuous light emission) and flash (noncontinuous light emission). The flash of the firefly has two sub-types, the simple and multi-pulsed flashes. The flash was previously represented using the illustration of a flash pattern in a simple diagram form. The diagram was constructed using three parameters of flash: the pulse duration, flash interval, and interpulse interval. The pulse duration is the time of flash occurrence. The flash interval is the period from the start of the current flash to the start of the next flash. The two mentioned parameters relate to constructing the pattern of simple flashes and multi-pulsed flashes. The interpulse interval is a period from the start of a current cluster to the next start of the next cluster of flashes [27] [28]. This parameter is only used to explain the sub-type of multi-pulsed flashes. Thus, the algorithm for flash data interpreting was written based on the mentioned criteria of these flash parameters.
After video analysis by the developed program, the output consists of three data sets: the flash areas (pixel 2 ), time of flash occurrence (millisecond, ms), and the coordinate position of the occurring flash. Only the data series of the flash area is interpreted to obtain the flash data. This data series represents a series of video frames. The interpretation process has four steps. The first, the light periods are counted (flash area ≥ 1 pixel 2 ). It may contain one frame or more until the flash area is equal to 0. Second, the counting number (frame no.) is converted to the pulse duration by equation 5. Third, the series of the dark periods are counted until the flash occurs again (flash area ≥ 1 pixel 2 ). Fourth, the calculation of the flash interval is the summation of the antecedent light period and the present dark period (6). After calculation, the process starts again with step 1 and repeatedly continues until the end of the video. However, the developed program is more suitable for simple flash patterns rather than multi-pulsed flash patterns.
where 'FPS' is the frame rate of video recording.

Validation of the developed program: sample collection and video recording
We captured male Sclerotia aquatilis from their habitat in the suburban area of Phitsanulok Province, Thailand. The S. aquatilis was previously classified as and referred to as Luciola aquatilis [29]. This study used S. aquatilis instead, which are the same species, previously referred to as Luciola aquatilis [30]. A single firefly was kept and observed in a transparent plastic box (7 × 10 × 5 cm, w × l × h); the observation box of multiple fireflies was 25 × 17 × 9 cm. In addition, the fireflies were allowed to fly in a room (425 × 300 × 326 cm, w × l × h) with the temperature-controlled at 25 degrees Celsius. During recording, the researcher tracked the free-flying firefly manually. The distance between the fireflies and the camera were approximately 50 centimeter (transparent plastic box observation) and 1-3 meters (flying observation). The recording of both observations proceeded under low-light intensity and visible-light wavelength conditions by using a digital camera (Canon Model 70D). The resolution of the video was 1920 × 1080 pixels with a frame rate = 30 frame per second (FPS), f-stop = 3.5, ISO = 6400, with manual focusing, with EF-S 18-135mm f/3.5-5.6 IS STM lens kit.

Validation of the developed program: comparison of the pulse duration, flash interval, and flash pattern to a reference study
We compared the results of the pulse duration and flash interval of male Sclerotia aquatilis obtained from the video analysis by the developed program with a reference study [30]. The maximum and minimum of the pulse duration and flash interval from the reference study were the benchmarks of comparison. The male S. aquatilis flash pattern was characterized by the time of flash occurrence (ms) and the flash area (pixel 2 ) on the x-axis and y-axis, respectively. The scale of the flash pattern chart was adjusted equally to the x-axis of the reference chart of the study of Thancharoen [30]. A similar comparison indicates the interpretation accuracy of the developed program.

Validation of the developed program: the firefly tracking algorithm
The process used the videos of box observant of 1-6 fireflies to evaluate the performance of the tracking algorithm of the developed program. The area of interest (I) was defined by distance (s) at 100, 200, 300, 400, and 500 pixels, which is related to parameter assignment in the developed program. The coordinate positions of the firefly flash were drawn to confirm the movement path with the video playback. The movement path and flash pattern were used for considering the ratio of tracking accuracy (A). This ratio was calculated following (7).
where A is the ratio of the tracking accuracy, f as a number of observed fireflies, C for a number of flash series that is correctly tracked, and F is a number of flash series that is a fragment, a part of a correct tracking of the flash series. P for a number of flash series that contains an overlapped tracking-path, W for the whole number of the flash series, and T is a number of tiny-fragments of a flash series, the flash data that contains only 1 or two flashes.

Validation of the developed program: time consumption comparison
The time necessary for the developed program during the analysis process, and the data recording process was compared to the stopwatch and videography methods. The protocol for the stopwatch and videography method, which were replicated follow the study of Iguchi [6] and Thancharoen [30], respectively. However, the flashing behavior was monitored with a firefly video instead of in the natural habitat by the researcher during the analysis using the stopwatch and the videography. The same video was then analyzed using the developed program. The program was run on a low-end (Intel® ATOM™ Z2760 at 1.8GHz, with 2 GB of RAM, a solid-state drive, OS Windows 8.1 32 bit) and high-end computer (Intel® Core TM i7-6700K at 4.00 GHz, with 16 GB of RAM, a solid-state drive, OS Windows 10 64 bit).
The time consumption is measured as two-component, each method measured during the conducting phase and recording phase. The conducting phase definition is from the beginning of video monitoring until the end. The time consumption of the recording phase is measured from the beginning of filling in the data until all data is entered. However, the time-consumption does not include file management, such as changing a file name, moving files, opening a program, and setting parameters.

Results and Discussion
The developed programs included flash video analysis (Fig. 2) and flash data interpretation (Fig. 3).

Validation by comparison of the pulse duration, flash interval and flash pattern
The reference method, according to the report of Thancharoen [30], firefly behavior was recorded in the laboratory using a video recorder. They classified four types of flash behavior. These are "Beginning flash" (type 1), "Advertising flash" (type 2), "Courtship flash" (type 3), and "Warning flash" (type 4). In our study, we used the video recording of male Sclerotia aquatilis, which were recorded under laboratory monitoring similar to the reference study. The results of the four types of flash patterns were similar to the reference report ( Fig. 4(a)).
Approximately 30-40 seconds after sunset, the male firefly awoke and advertised itself with a flash of type 1 (Fig. 4(a)). Three-to-five minutes later, the firefly started to walk around to find a female or to hold on to their position. While walking around, the male firefly emitted type 2 and 3 flash patterns ( Fig. 4(a)). Type 4 flashing was produced rarely compared to the other flashing types. In addition, the flying fireflies observed under laboratory conditions produced flash type 3. The average pulse duration was 190.35 ± 55.45ms (n=96), and the average flash interval was 392.87 ± 83.07ms (n=94). While the firefly flew near an obstacle such as a wall, it emitted a rapid flashing pattern ( Fig. 4(b)). The pulse duration and flash interval of type 2, 3, and 4 were similar to that report by Thancharoen [30] (Table 1). The flash pattern analysis in our study was similar to those reported by Thancharoen [30] (Fig. 4), including the pulse duration and the flash interval of type 2, 3, and 4 ( Table 1) under similar conditions: recording in the darkroom (0 lux), temperature, FPS of recording (30 FPS of the current; 25 FPS of the reference), distance of the recording, and the recording of a firefly within a plastic box [30]. It can be postulated that our method offers a convenient and reliable technique for studying flashing behavior in fireflies. Table 1. The flash type, pulse duration, and flash interval of the observed Sclerotia aquatilis in our study compared with the reference study [30]. Normally, type 4 flashing is found during courtship [30]. However, we found that only male fireflies perform this flash type in the plastic box, without female fireflies. The reason for type 4 flash emissions may be due to reflection of the flashes on the plastic box, and the males assumed that these were female flashes and responded to the flash. This behavior was also found when observing fireflies in their natural International Journal of Advances in Intelligent Informatics ISSN 2442-6571 Vol. 6, No. 1, March 2020, pp. 60-71 Tathawee et al. (Flash communication pattern analysis of fireflies based on computer vision) habitat. As our observation when a male Sclerotia aquatilis flew near the windshield of the car; and then they attempted to alight on the car due to the flash reflection on the windshield. However, when they found that it was not a female, they flew away from the windshield immediately.

Validation of the algorithm of firefly tracking
The fireflies spent more of their time walking than flying. The tracking algorithm can track the fireflies easily. The small area of interest was better to track multiple-fireflies. However, the large area of interest was more suitable to track a single firefly (Table 2). The example result was obtained from the analysis of a video of four fireflies. A large area of interest increases the overlapping rate of the tracking paths. Occasionally, the overlapping of tracking paths may lead to the combination of two or more datasets of firefly flashes by observing these flash in a single dataset (Fig 5; at a 500-pixel distance). Moreover, the green dots represent a new dataset of a single firefly (firefly no. 5), which is really the combination of firefly number 2 and 3 datasets. This combination caused a fragmentation of the dataset; i.e. the firefly datasets 2 and 3 were shorter due to the fragmentation. The shortened dataset may be the correct data, which depends on the overlapping rate among the dataset. In the case of the 200-pixel distance, the dataset was correct (firefly no. 2 and 3), but the dataset of firefly no. 1 at a distance of 300 pixels was an incomplete dataset (red arrow, Fig. 5). The firefly tracking capability of the developed program and associated an algorithm was able to track the flying fireflies and individually fireflies in a plastic box containing multiple fireflies. Using a large area of interest (I) caused the overlapping of the fireflies' paths, which caused a low accuracy ratio in this setting. Whereas the use of a small area of interest (I) was better than a large area. Even though the small area of interest (I) may provide an output where the data series are split. The split-series data was easy to check for the correctness of individual tracking by plotting a chart of the flight pattern after finishing the analysis. However, where the data overlapped, we cannot differentiate whether the signal comes from one firefly or another. Each flashing signal in the signal combination appears to be the same (Fig. 5). Therefore, we proposed to use a small area of interest rather than the large area in the multiple fireflies tracking, but for a single firefly, we suggest a large area of interest.
Interestingly, we found the synchronous-like flash behavior of Sclerotia aquatilis males, which were placed in the same box (2)(3)(4)(5)(6). This behavior has not been reported in previous studies of this species (Fig. 6). The synchronous-like flashing behavior of Sclerotia aquatilis may be performed for the sympatric speciation of competing males. The synchronous flash was reported in some species of Pteroptix, Photinus, and Luciola genera, but there are no other reports in S. aquatilis [31] [32]. Genetically, the S. aquatilis was thought to be closely related to Luciola [29]. This trait was a remarkable feature in this group, so, S. aquatilis perform similar synchronous flashing. Further study is suggested to investigate the synchronized flashing behavior in S. aquatilis.

Analysis of time consumption of the developed program
The measurement of decreasing in time consumption, the developed programs were run on a highend and low-end processor. These took less time to analyze data than the videography method. However, the low-end computer took a time similar to the stopwatch method. Thus, using the developed program can decrease the time consumption by up to 96.82% and 76.91% compared to the videography and the stopwatch method, respectively (Fig. 7). In addition, for interpreting the flash data, the developed program used the time for the analysis was equal to 'T' (second), where T = (0.0931 × l) + 3.6549, and 'l' = the length of the video (seconds) (when it was operated on a high-end computer).
The low-end computer method took a similar amount of time consumption as the stopwatch method. However, the stopwatch is limited in that it is prone to inaccuracies because the stopwatch operator has a fraction of a second delay in switching the watch on and off. At least 30 duplicated samplings should be done as the average flash duration is not statistically different, which compares with the photomultiplier tube technique [6]. Although the photomultiplier has a high resolution and is an accurate technique, it is limited in observing multiple fireflies. The video recording method can decrease this limitation by observing several fireflies at the same time. In addition, the video recording method can give the flash waveform similar to the photomultiplier method, which is different from the stopwatch method. It cannot provide a reliable flash intensity, which is important to draw the waveform. However, the video recording is limited by the time-consumption due to the frame by frame analysis that has manually done (Fig. 7). Each method detail was mentioned in the time consumption comparison method-section In conclusion, the video recording performs better than the photomultiplier and stopwatch method by providing an output of the flash waveform and allows multiple fireflies to be observed. However, time-consumption can be a limitation. Thus, the use of the video recording method combined with the developed program can decrease the time needed to analyze the flashing behavior video. Moreover, these programs provide benefits from automatic result recording, which can decrease human error while managing the output results.

Conclusion
The developed program can decrease routine tasks for biologists and entomologists when used as a substitute method for analyzing firefly flash behavior. The similarity of the results of the pulse duration and flash interval of Sclerotia aquatilis compared to a reference study supported this conclusion. The use of the developed program reduced the time spent on flash analysis compared to the reference method; it displayed the data of pulse duration and flash interval in a few minutes. For further study, machine learning can be applied to improve the firefly tracking or to recognize the unique flash patterns of different species. This improvement of the developed program will offer automatic firefly identification and automatic counting of firefly numbers in their natural habitat, which would benefit the study of firefly diversity.