10. Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types

Computer vision-based techniques were developed to overcome the limitations of visual inspection by trained human resources and to detect structural damage in images remotely, but most methods detect only specific types of damage, such as concrete or steel cracks. To provide quasi real-time simultaneous detection of multiple types of damages, a Faster Region-based Convolutional Neural Network (Faster R-CNN)-based structural visual inspection method is proposed. In order to realize this, a database including 2,366 images (with 500 × 375 pixels) labeled for five types of damages—concrete crack, steel corrosion with two levels (medium and high), bolt corrosion, and steel delamination—is developed. Then, the architecture of the Faster R-CNN is modified, trained, validated, and tested using this database. Results show 90.6%, 83.4%, 82.1%, 98.1%, and 84.7% average precision (AP) ratings for the five damage types, respectively, with a mean AP of 87.8%. The robustness of the trained Faster R-CNN is evaluated and demonstrated using 11 new 6,000 × 4,000-pixel images taken of different structures. Its performance is also compared to that of the traditional CNN-based method. Considering that the proposed method provides a remarkably fast test speed (0.03s per image with 500 × 375 resolution), a framework for quasi real-time damage detection on video using the trained networks is developed.

[Ref.] Cha Y.J.†, Choi W.*, Suh G.*,  and Mahmoudkhani S.* and Buyukozturk O.* (2018) “Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types,” Computer-Aided Civil and Infrastructure Engineering (IF: 5.786), DOI: 10.1111/mice.12334.  Top journal within Civil Engineering discipline).

US Patent was filed. Please contact Professor Cha, if you want to have any research collaboration (young.cha@umanitoba.ca).

 

 

9. Deep learning based crack damage detection using convolutional neural networks

A number of image processing techniques (IPTs) have been implemented for detecting civil infrastructure defects in order to partially replace human-conducted on-site inspections. These IPTs are primarily used to manipulate images in order to extract defect features, such as cracks in concrete and steel surfaces. However, the extensively varying real-world situations (e.g., lighting and shadow changes) can lead to challenges to the wide adoption of IPTs. To overcome these challenges, this research develops a vision-based method using a deep architecture of convolutional neural networks (CNNs) for detecting concrete cracks without calculating the defect features. As CNNs are capable of learning image features automatically, the proposed method works without the conjugation of IPTs for extracting features. The designed CNN is trained on 40K images of 256×256 pixel resolutions and, consequently, records with about 98% accuracy. The trained CNN is combined with a sliding window technique in order to scan any image size larger than 256×256 pixel resolutions. The robustness and adaptability of the proposed approach are tested on 55 images of 5888×3584 pixel resolutions taken from a different structure which is not used for training and validation processes under various conditions (e.g., strong light spot, shadows, and very thin cracks). Comparative studies are conducted to examine the performance of the proposed CNN using traditional Canny and Sobel edge detection methods. The results show that the proposed method shows quite better performances and can indeed find concrete cracks in realistic situations.

[Ref.] Cha Y.J.†, Choi W.*, and Buyukozturk, O. (2017) “Deep learning-based crack damage detection using convolutional neural networks,” Computer-Aided Civil and Infrastructure Engineering; 32(5) 361-378 (IF:5.288).

US Patent was filed. Please contact Professor Cha, if you want to have any research collaboration (young.cha@umanitoba.ca).

 

8. Unsupervised novelty detection based structural damage localization using density peaks-based fast clustering algorithm

Within machine learning, several structural damage detection and localization methods based on clustering and novelty detection methods have been proposed in the recent years in order to monitor mechanical and civil structures. In order to train a machine learning model, an unsupervised mode is preferred because it only requires sufficient normal data from the intact states of a structure for training, and the testing abnormal data from various damage states are generally quite rare. With an unsupervised training mode, the capability of detecting structural damage mainly depends on the identification of abnormal data from the testing data. This identification process is termed unsupervised novelty detection. The premise of unsupervised novelty detection is that a large volume of a normal dataset is available first to train a normal model that is established by machine learning algorithms. Then, the trained normal model can be used to identify abnormal data from future testing data. In this paper, a new structural damage detection and localization method is proposed using a density-peaks-based fast clustering algorithm (DPFCA). In order to realize damage detection, the original DPFCA is modified to an unsupervised machine learning method by adding training and testing processes. Furthermore, to improve the performance of the proposed method, the Gaussian kernel function of radius is introduced to calculate the local density of data points and a new damage-sensitive feature using a continuous wavelet transform is also proposed. Damage-sensitive features are extracted from the measured data through sensors installed on a laboratory-scale steel structure. Extensive experimental studies are carried out under various structural damage scenarios in order to validate the performance of the proposed method. The proposed DPFC method shows satisfactory performance with regard to damage localization under various damage scenarios as compared to a traditional approach.

[Ref.] Cha Y.J., and Wang Z.* (2017) “Unsupervised Novelty Detection Based Structural Damage Localization Using Density Peaks-Based Fast Clustering Algorithm,” Structural Health Monitoring (IF: 3.193), accepted.

 

 

7. Phase-based optical flow based structural system identification and damage detection

A damage detection methodology is proposed by integrating a nonlinear recursive filter and a non-contact computer vision based algorithm to measure structural dynamic responses. A phase-based optical flow processing inspired by motion magnification technique is used to measure structural displacements, and the unscented Kalman filter is used to predict structural properties such as stiffness and damping coefficients. This non-contact displacement measurement methodology does not require an intensive instrumentation process, does not add any additional mass to the structure which may skew measurements and can measures more signals compared to traditional methods. In order to detect structural damage using measured displacements from video, an unscented Kalman filter is used to remove noise from the displacement measurement and simultaneously detect damage by identifying the current stiffness and damping coefficient values given a known mass which are used to detect damage. To validate the proposed damage detection method, state-space equations are derived without external excitation input, and experimental tests are carried out. The experimental results show reasonable and accurate predictions of the stiffness and damping properties compared to dynamic analysis calculations.

 

 

4

[Ref.] Cha Y.J.† Chen J.G. and Büyüköztürk O. (2017), “Output-Only Computer Vision Based Damage Detection Using Phase-Based Optical Flow and Unscented Kalman Filters,” Engineering Structures, (IF: 1.893), 132, 300–313.

Cha Y.J.†, Chen J., and Buyukozturk O., “Motion Magnification Based Damage Detection Using High Speed Video10th International Workshop On Structural Health Monitoring (IWSHM), Stanford, USA, September 1-3, 2015.

 

 

6. Motion magnification based structural modal identification and response measurement using a video

Video cameras offer the unique capability of collecting high density spatial data from a distant scene of interest. They can be employed as remote monitoring or inspection sensors for structures because of their commonplace availability, simplicity, and potentially low cost. An issue is that video data is difficult to interpret into a format familiar to engineers such as displacement. A methodology called motion magnification has been developed for visualizing exaggerated versions of small displacements with an extension of the methodology to obtain the optical flow to measure displacements. These methods are extended to modal identification in structures and the measurement of structural vibrations.

[Ref]. Chen J., Wadhwa N., Cha Y.J., Durand F., Freeman F., and Buyukozturk O†. (2015), “Modal identification of simple structures with high-speed video using motion magnification,” Journal of Sound and Vibration, 345: 58-71 (IF: 1.857).

http://newsoffice.mit.edu/2015/magnifying-vibrations-bridges-and-buildings-0423

 

5. Air-coupled impact-echo damage detection in concrete structures using wavelet transforms

Internal damage detection of reinforced concrete (RC) structures is a challenging field that has garnered increasing attention over the past decades due to a decline in the state of infrastructure in North America. As a nondestructive testing mode, the impact-echo method is currently a promising approach. However, it requires intensive testing to cover large-scale civil RC structures with point-by-point inspection. In order to partially overcome this limitation, this study proposes a new impact-echo analysis method using wavelet transforms with dual microphones with 20 kHz resolution to improve damage detection capability. The signals recorded from the microphones are processed to recover spectral data that are further analyzed using percentage of energy information to determine the condition of the specimen and detect in-situ damages. In order to validate the performance of the proposed method, the results from traditional signal processing using FFT and wavelet transforms are compared. The proposed wavelet transform based approach showed better accuracy when covering broader areas, which can contribute to reduce testing time significantly when monitoring large-scale civil RC structures.

[Ref.] Epp T.*, and Cha Y.J.†, (2017) “Air-coupled microphone based concrete structure damage detection using wavelet transforms,” Smart Materials and Structures 26(2) 025018(IF: 2.769) .

4. Automated loosened bolt detection

 

Many contact-sensor-based methods for structural damage detection have been developed. However, the high possibility of false alarms due to noises, sensor malfunctions and complex environmental effects (temperature, humidity, and change of boundary condition) means that engineers would still have to make on-site visits to confirm that damage has occurred. To address this challenge, vision-based approaches have been developed to detect corrosions, cracks, delamination, and voids. However, there are few such approaches for loosened bolts. Therefore, we developed a novel vision-based detection method. Target images of loosened bolts were taken by a smartphone camera. From the images, damage-sensitive features, such as the horizontal and vertical lengths of the bolt head, were calculated automatically using the Hough transform and other image processing techniques. A linear support vector machine was trained with the aforementioned features, thereby building a robust classifier capable of automatically differentiating tight bolts from loose bolts. Leave-one-out cross-validation was adapted to analyze the performance of the proposed algorithm. The results highlight the excellent performance of the proposed approach to detecting loosened bolts, and that it can operate in quasi-real-time.

[Ref.] Cha Y.J.†, You K.S.*, and Choi, W.* (2016), “Vision-based detection of loosened bolts using the Hough transform and support vector machines,” Automation in Construction, 71(2): 181-188 (IF:2.442).

 

3. Tall building monitoring and system identification
1

Tall buildings are ubiquitous in major cities and house the homes and workplaces of many individuals. However, few studies have been carried out to study the dynamic characteristics of tall buildings based on field measurements. In this research, the dynamic behavior of the Green Building, a unique 20-story tall structure located on the campus of the Massachusetts Institute of Technology (MIT), was characterized and modeled as a simplified lumped-mass beam model (SLMM), using data from a network of accelerometers. The accelerometer network was used to record structural responses due to ambient vibrations, blast loading, and the October 16, 2012 earthquake near Hollis Center, Maine. Spectral and signal coherence analysis of the collected data was used to identify natural frequencies, modes, foundation rocking behavior, and structural asymmetries. A relation between foundation rocking and structural natural frequencies was also found. Natural frequencies and structural acceleration from the field measurements were compared with those predicted by the SLMM found to have good agreement.

[Ref] Cha Y.J.†, Trocha P., and Büyüköztürk, O. (2016), “Field measurement based system identification and dynamic response prediction of a unique MIT building,” Sensors MDPI,16(7), 1016 (IF: 2.033).

 

 

 

2. Damage localization and quantification using modal strain energy and advanced multi-objective optimization
2

In this research, a novel damage detection approach using hybrid multi-objective optimization algorithms based on modal strain energy is proposed to detect damages in various 3-dimensional (3-D) steel structures. Minor damages have little effect in the difference of the modal properties of the structure, and thus such damages with multiple locations in a structure are difficult to detect using traditional damage detection methods based on modal properties. Various minor damage scenarios are created for the 3-D structures to investigate the newly proposed multi-objective approach. The proposed hybrid multi-objective genetic algorithm detects the exact locations and extents of the induced minor damages in the structure. Even though the proposed method uses incomplete mode shapes lacking any measured information on the damaged element, it detects damages well. The robustness of the proposed method is investigated by adding 5% Gaussian random white noise as a noise effect to mode shapes which are used in the calculation of modal strain energy (Cha and Buyukozturk 2015).

[Ref] Cha Y.J. and Büyüköztürk O†. (2015), “Structural Damage Detection Using Modal Strain Energy And Hybrid Multi-Objective Optimization,” Computer-Aided Civil and Infrastructure Engineering, 30:347-358 (IF: 5.625).

 

1. Quasi real-time detection of damage using DWT

3

Discrete wavelet transformation (DWT) method is applied to detect changes of the structural frequency contents due to damage of the structures. The information provided from the sudden peak value of detail signal from the high pass filter of the DWT is used to find time and location of the damage in the system.