Original Investigation
DOI:
10.30827/ijrss.33247
Detection of similarities and differences within the same shot movement using artificial intelligence-based performance analysis: An example of a tennis service
Detección de similitudes y diferencias dentro de un mismo movimiento de golpeo mediante un análisis del rendimiento basado en inteligencia artificial: ejemplo del servicio en tenis
International Journal of Racket Sports Science, vol. 5(1) (Enero-Junio, 2023), Pag. 34-46. eISSN: 2695-4508
Received: 20-06-2023
Acepted: 07-10-2023
AUTHORS
Takashi Jindo
1 Division of Art, Music, and Physical Education, Osaka Kyoiku University, 4-698-1 Asahigaoka, Kashiwara, Osaka 582-8582, Japan.
Yusuke Satonaka
2 Information Services International-Dentsu, LTD, 2-17-1 Konan, Minato-ku, Tokyo 108-0075, Japan
Ryosuke Wakamoto
2 Information Services International-Dentsu, LTD, 2-17-1 Konan, Minato-ku, Tokyo 108-0075, Japan
Michitaka Iida
2 Information Services International-Dentsu, LTD, 2-17-1 Konan, Minato-ku, Tokyo 108-0075, Japan
Hikari Suzuki
3 Master’s Program in Physical Education, Health and Sport Sciences University of Tsukuba, 1-1-1 Tennodai, Tsukuba, Ibaraki, 305-8574, Japan.
Hirotaka Shiraishi
3 Master’s Program in Physical Education, Health and Sport Sciences University of Tsukuba, 1-1-1 Tennodai, Tsukuba, Ibaraki, 305-8574, Japan.
Daisuke Mitsuhashi
4 Faculty of Health and Sport Sciences, University of Tsukuba, 1-1-1 Tennodai, Tsukuba, Ibaraki, 305-8574, Japan.
Corresponding Author: Takashi Jindo, jindo-t93@cc.osaka-kyoiku.ac.jp
Cite this article as: Jindo, T., Satonaka, Y., Wakamoto, R., Iida, M., Suzuki, H., Shiraishi, H., & Mitsuhashi, D. (2023). Detection of similarities and differences within the same shot movement using artificial intelligence-based performance analysis: An example of a tennis service. International Journal of Racket Sports Science, 5(1), 34-46 10.30827/ijrss.33247
ABSTRACT
Abstract
Artificial intelligence (AI) -based performance analysis has the potential to support feedback in coaching; however, a useful method has not yet been proposed. This study aims to develop an AI-based performance analysis to support tennis coaching. Specifically, we investigate the accuracy of detecting similarities and differences within the same shot movement. The participants were two tennis players with more than ten years of tennis experience at the regional level. This study targeted service in tennis and videos of the 1st and 2nd service from both sides (number of services: 40 attempts) were recorded using a smartphone located on the fence behind the participant. The analysis code was executed in Python, and the main part involved the use of BlazePose, which estimates the X-, Y-, and Z-coordinates of a human pose. Video clips of 2 s were cut, with a 1 s overlap between each clip, and one of the clips was manually chosen as the standard clip. The clips were compared with the comparison clips, and the difference scores for the total and each body part were automatically calculated. An AI-based analysis was conducted considering 12 conditions combining the 1st and 2nd services from both sides and different players. As a result, a certain accuracy (≥ 70%) was confirmed for detecting overlapping phases between clips. Moreover, manually evaluated body parts that showed different movements by a certified coach corresponded to the top three different parts in the AI-based analysis for 8 of the 12 conditions. The proposed AI-based performance analysis can effectively extract similar or overlapping phases and suggest body parts exhibiting different movements.
Keywords: Performance analysis, motion analysis, artificial intelligence (AI), tennis, service.
Resumen
El análisis del rendimiento basado en inteligencia artificial (IA) tiene el potencial de apoyar la retroalimentación en el entrenamiento. Sin embargo, aún no se ha propuesto un método útil. El objetivo de este estudio es desarrollar un análisis del rendimiento basado en IA para apoyar el entrenamiento de tenis. En concreto, se investiga la precisión en la detección de similitudes y diferencias dentro de un mismo movimiento de golpeo. Los participantes fueron dos tenistas con más de diez años de experiencia en tenis a nivel regional. Este estudio se centró en el servicio en tenis y se grabaron videos de los dos primeros servicios desde ambos lados de la cancha (número de servicios: 40 intentos) con un teléfono inteligente situado en la valla detrás del participante. El código de análisis se ejecutó en Python, y la parte principal involucró el uso de BlazePose, que estima las coordenadas X, Y y Z de una posición humana. Se cortaron videos de 2 s, con un solapamiento de 1 s entre cada video, y se eligió manualmente uno de ellos como el video estándar. Los videos se compararon con los de comparación y se calcularon automáticamente las puntuaciones de diferencia para el total y para cada parte del cuerpo. Se realizó un análisis basado en IA que consideraba 12 condiciones y combinaba los dos primeros servicios desde ambos lados y de los diferentes jugadores. Como resultado, se confirmó cierta precisión (≥ 70%) en la detección de fases solapadas entre videos. Además, las partes del cuerpo evaluadas manualmente que mostraban movimientos diferentes por un entrenador certificado correspondían con las tres primeras partes diferentes del análisis basado en IA para 8 de las 12 condiciones. El análisis de rendimiento basado en IA propuesto puede extraer eficazmente fases similares o solapadas y sugerir partes del cuerpo que muestran movimientos diferentes.
Palabras clave: Análisis del rendimiento, análisis del movimiento, inteligencia artificial (IA), tenis, servicio.
Introduction
INTRODUCTION
Sports performance analyses using objective information are increasingly being conducted. In tennis, the target of this study, numerical data such as the success rates of various shots and rallies are often used as objective information (O’Donoghue, 2005). However, a performance analysis should adopt qualitative data, such as videos, to obtain detailed information. Qualitative data analysis takes the form of inputting specific events and labels using specialized software. Although this method can be useful for understanding the characteristics of each scene of play, the interpretation of the information obtained can be influenced by the experience and subjectivity of the players and coaches.
A method for automatically analyzing video data was developed based on the object-detection technology of artificial intelligence (AI), such as machine learning and its division deep learning (Brady et al., 2021; Cust et al., 2019). A systematic review stated that machine learning has been increasingly adopted in tennis tracking for analyzing both player and ball movements (Takahashi et al., 2022). More models are expected to feature deep learning owing to the development of better hardware and advantages of achieving more efficient model learning on large data inputs (Cust et al., 2019), which are suitable for racket sports because of the large number of attempted shots and movements. A systematic review of this research field (Cust et al., 2019) summarized inertial motion unit (IMU)-based and vision-based AI-based performance analyses for various sports. Different methods may be appropriate for different types of sports or situations (practice session or match). For example, in tennis, the vision-based method would be more useful because there is no need to attach devices to the player and there is potential for application in real matches. The systematic review (Cust et al., 2019) has summarized three studies that used vision-based analysis for tennis (Ó Conaire et al., 2010; Shah et al., 2007; Zhu et al., 2006). However, these studies only classified shot types such as service, forehand, or backhand strokes. Following a systematic review (Cust et al., 2019), Cai et al. (2020) conducted a more detailed AI-based analysis on 12 shots in tennis, and confirmed that there was significant confusion within the same shot, such as between a kick service and slice service.
Human pose estimation has significantly advanced. BlazePose, a lightweight convolutional neural network architecture for human pose estimation developed by Google Research (Bazarevsky et al., 2020) might be useful for tennis performance analysis. BlazePose has the ability to estimate the X-, Y-, and Z-coordinates. Therefore, this method is suitable for tennis analysis, which requires depth estimation for body direction or stance during shot movements. To date, shot classification for six-shot movements in cricket (Devanandan et al., 2021) and estimation of multiple joint angles during tennis service (Yoshida et al., 2021) using BlazePose have been reported. Although these studies suggest the application possibilities of BlazePose in racket sports, an accumulation of study findings is required to support sports coaching. Information on similarities or differences within the same shot is useful in sports coaching; therefore, AI-based analysis is expected to play a supportive role.
This study aimed to develop an AI-based performance analysis for tennis coaching applications. Specifically, we investigate the accuracy of detecting similarities and differences within the same shot movement.
Methods
METHODS
Study design and participant
An experimental study was conducted in a tennis court. The tennis court was blue and light blue in color, and the surface was hard. The study participants were two tennis players, who were also the authors of this study. The participants are experienced tennis players with more than ten years of experience at a regional level. The self-reported height and body mass of the two players were as follows: 173 cm, 62 kg for Player 1, and 170 cm, 68 kg for Player 2. This study was approved by the Ethics Committee of University of Tsukuba (approval number: Tai 022-80 and Tai 022-80-1).
Data preparation
This study focuses on tennis service as a target for performance analysis. The reason for focusing on service is that the first service points have a large impact on the win or loss of a match for both professional and junior players (Kovalchik & Reid, 2017). Thus, service is considered to have a high priority in improving tennis shot skills. In addition, the service would be a suitable target for the first step of the developed analysis because it can be performed by one player and does not need to consider the inclusion of opponents in the video.
The proposed method required two target videos. The first was a standard video that included one or more attempts at the target shot. A standard video is assumed to include an ideal movement or movement prior to specific training implementation. The second video was used for comparison. The comparison video included multiple attempts of the shot for comparison. In this study, some service videos of the two study participants were recorded, and certain conditions were set to evaluate the developed method (details are described below).
The videos were recorded at 1920 × 1080 pixels and 60 frame per second (fps) using an iPhone (Apple Inc.). The iPhone was located approximately 1.7 m high on the fence 8.5 m behind the participant and was the same for all conditions. The fence was stable, and leaning over did not affect the camera angle.
Development of AI-based performance analysis
This study developed an AI-based performance analysis method that uses tennis videos to automatically detect the differences and similarities within the same shot movement. The analysis was conducted using Google Colaboratory, an online execution environment for Python (https://www.python.org/). Target videos were uploaded to Google Drive’s cloud storage system and imported into the environment. The programming code was prototyped by researchers from Information Services International-Dentsu, LTD. The code was modified based on a preliminary study and discussions between the company and university researchers. Subsequently, the video data were gathered and analyzed by university researchers.
An overview of the AI-based performance analysis is shown in Figure 1. We adopted BlazePose (Bazarevsky et al., 2020), which is a lightweight convolutional neural network architecture for human pose estimation, for the main part of the analysis. BlazePose is one of the models in the Mediapipe framework developed by Google that offers customizable machine learning solutions for processing multimodal data. The technology is open source and available to the public. Although BlazePoze can estimate the coordinates of 33 body parts, the developed method targeted 13 body parts (nose, shoulders, elbows, wrists, left and right hips, knees, and ankles) that are important for tennis shot movements. The nose was included because its position would be useful for ascertaining the status of neck rotation, extension, and flexion.
The recorded video was converted from the MOV file to a GIF file and analyzed using developed functions, including BlazePose. In addition to the MOV file, the MP4 video format is also applicable for the analysis. In addition to converting to a GIF file, the GIF file was clipped to short-duration clips with an overlapping duration to conduct an analysis targeting the appropriate phase, such as before and after the impact. A phase indicates the entire or partial movement of a shot, whereas the entire phase indicates a shot. In this study, the clipped duration was set to 2 s, with a 1 s overlap between each clip. One of the clips that included the impact of the ball and racket was manually selected as the standard clip. A clip duration of 2 s mostly covered the entire service phase from toss-up to after impact.
These functions transform the skeletal coordinate information while excluding differences in the recording angle (i.e., between deuce and advantage sides) or position of the camera (i.e., this was not applicable to this study because of the same camera position). Specifically, a skeletal part (the left shoulder in this study) was set as the origin of the coordinates, and parallel shifts and rotations of all spatial coordinates were performed with respect to the origin. The skeletal coordinate information extracted by the functions was used to calculate the difference score between the videos.
To calculate the difference scores, the Dynamic Time Warping algorithm Sakoe & Chiba, 1978; https://tslearn.readthedocs.io/en/stable/user_guide/dtw.html) was used to consider the movement speed. Scores were calculated for each body part, with lower scores indicating smaller differences between the standard and comparison clips. The average of all body part scores was considered the total difference score for movement in this study. The scores ranged from zero to no upper limit.
These results were summarized in a score table that was sorted by the lowest total score and contained the scores of each body part. The score table is automatically output, and the results are interpreted by users.
The developed code can be demonstrated here (https://drive.google.com/drive/folders/11UG48uyxXfjIUTE6v5YGk9eZ109WRPdU?usp=sharing).
Evaluation
To achieve this goal, we conducted two evaluations. The first was to evaluate whether similar phase clips (i.e., from toss-up to immediately after the impact) were in the upper part of the score table to clarify whether the AI-based analysis extracted the overlapping phase of the standard clip from the comparison clips. To conduct the evaluation, the following conditions for service videos from the two study participants were gathered:1st and 2nd services for five shots on the deuce and advantage sides, respectively. The collected videos were used to conduct an AI-based analysis of comparisons within the same condition, between different conditions, and between different players; 12 conditions were conducted. The accuracy (%) was calculated by referring to previous studies (Cai et al., 2020; Ó Conaire et al., 2010). Notably, the top 12 clips within the same condition and top 15 clips between different conditions and players from the score table were considered for the calculation. Specifically, the accuracy was calculated by dividing the number of correct detections (i.e., overlapping the phase) by the total number of compared files. The top clips were considered because clipped comparison movies were assumed to contain at least 12 or 15 overlapping phase clips with the standard clip. The 2 s standard clip mostly contained the entire phase of service whilst the comparison movies contained four shots in the case of the same condition, and five shots in the case of different conditions and players. Therefore, although the number of clips for a single shot is dependent on the timing of clipping, a single shot can be clipped for three clips with 1 s overlap. Accordingly, the number of targeting clips was calculated by multiplying three movies by four or five shots. In addition, Spearman correlation coefficients (ρ) between correct detection and difference scores were calculated to understand these relationships.
The second objective was to evaluate the correspondence between manual and AI-based analyses for the detection of different movements in each body part. The target clips for the evaluation were set as standard clips, and one comparison clip showed the lowest total score in the first evaluation under each condition. When the clip did not have an overlapping phase between clips, the clip with the second-lowest total score was used. Manual evaluation was conducted by one of the authors who is a qualified Japan Sports Association Instructor of Tennis. The evaluator observed and compared the clips from the perspective of different movements and described applicable body parts that had different movements from the standard clip. As the left shoulder was set as the origin, this part was excluded from the evaluation. The evaluation was conducted under concealed conditions based on the results of the AI-based analysis. After completion of the manual evaluation, body parts with any description were compared with the scores and ranks of the AI-based analysis.
The evaluation of shot movement was conducted using the open-source movement analysis software Kinovia (https://www.kinovea.org/) and its functions, such as two playback screens, slow motion, and rewinding. IBM SPSS Statistics for Windows, Version 26.0. Armonk, NY: IBM Corp. was used for data analysis. The level of statistical significance was set at P < 0.05.
Results
RESULTS
Table 1 shows the accuracy of extracting the overlapping phase of the standard clip from the comparison clips. The duration of the target videos ranged from 34 to 45 s; thus, 34 to 45 clips were created. The average accuracy when comparing within the same condition was 79.2 % for both deuce and advantage sides. The average accuracy when comparing between different conditions were as follow: 73.3 % in case of side comparison, 70.0 % in case of the first and second service comparisons in both deuce and advantage sides. Among these, the accuracy in case of the side comparison for Player 2 was the lowest (46.7 %.), whilst that between different players was 80.0 and 40.0 % for the deuce and advantage sides, respectively.
Table 1 Accuracy for extracting overlapped phase of the standard clip from the comparison clips.
Table 2 shows the correlation between the correct detection and difference scores. The total and difference scores for each body part mostly showed a negative correlation with correct detection, which indicates that clips with lower difference scores are more likely to overlap. For the overall correlations, all difference scores showed a statistically significant correlation with correct detections. Moreover, side comparison for Player 2, which showed low accuracy of 46.7 %, had statistically positive correlation in the nose (ρ = 0.35), left elbow (ρ = 0.48), and left wrist (ρ = 0.40). For the different players’ comparison in the advantage side (accuracy: 40.0 %), some body parts showed statistically significant negative correlations (left elbow: ρ = -0.36 and left wrist: ρ = -0.35) while five body parts showed positive correlations.
Table 3 shows a comparison of the results between the manual and AI-based analyses for comparison within the same conditions. For the deuce side of Player 2, the manually evaluated body parts that showed different movements corresponded to the top three different parts in the AI-based analysis. However, for the deuce side for Player 1 and the advantage side for Player 2, manually evaluating different movements of the body parts had the lowest difference scores in the AI-based analysis (considered to be similar movements). For the advantage side of Player 1, there were no confirmed body parts that had different movements in the manual evaluation, whereas the left elbow showed the highest difference score.
Table 3. Comparison of the results between manual and AI-based analysis for the comparison within the same condition
Table 4 shows a comparison of the results of the manual and AI-based analyses for different conditions. One of the manually evaluated body parts that exhibited different movements corresponded to the top three different parts in the AI-based analysis under all conditions.
Table 4 Comparison of the results between manual and AI-based analysis for the comparison between different conditions
Table 5 shows a comparison of the results of the manual and AI-based analyses for different players. On the source side, two of the manually evaluated body parts that showed different movements corresponded to the top two or three parts in the AI-based analysis. For the advantage side, manually evaluated different movements of body parts were the top four difference scores in the AI-based analysis.
Table 5 Comparison of the results between manual and AI-based analysis for the comparison between different players
DISCUSSION
DISCUSSION
This study developed an AI-based performance analysis for tennis coaching and investigated its accuracy in detecting similarities and differences within the same shot movement. As a result, a certain accuracy (≥70 %) was confirmed for detecting overlapping phases between clips. Moreover, the manually evaluated body parts that showed different movements corresponded favorably to the results of the AI-based analysis. Based on these results, the developed analysis can play a supportive role in finding observation points in tennis coaching.
Previous studies (Cai et al., 2020; Ó Conaire et al., 2010; Shah et al., 2007; Zhu et al., 2006) have mostly focused on shot classification (service, forehand, or backhand stroke). This study was conducted from an original perspective that attempted to extract similar phases. Results show that the accuracy was as high as 79.2 % when comparing within the same conditions, in the 70.0 - 73.3 % range when comparing between different conditions, and 80.0 % when comparing between different players in the deuce side. In a recent study (Cai et al., 2020), it was reported that a machine learning-based analysis was able to recognize 12 tennis actions with an accuracy of 62 %. Accordingly, most of the conditions in this study showed equivalent or higher accuracy than the previous study. The developed AI-based analysis is expected to provide effective feedback in coaching by automatically extracting overlapping phases with a certain accuracy.
However, low accuracy was confirmed under some conditions. Overall, accuracy was low for the conditions targeting Player 2. The lowest accuracy was observed in the side comparison for Player 2 (46.7 %). As the correlation with correct detection in the left elbow and wrist were mostly negative coefficients, except for the comparison within the same condition for Player 2, weighting these body parts for total difference scores would be effective in improving accuracy. Actually, when doubled the weight of difference scores in the left elbow and wrist for total score, the accuracy was 53.3 %. The side comparison for Player 2 also showed low accuracy. It is possible that large differences in shot movements or positions between the sides affect the accuracy of extracting the overlapped phase. Adjusting or shortening the duration of clipping would be effective in improving accuracy. We attempted the solution for the side comparison for Player 2 by adjusting the clipping duration to 1 s with 0.5 s overlap. As the result, contrary to our expectations, the accuracy decreases to 6.6 %. Nevertheless, the adjustment of the clipping duration is also related to the phase length that the players and coaches want to focus on; thus, discretion of the user could be reflected. We assumed that the inclusion of extra movement after shots in the clips might have reduced accuracy, but this speculation was not applicable in this case. Another reason for the low accuracy could be that Player 2 wore a similarly colored shirt, and the estimator of the AI-based analysis was not able to estimate the human pose by changing the side and recording angle. Further investigation is required to determine the conditions that lead to low accuracy in the analysis.
This study attempts to extract the differences within the same shot movement using AI-based analysis. A previous study performed a comparison within the same shot in tennis (Cai et al., 2020), however, it reported that there was considerable confusion regarding the accuracy of the comparison. This study attempted to extract information on the differences in the movements of each body part that might be required for coaching. As a result, manually evaluated body parts that showed different movements by a certified coach corresponded to the top three different parts in the AI-based analysis for 8 of the 12 conditions. Although manual interpretation by users, such as players and coaches, automatically provides information on which body parts show different movements, it would help in effective and accurate motion analysis. However, some conditions showed contradictory results between manual and AI-based analyses. In this regard, the comparison within the same condition on the deuce side for Player 1 and the advantage side for Player 2 had the highest difference score for the right shoulder in the AI-based analysis. This may be because the internal and external rotation of the shoulder during service is fast, and the AI-based analysis would be judged as significantly different because of the video recording setting. In any case, no significant difference was observed between the clips because the services were performed consecutively in the comparison under the same conditions. It is expected that AI-based analysis would be useful for comparison in conditions that would have large differences, such as between early and late stages in a match or before and after a specific practice. Practical investigation of the application is needed. If rallies are targeted in the future, guards should be installed to protect cameras from ball hits. In addition, different scores in some clips with limited overlapping phases might have affected the movement included only in either clip. This may be an issue in practical applications. The proposed AI-based analysis should be treated as a supplemental tool to help coaching because selecting the reference video and interpreting the differences in movements are required to be conducted manually by the players or coaches. Even if this AI-based analysis underlines the differences in movement, the coach’s assistance is still necessary if the players themselves have difficulty interpreting these differences.
This study had several limitations. First, this study conducted video recordings under identical conditions (participants’ wear, camera location, type and setting, weather, court color, background, etc.). Therefore, further investigation is required to determine whether these conditions affect the analysis accuracy. Second, although variations in accuracy were observed in this study, this should be clarified by focusing on a variety of participants. Our aim was to compare services in various situations and investigate the perceptions of players and coaches to develop a useful AI-based analysis.
Concluding
CONCLUSIONS
This study attempted to develop an AI-based analysis that plays a supportive role in tennis coaching, and investigated its accuracy-targeting services. Consequently, a certain accuracy in detecting similarities and differences between movements was confirmed. Although there were some issues that needed to be solved, this AI analysis could effectively extract similar or overlapping phases and suggest body parts that might have different movements.
References