A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Volume 8 Issue 5
May  2021

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 6.171, Top 11% (SCI Q1)
    CiteScore: 11.2, Top 5% (Q1)
    Google Scholar h5-index: 51, TOP 8
Turn off MathJax
Article Contents
Shuangyi Wang, James Housden, Tianxiang Bai, Hongbin Liu, Junghwan Back, Davinder Singh, Kawal Rhode, Zeng-Guang Hou and Fei-Yue Wang, "Robotic Intra-Operative Ultrasound: Virtual Environments and Parallel Systems," IEEE/CAA J. Autom. Sinica, vol. 8, no. 5, pp. 1095-1106, May 2021. doi: 10.1109/JAS.2021.1003985
Citation: Shuangyi Wang, James Housden, Tianxiang Bai, Hongbin Liu, Junghwan Back, Davinder Singh, Kawal Rhode, Zeng-Guang Hou and Fei-Yue Wang, "Robotic Intra-Operative Ultrasound: Virtual Environments and Parallel Systems," IEEE/CAA J. Autom. Sinica, vol. 8, no. 5, pp. 1095-1106, May 2021. doi: 10.1109/JAS.2021.1003985

Robotic Intra-Operative Ultrasound: Virtual Environments and Parallel Systems

doi: 10.1109/JAS.2021.1003985
Funds:  This work was supported in part by the Key Research and Development Program 2020 of Guangzhou (202007050002), in part by the National Natural Science Foundation of China (62003339, U1811463), and in part by the Intel Collaborative Research Institute for Intelligent and Automated Connected Vehicles (“ICRI-IACV”)
More Information
  • Robotic intra-operative ultrasound has the potential to improve the conventional practice of diagnosis and procedure guidance that are currently performed manually. Working towards automatic or semi-automatic ultrasound, being able to define ultrasound views and the corresponding probe poses via intelligent approaches become crucial. Based on the concept of parallel system which incorporates the ingredients of artificial systems, computational experiments, and parallel execution, this paper utilized a recent developed robotic trans-esophageal ultrasound system as the study object to explore the method for developing the corresponding virtual environments and present the potential applications of such systems. The proposed virtual system includes the use of 3D slicer as the main workspace and graphic user interface (GUI), Matlab engine to provide robotic control algorithms and customized functions, and PLUS (Public software Library for UltraSound imaging research) toolkit to generate simulated ultrasound images. Detailed implementation methods were presented and the proposed features of the system were explained. Based on this virtual system, example uses and case studies were presented to demonstrate its capabilities when used together with the physical TEE robot. This includes standard view definition and customized view optimization for pre-planning and navigation, as well as robotic control algorithm evaluations to facilitate real-time automatic probe pose adjustments. To conclude, the proposed virtual system would be a powerful tool to facilitate the further developments and clinical uses of the robotic intra-operative ultrasound systems.


  • loading
  • [1]
    N. Magnavita, L. Bevilacqua, P. Mirk, A. Fileni, and N. Castellino, “Work-related musculoskeletal complaints in sonologists,” J. Occup. Environ. Med., vol. 41, no. 11, pp. 981–988, 1999. doi: 10.1097/00043764-199911000-00010
    L. LaGrone, V. Sadasivam, A. Kushner, and R. Groen, “A review of training opportunities for ultrasonography in low and middle income countries,” Tropical Med. Int. Health, vol. 17, no. 7, pp. 808–819, 2012. doi: 10.1111/j.1365-3156.2012.03014.x
    C. Kim, F. Schäfer, D. Chang, D. Petrisor, M. Han, and D. Stoianovici, “Robot for ultrasound-guided prostate imaging and intervention,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2011, pp. 943–948.
    M. Han, C. Kim, P. Mozer, F. Schäfer, S. Badaan, B. S, Vigaru, et al., “Tandem-robot Assisted Laparoscopic Radical Prostatectomy to Improve the Neurovascular Bundle Visualization: A Feasibility Study,” Urology, vol. 77, no. 2, pp. 502–506, 2011.
    T. Adebar, S. Salcudean, S. Mahdavi, M. Moradi, C. Nguan, and L. Goldenberg, “A robotic system for intra-operative trans-rectal ultrasound and ultrasound elastography in radical prostatectomy,” in Proc. Int. Conf. Inf. Process. Comput.-Assisted Interventions, 2011, pp. 79–89.
    T. Adebar, O. Mohareri, and S. Salcudean, “Instrument-based calibration and remote control of intraoperative ultrasound for robot-assisted surgery,” in Proc. Int. Conf. Biomed. Robot. Biomechatronics, 2012, pp. 38–43.
    C. M. Schneider, A. M. Okamura, and G. Fichtinger, “A robotic system for transrectal needle insertion into the prostate with integrated ultrasound,” in Proc. Int. Conf. Robot. Autom., 2004, pp. 365–370.
    T.Podder, I. Buzurovic, K. Yan, Y. Hu, R. Valicenti, A. Dicker, et al., “Robotic system for image-guided prostate seed implant,” Brachytherapy, vol. 7, no. 2, pp. 100–101, 2008.
    P. M. Loschak, L. J. Brattain, and R. D. Howe, “Automated pointing of cardiac imaging catheters,” in Proc. Int. Conf. Robot. Autom., 2013, pp. 5794–5799.
    P. Loschak, A. Degirmenci, Y. Tenzer, C. Tschabrunn, E. Anter, and R. Howe, “A four degree of freedom robot for positioning ultrasound imaging catheters,” J. Mech. Robot, vol. 8, no. 5, 2016.
    S. Wang, J. Housden, D. Singh, K. Althoefer, and K. Rhode, “Design, testing and modelling of a novel robotic system for trans‐oesophageal ultrasound,” Int. J. Medical Robot. Comput. Assist. Surg., vol. 12, no. 3, pp. 342–354, 2016. doi: 10.1002/rcs.1691
    S. Wang, D. Singh, D. Johnson, K. Althoefer, K. Rhode, and R. J. Housden, “Robotic ultrasound: View planning, tracking, and automatic acquisition of transesophageal echocardiography,” IEEE Robot. Autom. Mag., vol. 23, no. 4, pp. 118–127, 2016. doi: 10.1109/MRA.2016.2580478
    J. A. Goldstein, S. Balter, M. Cowley, J. Hodgson, and L. W. Klein, “Occupational hazards of interventional cardiologists: Prevalence of orthopedic health problems in contemporary practice,” Catheter. Cardiovasc. Intervent., vol. 63, pp. 407–411, 2004. doi: 10.1002/ccd.20201
    F.-Y. Wang, “Shadow systems: A new concept for nested and embedded co-simulation for intelligent systems”, University of Arizona, Tucson, USA, 1994.
    F.-Y. Wang, “On the modeling, analysis, control and management of complex systems,” Complex Syst. Complex. Sci., vol. 3, no. 2, pp. 26–34, 2006.
    F.-Y. Wang, “Parallel system methods for management and control of complex systems,” Control and Decision, vol. 19, no. 5, pp. 485–489, 514, 2004.
    F.-Y. Wang, N. N. Zheng, D. P. Cao, C. M. Martinez, L. Li, and T. Liu, “Parallel driving in CPSS: A unified approach for transport automation and vehicle intelligence,” IEEE/CAA J. Autom. Sinica, vol. 4, no. 4, pp. 577–587, 2017. doi: 10.1109/JAS.2017.7510598
    K. F. Wang, C. Gou, N. N. Zheng, J. M. Rehg, and F.-Y. Wang, “Parallel vision for perception and understanding of complex scenes: Methods, framework, and perspectives,” Artif. Intell. Rev., vol. 48, no. 3, pp. 299–329, 2017. doi: 10.1007/s10462-017-9569-z
    J. S. Shanewise, A. T. Cheung, S. Aronson, W. J. Stewart, R. L. Weiss, J. B. Mark, et al., “ASE/SCA guidelines for performing a comprehensive intraoperative multiplane transesophageal echocardiography examination: recommendations of the american society of echocardiography council for intraoperative echocardiography and the society of cardiovascular anesthesiologists task force for certification in perioperative transesophageal echocardiography,” Anesth. & Analg., vol. 89, pp. 870–84, 1999.
    A. Lasso, T. Heffter, A. Rankin, C. Pinter, T. Ungi, and G. Fichtinger, “PLUS: Open-source toolkit for ultrasound-guided intervention systems,” IEEE. Trans. Biomed. Eng., vol. 61, no. 10, pp. 2527–2537, 2014. doi: 10.1109/TBME.2014.2322864
    J. Peters, O. Ecabert, C. Meyer, H. Schramm, R. Kneser, A. Groth, and J.Weese, “Automatic whole heart segmentation in static magnetic resonanceimage volumes,” in Proc. Int. Conf. Med. Image Comput. Comput. Assist. Intervention, Lecture Notes Comput. Sci., 2007, pp. 402–410.
    L. Bartha, A. Lasso, C. Pinter, T. Ungi, Z. Keri, and G. Fichtinger, “Open-source surface mesh-based ultrasound-guided spinal intervention simulator,” Int. J. Comput. Assist. Radiol. Surg., vol. 8, no. 6, pp. 1043–1051, 2013. doi: 10.1007/s11548-013-0901-z
    J. Tokuda, G. S. Fischer, X. Papademetris, Z. Yaniv, L. Ibanez, P. Cheng, et al., “OpenIGTLink: an open network protocol for image-guided therapy environment,” Int. J. Medical Robot. Comput. Assist. Surg., vol. 5, no. 4, pp. 423–434, 2009.
    S. Wang, “Development of a robotic trans-oesophageal ultrasound system and its application in automatic acquisition,” Ph.D. dissertation, School of Biomedical Engineering & Imaging Sciences, King’s College London., London, UK, 2017.
    E. Catmull and R. Rom, “A class of local interpolating splines,” Comput. Aided Geom. Des., pp. 317–326, 1974.
    V. Malik, M. Hote, and A. Jha, “Minimally invasive cardiac surgery and transesophageal echocardiography,” Ann. Card. Anaesth., vol. 17, no. 2, Article No. 125, 2014. doi: 10.4103/0971-9784.129844
    M. Prabhu and A. George, “Transesophageal monitoring in anaesthesia: An update,” Curr. Anesthesiol. Rep., vol. 4, no. 3, pp. 261–273, 2014. doi: 10.1007/s40140-014-0071-8
    K. Deb, “An efficient constraint handling method for genetic algorithms,” Comput. Methods Appl. Mech. Eng., vol. 186, no. 2-4, pp. 311–338, 2000. doi: 10.1016/S0045-7825(99)00389-8
    K. Deep, K. Singh, M. Kansal, and C. Mohan, “A real coded genetic algorithm for solving integer and mixed integer optimization problems,” Appl. Math. Comput., vol. 212, no. 2, pp. 505–518, 2009.
    S. Wang, J. Housden, A. Zar, R. Gandecha, D. Singh, and K. Rhode, “Strategy for monitoring cardiac interventions with an intelligent robotic ultrasound device,” Micromachines, vol. 9, no. 2, Article No. 65, 2018. doi: 10.3390/mi9020065


    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索


    Article Metrics

    Article views (1459) PDF downloads(41) Cited by()


    • A virtual environment to represent the use of the robotic trans-esophageal ultrasound is presented.
    • Example uses are introduced to demonstrate its applications in surgical planning.
    • Concept of parallel systems is utilized for a specific medical robotic application.
    • The proposed virtual environment would assist the robotic-based automatic ultrasound.


    DownLoad:  Full-Size Img  PowerPoint