This paper presents an innovative investigation on prototyping a digital twin(DT)as the platform for human-robot interactive welding and welder behavior analysis.This humanrobot interaction(HRI)working style helps to ...This paper presents an innovative investigation on prototyping a digital twin(DT)as the platform for human-robot interactive welding and welder behavior analysis.This humanrobot interaction(HRI)working style helps to enhance human users'operational productivity and comfort;while data-driven welder behavior analysis benefits to further novice welder training.This HRI system includes three modules:1)a human user who demonstrates the welding operations offsite with her/his operations recorded by the motion-tracked handles;2)a robot that executes the demonstrated welding operations to complete the physical welding tasks onsite;3)a DT system that is developed based on virtual reality(VR)as a digital replica of the physical human-robot interactive welding environment.The DT system bridges a human user and robot through a bi-directional information flow:a)transmitting demonstrated welding operations in VR to the robot in the physical environment;b)displaying the physical welding scenes to human users in VR.Compared to existing DT systems reported in the literatures,the developed one provides better capability in engaging human users in interacting with welding scenes,through an augmented VR.To verify the effectiveness,six welders,skilled with certain manual welding training and unskilled without any training,tested the system by completing the same welding job;three skilled welders produce satisfied welded workpieces,while the other three unskilled do not.A data-driven approach as a combination of fast Fourier transform(FFT),principal component analysis(PCA),and support vector machine(SVM)is developed to analyze their behaviors.Given an operation sequence,i.e.,motion speed sequence of the welding torch,frequency features are firstly extracted by FFT and then reduced in dimension through PCA,which are finally routed into SVM for classification.The trained model demonstrates a 94.44%classification accuracy in the testing dataset.The successful pattern recognition in skilled welder operations should benefit to accelerate n展开更多
The broad application of virtual reality (VR) to medicine has been of great value. The virtual surgery is one of technically difficult applications. With the expansion of the increasingly fine and complicated ear micr...The broad application of virtual reality (VR) to medicine has been of great value. The virtual surgery is one of technically difficult applications. With the expansion of the increasingly fine and complicated ear microsurgery, new methods are required to train the doctors. It is necessary and of practical significance to apply VR to the ear micro-operation, which is a functional operation with high precision and great difficulties. In this article, medical VR applications were reviewed. The application of VR to the ear microsurgery was discussed and the virtual ear microsurgery system was designed.展开更多
360°videos enable viewers to watch freely from different directions but inevitably prevent them from perceiving all the helpful information.To mitigate this problem,picture-in-picture(PIP)guidance was proposed us...360°videos enable viewers to watch freely from different directions but inevitably prevent them from perceiving all the helpful information.To mitigate this problem,picture-in-picture(PIP)guidance was proposed using preview windows to show regions of interest(ROIs)outside the current view range.We identify several drawbacks of this representation and propose a new method for 360° film watching called AdaPIP.AdaPIP enhances traditional PIP by adaptively arranging preview windows with changeable view ranges and sizes.In addition,AdaPIP incorporates the advantage of arrow-based guidance by presenting circular windows with arrows attached to them to help users locate the corresponding ROIs more efficiently.We also adapted AdaPIP and Outside-In to HMD-based immersive virtual reality environments to demonstrate the usability of PIP-guided approaches beyond 2D screens.Comprehensive user experiments on 2D screens,as well as in VR environments,indicate that AdaPIP is superior to alternative methods in terms of visual experiences while maintaining a comparable degree of immersion.展开更多
The application of helicopter emergency rescue is becoming increasingly widespread,but the flight crew training in this area is still difficult due to high cost and risk.Nevertheless,with the development of Virtual Re...The application of helicopter emergency rescue is becoming increasingly widespread,but the flight crew training in this area is still difficult due to high cost and risk.Nevertheless,with the development of Virtual Reality(VR)technology,virtual simulation has become a significant role in crew training of helicopter rescue.During the implementation of VR-based training,how to transform complex real tasks into VR scenarios and how to evaluate the performance of crew are of great importance.To address these issues,a novel VR-based R-E-A-D(Report,Evaluate,Agree,Do)evaluation model for training is proposed,which is suitable for complex missions with multiple tasks,multiple scenarios,and multiple people.Then,a mapping method of VR scenarios is put forward,which can transform the real tasks into virtual scenarios to serve the virtual simulation training.Finally,an experiment is carried out to verify the feasibility of the evaluation method and virtual scenario mapping method.展开更多
Recently,virtual reality(VR)technology has been widely used in medical,military,manufacturing,entertainment,and other fields.These applications must simulate different complex material surfaces,various dynamic objects...Recently,virtual reality(VR)technology has been widely used in medical,military,manufacturing,entertainment,and other fields.These applications must simulate different complex material surfaces,various dynamic objects,and complex physical phenomena,increasing the complexity of VR scenes.Current computing devices cannot efficiently render these complex scenes in real time,and delayed rendering makes the content observed by the user inconsistent with the user’s interaction,causing discomfort.Foveated rendering is a promising technique that can accelerate rendering.It takes advantage of human eyes’inherent features and renders different regions with different qualities without sacrificing perceived visual quality.Foveated rendering research has a history of 31 years and is mainly focused on solving the following three problems.The first is to apply perceptual models of the human visual system into foveated rendering.The second is to render the image with different qualities according to foveation principles.The third is to integrate foveated rendering into existing rendering paradigms to improve rendering performance.In this survey,we review foveated rendering research from 1990 to 2021.We first revisit the visual perceptual models related to foveated rendering.Subsequently,we propose a new foveated rendering taxonomy and then classify and review the research on this basis.Finally,we discuss potential opportunities and open questions in the foveated rendering field.We anticipate that this survey will provide new researchers with a high-level overview of the state-of-the-art in this field,furnish experts with up-to-date information,and offer ideas alongside a framework to VR display software and hardware designers and engineers.展开更多
In this paper we briefly describe our effort in developing standards for immersive display interfaces in virtual manufacturing.This work is being conducted in collaboration with the National Institute of Standards and...In this paper we briefly describe our effort in developing standards for immersive display interfaces in virtual manufacturing.This work is being conducted in collaboration with the National Institute of Standards and Technology(NIST).First we review the concept of virtual manufacturing,discuss its scope and lay the contextual foundation for describing our application.Next we describe a prototype implementation of immersive display interfaces in virtual manufacturing.This prototype is based on processing the output of a factory simulation model.Finally we describe the standards emerging from this prototype.展开更多
With increasing demands of virtual reality(VR) applications, efficient VR rendering techniques are becoming essential. Because VR stereo rendering has increased computational costs to separately render views for the l...With increasing demands of virtual reality(VR) applications, efficient VR rendering techniques are becoming essential. Because VR stereo rendering has increased computational costs to separately render views for the left and right eyes, to reduce the rendering cost in VR applications, we present a novel traversal order for tile-based mobile GPU architectures: Z^2 traversal order. In tile-based mobile GPU architectures,a tile traversal order that maximizes spatial locality can increase GPU cache efficiency. For VR applications, our approach improves upon the traditional Z order curve.We render corresponding screen tiles in left and right views in turn, or simultaneously, and as a result, we can exploit spatial adjacency of the two tiles. To evaluate our approach, we conducted a trace-driven hardware simulation using Mesa and a hardware simulator. Our experimental results show that Z^2 traversal order can reduce external memory bandwidth requirements and increase rendering performance.展开更多
This study explored the use of multi-physiological signals and simultaneously recorded high-density electroencephalography(EEG),electrocardiogram(ECG),and eye movements to better understand pilots’cognitive behaviour...This study explored the use of multi-physiological signals and simultaneously recorded high-density electroencephalography(EEG),electrocardiogram(ECG),and eye movements to better understand pilots’cognitive behaviour during flight simulator manoeuvres.Multimodal physiological signals were collected from 12 experienced pilots with international aviation qualifications under the wide-angle and impressive vision simulation.The data collection spanned two flight strike missions,each with three mission intensities,resulting in a data set of EEG,ECG,and eye movement signals from six subtasks.The multimodal data were analysed using signal processing methods.The results indicated that,when the flight missions were performed,the pilots’physiological characteristics exhibited rhythmic changes in the power spectrum ofθwaves in the EEG,r-MSSD in the ECG,and average gaze duration.Furthermore,the pilots’physiological signals were more sensitive during the target mission than during the empty target mission.The results also showed correlations between different physiological characteristics.We showed that specific multimodal features are useful for advancing neuroscience research into pilots’cognitive behaviour and processes related to brain activity,psychological rhythms,and eye movement.展开更多
An Avatar-like robot in a virtual museum environment was designed to perform the function of telepresence and teleoperation,and make the three-dimensional(3 D) effect through a binocular camera and a virtual reality(V...An Avatar-like robot in a virtual museum environment was designed to perform the function of telepresence and teleoperation,and make the three-dimensional(3 D) effect through a binocular camera and a virtual reality(VR) head-mounted display(HMD).This robot supports users to participate in the exhibition remotely in a new and interactive way in multiple scenarios.The results show that the system has good usability and is worth further optimizing.展开更多
This paper investigates user preferences and behaviour associated with 2D and 3D modes of urban representation within a novel Topographic Immersive Virtual Environment(TopoIVE)created from official 1:10,000 mapping.Si...This paper investigates user preferences and behaviour associated with 2D and 3D modes of urban representation within a novel Topographic Immersive Virtual Environment(TopoIVE)created from official 1:10,000 mapping.Sixty participants were divided into two groups:the first were given a navigational task within a simulated city and the second were given the freedom to explore it.A Head-Mounted Display(HMD)Virtual Reality(VR)app allowed participants to switch between 2D and 3D representations of buildings with a remote controller and their use of these modes during the experiment was recorded.Participants performed mental rotation tests before entering the TopoIVE and were interviewed afterwards about their experiences using the app.The results indicate that participants preferred the 3D mode of representation overall,although preference for the 2D mode was slightly higher amongst those undertaking the navigational task,and reveal that different wayfinding solutions were adopted by participants according to their gender.Overall,the findings suggest that users exploit different aspects of 2D and 3D modes of visualization in their wayfinding strategy,regardless of their task.The potential to combine the functionality of 2D and 3D modes therefore offers substantial opportunities for the development of immersive virtual reality products derived from topographic datasets.展开更多
The growing and evolving use of emerging technology including Artificial Intelligence(AI)and Virtual Reality(VR),has significantly impacted the lives of two specific groups-the elderly and the disabled.This paper inve...The growing and evolving use of emerging technology including Artificial Intelligence(AI)and Virtual Reality(VR),has significantly impacted the lives of two specific groups-the elderly and the disabled.This paper investigates potential reasons for this phenomenon.Clearly,AI and VR Technology alters the everyday lives of people with disabilities and how they navigate the world.Technological developments increasingly work to address the isolation that people with disabilities as well as the elderly experience for they are often unable or limited in how they engage with their communities.This research paper outlines the way technology has improved social communication,information distribution,and day-to-day living for those with disabilities and the elderly.Undoubtedly,the internet has transformed social communication and interaction for most people.Socially isolated individuals with disabilities have gained exposure to social environments through social media.Moreover,the broad range of information available on the internet has increased access to resources such as government services,health services,and social services support.On a related point,assistive devices have enabled disabled people including many seniors to overcome motor,sensory,or cognitive difficulties that may have previously hindered them from performing daily tasks.However,although AI and VR technology has been effectively integrated in the lives of those with disabilities,many such individuals lack access to commonplace technologies,like a personal computer.This paper examines how AI and VR technology has enhanced communication,information access,and everyday activities for the disabled and aging communities despite such socio-economic limitations.展开更多
Training in residency programs is highly competitive,it requires the formation of competent physicians that achieve the performance standards that were declared for their technical skills,attitudes and interpersonal a...Training in residency programs is highly competitive,it requires the formation of competent physicians that achieve the performance standards that were declared for their technical skills,attitudes and interpersonal abilities.The use of simulation and technology on the medical education has increased considerably.Particularly in ophthalmology the simulators used are:live models from animal or cadavers,mannequins,wet laboratories,simulated patients,part-task moles,laser or surgical models,and more recently,virtual reality(VR).VR places a person in a simulated environment that has a specific sense of self-location,where the participant interacts with the objects within the setting.Teaching with VR refers to the use of the available resources in technology and visualization of structures to improve the educational experience of medical students,residents and physicians in professional continuous development programs.Several authors highlight the benefits of assessing trainees with the tools,they argue that the key contribution of this model is in the formative assessment.Rather than evaluating and putting a score on student’s grades,VR provides a powerful experience for the acquisition of skills.A conclusion is the need to develop studies to document the effects that it has on knowledge,skills and behaviors,and to patient related outcomes.展开更多
文摘This paper presents an innovative investigation on prototyping a digital twin(DT)as the platform for human-robot interactive welding and welder behavior analysis.This humanrobot interaction(HRI)working style helps to enhance human users'operational productivity and comfort;while data-driven welder behavior analysis benefits to further novice welder training.This HRI system includes three modules:1)a human user who demonstrates the welding operations offsite with her/his operations recorded by the motion-tracked handles;2)a robot that executes the demonstrated welding operations to complete the physical welding tasks onsite;3)a DT system that is developed based on virtual reality(VR)as a digital replica of the physical human-robot interactive welding environment.The DT system bridges a human user and robot through a bi-directional information flow:a)transmitting demonstrated welding operations in VR to the robot in the physical environment;b)displaying the physical welding scenes to human users in VR.Compared to existing DT systems reported in the literatures,the developed one provides better capability in engaging human users in interacting with welding scenes,through an augmented VR.To verify the effectiveness,six welders,skilled with certain manual welding training and unskilled without any training,tested the system by completing the same welding job;three skilled welders produce satisfied welded workpieces,while the other three unskilled do not.A data-driven approach as a combination of fast Fourier transform(FFT),principal component analysis(PCA),and support vector machine(SVM)is developed to analyze their behaviors.Given an operation sequence,i.e.,motion speed sequence of the welding torch,frequency features are firstly extracted by FFT and then reduced in dimension through PCA,which are finally routed into SVM for classification.The trained model demonstrates a 94.44%classification accuracy in the testing dataset.The successful pattern recognition in skilled welder operations should benefit to accelerate n
基金The Scientific Research Fund of ShanghaiScience Committee( 0 3 411980 8) and the Science and TechnologySpecial Fund of Pudong New District( PKJ2 0 0 3 -4 1)
文摘The broad application of virtual reality (VR) to medicine has been of great value. The virtual surgery is one of technically difficult applications. With the expansion of the increasingly fine and complicated ear microsurgery, new methods are required to train the doctors. It is necessary and of practical significance to apply VR to the ear micro-operation, which is a functional operation with high precision and great difficulties. In this article, medical VR applications were reviewed. The application of VR to the ear microsurgery was discussed and the virtual ear microsurgery system was designed.
基金supported by the National Natural Science Foundation of China(Project Number 62132012)the Beijing Science and Technology Program(Project Number Z221100007722001)the Tsinghua-Tencent Joint Laboratory for Internet Innovation Technology.
文摘360°videos enable viewers to watch freely from different directions but inevitably prevent them from perceiving all the helpful information.To mitigate this problem,picture-in-picture(PIP)guidance was proposed using preview windows to show regions of interest(ROIs)outside the current view range.We identify several drawbacks of this representation and propose a new method for 360° film watching called AdaPIP.AdaPIP enhances traditional PIP by adaptively arranging preview windows with changeable view ranges and sizes.In addition,AdaPIP incorporates the advantage of arrow-based guidance by presenting circular windows with arrows attached to them to help users locate the corresponding ROIs more efficiently.We also adapted AdaPIP and Outside-In to HMD-based immersive virtual reality environments to demonstrate the usability of PIP-guided approaches beyond 2D screens.Comprehensive user experiments on 2D screens,as well as in VR environments,indicate that AdaPIP is superior to alternative methods in terms of visual experiences while maintaining a comparable degree of immersion.
文摘The application of helicopter emergency rescue is becoming increasingly widespread,but the flight crew training in this area is still difficult due to high cost and risk.Nevertheless,with the development of Virtual Reality(VR)technology,virtual simulation has become a significant role in crew training of helicopter rescue.During the implementation of VR-based training,how to transform complex real tasks into VR scenarios and how to evaluate the performance of crew are of great importance.To address these issues,a novel VR-based R-E-A-D(Report,Evaluate,Agree,Do)evaluation model for training is proposed,which is suitable for complex missions with multiple tasks,multiple scenarios,and multiple people.Then,a mapping method of VR scenarios is put forward,which can transform the real tasks into virtual scenarios to serve the virtual simulation training.Finally,an experiment is carried out to verify the feasibility of the evaluation method and virtual scenario mapping method.
基金supported by National Key R&D Program of China(2019YFC1521102)the National Natural Science Foundation of China(61932003)Beijing Science and Technology Plan(Z221100007722004).
文摘Recently,virtual reality(VR)technology has been widely used in medical,military,manufacturing,entertainment,and other fields.These applications must simulate different complex material surfaces,various dynamic objects,and complex physical phenomena,increasing the complexity of VR scenes.Current computing devices cannot efficiently render these complex scenes in real time,and delayed rendering makes the content observed by the user inconsistent with the user’s interaction,causing discomfort.Foveated rendering is a promising technique that can accelerate rendering.It takes advantage of human eyes’inherent features and renders different regions with different qualities without sacrificing perceived visual quality.Foveated rendering research has a history of 31 years and is mainly focused on solving the following three problems.The first is to apply perceptual models of the human visual system into foveated rendering.The second is to render the image with different qualities according to foveation principles.The third is to integrate foveated rendering into existing rendering paradigms to improve rendering performance.In this survey,we review foveated rendering research from 1990 to 2021.We first revisit the visual perceptual models related to foveated rendering.Subsequently,we propose a new foveated rendering taxonomy and then classify and review the research on this basis.Finally,we discuss potential opportunities and open questions in the foveated rendering field.We anticipate that this survey will provide new researchers with a high-level overview of the state-of-the-art in this field,furnish experts with up-to-date information,and offer ideas alongside a framework to VR display software and hardware designers and engineers.
文摘In this paper we briefly describe our effort in developing standards for immersive display interfaces in virtual manufacturing.This work is being conducted in collaboration with the National Institute of Standards and Technology(NIST).First we review the concept of virtual manufacturing,discuss its scope and lay the contextual foundation for describing our application.Next we describe a prototype implementation of immersive display interfaces in virtual manufacturing.This prototype is based on processing the output of a factory simulation model.Finally we describe the standards emerging from this prototype.
文摘With increasing demands of virtual reality(VR) applications, efficient VR rendering techniques are becoming essential. Because VR stereo rendering has increased computational costs to separately render views for the left and right eyes, to reduce the rendering cost in VR applications, we present a novel traversal order for tile-based mobile GPU architectures: Z^2 traversal order. In tile-based mobile GPU architectures,a tile traversal order that maximizes spatial locality can increase GPU cache efficiency. For VR applications, our approach improves upon the traditional Z order curve.We render corresponding screen tiles in left and right views in turn, or simultaneously, and as a result, we can exploit spatial adjacency of the two tiles. To evaluate our approach, we conducted a trace-driven hardware simulation using Mesa and a hardware simulator. Our experimental results show that Z^2 traversal order can reduce external memory bandwidth requirements and increase rendering performance.
文摘This study explored the use of multi-physiological signals and simultaneously recorded high-density electroencephalography(EEG),electrocardiogram(ECG),and eye movements to better understand pilots’cognitive behaviour during flight simulator manoeuvres.Multimodal physiological signals were collected from 12 experienced pilots with international aviation qualifications under the wide-angle and impressive vision simulation.The data collection spanned two flight strike missions,each with three mission intensities,resulting in a data set of EEG,ECG,and eye movement signals from six subtasks.The multimodal data were analysed using signal processing methods.The results indicated that,when the flight missions were performed,the pilots’physiological characteristics exhibited rhythmic changes in the power spectrum ofθwaves in the EEG,r-MSSD in the ECG,and average gaze duration.Furthermore,the pilots’physiological signals were more sensitive during the target mission than during the empty target mission.The results also showed correlations between different physiological characteristics.We showed that specific multimodal features are useful for advancing neuroscience research into pilots’cognitive behaviour and processes related to brain activity,psychological rhythms,and eye movement.
文摘An Avatar-like robot in a virtual museum environment was designed to perform the function of telepresence and teleoperation,and make the three-dimensional(3 D) effect through a binocular camera and a virtual reality(VR) head-mounted display(HMD).This robot supports users to participate in the exhibition remotely in a new and interactive way in multiple scenarios.The results show that the system has good usability and is worth further optimizing.
文摘This paper investigates user preferences and behaviour associated with 2D and 3D modes of urban representation within a novel Topographic Immersive Virtual Environment(TopoIVE)created from official 1:10,000 mapping.Sixty participants were divided into two groups:the first were given a navigational task within a simulated city and the second were given the freedom to explore it.A Head-Mounted Display(HMD)Virtual Reality(VR)app allowed participants to switch between 2D and 3D representations of buildings with a remote controller and their use of these modes during the experiment was recorded.Participants performed mental rotation tests before entering the TopoIVE and were interviewed afterwards about their experiences using the app.The results indicate that participants preferred the 3D mode of representation overall,although preference for the 2D mode was slightly higher amongst those undertaking the navigational task,and reveal that different wayfinding solutions were adopted by participants according to their gender.Overall,the findings suggest that users exploit different aspects of 2D and 3D modes of visualization in their wayfinding strategy,regardless of their task.The potential to combine the functionality of 2D and 3D modes therefore offers substantial opportunities for the development of immersive virtual reality products derived from topographic datasets.
文摘The growing and evolving use of emerging technology including Artificial Intelligence(AI)and Virtual Reality(VR),has significantly impacted the lives of two specific groups-the elderly and the disabled.This paper investigates potential reasons for this phenomenon.Clearly,AI and VR Technology alters the everyday lives of people with disabilities and how they navigate the world.Technological developments increasingly work to address the isolation that people with disabilities as well as the elderly experience for they are often unable or limited in how they engage with their communities.This research paper outlines the way technology has improved social communication,information distribution,and day-to-day living for those with disabilities and the elderly.Undoubtedly,the internet has transformed social communication and interaction for most people.Socially isolated individuals with disabilities have gained exposure to social environments through social media.Moreover,the broad range of information available on the internet has increased access to resources such as government services,health services,and social services support.On a related point,assistive devices have enabled disabled people including many seniors to overcome motor,sensory,or cognitive difficulties that may have previously hindered them from performing daily tasks.However,although AI and VR technology has been effectively integrated in the lives of those with disabilities,many such individuals lack access to commonplace technologies,like a personal computer.This paper examines how AI and VR technology has enhanced communication,information access,and everyday activities for the disabled and aging communities despite such socio-economic limitations.
文摘Training in residency programs is highly competitive,it requires the formation of competent physicians that achieve the performance standards that were declared for their technical skills,attitudes and interpersonal abilities.The use of simulation and technology on the medical education has increased considerably.Particularly in ophthalmology the simulators used are:live models from animal or cadavers,mannequins,wet laboratories,simulated patients,part-task moles,laser or surgical models,and more recently,virtual reality(VR).VR places a person in a simulated environment that has a specific sense of self-location,where the participant interacts with the objects within the setting.Teaching with VR refers to the use of the available resources in technology and visualization of structures to improve the educational experience of medical students,residents and physicians in professional continuous development programs.Several authors highlight the benefits of assessing trainees with the tools,they argue that the key contribution of this model is in the formative assessment.Rather than evaluating and putting a score on student’s grades,VR provides a powerful experience for the acquisition of skills.A conclusion is the need to develop studies to document the effects that it has on knowledge,skills and behaviors,and to patient related outcomes.
