We present a novel approach for automatically detecting and tracking facial landmarks acrossposesandexpressionsfromin-the-wild monocular video data,e.g.,You Tube videos and smartphone recordings.Our method does not re...We present a novel approach for automatically detecting and tracking facial landmarks acrossposesandexpressionsfromin-the-wild monocular video data,e.g.,You Tube videos and smartphone recordings.Our method does not require any calibration or manual adjustment for new individual input videos or actors.Firstly,we propose a method of robust 2D facial landmark detection across poses,by combining shape-face canonical-correlation analysis with a global supervised descent method.Since 2D regression-based methods are sensitive to unstable initialization,and the temporal and spatial coherence of videos is ignored,we utilize a coarse-todense 3D facial expression reconstruction method to refine the 2D landmarks.On one side,we employ an in-the-wild method to extract the coarse reconstruction result and its corresponding texture using the detected sparse facial landmarks,followed by robust pose,expression,and identity estimation.On the other side,to obtain dense reconstruction results,we give a face tracking flow method that corrects coarse reconstruction results and tracks weakly textured areas;this is used to iteratively update the coarse face model.Finally,a dense reconstruction result is estimated after it converges.Extensive experiments on a variety of video sequences recorded by ourselves or downloaded from You Tube show the results of facial landmark detection and tracking under various lighting conditions,for various head poses and facial expressions.The overall performance and a comparison with state-of-art methods demonstrate the robustness and effectiveness of our method.展开更多
A virtual cosmetics try-on system provides a realistic try-on experience for consumers and helps them efficiently choose suitable cosmetics.In this article,we propose a real-time augmented reality virtual cosmetics tr...A virtual cosmetics try-on system provides a realistic try-on experience for consumers and helps them efficiently choose suitable cosmetics.In this article,we propose a real-time augmented reality virtual cosmetics try-on system for smartphones(ARCosmetics),taking speed,accuracy,and stability into consideration at each step to ensure a better user experience.A novel and very fast face tracking method utilizes the face detection box and the average position of facial landmarks to estimate the faces in continuous frames.A dynamic weight Wing loss is introduced to assign a dynamic weight to every landmark by the estimated error during training.It balances the attention between small,medium,and large range error and thus increases the accuracy and robustness.We also designed a weighted average method to utilize the information of the adjacent frame for landmark refinement,guaranteeing the stability of the generated landmarks.Extensive experiments conducted on a large 106-point facial landmark dataset and the 300-VW dataset demonstrate the superior performance of the proposed method compared to other state-of-the-art methods.We also conducted user satisfaction studies further to verify the efficiency and effectiveness of our ARCosmetics system.展开更多
基金supported by the Harbin Institute of Technology Scholarship Fund 2016the National Centre for Computer Animation,Bournemouth University
文摘We present a novel approach for automatically detecting and tracking facial landmarks acrossposesandexpressionsfromin-the-wild monocular video data,e.g.,You Tube videos and smartphone recordings.Our method does not require any calibration or manual adjustment for new individual input videos or actors.Firstly,we propose a method of robust 2D facial landmark detection across poses,by combining shape-face canonical-correlation analysis with a global supervised descent method.Since 2D regression-based methods are sensitive to unstable initialization,and the temporal and spatial coherence of videos is ignored,we utilize a coarse-todense 3D facial expression reconstruction method to refine the 2D landmarks.On one side,we employ an in-the-wild method to extract the coarse reconstruction result and its corresponding texture using the detected sparse facial landmarks,followed by robust pose,expression,and identity estimation.On the other side,to obtain dense reconstruction results,we give a face tracking flow method that corrects coarse reconstruction results and tracks weakly textured areas;this is used to iteratively update the coarse face model.Finally,a dense reconstruction result is estimated after it converges.Extensive experiments on a variety of video sequences recorded by ourselves or downloaded from You Tube show the results of facial landmark detection and tracking under various lighting conditions,for various head poses and facial expressions.The overall performance and a comparison with state-of-art methods demonstrate the robustness and effectiveness of our method.
基金supported in part by the National Key R&D Program of China(2021ZD0140407)in part by the National Natural Science Foundation of China(Grant No.U21A20523).
文摘A virtual cosmetics try-on system provides a realistic try-on experience for consumers and helps them efficiently choose suitable cosmetics.In this article,we propose a real-time augmented reality virtual cosmetics try-on system for smartphones(ARCosmetics),taking speed,accuracy,and stability into consideration at each step to ensure a better user experience.A novel and very fast face tracking method utilizes the face detection box and the average position of facial landmarks to estimate the faces in continuous frames.A dynamic weight Wing loss is introduced to assign a dynamic weight to every landmark by the estimated error during training.It balances the attention between small,medium,and large range error and thus increases the accuracy and robustness.We also designed a weighted average method to utilize the information of the adjacent frame for landmark refinement,guaranteeing the stability of the generated landmarks.Extensive experiments conducted on a large 106-point facial landmark dataset and the 300-VW dataset demonstrate the superior performance of the proposed method compared to other state-of-the-art methods.We also conducted user satisfaction studies further to verify the efficiency and effectiveness of our ARCosmetics system.
