Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Facelandmarks 3D position in IOS Scenekit #5801

Open
atilagulers opened this issue Dec 31, 2024 · 0 comments
Open

Facelandmarks 3D position in IOS Scenekit #5801

atilagulers opened this issue Dec 31, 2024 · 0 comments
Assignees
Labels
platform:python MediaPipe Python issues task:face landmarker Issues related to Face Landmarker: Identify facial features for visual effects and avatars. type:support General questions

Comments

@atilagulers
Copy link

atilagulers commented Dec 31, 2024

Hello, i'm trying to convert facelandmarks to 3d points in scenekit.

Even with incorrect depth value face tracks correctly but it is not useful as occluder. because worldpoint is wrong. and face mesh always stays front of every sceneobject or back of every sceneobject.

I've tried to use faceNode.transform = matrix (from mediapipe) it works great but it does not track landmarks.

right now i update geometry vertices by using these landmarks and update geometry uv with uv.json data that mediapipe provided.

How can i get correct world position of landmarks in scenekit?

Thank you.

        if let normalizedLandmarkArray = firstFaceNormalizedLandmarks {
            
            
            let vertexPositionArray = normalizedLandmarkArray.enumerated().map { index, landmark -> simd_float3 in

                let screenX = CGFloat(landmark.x) * imageWidth
                let screenY = CGFloat(landmark.y) * imageHeight
                

                let screenPoint = SCNVector3(screenX, screenY, 0.5)
                
                let worldPoint = sceneView.unprojectPoint(screenPoint)



                return simd_float3(Float(worldPoint.x), Float(worldPoint.y), Float(worldPoint.z))


            }
            


6026270812497103466

@atilagulers atilagulers added the type:others issues not falling in bug, perfromance, support, build and install or feature label Dec 31, 2024
@kuaashish kuaashish added type:support General questions task:face landmarker Issues related to Face Landmarker: Identify facial features for visual effects and avatars. platform:python MediaPipe Python issues and removed type:others issues not falling in bug, perfromance, support, build and install or feature labels Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
platform:python MediaPipe Python issues task:face landmarker Issues related to Face Landmarker: Identify facial features for visual effects and avatars. type:support General questions
Projects
None yet
Development

No branches or pull requests

2 participants