Feature: Add a camera and microphone so TARS can see you and hear you.

[rkeshwani]

GPT models are now multi-modal so would be nice if the cad file had a spot for a camera that could be connected. Same goes for the microphone.

[poboisvert]

You can find on Youtube tutorial to extrude the"spots" required to add microphone and camera

[JFerguson576]

Hi guys, Has anyone got working code for TARs CHATGPT integration (with voice and mobility control) that they are willing to share please? Thank you. John John Ferguson

…

________________________________ From: Pierre-Olivier ***@***.***> Sent: Monday, August 12, 2024 2:33:59 AM To: poboisvert/GPTARS_Interstellar ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [poboisvert/GPTARS_Interstellar] Feature: Add a camera and microphone so TARS can see you and hear you. (Issue #2) You can find on Youtube tutorial to extrude the"spots" required to add microphone and camera — Reply to this email directly, view it on GitHub<#2 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACE2SAAOTWGB7PYQCT5SNK3ZQ5Y5PAVCNFSM6AAAAABMKISFOKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBSG44DCNJVHE>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[rkeshwani]

You could use function calling from large language models to call python functions to connect to movement and other functionality. For voice I suggest taking a look at https://github.com/coqui-ai/TTS.

[rkeshwani]

@poboisvert I'll take a look, not much recent experience with CAD. Do you have any free software suggestions? I have FreeCAD installed but I find the navigation a little clunky. Also, I was unable to download the file linked here but I was able to access the original. It seems at first glance that the hands are partially completed but might be due to my own ignorance around how the CAD software works. I see it is completed piecemeal. Example, no idea where smaller servos go. The arms appear to be partially completed but not linked to any servo. I am wanting to use aluminum for the outside and plastic for the inside but trying to figure out the best way to sort out the internal components.

[SAMSAMPOP]

I too am struggling with the code for TARs CHATGPT integration. Am currently working through the python scripts. I have the internals assembled and just calling the tars_runner.py file. I did get the servos working but it's stopped for some reason. Anyway, if anyone has success with the TARS voice and is happy to share I would be super grateful. Thank you

[rkeshwani]

I've got a working prototype on the voice to text to AI on this except for the TARS voice. I found this library that could be used but unsure of copywrite rules about voice clips. https://docs.cartesia.ai/getting-started/using-the-api

For the microphone. I'm using what I have for now but here is a potential device:
https://www.amazon.com/DEWIN-Microphone-Portable-Household-Recording/dp/B086DRRP79/ref=sr_1_4?crid=2MWJ0DR7IZCN3&dib=eyJ2IjoiMSJ9.mMEXdxDyLwei6orkRikf2i9utuskE-QfhPpD5qbiqOg8TilnPwnQWio-JE7UqNmZ4KMpNg4CTbgnR_sOPbYEW0rpVCSI4gf2ROEi_2Lnisc32GCPYuCJCNRI8uYeHA2rDAiqEJzS2wvM81L5FafZ0ok0pGnLtmjW-Rkdi4_BQUleUct-kFcJjY81I7aIJk2dVvDKsyJHUbwChVeKltMqGHL2gSJ-UXe00ycY4L2d_kg.MLqbxDm9ERU84-7O-lVsnN73dl8xhkSy1qp_1YpaiY4&dib_tag=se&keywords=usb+microphone+for+raspberry+pi&qid=1725731111&sprefix=usb+microphone+for+%2Caps%2C135&sr=8-4

Use pyaudio to send voice to https://console.groq.com/docs/speech-text
If you have a powerful enough board, you could run the tiny-whisper model locally on device.
Then send that to your favorite LLM. Then send that text to cartesia.

The https://github.com/coqui-ai/TTS. I mentioned above is too heavy of a library and won't run on my sbc board but it could run locally if you have a nvidia jetson nano or a coral tpu.

Once I have something more refined and a camera working I will create a pull request.

[pyrater]

Has anyone been able to confirm the parts? the Step file shows https://www.amazon.com/gp/product/B083ZMZZCB/ref=ox_sc_act_title_1?smid=A333XEUDSX4WY3&psc=1 but the parts list here shows https://www.amazon.com/gp/product/B073F4TRSK/ref=ox_sc_act_title_12?smid=A1K1UK7O5KP6WQ&psc=1