Fix/ twitter Image Format Validation (#1694)

[AIFlowML]

PR: Fix Image Format Validation (#1694)

Changes Made

Client Twitter Package

1. Enhanced Base Client (base.ts):

   • Added initialization of ImageDescriptionService
   • Improved error handling for service retrieval
   • Added type safety for service interactions

2. Post Client (post.ts):

   • Added image format validation before processing
   • Integrated image description generation
   • Enhanced error handling for image processing
   • Added logging for better debugging

3. Tests:

   • Added comprehensive test suite (twitter-client-image.test.ts)
   • Test cases cover:
     • Single image processing
     • Multiple images in one tweet
     • Error handling for invalid images
     • Network error handling
     • Long tweet handling with images
   • Added real test images (PNG, JPEG, GIF, BMP)

Plugin Node Package

1. Image Service (image.ts):

   • Added format validation before OpenAI API calls
   • Improved error handling and logging
   • Added support for multiple image formats
   • Enhanced type safety

2. Tests:

   • Added real image test suite (image.service.real.test.ts)
   • Test cases cover:
     • PNG image processing
     • JPEG image processing
     • GIF image processing
     • BMP format processing
     • Network image handling
     • Error cases with malformed data
   • Added test fixtures for all supported formats

Testing

• All tests pass successfully
• Tested with real images of various formats (PNG, JPEG, GIF, BMP)
• Verified error handling with invalid formats
• Confirmed logging works as expected
• Tested network resilience

Notes

• No breaking changes introduced
• All changes are backward compatible
• Added comprehensive logging for better debugging
• Test images are included in the repository for consistent testing

[odilitime]

not sure I like two copies of the same images, wonder if we should make a global directory for those

[AIFlowML]

not sure I like two copies of the same images, wonder if we should make a global directory for those

I kept them separated because on 2 different packages.
We can think a common test strategy for the whole set of plugs/client once there is less backlog

[AIFlowML]

You want me to remove completely the tests here too ?
I can do it no issues

[AIFlowML]

@odilitime i fixed the conflics in the image.ts

[rchak007]

FYI -
import { dominosPlugin } from "@elizaos/plugin-dominos";
is missing in index.ts

i replied to the AI Agent tweet with another user to give a picture and it failed now - A;lthought the text reply came. I see DallE was invoked to get image too.

Received response from OpenAI model.
◎ LOGS
Image prompt received:
Main subject: A futuristic temple with a serene monk meditating. Environment: Lush, vibrant garden. Lighting: Ethereal glow from above. Colors: Vivid blues, greens, and golds. Mood: Tranquil and sacred. Composition: Central focus on monk, surrounded by nature. Style: Futuristic with vibrant colors.

◎ LOGS
Image settings:
{}

◎ LOGS
Generating image with prompt:
Main subject: A futuristic temple with a serene monk meditating. Environment: Lush, vibrant garden. Lighting: Ethereal glow from above. Colors: Vivid blues, greens, and golds. Mood: Tranquil and sacred. Composition: Central focus on monk, surrounded by nature. Style: Futuristic with vibrant colors.

ℹ INFORMATIONS
Generating image with options:
{"imageModelProvider":"dall-e-3"}

["◎ Generating new tweet"]

["◎ Matched fragment: promotes kriya yoga as a spiritual awakening tool thats better than your morning espressoinstant energy no caffeine jitters. with similarity: 0.6251623630523682"]

["◎ Matched fragment: represents the ideals of self-realization fellowship and the yoga sutras of patanjalithink ancient wisdom with a cosmic splash of sass. with similarity: 0.6848316192626953"]

["◎ Matched fragment: encourages the balance of meditation and actionbecause even enlightened beings cant skip leg day or laundry day. with similarity: 0.7110828757286072"]

["◎ Matched fragment: draws from the teachings of paramahansa yogananda author of autobiography of a yogi and patanjali the father of classical yogabecause the cosmos clearly loves a best-seller. with similarity: 0.7469132542610168"]

["◎ Matched fragment: teaches that life is a divine play urging you to ditch the drama and embrace the peaceunless youre auditioning for an oscar of course. with similarity: 0.7749838829040527"]

["◎ Generating text..."]

ℹ INFORMATIONS
Generating text with options:
{"modelProvider":"openai","model":"small","verifiableInference":false}

◎ LOGS
Using provider:
openai

ℹ INFORMATIONS
Selected model:
gpt-4o-mini

◎ LOGS
Image generation successful, number of images:
1

◎ LOGS
Processing image 1:
generated_1736573510427_0

◎ LOGS
Generated caption for image 1:
...

/home/rchak007/github/AI-Agents/selfRealize/PullTests/eliza-pr-test1756/node_modules/solc/soljson.js:133
process["on"]("unhandledRejection", function (reason) { throw reason; });
^

Error: {"errors":[{"code":38,"message":"media parameter is missing."}]}
at uploadMedia (file:///home/rchak007/github/AI-Agents/selfRealize/PullTests/eliza-pr-test1756/node_modules/agent-twitter-client/dist/node/esm/index.mjs:2334:13)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async Promise.all (index 0)
at async createCreateTweetRequest (file:///home/rchak007/github/AI-Agents/selfRealize/PullTests/eliza-pr-test1756/node_modules/agent-twitter-client/dist/node/esm/index.mjs:1975:22)
at async Scraper.sendTweet (file:///home/rchak007/github/AI-Agents/selfRealize/PullTests/eliza-pr-test1756/node_modules/agent-twitter-client/dist/node/esm/index.mjs:3503:12)
at async file:///home/rchak007/github/AI-Agents/selfRealize/PullTests/eliza-pr-test1756/packages/client-twitter/dist/index.js:14:36
at async RequestQueue.processQueue (file:///home/rchak007/github/AI-Agents/selfRealize/PullTests/eliza-pr-test1756/packages/client-twitter/dist/index.js:32:17)

Node.js v23.3.0
/home/rchak007/github/AI-Agents/selfRealize/PullTests/eliza-pr-test1756/agent:
 ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL  @elizaos/[email protected] start: node --loader ts-node/esm src/index.ts "--isRoot"
Exit status 1
 ELIFECYCLE  Command failed with exit code 1.
 
 i asked ChatGpt- but need to debug still. But just letting you know it did not work for me yet..
 The error media parameter is missing indicates that the image URL or the media parameter required to attach the image to the tweet was not properly passed to the Twitter API. Here’s a step-by-step guide to resolve this issue:

Root Cause
The image upload to Twitter failed.
uploadMedia function: The OpenAI DALL-E integration seems to have generated the image correctly, but the image data or URL wasn't properly included when sending the tweet.