Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LocalAI: Stable Diffusion image generation broken: 500 Internal Server Error #88

Closed
v3DJG6GL opened this issue May 22, 2024 · 8 comments
Closed

LocalAI: Stable Diffusion image generation broken: 500 Internal Server Error #88

v3DJG6GL opened this issue May 22, 2024 · 8 comments
Labels
bug Something isn't working

Comments

@v3DJG6GL
Copy link

v3DJG6GL commented May 22, 2024 •
edited
Loading

Which version of assistant are you using?

v1.0.9

Which version of Nextcloud are you using?

v29.0.0

Which browser are you using? In case you are using the phone App, specify the Android or iOS version and device please.

Firefox

Describe the Bug

I cannot get Stable Diffusion image generation to work with Nextcloud Assistant: I have my LocalAI instance attached and using the own LocalAI UI, image generation works flawlessly.

  • Here are the LocalAI logfiles when trying to generate a image with Nextcoud Assistant:
1:10PM DBG Request received: {"model":"","language":"","n":0,"top_p":null,"top_k":null,"temperature":null,"max_tokens":null,"echo":false,"batch":0,"ignore_eos":false,"repeat_penalty":0,"n_keep":0,"frequency_penalty":0,"presence_penalty":0,"tfz":null,"typical_p":null,"seed":null,"negative_prompt":"","rope_freq_base":0,"rope_freq_scale":0,"negative_prompt_scale":0,"use_fast_tokenizer":false,"clip_skip":0,"tokenizer":"","file":"","response_format":{},"size":"1024x1024","prompt":"a man on the moon","instruction":"","input":null,"stop":null,"messages":null,"functions":null,"function_call":null,"stream":false,"mode":0,"step":0,"grammar":"","grammar_json_functions":null,"backend":"","model_base_name":""}
1:10PM DBG Loading model: stablediffusion
1:10PM DBG Parameter Config: &{PredictionOptions:{Model:stablediffusion_assets Language: N:0 TopP:0xc000b96790 TopK:0xc000b96798 Temperature:0xc000b967a0 Maxtokens:0xc000b967e0 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc000b967d8 TypicalP:0xc000b967d0 Seed:0xc000b967f8 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:stablediffusion F16:0xc000b96788 Threads:0xc000b96780 Debug:0xc000ace420 Roles:map[] Embeddings:false Backend:stablediffusion TemplateConfig:{Chat: ChatMessage: Completion: Edit: Functions: UseTokenizerTemplate:false} PromptStrings:[a man on the moon] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: FunctionsConfig:{DisableNoAction:false NoActionFunctionName: NoActionDescriptionName: ParallelCalls:false NoGrammar:false ResponseRegex:} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc000b967b8 MirostatTAU:0xc000b967b0 Mirostat:0xc000b967a8 NGPULayers:0xc000b967e8 MMap:0xc000b967f0 MMlock:0xc000b967f1 LowVRAM:0xc000b967f1 Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:0xc000b96768 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} VallE:{AudioPath:} CUDA:false DownloadFiles:[] Description: Usage:}
1:10PM INF Loading model 'stablediffusion_assets' with backend stablediffusion
1:10PM DBG Stopping all backends except 'stablediffusion_assets'
1:10PM DBG Model already loaded in memory: stablediffusion_assets
1:10PM WRN GRPC Model not responding: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:46777: connect: connection refused"
1:10PM WRN Deleting the process in order to recreate it
1:10PM DBG GRPC Process is not responding: stablediffusion_assets
1:10PM ERR error stopping process error="process does not exist" process=stablediffusion_assets
1:10PM DBG Loading model in memory from file: /models/stablediffusion_assets
1:10PM DBG Loading Model stablediffusion_assets with gRPC (file: /models/stablediffusion_assets) (backend: stablediffusion): {backendString:stablediffusion model:stablediffusion_assets threads:4 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc0004bd000 externalBackends:map[autogptq:/build/backend/python/autogptq/run.sh bark:/build/backend/python/bark/run.sh coqui:/build/backend/python/coqui/run.sh diffusers:/build/backend/python/diffusers/run.sh exllama:/build/backend/python/exllama/run.sh exllama2:/build/backend/python/exllama2/run.sh huggingface-embeddings:/build/backend/python/sentencetransformers/run.sh mamba:/build/backend/python/mamba/run.sh parler-tts:/build/backend/python/parler-tts/run.sh petals:/build/backend/python/petals/run.sh rerankers:/build/backend/python/rerankers/run.sh sentencetransformers:/build/backend/python/sentencetransformers/run.sh transformers:/build/backend/python/transformers/run.sh transformers-musicgen:/build/backend/python/transformers-musicgen/run.sh vall-e-x:/build/backend/python/vall-e-x/run.sh vllm:/build/backend/python/vllm/run.sh] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:true parallelRequests:false}
1:10PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/stablediffusion
1:10PM DBG GRPC Service for stablediffusion_assets will be running at: '127.0.0.1:46769'
1:10PM DBG GRPC Service state dir: /tmp/go-processmanager247863740
1:10PM DBG GRPC Service Started
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr 2024/05/22 13:10:22 gRPC Server listening at 127.0.0.1:46769
1:10PM DBG GRPC Service Ready
1:10PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:stablediffusion_assets ContextSize:512 Seed:1751620475 NBatch:512 F16Memory:false MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:4 LibrarySearchPath: RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/models/stablediffusion_assets Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type:}
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stdout ----------------[start generation upscaled image]------------------
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stdout positive_prompt: a man on the moon
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stdout output_png_path: /images/b643007357863.png
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stdout negative_prompt: 
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stdout step: 15
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stdout seed: -1
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stdout ----------------[prompt]------------------
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr fopen /models/stablediffusion_assets/UNetModel-fp16.param failed
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr fopen /models/stablediffusion_assets/UNetModel-fp16.bin failed
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stdout ----------------[diffusion]---------------
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr find_blob_index_by_name in0 failed
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr Try
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr find_blob_index_by_name in1 failed
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr Try
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr find_blob_index_by_name in2 failed
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr Try
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr find_blob_index_by_name c_in failed
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr Try
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr find_blob_index_by_name c_out failed
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr Try
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr find_blob_index_by_name outout failed
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr Try
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr find_blob_index_by_name in0 failed
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr Try
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr find_blob_index_by_name in1 failed
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr Try
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr find_blob_index_by_name in2 failed
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr Try
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr find_blob_index_by_name c_in failed
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr Try
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr find_blob_index_by_name c_out failed
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr Try
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr find_blob_index_by_name outout failed
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr Try
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr SIGSEGV: segmentation violation
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr PC=0x87e8b0 m=0 sigcode=1 addr=0x0
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr signal arrived during cgo execution
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr 
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr goroutine 38 gp=0xc0003b8380 m=0 mp=0x158c760 [syscall]:
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.cgocall(0x8676a0, 0xc0004317a0)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/cgocall.go:157 +0x4b fp=0xc000431778 sp=0xc000431740 pc=0x43e9ab
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr github.com/mudler/go-stable-diffusion._Cfunc_generate_image_upscaled(0x400, 0x400, 0xf, 0xffffffff, 0x1e10480, 0x1e104a0, 0x1e0a970, 0x1e0ef70)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         _cgo_gotypes.go:107 +0x4b fp=0xc0004317a0 sp=0xc000431778 pc=0x86608b
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr github.com/mudler/go-stable-diffusion.GenerateImageUpscaled(0x400, 0x400, 0xf, 0xffffffffffffffff, {0xc000254078?, 0x41?}, {0x0, 0x0}, {0xc0003be0a0, 0x19}, ...)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /build/sources/go-stable-diffusion/stablediffusion.go:34 +0xf0 fp=0xc000431800 sp=0xc0004317a0 pc=0x866550
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr github.com/go-skynet/LocalAI/pkg/stablediffusion.GenerateImage(0xc00041e280?, 0xc000431998?, 0x846b43?, 0x1533520?, 0xc0004220f0?, {0xc000254078?, 0x5?}, {0x0?, 0xc00041e280?}, {0xc0003be0a0, ...}, ...)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /build/pkg/stablediffusion/generate.go:12 +0x7d fp=0xc000431888 sp=0xc000431800 pc=0x86667d
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr github.com/go-skynet/LocalAI/pkg/stablediffusion.(*StableDiffusion).GenerateImage(...)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /build/pkg/stablediffusion/stablediffusion.go:19
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr main.(*Image).GenerateImage(0x41?, 0x41?)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /build/backend/go/image/stablediffusion/stablediffusion.go:24 +0x65 fp=0xc000431900 sp=0xc000431888 pc=0x866aa5
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr github.com/go-skynet/LocalAI/pkg/grpc.(*server).GenerateImage(0xc000037750, {0x1070260?, 0xc00044e9e8?}, 0xc00041e280)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /build/pkg/grpc/server.go:71 +0xe7 fp=0xc0004319a8 sp=0xc000431900 pc=0x8628e7
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr github.com/go-skynet/LocalAI/pkg/grpc/proto._Backend_GenerateImage_Handler({0x1070260, 0xc000037750}, {0x1152640, 0xc00025a510}, 0xc0003c0080, 0x0)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /build/pkg/grpc/proto/backend_grpc.pb.go:409 +0x1a6 fp=0xc0004319f8 sp=0xc0004319a8 pc=0x85ed26
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr google.golang.org/grpc.(*Server).processUnaryRPC(0xc0002001e0, {0x1152640, 0xc00025a450}, {0x1155c60, 0xc000424000}, 0xc00043e000, 0xc000228300, 0x153f040, 0x0)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/google.golang.org/[email protected]/server.go:1343 +0xdd1 fp=0xc000431df0 sp=0xc0004319f8 pc=0x845411
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr google.golang.org/grpc.(*Server).handleStream(0xc0002001e0, {0x1155c60, 0xc000424000}, 0xc00043e000)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/google.golang.org/[email protected]/server.go:1737 +0xc47 fp=0xc000431f78 sp=0xc000431df0 pc=0x84a3c7
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr google.golang.org/grpc.(*Server).serveStreams.func1.1()
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/google.golang.org/[email protected]/server.go:986 +0x86 fp=0xc000431fe0 sp=0xc000431f78 pc=0x8432e6
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.goexit({})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000431fe8 sp=0xc000431fe0 pc=0x4a6c81
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 37
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/google.golang.org/[email protected]/server.go:997 +0x136
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr 
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr goroutine 1 gp=0xc0000061c0 m=nil [IO wait]:
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.gopark(0x1?, 0xc000217ba8?, 0x91?, 0x13?, 0xc000217b60?)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:402 +0xce fp=0xc000217b28 sp=0xc000217b08 pc=0x474fce
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.netpollblock(0x0?, 0x43e146?, 0x0?)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:573 +0xf7 fp=0xc000217b60 sp=0xc000217b28 pc=0x46ddd7
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr internal/poll.runtime_pollWait(0x7ff6dad16f20, 0x72)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:345 +0x85 fp=0xc000217b80 sp=0xc000217b60 pc=0x4a1b45
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr internal/poll.(*pollDesc).wait(0x3?, 0x45d365?, 0x0)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000217ba8 sp=0xc000217b80 pc=0x50c367
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr internal/poll.(*pollDesc).waitRead(...)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:89
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr internal/poll.(*FD).Accept(0xc0001c4e00)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc000217c50 sp=0xc000217ba8 pc=0x51170c
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr net.(*netFD).accept(0xc0001c4e00)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/net/fd_unix.go:172 +0x29 fp=0xc000217d08 sp=0xc000217c50 pc=0x598c29
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr net.(*TCPListener).accept(0xc00012c5a0)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/net/tcpsock_posix.go:159 +0x1e fp=0xc000217d30 sp=0xc000217d08 pc=0x5affde
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr net.(*TCPListener).Accept(0xc00012c5a0)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/net/tcpsock.go:327 +0x30 fp=0xc000217d60 sp=0xc000217d30 pc=0x5af1d0
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr google.golang.org/grpc.(*Server).Serve(0xc0002001e0, {0x1151bf0, 0xc00012c5a0})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/google.golang.org/[email protected]/server.go:852 +0x469 fp=0xc000217e98 sp=0xc000217d60 pc=0x841f69
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr github.com/go-skynet/LocalAI/pkg/grpc.StartServer({0x7fff19b6655e?, 0xc000024280?}, {0x1158330, 0xc0000376a0})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /build/pkg/grpc/server.go:226 +0x170 fp=0xc000217f20 sp=0xc000217e98 pc=0x864a70
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr main.main()
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /build/backend/go/image/stablediffusion/main.go:18 +0x85 fp=0xc000217f50 sp=0xc000217f20 pc=0x866925
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.main()
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:271 +0x29d fp=0xc000217fe0 sp=0xc000217f50 pc=0x474b9d
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.goexit({})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000217fe8 sp=0xc000217fe0 pc=0x4a6c81
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr 
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:402 +0xce fp=0xc00010cfa8 sp=0xc00010cf88 pc=0x474fce
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.goparkunlock(...)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:408
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.forcegchelper()
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:326 +0xb3 fp=0xc00010cfe0 sp=0xc00010cfa8 pc=0x474e53
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.goexit({})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00010cfe8 sp=0xc00010cfe0 pc=0x4a6c81
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr created by runtime.init.6 in goroutine 1
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:314 +0x1a
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr 
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:402 +0xce fp=0xc00010d780 sp=0xc00010d760 pc=0x474fce
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.goparkunlock(...)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:408
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.bgsweep(0xc000136000)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00010d7c8 sp=0xc00010d780 pc=0x4605d4
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.gcenable.gowrap1()
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:203 +0x25 fp=0xc00010d7e0 sp=0xc00010d7c8 pc=0x454f25
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.goexit({})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00010d7e8 sp=0xc00010d7e0 pc=0x4a6c81
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr created by runtime.gcenable in goroutine 1
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:203 +0x66
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr 
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]:
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.gopark(0xc000136000?, 0x114a170?, 0x1?, 0x0?, 0xc000007340?)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:402 +0xce fp=0xc00010df78 sp=0xc00010df58 pc=0x474fce
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.goparkunlock(...)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:408
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.(*scavengerState).park(0x158bf40)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00010dfa8 sp=0xc00010df78 pc=0x45dfc9
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.bgscavenge(0xc000136000)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00010dfc8 sp=0xc00010dfa8 pc=0x45e55c
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.gcenable.gowrap2()
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:204 +0x25 fp=0xc00010dfe0 sp=0xc00010dfc8 pc=0x454ec5
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.goexit({})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00010dfe8 sp=0xc00010dfe0 pc=0x4a6c81
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr created by runtime.gcenable in goroutine 1
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:204 +0xa5
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr 
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr goroutine 5 gp=0xc000007c00 m=nil [finalizer wait]:
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.gopark(0xc00010c648?, 0x448605?, 0xa8?, 0x1?, 0x108fb80?)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:402 +0xce fp=0xc00010c620 sp=0xc00010c600 pc=0x474fce
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.runfinq()
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/mfinal.go:194 +0x107 fp=0xc00010c7e0 sp=0xc00010c620 pc=0x453f67
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.goexit({})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00010c7e8 sp=0xc00010c7e0 pc=0x4a6c81
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr created by runtime.createfing in goroutine 1
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/mfinal.go:164 +0x3d
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr 
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr goroutine 35 gp=0xc000434000 m=nil [select]:
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.gopark(0xc0002a1f00?, 0x2?, 0x1e?, 0x0?, 0xc0002a1ed4?)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:402 +0xce fp=0xc0002a1d80 sp=0xc0002a1d60 pc=0x474fce
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.selectgo(0xc0002a1f00, 0xc0002a1ed0, 0x7dcbf6?, 0x0, 0xc000412000?, 0x1)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/select.go:327 +0x725 fp=0xc0002a1ea0 sp=0xc0002a1d80 pc=0x486345
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000410050, 0x1)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:418 +0x113 fp=0xc0002a1f30 sp=0xc0002a1ea0 pc=0x7bbb13
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000292070)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:552 +0x86 fp=0xc0002a1f90 sp=0xc0002a1f30 pc=0x7bc266
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr google.golang.org/grpc/internal/transport.NewServerTransport.func2()
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:336 +0xd5 fp=0xc0002a1fe0 sp=0xc0002a1f90 pc=0x7d2b55
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.goexit({})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0002a1fe8 sp=0xc0002a1fe0 pc=0x4a6c81
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 34
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:333 +0x1a8c
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr 
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr goroutine 36 gp=0xc0004341c0 m=nil [select]:
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.gopark(0xc000404f40?, 0x4?, 0xf0?, 0x4d?, 0xc000404ec0?)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:402 +0xce fp=0xc000404d58 sp=0xc000404d38 pc=0x474fce
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.selectgo(0xc000404f40, 0xc000404eb8, 0x0?, 0x0, 0x0?, 0x1)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/select.go:327 +0x725 fp=0xc000404e78 sp=0xc000404d58 pc=0x486345
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000424000)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:1152 +0x205 fp=0xc000404fc8 sp=0xc000404e78 pc=0x7d9ca5
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr google.golang.org/grpc/internal/transport.NewServerTransport.gowrap1()
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:339 +0x25 fp=0xc000404fe0 sp=0xc000404fc8 pc=0x7d2a45
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.goexit({})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000404fe8 sp=0xc000404fe0 pc=0x4a6c81
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 34
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:339 +0x1ace
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr 
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr goroutine 37 gp=0xc000434380 m=nil [IO wait]:
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.gopark(0x10523e0?, 0xc00025a420?, 0xe8?, 0xdb?, 0xb?)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:402 +0xce fp=0xc00011cab0 sp=0xc00011ca90 pc=0x474fce
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.netpollblock(0x4f2438?, 0x43e146?, 0x0?)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:573 +0xf7 fp=0xc00011cae8 sp=0xc00011cab0 pc=0x46ddd7
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr internal/poll.runtime_pollWait(0x7ff6dad16e28, 0x72)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:345 +0x85 fp=0xc00011cb08 sp=0xc00011cae8 pc=0x4a1b45
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr internal/poll.(*pollDesc).wait(0xc0003c0000?, 0xc000408000?, 0x0)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc00011cb30 sp=0xc00011cb08 pc=0x50c367
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr internal/poll.(*pollDesc).waitRead(...)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:89
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr internal/poll.(*FD).Read(0xc0003c0000, {0xc000408000, 0x8000, 0x8000})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc00011cbc8 sp=0xc00011cb30 pc=0x50d65a
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr net.(*netFD).Read(0xc0003c0000, {0xc000408000?, 0x1060100000000?, 0x8?})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/net/fd_posix.go:55 +0x25 fp=0xc00011cc10 sp=0xc00011cbc8 pc=0x596c45
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr net.(*conn).Read(0xc0003c2000, {0xc000408000?, 0x800010601?, 0x0?})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/net/net.go:179 +0x45 fp=0xc00011cc58 sp=0xc00011cc10 pc=0x5a71e5
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr net.(*TCPConn).Read(0x0?, {0xc000408000?, 0xc00011ccb0?, 0x4949ad?})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         <autogenerated>:1 +0x25 fp=0xc00011cc88 sp=0xc00011cc58 pc=0x5b9305
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr bufio.(*Reader).Read(0xc0003c4000, {0xc00041a040, 0x9, 0xc000280008?})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/bufio/bufio.go:241 +0x197 fp=0xc00011ccc0 sp=0xc00011cc88 pc=0x55c557
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr io.ReadAtLeast({0x114f2a0, 0xc0003c4000}, {0xc00041a040, 0x9, 0x9}, 0x9)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/io/io.go:335 +0x90 fp=0xc00011cd08 sp=0xc00011ccc0 pc=0x4ec270
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr io.ReadFull(...)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/io/io.go:354
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr golang.org/x/net/http2.readFrameHeader({0xc00041a040, 0x9, 0xc0002a2000?}, {0x114f2a0?, 0xc0003c4000?})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x65 fp=0xc00011cd58 sp=0xc00011cd08 pc=0x7a8c85
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr golang.org/x/net/http2.(*Framer).ReadFrame(0xc00041a000)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x85 fp=0xc00011ce00 sp=0xc00011cd58 pc=0x7a93c5
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000424000, 0xc00025a2d0)
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:636 +0x145 fp=0xc00011cf08 sp=0xc00011ce00 pc=0x7d5be5
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr google.golang.org/grpc.(*Server).serveStreams(0xc0002001e0, {0x1155c60, 0xc000424000})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/google.golang.org/[email protected]/server.go:979 +0x1aa fp=0xc00011cf80 sp=0xc00011cf08 pc=0x8430aa
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr google.golang.org/grpc.(*Server).handleRawConn.func1()
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/google.golang.org/[email protected]/server.go:920 +0x45 fp=0xc00011cfe0 sp=0xc00011cf80 pc=0x842905
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr runtime.goexit({})
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00011cfe8 sp=0xc00011cfe0 pc=0x4a6c81
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 34
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr         /root/go/pkg/mod/google.golang.org/[email protected]/server.go:919 +0x15b
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr 
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr rax    0x0
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr rbx    0x7fff19b64920
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr rcx    0x0
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr rdx    0x0
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr rdi    0x0
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr rsi    0x4
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr rbp    0x7fff19b636f0
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr rsp    0x7fff19b632e0
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr r8     0x0
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr r9     0x0
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr r10    0x0
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr r11    0x118551543e87248d
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr r12    0x7fff19b64968
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr r13    0x7fff19b633e0
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr r14    0x7fff19b63340
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr r15    0x7fff19b634d0
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr rip    0x87e8b0
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr rflags 0x10246
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr cs     0x33
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr fs     0x0
1:10PM DBG GRPC(stablediffusion_assets-127.0.0.1:46769): stderr gs     0x0
1:10PM ERR Server error error="rpc error: code = Unavailable desc = error reading from server: EOF" ip=172.16.122.157 latency=4.447314403s method=POST status=500 url=/v1/images/generations
  • These are the correspondending Nextcloud log entries:
[integration_openai] Warning: OpenAI/LocalAI's text to image generation failed with: API request error: rpc error: code = Unavailable desc = error reading from server: EOF
	POST /ocs/v2.php/apps/assistant/api/v1/i/process_prompt
	from XXX.XXX.XX.XX by XXXX at 22.05.2024, 13:10:27

[integration_openai] Warning: API request error : Server error: POST http://local-ai.ix-local-ai.svc.cluster.local:8080/v1/images/generations resulted in a 500 Internal Server Error response:
{"error":{"code":500,"message":"rpc error: code = Unavailable desc = error reading from server: EOF","type":""}}

	POST /ocs/v2.php/apps/assistant/api/v1/i/process_prompt
	from XXX.XXX.XX.XX by XXXX at 22.05.2024, 13:10:27
  • And here are the LocalAI logfiles when generating an image directly with the LocalAI UI:
1:11PM DBG Request received: {"model":"stablediffusion","language":"","n":1,"top_p":null,"top_k":null,"temperature":null,"max_tokens":null,"echo":false,"batch":0,"ignore_eos":false,"repeat_penalty":0,"n_keep":0,"frequency_penalty":0,"presence_penalty":0,"tfz":null,"typical_p":null,"seed":null,"negative_prompt":"","rope_freq_base":0,"rope_freq_scale":0,"negative_prompt_scale":0,"use_fast_tokenizer":false,"clip_skip":0,"tokenizer":"","file":"","response_format":{},"size":"512x512","prompt":"A person walking on the moon, their hair is flowing in the windy athmosphere.","instruction":"","input":null,"stop":null,"messages":null,"functions":null,"function_call":null,"stream":false,"mode":0,"step":0,"grammar":"","grammar_json_functions":null,"backend":"","model_base_name":""}
1:11PM DBG Loading model: stablediffusion
1:11PM DBG Parameter Config: &{PredictionOptions:{Model:stablediffusion_assets Language: N:0 TopP:0xc000b96790 TopK:0xc000b96798 Temperature:0xc000b967a0 Maxtokens:0xc000b967e0 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc000b967d8 TypicalP:0xc000b967d0 Seed:0xc000b967f8 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:stablediffusion F16:0xc000b96788 Threads:0xc000b96780 Debug:0xc000729fd0 Roles:map[] Embeddings:false Backend:stablediffusion TemplateConfig:{Chat: ChatMessage: Completion: Edit: Functions: UseTokenizerTemplate:false} PromptStrings:[A person walking on the moon, their hair is flowing in the windy athmosphere.] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: FunctionsConfig:{DisableNoAction:false NoActionFunctionName: NoActionDescriptionName: ParallelCalls:false NoGrammar:false ResponseRegex:} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc000b967b8 MirostatTAU:0xc000b967b0 Mirostat:0xc000b967a8 NGPULayers:0xc000b967e8 MMap:0xc000b967f0 MMlock:0xc000b967f1 LowVRAM:0xc000b967f1 Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:0xc000b96768 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} VallE:{AudioPath:} CUDA:false DownloadFiles:[] Description: Usage:}
1:11PM INF Loading model 'stablediffusion_assets' with backend stablediffusion
1:11PM DBG Stopping all backends except 'stablediffusion_assets'
1:11PM DBG Model already loaded in memory: stablediffusion_assets
1:11PM WRN GRPC Model not responding: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:46769: connect: connection refused"
1:11PM WRN Deleting the process in order to recreate it
1:11PM DBG GRPC Process is not responding: stablediffusion_assets
1:11PM ERR error stopping process error="process does not exist" process=stablediffusion_assets
1:11PM DBG Loading model in memory from file: /models/stablediffusion_assets
1:11PM DBG Loading Model stablediffusion_assets with gRPC (file: /models/stablediffusion_assets) (backend: stablediffusion): {backendString:stablediffusion model:stablediffusion_assets threads:4 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc000809800 externalBackends:map[autogptq:/build/backend/python/autogptq/run.sh bark:/build/backend/python/bark/run.sh coqui:/build/backend/python/coqui/run.sh diffusers:/build/backend/python/diffusers/run.sh exllama:/build/backend/python/exllama/run.sh exllama2:/build/backend/python/exllama2/run.sh huggingface-embeddings:/build/backend/python/sentencetransformers/run.sh mamba:/build/backend/python/mamba/run.sh parler-tts:/build/backend/python/parler-tts/run.sh petals:/build/backend/python/petals/run.sh rerankers:/build/backend/python/rerankers/run.sh sentencetransformers:/build/backend/python/sentencetransformers/run.sh transformers:/build/backend/python/transformers/run.sh transformers-musicgen:/build/backend/python/transformers-musicgen/run.sh vall-e-x:/build/backend/python/vall-e-x/run.sh vllm:/build/backend/python/vllm/run.sh] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:true parallelRequests:false}
1:11PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/stablediffusion
1:11PM DBG GRPC Service for stablediffusion_assets will be running at: '127.0.0.1:45145'
1:11PM DBG GRPC Service state dir: /tmp/go-processmanager2513190298
1:11PM DBG GRPC Service Started
1:11PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stderr 2024/05/22 13:11:41 gRPC Server listening at 127.0.0.1:45145
1:11PM DBG GRPC Service Ready
1:11PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:stablediffusion_assets ContextSize:512 Seed:38130937 NBatch:512 F16Memory:false MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:4 LibrarySearchPath: RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/models/stablediffusion_assets Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type:}
1:11PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout ----------------[start stable diffusion]--------------------
1:11PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout ----------------[init]--------------------
1:11PM INF Success ip=172.16.0.1 latency="85.154µs" method=GET status=200 url=/readyz
1:11PM INF Success ip=172.16.0.1 latency="58.234µs" method=GET status=200 url=/readyz
1:11PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stderr fopen /models/stablediffusion_assets/AutoencoderKL-encoder-512-512-fp16.param failed
1:11PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stderr network graph not ready
1:11PM DBG [WatchDog] Watchdog checks for idle connections
1:11PM DBG [WatchDog] 127.0.0.1:33561: idle connection
1:11PM DBG [WatchDog] 127.0.0.1:46777: idle connection
1:11PM DBG [WatchDog] 127.0.0.1:46769: idle connection
1:11PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout  3.49G / 3.63G
1:11PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout ----------------[prompt]------------------ 3.49G / 3.63G
1:11PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout ----------------[diffusion]---------------
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout step: 1/15      8590.86ms
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout step: 2/15      5753.00ms
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout step: 3/15      6981.75ms
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout step: 4/15      5995.39ms
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout step: 5/15      7181.03ms
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout step: 6/15      6246.80ms
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout step: 7/15      5298.99ms
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout step: 8/15      6326.69ms
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout step: 9/15      6284.52ms
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout step:10/15      6408.69ms
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout step:11/15      6838.78ms
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout step:12/15      6402.75ms
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout step:13/15      5954.36ms
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout step:14/15      6347.13ms
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout step:15/15      6097.85ms
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout ----------------[diffusion]--------------- 6.42G / 6.42G
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout ----------------[decode]------------------ 7.87G / 7.87G
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout ----------------[save]--------------------
1:13PM DBG GRPC(stablediffusion_assets-127.0.0.1:45145): stdout ----------------[close]-------------------
1:13PM DBG Response: {"created":1716376423,"id":"a4e07773-eb6b-4745-82b8-c00b9a777a89","data":[{"embedding":null,"index":0,"url":"https://localai.DOMAIN.TLD/generated-images/b644044860645.png"}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
1:13PM INF Success ip=172.16.134.13 latency=2m1.865136232s method=POST status=200 url=/v1/images/generations
1:13PM INF Success ip=172.16.134.13 latency="183.569µs" method=GET status=200 url=/generated-images/b644044860645.png
1:13PM INF Success ip=172.16.0.1 latency="26.87µs" method=GET status=200 url=/readyz

Expected Behavior

I should be able to generate an image with Nextcloud Assistant

To Reproduce

See above.
@v3DJG6GL v3DJG6GL added the bug Something isn't working label May 22, 2024
@v3DJG6GL v3DJG6GL changed the title LocalAI: Stable Diffusion image generation broken LocalAI: Stable Diffusion image generation broken: 500 Internal Server Error May 22, 2024
@S-I-M-O-N
Copy link

I have a similar behavior. I think it is related to the image size requested by the assistant, which is 1024x1024. See line 46 in Application.php of the integration_openai repo. In the past there was a picker for this in the assistant or text2image helper. See also TextToImageProvider.php line 56. The default stablediffusion of LocalAI can only deal with resolutions up to 512x512.

@Killerjoe
Copy link

@S-I-M-O-N thank you for this hint!
I was wonder why I could not generate an image with Nextcloud and localai and could not find a solution for this.
You just gave the answer.
I have changed the entry DEFAULT_IMAGE_SIZE from 1024x1024 to 512x512 in file nextcloud/apps/integration_openai/lib/AppInfo/Application.php and now I´m able to generate images with nextcloud assistant :-)
public const DEFAULT_IMAGE_SIZE = '512x512';
I don´t understand how this could ever work with localai?
Does openai deal with higher resolutions than 512x512?

@S-I-M-O-N
Copy link

@Killerjoe It depends on the model used. I stumbled across this previously, because earlier versions of the Nextcloud integration only allowed 1024x1024 to be selected for OpenAI (thus I assume it can handle larger than 512x512). I used LocalAI with stable-diffusion-xl-base-1.0 which required at least 1024x1024 otherwise it would just produce garbage.

@julien-nc
Copy link
Member

The latest assistant (NC >= 30) + the latest integration_openai app allow users to define the desired image size in the assistant's input form.
The app declares an optional input field for image generation and the assistant renders in in the input form.
image
Wdyt about adding a field in the app's admin settings to change the default image size? It would be more convenient than asking users to set 512x512 for each task they launch.

@julien-nc
Copy link
Member

And the default size has been changed to 512x512 in integration_openai as well (>= v3.1.2).

@S-I-M-O-N
Copy link

@julien-nc Thank you for fixing the default value. LocalAI should work now out of the box.
Confirmed with assistent 2.1.0 and integration_openai 3.1.2.
Setting the default value in Admin settings would certainly be convenient.
Somehow I miss the additional fields on the form though.
grafik

@julien-nc
Copy link
Member

If OpenAI is selected as provider for image generation, you should be able to enable "advanced options" next to the submit button in the assistant.

image

@S-I-M-O-N
Copy link

Thank you, that was an important hint to use LocalAI as a provider. For some reason LocalAI ImageGeneration is also in the picklist which is a bit misleading:
grafik

@janepie janepie closed this as completed Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@S-I-M-O-N @julien-nc @Killerjoe @janepie @v3DJG6GL