lua incompatibility issue with latest Nginx version 1.24 #2218

AnitaThakur671 · 2023-08-08T18:11:16Z

Hi team,

We are trying to upgrade Nginx version from v1.22 to v.124 and facing compatibility issues with lua version.

After Nginx version upgrade to v1.24 it keeps crashing with errors like this:

|LOG_LINE| 26/Jul/2023:12:43:22 +0000 [edge.access.marriott_default:-:REQUEST ID=] { host: "10.221.70.200", remote_addr: "10.221.70.1", request: "GET /healthCheck HTTP/1.1", status: 200, referer: "-", request_time: 0.000, connection: "1342", connection_requests: "1", location: "", upstream_location_response: "-", bytes_sent: "247", client_ip: "-", x_dynaTrace: "-" }
2023/07/26 12:43:22 [alert] 184#0: worker process 201 exited on signal 11 (core dumped)
|LOG_LINE| 26/Jul/2023:12:43:32 +0000 [edge.access.marriott_default:-:REQUEST ID=] { host: "10.221.70.200", remote_addr: "10.221.70.1", request: "GET /healthCheck HTTP/1.1", status: 200, referer: "-", request_time: 0.000, connection: "1344", connection_requests: "1", location: "", upstream_location_response: "-", bytes_sent: "247", client_ip: "-", x_dynaTrace: "-" }

2023/07/26 12:43:25 [alert] 184#0: worker process 198 exited on signal 9

Suspecting this to be a lua issue, we have tried different versions of lua, but to no avail.
Following table shows key combinations we have tried:
nginx lua-nginx-module resty core status
1.24.0 0.10.22 v.0.1.24 not working
1.24.0 0.10.23 v0.1.25 not working
1.24.0 0.10.24 v0.1.25 not working
1.24.0 0.10.24 v0.1.26 not working

We also tried to analyze the core files which are dumped into the working directory of Nginx and we are getting the
below errors -

Core was generated by `nginx: worker process'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 ngx_http_variable_headers_internal (r=0x1f414a0, v=0x1f42990, data=, sep=sep@entry=44 ',') at src/http/ngx_http_variables.c:840
840 src/http/ngx_http_variables.c: No such file or directory.
Ensure you have provided the following details while reporting a problem:

Due to this issue we are blocked from upgrading Nginx version.

requesting help to resolve this issue.
Thanks.

zhuizhuhaomeng · 2023-08-09T01:42:47Z

The latest release of openresty is 1.21.4.2.
nginx 1.24 was not thoroughly tested by the Openresty team.

You can use https://xray.openresty.com to get more info from the core dump file.

AnitaThakur671 · 2023-08-21T17:36:59Z

As per lua community update(e2d0505) lua version is supporting latest Nginx v1.25+. So we tried with Nginx v1.25.1 but still we are seeing the same issue(worker process <process_number>exited on signal 11 (core dumped)) .
Combinations tried -
1.
Nginx Version - 1.25.1
lua-nginx-module - 0.10.24
lua-resty-core - v0.1.27 (latest)
Result - Compatibility Issues(pod doesn't start up)

Nginx Version - 1.25.1
lua-nginx-module-0.10.25
lua-resty-core - v0.1.27 (latest)
Result - Same issue ( child processes exiting with signal 11)

Any help related to this issue?

zhuizhuhaomeng · 2023-08-23T00:15:55Z

@AnitaThakur671 we need a minimal config that can reproduce this problem.

danilutkin · 2023-09-02T20:57:53Z

@zhuizhuhaomeng Hello! I have met the same issue. I believe.
This problem is reproducible.
I've prepared a Dockerfile to build nginx with needed versions and I have added a special config that uses rewrite_by_lua_block, you can find the nginx config inside Dockerfile.

NGINX_VERSION=1.25.1 \ 
LUA_MODULE_VERSION=0.10.25 \ 
LUA_RESTY_CORE_VERSION=0.1.27

Dockerfile:

# syntax = docker/dockerfile:1.4.0
FROM alpine:3.18

ENV NGINX_VERSION=1.25.1 \ 
    LUA_MODULE_VERSION=0.10.25 \ 
    LUA_RESTY_CORE_VERSION=0.1.27 \ 
    LUA_RESTY_LRUCACHE_VERSION=0.13 \ 
    NGINX_DEVEL_KIT_VERSION=0.3.2 \ 
    LUAJIT_VERSION=2.1-20230410 \ 
    LUA_VERSION=5.1 \ 
    LUA_PATH="/usr/local/lib/lua/5.1/?.lua;;" \
    BUILD_DEPS=" \
            build-base ca-certificates patch cmake \
            autoconf automake curl luajit-dev libtool \
            git gcc libc-dev make openssl-dev pcre-dev \
            zlib-dev linux-headers libxslt-dev gd-dev \
            perl-dev libedit-dev mercurial \
            bash alpine-sdk findutils gnupg" \
    NGINX_SETUP_DIR=/usr/src/nginx  \
    NGINX_CACHE_DIR=/var/cache/nginx  \
    NGINX_LOG_DIR=/var/log/nginx  \
    NGINX_CONF_DIR=/etc/nginx  \
    NGINX_MODULES=""
RUN set -x \
    && mkdir -p \
    ${NGINX_SETUP_DIR} \
    ${NGINX_CACHE_DIR} \
    ${NGINX_LOG_DIR} \
    ${NGINX_CONF_DIR} \
    && addgroup -g 101 -S nginx \
    && adduser -S -D -H -u 101 -h ${NGINX_CACHE_DIR} -s /sbin/nologin -G nginx -g nginx nginx \
    && apk add --no-cache --virtual .build-deps ${BUILD_DEPS} \
    && \
    echo "${NGINX_MODULES}" \
    && \
    NGINX_MODULES="${NGINX_MODULES} --add-module=${NGINX_SETUP_DIR}/ngx_devel_kit-${NGINX_DEVEL_KIT_VERSION}"; \
    curl -fSL https://github.com/simpl/ngx_devel_kit/archive/v${NGINX_DEVEL_KIT_VERSION}.tar.gz -o "${NGINX_SETUP_DIR}/ngx_devel_kit.tar"; \
    tar -zxC  "${NGINX_SETUP_DIR}" -f "${NGINX_SETUP_DIR}/ngx_devel_kit.tar"; \
    curl -fSL https://github.com/openresty/luajit2/archive/v${LUAJIT_VERSION}.tar.gz -o /tmp/luajit2-module.tar.gz; \
    tar -zxC /tmp/ -f /tmp/luajit2-module.tar.gz; \
    cd /tmp/luajit2-${LUAJIT_VERSION} && make && make install; \
    curl -fSL https://github.com/openresty/lua-resty-core/archive/v${LUA_RESTY_CORE_VERSION}.tar.gz -o /tmp/lua-resty-core.tar.gz; \
    curl -fSL https://github.com/openresty/lua-resty-lrucache/archive/v${LUA_RESTY_LRUCACHE_VERSION}.tar.gz -o /tmp/lua-resty-lrucache.tar.gz;\
    tar -zxC /tmp/ -f /tmp/lua-resty-core.tar.gz; \
    tar -zxC /tmp/ -f /tmp/lua-resty-lrucache.tar.gz; \
    cd /tmp/lua-resty-core-${LUA_RESTY_CORE_VERSION} ; make install ; cd - ;\
    cd /tmp/lua-resty-lrucache-${LUA_RESTY_LRUCACHE_VERSION} ; make install  ; cd - ;\
    NGINX_MODULES="${NGINX_MODULES} --add-module=${NGINX_SETUP_DIR}/lua-nginx-module-${LUA_MODULE_VERSION}"; \
    curl -fSL https://github.com/openresty/lua-nginx-module/archive/v${LUA_MODULE_VERSION}.tar.gz -o "${NGINX_SETUP_DIR}/lua_module.tar"; \
    tar -zxC "${NGINX_SETUP_DIR}" -f "${NGINX_SETUP_DIR}/lua_module.tar"; \
    export LUAJIT_LIB=/usr/local/lib; \
    export LUAJIT_INC=/usr/local/include/luajit-2.1; \
    BASE_CONFIGURE_ARGS="\
        --conf-path=${NGINX_CONF_DIR}/nginx.conf \
        --error-log-path=${NGINX_LOG_DIR}/error.log \
        --group=nginx \
        --http-client-body-temp-path=${NGINX_CACHE_DIR}/client_temp \
        --http-fastcgi-temp-path=${NGINX_CACHE_DIR}/fastcgi_temp \
        --http-log-path=${NGINX_LOG_DIR}/access.log \
        --http-proxy-temp-path=${NGINX_CACHE_DIR}/proxy_temp \
        --http-scgi-temp-path=${NGINX_CACHE_DIR}/scgi_temp \
        --http-uwsgi-temp-path=${NGINX_CACHE_DIR}/uwsgi_temp \
        --lock-path=/var/run/nginx.lock \
        --modules-path=/usr/lib/nginx/modules \
        --pid-path=/var/run/nginx.pid \
        --prefix=${NGINX_CONF_DIR} \
        --sbin-path=/usr/sbin/nginx \
        --user=nginx \
        --with-compat \
        --with-file-aio \
        --with-http_addition_module \
        --with-http_auth_request_module \
        --with-http_dav_module \
        --with-http_flv_module \
        --with-http_gunzip_module \
        --with-http_gzip_static_module \
        --with-http_degradation_module \
        --with-http_mp4_module \
        --with-http_random_index_module \
        --with-http_realip_module \
        --with-http_secure_link_module \
        --with-http_slice_module \
        --with-http_ssl_module \
        --with-http_stub_status_module \
        --with-http_sub_module \
        --with-http_v2_module \
        --with-ld-opt=-Wl,--as-needed \
        --with-ld-opt=-Wl,-rpath,$LUAJIT_LIB \
        --with-mail \
        --with-mail_ssl_module \
        --with-perl_modules_path=/usr/lib/perl5/vendor_perl \
        --with-stream \
        --with-stream_realip_module \
        --with-stream_ssl_module \
        --with-stream_ssl_preread_module \
        --with-threads \
        --with-http_image_filter_module=dynamic \
        --with-http_xslt_module=dynamic \
        ${NGINX_MODULES} \
        " \
    && curl -fSL http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz -o "${NGINX_SETUP_DIR}/nginx.tar.gz" \
    && tar -zxC "${NGINX_SETUP_DIR}" -f "${NGINX_SETUP_DIR}/nginx.tar.gz" \
    && rm ${NGINX_SETUP_DIR}/nginx.tar.gz \
    && cd ${NGINX_SETUP_DIR}/nginx-${NGINX_VERSION} \
    && ls -lah /usr/src/nginx/ \
    && ./configure $BASE_CONFIGURE_ARGS --with-cc-opt='-Os -fomit-frame-pointer'  --with-debug \
    && ls -lah objs/ \
    && make \
    && make install \
    && rm -rf ${NGINX_CONF_DIR}/html/ \
    && mkdir ${NGINX_CONF_DIR}/conf.d/ \
    && mkdir -p /usr/share/nginx/html/ \
    && install -m644 html/index.html /usr/share/nginx/html/ \
    && install -m644 html/50x.html /usr/share/nginx/html/ \
    && ln -s /usr/lib/nginx/modules ${NGINX_CONF_DIR}/modules \
    && strip  -v /usr/sbin/nginx* \
    && strip  -v /usr/lib/nginx/modules/*.so \
    && rm -rf ${NGINX_SETUP_DIR}/ \
    && rm -rf /tmp/* \
    && apk add --no-cache --virtual .gettext gettext \
    && mv /usr/bin/envsubst /tmp/ \
    && runDeps="$(scanelf --needed --nobanner --format '%n#p' /usr/sbin/nginx /usr/lib/nginx/modules/*.so /tmp/envsubst | tr ',' '\n' | sort -u | awk 'system("[ -e /usr/local/lib/" $1 " ]") == 0 { next } { print "so:" $1 }')" \
    && apk add --no-cache $runDeps \
    && apk add --no-cache tzdata libgcc \
    && ln -sf /dev/stdout ${NGINX_LOG_DIR}/access.log \
    && ln -sf /dev/stderr ${NGINX_LOG_DIR}/error.log

RUN <<EOF cat > /etc/nginx/nginx.conf
worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '\$remote_addr - \$remote_user [\$time_local] "\$request" '
                     '\$status \$body_bytes_sent "\$http_referer" '
                     '"\$http_user_agent" "\$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    error_log /dev/stdout debug;

    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;
        default_type text/plain;
        log_subrequest on;
        location /test {
            set \$a \$arg_a;
            rewrite_by_lua_block {
                local res = ngx.location.capture('/attr/',
                    { args = { a = ngx.var.a } }
                )
                if res.body == "1" then
                    return ngx.exec('@golocation_1')
                elseif res.body == "2" then
                    return ngx.exec('@golocation_2')
                end
            }
            alias /index;
        }
        location @golocation_1 {
            content_by_lua_block {
                ngx.say("You are in golocation_1")
            }
        }
        location @golocation_2 {
            content_by_lua_block {
                ngx.say("You are in golocation_2")
            }
        }
        location /attr/ {
            add_header "attr" \$arg_a;
            return 200 "\$arg_a"; 
        }
    }
}
EOF

RUN mkdir -p /index && echo "I am index.html" > /index/index.html && chown -R nginx.nginx /index

CMD ["nginx", "-g", "daemon off;"]

To build the test image:

docker build -t nginx-test-lua -f Dockerfile .

After building this image you can just run this sequence of commands to reproduce the problem.

We start a new container with nginx in debug mode with some Lua config and make the first 10 requests to the location. 🆗

No problems

We reloaded the Nginx once and waited 2 seconds to ensure that nginx reloaded and made the second 10 requests to the location. 🆗

No problems

We reloaded nginx a second time and we got the problem.

Approximately 50% of requests to the same location receive Empty reply from server and for each Empty reply from server you can see that worker process N exited on signal 11 in docker logs nginx-test-lua.

docker run --rm --name nginx-test-lua  -p  8080:80 -tid nginx-test-lua; 
for i in $(seq 10); do curl localhost:8080/test/\?a=1; done; 
docker exec nginx-test-lua nginx -s reload; sleep 2;
for i in $(seq 10); do curl localhost:8080/test/\?a=1 ; done
docker exec nginx-test-lua nginx -s reload; sleep 2;
for i in $(seq 10); do curl localhost:8080/test/\?a=1 ; done
docker logs nginx-test-lua  | grep "signal 11";
docker stop nginx-test-lua

My stdout for this command ⬆️ as an example:

You are in golocation_1
You are in golocation_1
You are in golocation_1
You are in golocation_1
You are in golocation_1
You are in golocation_1
You are in golocation_1
You are in golocation_1
You are in golocation_1
You are in golocation_1
2023/09/02 20:28:56 [notice] 9#9: signal process started
You are in golocation_1
You are in golocation_1
You are in golocation_1
You are in golocation_1
You are in golocation_1
You are in golocation_1
You are in golocation_1
You are in golocation_1
You are in golocation_1
You are in golocation_1
2023/09/02 20:28:58 [notice] 17#17: signal process started
You are in golocation_1
curl: (52) Empty reply from server
You are in golocation_1
curl: (52) Empty reply from server
You are in golocation_1
curl: (52) Empty reply from server
You are in golocation_1
curl: (52) Empty reply from server
You are in golocation_1
curl: (52) Empty reply from server
2023/09/02 20:29:00 [alert] 1#1: worker process 24 exited on signal 11
2023/09/02 20:29:00 [alert] 1#1: worker process 25 exited on signal 11
2023/09/02 20:29:00 [alert] 1#1: worker process 26 exited on signal 11
2023/09/02 20:29:00 [alert] 1#1: worker process 27 exited on signal 11
2023/09/02 20:29:00 [alert] 1#1: worker process 28 exited on signal 11

Also maybe this link will be helpful - https://trac.nginx.org/nginx/ticket/2521.
Hope this helps solve the problem. Thanks.

swananan · 2023-09-06T01:08:53Z

@danilutkin Thanks for your information. I have successfully reproduced this crash. While attempting to fix this bug, I discovered that someone has already raised a PR and resolved the issue (#2177). However, the lua-nginx-module-0.10.25 version does not include the bugfix. Perhaps you can try using the lua-nginx-module master branch to resolve this issue.
FYI, @zhuizhuhaomeng @AnitaThakur671

vkhulbey · 2023-10-10T14:04:56Z

@danilutkin Thanks for your information. I have successfully reproduced this crash. While attempting to fix this bug, I discovered that someone had already raised a PR and resolved the issue (#2177). However, the lua-nginx-module-0.10.25 version does not include the bug fix. Perhaps you can try using the lua-nginx-module master branch to resolve this issue. FYI, @zhuizhuhaomeng @AnitaThakur671

Will this fix work(using the lua-nginx-module master branch) with Nginx v1.24.0 as well?

swananan · 2023-10-12T03:29:53Z

@danilutkin Thanks for your information. I have successfully reproduced this crash. While attempting to fix this bug, I discovered that someone had already raised a PR and resolved the issue (#2177). However, the lua-nginx-module-0.10.25 version does not include the bug fix. Perhaps you can try using the lua-nginx-module master branch to resolve this issue. FYI, @zhuizhuhaomeng @AnitaThakur671

Will this fix work(using the lua-nginx-module master branch) with Nginx v1.24.0 as well?

Based on the information available at #2177, it appears that a Nginx version higher than 1.24.0 is required. Therefore, the answer is negative. Currently, OpenResty has extended support for Nginx version 1.25.1, making it a suitable choice for this issue.

oxpa · 2023-10-12T14:03:39Z

@swananan could you please explain me what is the issue? The #2177 seems to be included in 0.10.25 and lua should be able to handle "new style" variables.
Is it about the last thing that is "fixed in nginx" ? I have #2182 which I didn't bother finishing as I thought it all works.

I'm happy to provide a fix (or, say, a better fix) i just don't quite understand the problem =)

swananan · 2023-10-13T08:18:15Z

@swananan could you please explain me what is the issue? The #2177 seems to be included in 0.10.25 and lua should be able to handle "new style" variables. Is it about the last thing that is "fixed in nginx" ? I have #2182 which I didn't bother finishing as I thought it all works.

I'm happy to provide a fix (or, say, a better fix) i just don't quite understand the problem =)

Hi @oxpa , v0.10.25 branch doesn't include #2177, which might seem a bit unusual, but it's true, and I am not familiar with lua-nginx-module release plan either.
I suggested nginx version should be higher than 1.24.0, because of this fix https://hg.nginx.org/nginx/rev/b71e69247483, which was mentioned by #2177 .

oxpa · 2023-10-13T10:09:11Z

@swananan i think there is slight misunderstanding here.
#2177 is to keep up with this change: nginx/nginx@3aef1d6
And there is #2182 which aims at having a fix corresponding to https://hg.nginx.org/nginx/rev/b71e69247483 in lua.
I didn't bother finishing it as I didn't think anyone is using an old nginx with new lua: it doesn't make much sense.

I'll have another go at #2182 then.

swananan · 2023-10-15T11:38:50Z

@swananan i think there is slight misunderstanding here. #2177 is to keep up with this change: nginx/nginx@3aef1d6 And there is #2182 which aims at having a fix corresponding to https://hg.nginx.org/nginx/rev/b71e69247483 in lua. I didn't bother finishing it as I didn't think anyone is using an old nginx with new lua: it doesn't make much sense.

I'll have another go at #2182 then.

Hi @oxpa . The crash mentioned in this issue has been resolved by #2177. However, it was also noted in #2177 that another memory overrun case needed attention. In my opition, this memory overrun case was addressed in https://hg.nginx.org/nginx/rev/b71e69247483. I guess I may have some misunderstanding here.

I took a look at #2182, but I didn't quite get the point. It appears that ngx_http_variable_headers_internal iterates through the linked list, so inserting into either the tail or the header should be acceptable. Could you please provide more details regarding the specific issue that #2182 aims to address? Thank you!

oxpa · 2023-10-15T12:17:48Z

I looked through commits again and I think I'm wrong and the patch is, indeed, not included nor in 0.10.24 nor in 0.10.25. I'm truly surprised.
Sorry for the confusion i brought in.

ngx_http_variable_headers_internal would iterate over a list adding a separator and a space each time it adds a value.
Assume you have a list of 5 elements where only the first one is set and all others have hash==0 (but still present in the list).
Before the fix nginx would go over the list of headers, correctly calculate resulting length and allocate p to match existing headers (the one from the first element of the list).
Nginx then would go over the same list copying values of the list elements into p.
If the list element is not the last - nginx adds a separator and a space. Our sole element is the first in the list. So nginx would copy the value into p and then wrongly add a separator and a space.
Nginx fix is to check that p is full and stop going through the list.
My fix in #2182 is to make sure all values are added to the tail of the list.
My fix is not that good: it's better to recreate the whole list of headers. I got told that this is an OK approach. And so I have to rewrite my PR properly =)

Anyways, #2182 indeed is not related to this issue. Also, if #2177 is not merged into lua then nginx should be pre 1.23.0. Nginx 1.23.0, afaik, introduces lists of headers which break lua. But as we saw previously - I can be wrong so don't hesitate to correct me ;)

Kind regards.

swananan · 2023-10-15T12:57:08Z

I looked through commits again and I think I'm wrong and the patch is, indeed, not included nor in 0.10.24 nor in 0.10.25. I'm truly surprised. Sorry for the confusion i brought in.

ngx_http_variable_headers_internal would iterate over a list adding a separator and a space each time it adds a value. Assume you have a list of 5 elements where only the first one is set and all others have hash==0 (but still present in the list). Before the fix nginx would go over the list of headers, correctly calculate resulting length and allocate p to match existing headers (the one from the first element of the list). Nginx then would go over the same list copying values of the list elements into p. If the list element is not the last - nginx adds a separator and a space. Our sole element is the first in the list. So nginx would copy the value into p and then wrongly add a separator and a space. Nginx fix is to check that p is full and stop going through the list. My fix in #2182 is to make sure all values are added to the tail of the list. My fix is not that good: it's better to recreate the whole list of headers. I got told that this is an OK approach. And so I have to rewrite my PR properly =)

Anyways, #2182 indeed is not related to this issue. Also, if #2177 is not merged into lua then nginx should be pre 1.23.0. Nginx 1.23.0, afaik, introduces lists of headers which break lua. But as we saw previously - I can be wrong so don't hesitate to correct me ;)

Kind regards.

Oh, I understand now. #2182 changes the insertion from the head to the tail to address the memory overrun issue, and that makes sense. I didn't realize this because I saw that Nginx had already resolved this problem. Thanks for your explanation!

Possibly a bug with nginx, there were some segfault fixes in 1.25.4 release: https://nginx.org/en/CHANGES openresty/lua-nginx-module#2218 Signed-off-by: Alex Kalenyuk <[email protected]>

* Rollback nginx from 1.24.0->1.22.1 to avoid segfaults when pulling https Possibly a bug with nginx, there were some segfault fixes in 1.25.4 release: https://nginx.org/en/CHANGES openresty/lua-nginx-module#2218 Signed-off-by: Alex Kalenyuk <[email protected]> * Run make rpm-deps to udpate dependencies for all arches Ensure the dependencies are up to date so we can easily merge with s390x PRs in the future. Signed-off-by: Alex Kalenyuk <[email protected]> --------- Signed-off-by: Alex Kalenyuk <[email protected]> Co-authored-by: Alex Kalenyuk <[email protected]>

swananan mentioned this issue Sep 19, 2023

lua incompatibility issue with latest Nginx version 1.24 openresty/openresty#922

Closed

zhuizhuhaomeng closed this as completed Apr 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lua incompatibility issue with latest Nginx version 1.24 #2218

lua incompatibility issue with latest Nginx version 1.24 #2218

AnitaThakur671 commented Aug 8, 2023

zhuizhuhaomeng commented Aug 9, 2023

AnitaThakur671 commented Aug 21, 2023

zhuizhuhaomeng commented Aug 23, 2023

danilutkin commented Sep 2, 2023

swananan commented Sep 6, 2023

vkhulbey commented Oct 10, 2023

swananan commented Oct 12, 2023

oxpa commented Oct 12, 2023

swananan commented Oct 13, 2023

oxpa commented Oct 13, 2023

swananan commented Oct 15, 2023

oxpa commented Oct 15, 2023

swananan commented Oct 15, 2023

lua incompatibility issue with latest Nginx version 1.24 #2218

lua incompatibility issue with latest Nginx version 1.24 #2218

Comments

AnitaThakur671 commented Aug 8, 2023

zhuizhuhaomeng commented Aug 9, 2023

AnitaThakur671 commented Aug 21, 2023

zhuizhuhaomeng commented Aug 23, 2023

danilutkin commented Sep 2, 2023

swananan commented Sep 6, 2023

vkhulbey commented Oct 10, 2023

swananan commented Oct 12, 2023

oxpa commented Oct 12, 2023

swananan commented Oct 13, 2023

oxpa commented Oct 13, 2023

swananan commented Oct 15, 2023

oxpa commented Oct 15, 2023

swananan commented Oct 15, 2023