Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Text input bugs of the Android backend and the Skia renderer #7203

Open
wuwbobo2021 opened this issue Dec 23, 2024 · 5 comments
Open

Text input bugs of the Android backend and the Skia renderer #7203

wuwbobo2021 opened this issue Dec 23, 2024 · 5 comments
Assignees
Labels
a:renderer-skia Skia Renderer (mS) bug Something isn't working

Comments

@wuwbobo2021
Copy link
Contributor

wuwbobo2021 commented Dec 23, 2024

Bug Description

Tested on Android 11.0, and Android 6.0 (the later requires my modified version to run it), without SLINT_STYLE override.

LineEdit is almost unusable because the left side of the text selector often appears and sticks at the left side, then anything typed into it may disappear after losing focus.

When editing text in TextEdit, it is easy to trigger application crashing (the log is provided below). Even if it does not crash, it may still produce strange insertion beyond the cursor location. It seems like a bug of character index calculation.

Reproducible Code (if applicable)

[package]
name = "slint-test"
version = "0.1.0"
edition = "2021"
publish = false

[dependencies]
slint = { version = "1.9.1", features = ["backend-android-activity-06"] }

[lib]
crate-type = ["cdylib"]
path = "lib.rs"
#[cfg(target_os = "android")]
#[no_mangle]
fn android_main(app: slint::android::AndroidApp) {
    slint::android::init(app).unwrap();
    slint::slint!{
        import {Button, GroupBox, TextEdit, LineEdit} from "std-widgets.slint";
        export component MainWindow inherits Window {
            VerticalLayout {
                LineEdit { text: "Hello"; }
                TextEdit { text: "Line 1\nLine2\nLine3\n"; }
            }
        }
    }
    let window = MainWindow::new().unwrap();
    window.run().unwrap();
}
12-23 22:55:40.536  2828  2850 I RustStdoutStderr: thread '<unnamed>' panicked at C:\Users\net\.cargo\registry\src\index.crates.io-6f17d22bba15001f\unicode-segmentation-1.12.0\src\grapheme.rs:697:29:
12-23 22:55:40.536  2828  2850 I RustStdoutStderr: byte index 20 is out of bounds of `Line 1
12-23 22:55:40.536  2828  2850 I RustStdoutStderr:
12-23 22:55:40.537  2828  2850 I RustStdoutStderr: Line3
12-23 22:55:40.537  2828  2850 I RustStdoutStderr: `
12-23 22:55:40.537  2828  2850 I RustStdoutStderr: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
12-23 22:55:40.537  2828  2850 I RustStdoutStderr: thread '<unnamed>' panicked at core/src/panicking.rs:221:5:
12-23 22:55:40.537  2828  2850 I RustStdoutStderr: panic in a function that cannot unwind
12-23 22:55:40.537  2828  2850 I RustStdoutStderr: stack backtrace:
12-23 22:55:40.542  2828  2850 I RustStdoutStderr:    0:     0x781ca708f01a - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hc23b1ffd781af1d9
12-23 22:55:40.542  2828  2850 I RustStdoutStderr:    1:     0x781ca70b1e9b - core::fmt::write::h37d92ec90f9aac89
12-23 22:55:40.543  2828  2850 I RustStdoutStderr:    2:     0x781ca708cab3 - std::io::Write::write_fmt::h0a0011a72def67f2
12-23 22:55:40.543  2828  2850 I RustStdoutStderr:    3:     0x781ca7090122 - std::panicking::default_hook::{{closure}}::h18da23b1edd5cb42
12-23 22:55:40.543  2828  2850 I RustStdoutStderr:    4:     0x781ca708fce2 - std::panicking::default_hook::h7dfb0b482a149ed6
12-23 22:55:40.544  2828  2850 I RustStdoutStderr:    5:     0x781ca7090d0c - std::panicking::rust_panic_with_hook::he8de8b0cc5a28393
12-23 22:55:40.546  2828  2850 I RustStdoutStderr:    6:     0x781ca7090b73 - std::panicking::begin_panic_handler::{{closure}}::hdad7fa2424cd4020
12-23 22:55:40.547  2828  2850 I RustStdoutStderr:    7:     0x781ca708f509 - std::sys::backtrace::__rust_end_short_backtrace::hc171f97f78737681
12-23 22:55:40.547  2828  2850 I RustStdoutStderr:    8:     0x781ca7090834 - rust_begin_unwind
12-23 22:55:40.547  2828  2850 I RustStdoutStderr:    9:     0x781ca70af915 - core::panicking::panic_nounwind_fmt::h4e1f8f7fae34bb56
12-23 22:55:40.547  2828  2850 I RustStdoutStderr:   10:     0x781ca70af9a2 - core::panicking::panic_nounwind::h837dcbb75a233ddb
12-23 22:55:40.547  2828  2850 I RustStdoutStderr:   11:     0x781ca70afa86 - core::panicking::panic_cannot_unwind::hfb2138252c96e0b4
12-23 22:55:40.547  2828  2850 I RustStdoutStderr:   12:     0x781ca6fe3c57 - i_slint_core::items::TextInputVTable::input_event::h8c8dc73ab49d2b37
12-23 22:55:40.548  2828  2850 I RustStdoutStderr:   13:     0x781ca6fe0a4b - i_slint_core::input::send_mouse_event_to_item::hffe80224059ffd2b
12-23 22:55:40.549  2828  2850 I RustStdoutStderr:   14:     0x781ca6ff2238 - i_slint_core::item_tree::ItemVisitor_vtable_mod::ItemVisitorVTable::new::visit_item::h98ff8f1fd074fe91
12-23 22:55:40.550  2828  2850 I RustStdoutStderr:   15:     0x781ca6a17db7 - i_slint_core::item_tree::visit_item_tree::{{closure}}::hb25308458c637868
12-23 22:55:40.550  2828  2850 I RustStdoutStderr:   16:     0x781ca6a17cdd - i_slint_core::item_tree::visit_item_tree::ha8d6beb9913a4f64
12-23 22:55:40.550  2828  2850 I RustStdoutStderr:   17:     0x781ca6a0a3c9 - <slint_test::android_main::slint_generatedMainWindow::InnerMainWindow as i_slint_core::item_tree::ItemTree_vtable_mod::ItemTree>::visit_children_item::h54b132042a38f159
12-23 22:55:40.550  2828  2850 I RustStdoutStderr:   18:     0x781ca6a0bbe4 - slint_test::android_main::slint_generatedMainWindow::_::VT::visit_children_item::h5f3ee95c73300a5b
12-23 22:55:40.550  2828  2850 I RustStdoutStderr:   19:     0x781ca6fe096c - i_slint_core::input::send_mouse_event_to_item::hffe80224059ffd2b
12-23 22:55:40.550  2828  2850 I RustStdoutStderr:   20:     0x781ca6ff21c2 - i_slint_core::item_tree::ItemVisitor_vtable_mod::ItemVisitorVTable::new::visit_item::h565d89adb8fda1e9
12-23 22:55:40.550  2828  2850 I RustStdoutStderr:   21:     0x781ca6a17db7 - i_slint_core::item_tree::visit_item_tree::{{closure}}::hb25308458c637868
12-23 22:55:40.550  2828  2850 I RustStdoutStderr:   22:     0x781ca6a17cdd - i_slint_core::item_tree::visit_item_tree::ha8d6beb9913a4f64
12-23 22:55:40.550  2828  2850 I RustStdoutStderr:   23:     0x781ca6a0a3c9 - <slint_test::android_main::slint_generatedMainWindow::InnerMainWindow as i_slint_core::item_tree::ItemTree_vtable_mod::ItemTree>::visit_children_item::h54b132042a38f159
12-23 22:55:40.550  2828  2850 I RustStdoutStderr:   24:     0x781ca6a0bbe4 - slint_test::android_main::slint_generatedMainWindow::_::VT::visit_children_item::h5f3ee95c73300a5b
12-23 22:55:40.550  2828  2850 I RustStdoutStderr:   25:     0x781ca6fe0623 - i_slint_core::input::process_delayed_event::h1654dc7b2a1b17dd
12-23 22:55:40.551  2828  2850 I RustStdoutStderr:   26:     0x781ca6ffaf31 - i_slint_core::window::WindowInner::process_mouse_input::h13eed52c34c85f9e
12-23 22:55:40.553  2828  2850 I RustStdoutStderr:   27:     0x781ca6ff9185 - i_slint_core::api::Window::dispatch_event::h70d80f65e2daf061
12-23 22:55:40.555  2828  2850 I RustStdoutStderr:   28:     0x781ca6a3a960 - android_activity::activity_impl::InputIteratorInner::next::h36cb1c07a18ab524
12-23 22:55:40.555  2828  2850 I RustStdoutStderr:   29:     0x781ca6a3864b - i_slint_backend_android_activity::androidwindowadapter::AndroidWindowAdapter::process_event::h1dcf50515d5093fb
12-23 22:55:40.555  2828  2850 I RustStdoutStderr:   30:     0x781ca6a3b5ef - <i_slint_backend_android_activity::AndroidPlatform as i_slint_core::platform::Platform>::run_event_loop::{{closure}}::hbbd10b728723c2cc
12-23 22:55:40.555  2828  2850 I RustStdoutStderr:   31:     0x781ca6a39e6a - android_activity::activity_impl::AndroidAppInner::poll_events::h93d9c87f5b7ff86d
12-23 22:55:40.555  2828  2850 I RustStdoutStderr:   32:     0x781ca6a4753b - android_activity::AndroidApp::poll_events::h4a3921e8ca378a36
12-23 22:55:40.555  2828  2850 I RustStdoutStderr:   33:     0x781ca6a3d935 - <i_slint_backend_android_activity::AndroidPlatform as i_slint_core::platform::Platform>::run_event_loop::h0a99430e906120e0
12-23 22:55:40.555  2828  2850 I RustStdoutStderr:   34:     0x781ca6a18a23 - slint::run_event_loop::hceede304fe64f7f4
12-23 22:55:40.555  2828  2850 I RustStdoutStderr:   35:     0x781ca6a031ec - android_main
12-23 22:55:40.555  2828  2850 I RustStdoutStderr:   36:     0x781ca6a50576 - std::sys::backtrace::__rust_begin_short_backtrace::h9251ec0cb18eee3d
12-23 22:55:40.555  2828  2850 I RustStdoutStderr:   37:     0x781ca6a5191f - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc211e3815c8a0379
12-23 22:55:40.555  2828  2850 I RustStdoutStderr:   38:     0x781ca70927cb - std::sys::pal::unix::thread::Thread::new::thread_start::h7fe747499d12117e
12-23 22:55:40.557  2828  2850 I RustStdoutStderr:   39:     0x781f9398fd2b - _ZL15__pthread_startPv
12-23 22:55:40.558  2828  2850 I RustStdoutStderr:   40:     0x781f939270c8 - __start_thread
12-23 22:55:40.558  2828  2850 I RustStdoutStderr: thread caused non-unwinding panic. aborting.


### Environment Details

- Slint Version: 1.9.1
- Platform/OS: Android 11.0
- Programming Language: Rust
- Backend/Renderer: android-activity


### Product Impact

_No response_
@wuwbobo2021
Copy link
Contributor Author

wuwbobo2021 commented Dec 25, 2024

This issue can be split into two parts. One part is what I'm fixing in the PR, it consists of the wrong cursor index calculation bug and java.lang.RuntimeException: Can't create handler inside thread that has not called Looper.prepare() crashing in SlintAndroidJavaHelper.get_clipboard and SlintAndroidJavaHelper.set_clipboard.

Another problem is related to the panic logging in the initial post: unicode-segmentation-1.12.0\src\grapheme.rs:697:29: byte index is out of bounds. I did a lot of tests with the Android backend crate. Java_SlintAndroidJavaHelper_updateText's UpdateComposition looks spicious, and setting cursor_position and anchor_position to some invalid large value produces crashing with the same error message; but the original problem still exists even if they are both set to 0. I asked a lot of questions to the Copilot which has become free on Github recently, but it doesn't help.

Eventually I discovered that it is a bug of the skia renderer (which is the only renderer for Android), not the Android backend. I've reproduced it on Windows. Add the renderer-skia feature to the slint dependency, and have the below code in main.rs:

fn main() -> Result<(), slint::PlatformError> {
    let selector = slint::BackendSelector::new().renderer_name("skia".to_string());
    if let Err(err) = selector.select() {
        panic!("Unable to select the skia render: {err:?}");
    }

    slint::slint!{
        import {Button, GroupBox, TextEdit, LineEdit} from "std-widgets.slint";
        export component MainWindow inherits Window {
            width: 300px;
            height: 200px;
            VerticalLayout {
                LineEdit { text: "Hello"; }
                TextEdit { text: "Line 1\nLine2\nLine3\n"; }
            }
        }
    }

    let main_window = MainWindow::new()?;

    main_window.run()
}

Run it and switch to some input method that distinguishes "preedit" text from existing text, type a few characters at the middle of existing text, some mark like underline will appear for the "preedit" text; then click on a position beyond the end of the whole text to produce the crashing.

This can be found easily on Android because the input method exists for English. On PC, some input method of other languages may be installed to reproduce the problem.

Note: unicode-segmentation crate is used in:

wuwbobo2021 added a commit to wuwbobo2021/slint that referenced this issue Dec 25, 2024
@wuwbobo2021
Copy link
Contributor Author

wuwbobo2021 commented Dec 25, 2024

I added an index checker in cursor_rect function of the skia renderer to stop this case from crashing (cursor_pos exceeds string.len(), probably by 1).

I have found another possible Unicode problem with the preedit feature (while changing cursor position).

12-26 06:31:54.945 12591 12604 I RustStdoutStderr: thread '<unnamed>' panicked at C:\Users\net\.cargo\registry\src\index.crates.io-6f17d22bba15001f\unicode-segmentation-1.12.0\src\grapheme.rs:697:29:
12-26 06:31:54.945 12591 12604 I RustStdoutStderr: byte index 23 is not a char boundary; it is inside '程' (bytes 22..25) of `Line 1
12-26 06:31:54.945 12591 12604 I RustStdoutStderr: Line2
12-26 06:31:54.945 12591 12604 I RustStdoutStderr: Line3
12-26 06:31:54.945 12591 12604 I RustStdoutStderr: 过程`

grapheme_cursor.is_boundary() panics too, because it's just a function checking for Unicode grapheme bounds. So I added a checker with the standard str::is_char_boundary (dismiss if it isn't).

But this is not eliminating the invalid index from its source, and the text input is still buggy without crashing (try to modify some text and you'll realize some problems), I am frustrated...

@wuwbobo2021
Copy link
Contributor Author

wuwbobo2021 commented Dec 26, 2024

Sorry. I have tested some emoji (stored by a pair of Java primitive char) and printed cursor indexes. My previous modification for fixing index convertion is wrong. It's not fixing problems, but introducing some problem. The Copilot keeps cheating me with detailed answers for my clear questions, and I will not use it anymore.

After recovering the previous convert_utf16_index_to_utf8 and convert_utf8_index_to_utf16:

The crashing is still produced when editing non-ASCII text if my character boundary checker is removed from cursor_rect. With this checker, another problem may occur while editing non-ASCII text (it's easier to reproduce it with Google Pinyin Input 4.5.2 instead of the English input method): somehow characters cannot be typed into the text box anymore, switch to another line input box and you can find the problem still exists. Copying and pasting still work in this case. This problem cannot be produced on PC with the Skia renderer.

Note for the description "anything typed into it may disappear after losing focus" in the first post: it's probably not a problem, because the text that disappears is always the preedit text in the input method. Sometimes the underline isn't shown, though.

wuwbobo2021 added a commit to wuwbobo2021/slint that referenced this issue Dec 27, 2024
@wuwbobo2021
Copy link
Contributor Author

wuwbobo2021 commented Dec 27, 2024

Added notes:

This bug can be produced after removing the character boundary checker if !string.is_char_boundary(cursor_pos) (added in skia's cursor_rect) from the PR, on Android and on PC with the Skia renderer and an input method with preedit feature:

Put "úī " in the text input, type "a" in the default (English) input method as the preedit text, set the cursor before 'ú' then after 'ú', then it will panic at unicode-segmentation-1.12.0\src\grapheme.rs:697:29:

RustStdoutStderr: byte index 3 is not a char boundary; it is inside 'ì' (bytes 2..4) of `úì `

The "characters cannot be typed into the text input anymore" problem should be seperated from this bug: it can be observed without triggering the panicking described above while using Google Pinyin input method (either Chinese mode or English mode, difficult to produce under English input method); the problem remains after switching to the English input method, before forced-stopping and restarting the application. It cannot be reproduced on PC with the Skia renderer. I am not sure whether it is a bug inside Slint.

Observed with the character boundary checker added in the PR:

My assertion added after let text_pos = text_input.as_pin_ref().byte_offset_for_position(pos, &adaptor); in javahelper.rs may fail while instantly moving the preedit text (in the English input method) inside some European text: text_input.as_pin_ref().text().to_string().is_char_boundary(text_pos) (it is probably related to the len + 1 index problem). It is difficult to reproduce. This assertion isn't present in my PR.

@wuwbobo2021 wuwbobo2021 changed the title Buggy LineEdit and TextEdit on Android Text input bugs of the Android backend and the Skia renderer Dec 28, 2024
@wuwbobo2021
Copy link
Contributor Author

wuwbobo2021 commented Jan 1, 2025

Without the PR #7204, the clipboard operation crashing problem and the "soft input always prompt out" exist on Android 6.0 and 8.0, but they may not be reproduced on Android 11.0. Both problems aren't related to the skia renderer.

Panic: JNI error: JavaException
--------- beginning of crash
01-01 16:04:21.163  3297  3314 E AndroidRuntime: FATAL EXCEPTION: Thread-2
01-01 16:04:21.163  3297  3314 E AndroidRuntime: Process: rust.slint_test, PID: 3297
01-01 16:04:21.163  3297  3314 E AndroidRuntime: java.lang.RuntimeException: Can't create handler inside thread that has not called Looper.prepare()
01-01 16:04:21.163  3297  3314 E AndroidRuntime:        at android.os.Handler.<init>(Handler.java:203)
01-01 16:04:21.163  3297  3314 E AndroidRuntime:        at android.os.Handler.<init>(Handler.java:117)
01-01 16:04:21.163  3297  3314 E AndroidRuntime:        at android.content.ClipboardManager$2.<init>(ClipboardManager.java:62)
01-01 16:04:21.163  3297  3314 E AndroidRuntime:        at android.content.ClipboardManager.<init>(ClipboardManager.java:62)
01-01 16:04:21.163  3297  3314 E AndroidRuntime:        at android.app.SystemServiceRegistry$11.createService(SystemServiceRegistry.java:249)
01-01 16:04:21.163  3297  3314 E AndroidRuntime:        at android.app.SystemServiceRegistry$11.createService(SystemServiceRegistry.java:247)
01-01 16:04:21.163  3297  3314 E AndroidRuntime:        at android.app.SystemServiceRegistry$CachedServiceFetcher.getService(SystemServiceRegistry.java:928)
01-01 16:04:21.163  3297  3314 E AndroidRuntime:        at android.app.SystemServiceRegistry.getSystemService(SystemServiceRegistry.java:880)
01-01 16:04:21.163  3297  3314 E AndroidRuntime:        at android.app.ContextImpl.getSystemService(ContextImpl.java:1651)
01-01 16:04:21.163  3297  3314 E AndroidRuntime:        at android.view.ContextThemeWrapper.getSystemService(ContextThemeWrapper.java:171)
01-01 16:04:21.163  3297  3314 E AndroidRuntime:        at android.app.Activity.getSystemService(Activity.java:5893)
01-01 16:04:21.163  3297  3314 E AndroidRuntime:        at SlintAndroidJavaHelper.set_clipboard(SlintAndroidJavaHelper.java:527)

Line 527 indicated by the stack trace does mActivity.getSystemService(Context.CLIPBOARD_SERVICE).

I've found a similar issue: xamarin/Essentials#820, https://github.com/MicrosoftDocs/xamarin-docs/pull/1812/files.

@tronical tronical self-assigned this Jan 2, 2025
@ogoffart ogoffart added a:renderer-skia Skia Renderer (mS) and removed need triaging Issue that the owner of the area still need to triage labels Jan 2, 2025
ogoffart pushed a commit that referenced this issue Jan 3, 2025
* Fixes for older Android versions (6.0 to 9.0)

* Fix build script for Android (#4920)

* Fix clipboard operations on Android

* Fix TextEdit crashing on Android (#7203)

* Remove unsafe calls added in 'Fixes for older Android versions'

* Update a comment in androidwindowadapter.rs

Fixes #7182
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:renderer-skia Skia Renderer (mS) bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants