[cscore] Use frame time in Linux UsbCameraImpl #7609

mcm001 · 2024-12-30T17:15:55Z

Previously, we used wpi::Now in the USBCamera thread. We can instead use the frametime v4l hands us.

TODO:

Test on hardware with logging enabled and see what flags we get

PeterJohnson · 2024-12-30T17:19:11Z

At best, it will give you system time, which is a different time base than wpi::Now, so we will need a correction factor. I see you’ve implemented one, but yeah, we need to test with real hardware to see if it’s garbage or useful.

mcm001 · 2024-12-30T17:20:36Z

"It" meaning v4l? I believe if V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC is set it should give us montonic time, but we'll still need a correction factor anyways coz wpi::Now has a different offset (which -should- be implemented here)

cscore/src/main/native/linux/UsbCameraImpl.cpp

mcm001 · 2024-12-30T17:30:41Z

Re: discord -- "What I don’t know is whether the timestamp is done in the kernel for USB cameras or comes from the camera itself. If the former, it’s more likely to actually work (if the latter, there will be like 2 vendors who do it right and all the rest will be zero or garbage)"
This also scares me

mcm001 · 2024-12-30T17:50:38Z

some sample output from my OV2311 at 1600x1200x50fps, mjpg:

CS: DEBUG: USB Camera 0: grabbing image (UsbCameraImpl.cpp:522)
CS: DEBUG: USB Camera 0: got image size=105000 index=1 (UsbCameraImpl.cpp:539)
CS: DEBUG: USB Camera 0: Flags 8192 (UsbCameraImpl.cpp:562)
CS: DEBUG: USB Camera 0: Got valid monotonic time for frame (UsbCameraImpl.cpp:567)
CS: DEBUG: USB Camera 0: Frame was 16179 uS old (UsbCameraImpl.cpp:581)
CS: DEBUG: USB Camera 0: End Of Frame false Start Of Exposure true (UsbCameraImpl.cpp:586)
CS: DEBUG: USB Camera 0: Copying data to 0x72bf082f12a0 from 0x72bf1c056000 (105000 bytes) (SourceImpl.cpp:479)
CS: DEBUG: converting image from 1600x1200 type 1 to 1600x1200 type 4 (Frame.cpp:749)
CS: DEBUG: serve_USB Camera 0: sending frame size=105000 addDHT=false (MjpegServerImpl.cpp:768)
CS: DEBUG: serve_USB Camera 0: waiting for frame (MjpegServerImpl.cpp:703)
Robot: latency 27.477 ms
CS: DEBUG: USB Camera 0: grabbing image (UsbCameraImpl.cpp:522)
CS: DEBUG: USB Camera 0: got image size=104912 index=2 (UsbCameraImpl.cpp:539)
CS: DEBUG: USB Camera 0: Flags 8192 (UsbCameraImpl.cpp:562)
CS: DEBUG: USB Camera 0: Got valid monotonic time for frame (UsbCameraImpl.cpp:567)
CS: DEBUG: USB Camera 0: Frame was 16114 uS old (UsbCameraImpl.cpp:581)
CS: DEBUG: USB Camera 0: End Of Frame false Start Of Exposure true (UsbCameraImpl.cpp:586)
CS: DEBUG: USB Camera 0: Copying data to 0x72bf0830b1f0 from 0x72bf12856000 (104912 bytes) (SourceImpl.cpp:479)
CS: DEBUG: converting image from 1600x1200 type 1 to 1600x1200 type 4 (Frame.cpp:749)
CS: DEBUG: serve_USB Camera 0: sending frame size=104912 addDHT=false (MjpegServerImpl.cpp:768)
CS: DEBUG: serve_USB Camera 0: waiting for frame (MjpegServerImpl.cpp:703)
Robot: latency 28.161 ms
CS: DEBUG: USB Camera 0: grabbing image (UsbCameraImpl.cpp:522)
CS: DEBUG: USB Camera 0: got image size=104992 index=3 (UsbCameraImpl.cpp:539)
CS: DEBUG: USB Camera 0: Flags 8192 (UsbCameraImpl.cpp:562)
CS: DEBUG: USB Camera 0: Got valid monotonic time for frame (UsbCameraImpl.cpp:567)
CS: DEBUG: USB Camera 0: Frame was 16193 uS old (UsbCameraImpl.cpp:581)
CS: DEBUG: USB Camera 0: End Of Frame false Start Of Exposure true (UsbCameraImpl.cpp:586)
CS: DEBUG: USB Camera 0: Copying data to 0x72bf082f12a0 from 0x72bf124ac000 (104992 bytes) (SourceImpl.cpp:479)
CS: DEBUG: converting image from 1600x1200 type 1 to 1600x1200 type 4 (Frame.cpp:749)
CS: DEBUG: serve_USB Camera 0: sending frame size=104992 addDHT=false (MjpegServerImpl.cpp:768)
CS: DEBUG: serve_USB Camera 0: waiting for frame (MjpegServerImpl.cpp:703)
Robot: latency 29.574 ms
CS: DEBUG: USB Camera 0: grabbing image (UsbCameraImpl.cpp:522)
CS: DEBUG: USB Camera 0: got image size=104880 index=0 (UsbCameraImpl.cpp:539)
CS: DEBUG: USB Camera 0: Flags 8192 (UsbCameraImpl.cpp:562)
CS: DEBUG: USB Camera 0: Got valid monotonic time for frame (UsbCameraImpl.cpp:567)
CS: DEBUG: USB Camera 0: Frame was 16150 uS old (UsbCameraImpl.cpp:581)
CS: DEBUG: USB Camera 0: End Of Frame false Start Of Exposure true (UsbCameraImpl.cpp:586)
CS: DEBUG: USB Camera 0: Copying data to 0x72bf0830b1f0 from 0x72bf24256000 (104880 bytes) (SourceImpl.cpp:479)
CS: DEBUG: serve_USB Camera 0: sending frame size=104880 addDHT=false (MjpegServerImpl.cpp:768)
CS: DEBUG: converting image from 1600x1200 type 1 to 1600x1200 type 4 (Frame.cpp:749)
CS: DEBUG: serve_USB Camera 0: waiting for frame (MjpegServerImpl.cpp:703)
Robot: latency 29.85 ms

The scene in question:

cscore/src/main/native/cpp/Frame.h

mcm001 · 2024-12-30T21:10:50Z

ov9782_2311_cscore_latency.txt
I did some more testing on an Orange Pi 5. I had an arducam OV9782 on /dev/video0 and an arducam ov2311 on /dev/video4. the 2311 was averaging something like 16-20ms, and the 9782 more like 28-32ms extra latency. Resolutions for both were 1600x1200 @ 50fps mjpeg. And some raw data:

Robot: camera 0: capture latency 41.45 ms from timesource 3, @ 29 fps
Robot: camera 4: capture latency 29.519 ms from timesource 3, @ 24 fps
Robot: camera 0: capture latency 37.179 ms from timesource 3, @ 36 fps
Robot: camera 4: capture latency 33.368 ms from timesource 3, @ 23 fps
Robot: camera 0: capture latency 37.404 ms from timesource 3, @ 30 fps
Robot: camera 4: capture latency 33.328 ms from timesource 3, @ 25 fps
Robot: camera 0: capture latency 37.387 ms from timesource 3, @ 33 fps
Robot: camera 0: capture latency 37.407 ms from timesource 3, @ 29 fps
Robot: camera 4: capture latency 29.645 ms from timesource 3, @ 24 fps
Robot: camera 0: capture latency 37.303 ms from timesource 3, @ 32 fps
Robot: camera 4: capture latency 29.362 ms from timesource 3, @ 26 fps
Robot: camera 0: capture latency 41.227 ms from timesource 3, @ 29 fps
Robot: camera 4: capture latency 29.589 ms from timesource 3, @ 24 fps
Robot: camera 0: capture latency 37.269 ms from timesource 3, @ 36 fps
Robot: camera 4: capture latency 29.585 ms from timesource 3, @ 25 fps
Robot: camera 0: capture latency 37.382 ms from timesource 3, @ 29 fps

mcm001 · 2024-12-30T21:55:23Z

Just saw pybind11_stubgen - [ ERROR] In cscore._cscore.CvSink.lastFrameTimeSource : Can't find/import 'WPI_TimestampSource' - trying to include RawFrame, but does pybind support c enums?

mcm001 · 2024-12-30T22:20:25Z

And the same test on a Pi. Note that this is at 30fps and 10fps, respectively, for the cameras, and we are in fact getting v4l timestamps in the monotonic clock timebase ;)
timestamps_pi3_ov9281_ov2311.txt

auscompgeek · 2024-12-30T23:51:28Z

but does pybind support c enums?

It does. I don't think RobotPy currently exposes the RawFrame header.

mcm001 · 2025-01-01T09:53:39Z

Downstream fix for robotpy will be merged in robotpy/mostrobotpy#124

PeterJohnson · 2025-01-01T16:28:07Z

This needs to be added to the Java/JNI layer.

mcm001 · 2025-01-01T17:11:17Z

Right now, Java's RawFrame doesn't store/expose time at all. Do you want me to add that? Or what makes the most sense?

mcm001 · 2025-01-01T19:09:12Z

cscore/src/test/java/edu/wpi/first/cscore/FrameTimeTest.java

+    // (Note: the frame time and time source are overridden by RawSourceImpl, and changing this is
+    // spooky)


Could consider updating cscore to make this more testable, but i'm happy with this for now.

PeterJohnson · 2025-01-01T22:12:23Z

Needs some doc fixes

mcm001 · 2025-01-01T22:39:36Z

Yeah and the AprilTagGenerationTest failure looks real as well. Will repro and fix up tn

mcm001 · 2025-01-02T00:50:58Z

fixed docs and default initialized time in RawFrame

mcm001 requested a review from a team as a code owner December 30, 2024 17:15

github-actions bot added the component: cscore CameraServer library label Dec 30, 2024

mcm001 changed the title ~~Use frame time in Linux UsbCameraImpl~~ [cscore] Use frame time in Linux UsbCameraImpl Dec 30, 2024

ThadHouse reviewed Dec 30, 2024

View reviewed changes

cscore/src/main/native/linux/UsbCameraImpl.cpp Show resolved Hide resolved

mcm001 requested a review from PeterJohnson as a code owner December 30, 2024 20:12

github-actions bot added the component: wpiutil WPI utility library label Dec 30, 2024

mcm001 commented Dec 30, 2024

View reviewed changes

cscore/src/main/native/cpp/Frame.h Show resolved Hide resolved

mcm001 mentioned this pull request Jan 1, 2025

Add RawFrame to autogen_headers robotpy/mostrobotpy#124

Open

mcm001 added 11 commits January 1, 2025 10:43

Use frame time in Linux UsbCameraImpl

81f509f

Don't narrow lol

df32398

Address review

caa2234

fixup units

53c4121

update debug prints

5086ea8

Remove unused variable

967259d

pipe frame time source up

ff589b8

run lint on wpiutil

4f10089

actually pipe timetssamp up

e8ddd5a

mabe if I include what i use?

5d84182

Expose frame time through JNI

fc49c3b

mcm001 force-pushed the cscore-frametime-linux branch from 7748911 to fc49c3b Compare January 1, 2025 18:43

mcm001 added 2 commits January 1, 2025 11:08

Add jni for frame time

4422234

Create test

f32c283

mcm001 commented Jan 1, 2025

View reviewed changes

update defaults

944cb58

mcm001 added 2 commits January 1, 2025 16:50

Fixup docs

e09e1fb

default init time

9ce9197

mcm001 added 4 commits January 1, 2025 17:06

run lint and add javadocs

c73d464

spotbugs? more like spot periods

8ccc844

= 0 is apparently redundant

2db2871

make public

80b45c3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[cscore] Use frame time in Linux UsbCameraImpl #7609

[cscore] Use frame time in Linux UsbCameraImpl #7609

mcm001 commented Dec 30, 2024 •

edited

Loading

PeterJohnson commented Dec 30, 2024 •

edited

Loading

mcm001 commented Dec 30, 2024 •

edited

Loading

mcm001 commented Dec 30, 2024

mcm001 commented Dec 30, 2024 •

edited

Loading

mcm001 commented Dec 30, 2024 •

edited

Loading

mcm001 commented Dec 30, 2024

mcm001 commented Dec 30, 2024 •

edited

Loading

auscompgeek commented Dec 30, 2024

mcm001 commented Jan 1, 2025

PeterJohnson commented Jan 1, 2025

mcm001 commented Jan 1, 2025

mcm001 Jan 1, 2025

PeterJohnson commented Jan 1, 2025

mcm001 commented Jan 1, 2025

mcm001 commented Jan 2, 2025

		// (Note: the frame time and time source are overridden by RawSourceImpl, and changing this is
		// spooky)

[cscore] Use frame time in Linux UsbCameraImpl #7609

Are you sure you want to change the base?

[cscore] Use frame time in Linux UsbCameraImpl #7609

Conversation

mcm001 commented Dec 30, 2024 • edited Loading

PeterJohnson commented Dec 30, 2024 • edited Loading

mcm001 commented Dec 30, 2024 • edited Loading

mcm001 commented Dec 30, 2024

mcm001 commented Dec 30, 2024 • edited Loading

mcm001 commented Dec 30, 2024 • edited Loading

mcm001 commented Dec 30, 2024

mcm001 commented Dec 30, 2024 • edited Loading

auscompgeek commented Dec 30, 2024

mcm001 commented Jan 1, 2025

PeterJohnson commented Jan 1, 2025

mcm001 commented Jan 1, 2025

mcm001 Jan 1, 2025

Choose a reason for hiding this comment

PeterJohnson commented Jan 1, 2025

mcm001 commented Jan 1, 2025

mcm001 commented Jan 2, 2025

mcm001 commented Dec 30, 2024 •

edited

Loading

PeterJohnson commented Dec 30, 2024 •

edited

Loading

mcm001 commented Dec 30, 2024 •

edited

Loading

mcm001 commented Dec 30, 2024 •

edited

Loading

mcm001 commented Dec 30, 2024 •

edited

Loading

mcm001 commented Dec 30, 2024 •

edited

Loading