This is happening because when usePrefillCheckoutContactForm finds cached contact details, it updates the state via setStepCompleteStatus and causes the form to change from ready to validating which is how the WPCheckoutOrderSummary determines whether the cart is loading.

Ideally, we can change it so it only loads when products change. Since the total also relies on the same isCartUpdating logic and is affected by contact information (tax), it's probably safer to make the change for the CheckoutSummaryFeaturedList only. I'll look more closely at the feature list to make sure it doesn't rely on contact information as well before.

Noting, I opened a separate ticket for the contact details section loading. I'm hoping it may be possible to prefill contact information before page load to avoid this loading stage entirely but either way this should probably be separated out.