Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple’s updated Vision framework, as explained in this WWDC 2020 session.
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.
Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple’s demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.
Additionally, apps could use the framework to overlay emoji or graphics on a user’s hands that mirror the specific gesture, such as a peace sign.
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.
The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.
Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.
Apple today announced a small but long-awaited change being made in iOS 14 and iPadOS 14: incoming phone and FaceTime calls will no longer take up the entire screen, providing for a much less disruptive experience. iOS 14 is available in beta for registered Apple developers today, with a public beta to follow next month. The software update will be released to all users with an iPhone 6s or …
Apple’s first-ever all-online Worldwide Developers Conference (WWDC) starts today with the traditional keynote kicking things off at 10:00 a.m. Pacific Time.
We’re expecting to see a number of announcements, including iOS 14, macOS 10.16, watchOS 7, and tvOS 14, as well as an announcement about Apple’s long-rumored transition of its Mac lineup from Intel processors to Apple-designed…
Apple today released a new firmware update for the AirPods Pro, upgrading the existing 2D15 firmware to 2D27.
No details are available on what’s included in the refreshed firmware at this time, but there have been ongoing complaints from AirPods Pro users about issues with Active Noise Cancellation and crackling or popping sounds.
Apple is also adding a new spatial audio feature to the…
Today’s digital WWDC keynote event was jam packed with announcements, with Apple debuting iOS 14, iPadOS 14, macOS Big Sur, watchOS 7, tvOS 14, and, most notably, its work on custom-designed chips that will be used in Macs in the future.
Subscribe to the MacRumors YouTube channel for more videos. For those that didn’t have a chance to watch the keynote event and don’t want to spend two hours…
Apple yesterday unveiled iOS 14, the newest version of the operating system designed for the iPhone, and we thought we’d go hands on with the new software to give MacRumors readers a sneak peek at all the features.
Subscribe to the MacRumors YouTube channel for more videos. iOS 14 brings some useful new interface changes, including a revamped Home screen. Widgets have been redesigned, and for …
Apple today previewed iOS 14, which features an all-new home screen design with widgets and a new App Library view, plus much more.
Widgets can be pinned in different sizes on any home screen page, providing useful information at a glance. Users can also create a Smart Stack of widgets, which uses machine learning to surface the right widget based on time, location, and activity….
Apple today previewed macOS Big Sur with redesigned interfaces such as the Dock and Notification Center, the addition of a Control Center, several improvements to core apps like Safari and Messages, and much more.
macOS Big Sur is available in beta for registered Apple developers starting today, with a public beta to follow next month. The free software update will be released to all users…