Street scenes and pedestrian intent for a driver-assistance developer

The client develops perception software for driver-assistance and automated-driving systems and licenses it to vehicle and truck manufacturers. Its software has to find and follow every road user around a vehicle, in cities that look nothing like the test routes it was first built on.

It began working with us in September 2019 to label drives from cities its existing data barely covered, and now sends us drives from four continents.

Dense, mixed traffic is hard to label. A single junction can hold minibuses that stop anywhere, motorcycles carrying passengers or goods, handcarts, livestock, cyclists and pedestrians walking in the road, many of them partly hidden behind one another. Each has to be boxed in every keyframe and keep the same identity through the clip, including when it disappears behind a bus and comes out the other side.

The client also needed pedestrian intent: whether a person at the kerb is waiting, about to cross, crossing or walking along the road. Intent cannot be read from one frame. A labeller has to watch several seconds, and two careful people can disagree.

The work has to keep pace with the client’s test fleet, with clips back within three working days, at night and in rain as well as in daylight.

Team

About 120 labellers, four quality reviewers and two team leads, across both centres, with a project lead in Toronto. A little under half of the team works through Rungwell at Home, in Kiambu, Machakos and Nakuru and in Bulacan, Rizal and Cavite.

Tools

Clips are about twenty seconds long. In Rungwell Tasks labellers box each road user on keyframes every half-second and the tool interpolates between them; the labeller corrects the frames where the interpolation drifts. Each object keeps its track number through the clip, and every box carries occlusion and truncation flags. Pedestrian intent is labelled as an event with a start and end time. The street frame on our Image and video page is drawn from this project’s screen.

Guidelines

The guideline is in its fourth major version. Version 1 (2019) covered 14 classes in daylight; version 2 (2021) added night, rain and tracking; version 3 (2023) added pedestrian intent; version 4 (2025) brought the taxonomy to 38 classes. Version 4.2 came into force in May 2026.

Quality

Gold clips make up 3% of each queue. The whole team, including Rungwell at Home labellers by video, calibrates every Thursday. Reviewers check at least 5% of keyframes in every batch and all of a labeller’s first week. Boxes must overlap the reference by at least 0.85 (intersection over union), and tracking is measured by identity switches per 1,000 frames, against a target of no more than one.

Remote work

Until 2023 the client allowed its work to be done only in our centres. When the centres closed to labellers in March 2020, it paused the project for eleven weeks rather than let its data leave them. The client let a small group of Rungwell at Home labellers work on the project during the Focus Assurance pilot, from December 2022 to March 2023, its first work outside our centres; when Focus Assurance was introduced, it agreed to Rungwell at Home for the project as a whole. Items completed while a session’s focus score is below 70 are reviewed a second time before delivery.

We have delivered 244,650 accepted clips since 2019. Identity switches have fallen from 1.4 per 1,000 frames when tracking began in 2021 to 0.6 this year, and the share of items the client returns has fallen in every year, from 2.60% in 2019 to 0.88% in 2026.

Opening the project to Rungwell at Home let it grow without new desks, and let us offer the work to labellers who cannot travel to Kilimani or Cubao. 48% of its labelling hours this year were worked from home, and items done at home are returned no more often than items done on site.

Many of the project’s labellers travel to work by matatu and jeepney through the kind of traffic they label, and the guideline’s classes for minibuses, handcarts and loaded motorcycles were written with their help. Its chapter on night scenes was revised for version 4 with Rungwell at Home labellers in Kiambu, who had asked for clearer examples.

Since 2023 the project has been one of the largest sources of work for Rungwell at Home, including for parents of young children and for labellers who live too far from either centre to commute.