Validity and reliability of naturalistic driving scene categorization judgments from crowdsourcing

Christopher D. Cabrall
Zhenji Lu
Miltos Kyriakidis
Laura Manca
Chris Dijksterhuis
Riender Happee
Joost de Winter

A common challenge with processing naturalistic driving data for many different possible driving research interests or applications is that humans may need to categorize great volumes of recorded visual information until automated algorithms might be trained to do so alone.

This study, by means of the online platform CrowdFlower, investigated the potential of crowdsourcing to provide content identification categorizations of driving scene features (e.g., presence of another vehicle, straight road segments, etc.) at greater scale than a single person or a small team of researchers would be capable of. The validity and reliability of CrowdFlower results were examined, both with and without employing a set of randomly embedded controlled questions (Gold Test Questions) intermixed with experimental questions (Work Mode). In total, 200 workers from 46 countries participated in this study, and the collection of data lasted one and a half days.

By employing Gold Test Questions, we found significantly more accurate and consistent responses from external workers at both a smaller and larger scale of video segment categorizations for the identification of common driving scene elements (e.g., position and behavior of other vehicles, road and signage characteristics, etc.). In terms of validity and at the small scale, an average accuracy of 91% on paired items was found with the controlled questions compared to 78% without. A difference in bias was found where without Gold Test Questions external workers returned more false positives than false negatives whereas the opposite was found true of the condition with Gold Test Questions. At the large scale (making use of the controlled questions), a random subset of categorizations returned similar levels of accuracy (95%) and a similar pattern of error bias. In terms of reliability and at the small scale, where segments were rated in triplicate redundancy, the percentage of unanimous agreement was found significantly higher when using controlled questions (90%) than without them (65%). Across the small scale of internally validated answers, more than two-thirds of any correct categorization were unanimously returned and 86% or more of any correct categorization was returned by a majority vote. Where it would be infeasible to validate every response for accuracy, similar voting reliability results were found to exist across the responses of the large scale.

Overall results support compelling evidence for CrowdFlower as being able to yield valid and reliable crowdsourced categorizations of naturalistic driving scene contents in a short period of time and thus a potentially powerful and as-of-yet under-utilized resource in the toolbox of driving research and driving automation development.

MEET US


25-26
Apr

Cost Benefit Analysis (CBA) workshop in Stockholm

An open seminar and workshop in Stockholm will be held on 25-26 April 2018. The workshop deals with the use of CBA as a basis for decision-making in the public sector. The workshop is organized by, among others, Professor Jan-Eric Nilsson, VTI.

LATEST NEWS


2018-02-19

Modal shift for an environmental lift?

Investigations in Sweden and other countries suggest a shift of goods transport from road to rail and waterborne transport to reach environmental and climate objectives. VTI is leading a new project to investigate how the modal shift can contribute and what...


2018-02-13

Automation and digitalisation are making rail competitive

Road transport is developing rapidly and its productivity has increased sharply. Rail transport, however, has not developed at the same rate. Automation and digitalisation are essential if rail freight in Europe is to survive.


2018-02-08

New research is creating a driverless logistics chain

The research project Born to Drive has come up with a system that allows new cars to move, without a driver, from the production line out to the parking area prior to being transported elsewhere. The vision is to automate the entire logistics chain from...


2018-02-05

VTI testing automation in EU project

VTI is leading a series of tests in a major EU project on automated driving. The first driving tests were carried out n a test track in Slovenia in December. The project will focus in part on acceptance among different groups in society, in part on...


2018-02-02

Freight transportation on road and rail analysed

Freight transport accounts for a large proportion of the emissions, noise and congestion produced by road traffic. Transporting freight in larger but fewer lorries could reduce the problem. At the same time it might entail freight being diverted from more...


2017-11-30

Millions for research into maritime transport and the environment

Maritime transport is a major source of emissions of harmful air pollutants and carbon dioxide. In a new project, a research team from the Swedish National Road and Transport Research Institute (VTI) and the University of Gothenburg has received SEK 6.4...