FOT-Net Data Webinar: “Performance indicators, Study design and Measures and Sensors”, 19 November 2014

The objective of this second webinar was to transfer knowledge about the FESTA methodology for designing and conducting Field Operational Tests. The FESTA handbook can be downloaded from: http://fot-net.eu/Documents/festa-handbook-version-5-2014/.

Yvonne Barnard, of the Institute for Transport Studies of the University of Leeds and seconded to ERTICO, presented the FESTA “V”[1]. The FESTA handbook establishes the process to guarantee the best quality possible during each step when designing and conducting a FOT or NDS.). The handbook went through a series of thorough revisions; the most recent one, revision 5, also includes NDS, cooperative systems etc.

Yvonne Barnard talked about performance indicators (PI), experimental procedures and sensors, and defining and arranging things to get the vehicles out on the road and collect data.. Yvonne highlighted that PIs can be both objective (e.g. messages transmitted, speed, lateral position) and subjective (e.g. acceptance, workload, willingness to pay). PIs are usually designed for comparison because often the hypotheses are also defined in a qualitative way. From definition to measures, Yvonne explained five different types of measures, namely direct measures (raw data, e.g. distance to lead vehicle), indirect measures (pre-processed data, e.g. time-to-collision), events (e.g. overtaking manoeuvre), self-reported measures (e.g. questionnaire), and situational variables (e.g. weather data).

Yvonne highlighted that identifying crash relevant events is difficult in practice as no specific recommendations or commonly agreed trigger values are available. She presented four commonly used methods for the identification, which are driver response, safety function response, driving context and driving history, recommended that a combined approach could be the most appropriate.

Yvonne presented a matrix in which PIs, measures and sensors are all related to each other and informed that there is a link in the FESTA handbook where readers can find the matrix excel sheet. PIs can also related to impact areas such as driving performance and safety (e.g. driver’s glance, speed), system performance and influence on driver behavior (e.g. false alarms, interaction system-driver), environmental aspects (e.g. exhaust emissions, fuel consumption), traffic efficiency (e.g. travel time, mean speed), acceptance and trust (e.g. perceived usefulness, rate of use).

When talking about experimental procedures, Yvonne emphasized that a number of issues need to be covered for example participants, study design, experimental environment, and that conducting a pilot study to test the evaluation process is absolutelty necessary. Sometimes controlled testing is the best method, for example for testing cooperative systems. Experimental rigour and scientific quality always need to be ensured.

Sensors are the instruments to help find out what we want to know and how to measure it. Yvonne explained different types of sensors to be used for different purposes and mentioned some specific sensors (e.g. video, vehicle-bus data, head/eye trackers, radar, map matching).

Yvonne finished her presentation by reiterating that the FESTA steps are not always that easy and need to consider iteration, resources, practicalities, ethical & legal objections, and data analysis issues. She reassures the audience that it is complicated but they can get help from the FOT implementation plan which is detailed in the FESTA handbook.

The webinar was concluded by a series of questions as listed below:

1. The experimental procedures in an FOT take time and resources to be carried out. If my project has limited budget, which of these elements (e.g. participants, study design, experimental environment, a pilot study to test the evaluation process, controlled testing) should I focus first? Yvonne responded that the most important thing is to be sure what you want to know if you have limited resources. You can limit your research questions, simplify hypotheses, and make sure that you don’t run into too many technical problems. Perhaps you can start with easy or cheap solutions for example by using of smartphones.
2. How has the FOT methodology been implemented in practice such as in the CIP projects? Is the FOT implementation plan too general or too specific? Yvonne said that “we’ve actually hear back from more specialized projects which followed part of the methodology. They take what is useful to them and follow a systematic approach. If you go to the FOT-Net website library (http://fot-net.eu/library/), you will find all the previous stakeholder meetings, international workshops, and seminars. You will find all kind of experiences that people report back from those projects. We try to collect and store those experiences and make them available for everyone so that we can learn from each other”.

Finally, webinar participants were invited to join the next webinar that will be dedicated to ethical and legal issues and data gathering, 26 November 2014, by Helena Helena Gellerman at SAFER.

[1] http://wiki.fot-net.eu/index.php?title=FESTA_handbook#FESTAV

Webinar presentations:

– Download the introduction from Haibo Chen
– Download the presentation from Yvonne Barnard