Augmenting neural networks with additional knowledge

The last few years have demonstrated that neural networks are remarkably good at processing and drawing conclusions from raw data like images or text.

An unfortunate side effect of this is that higher level knowledge on problems and inputs has progressively fallen to the wayside as end-to-end model become ever more widespread. Indeed, modern neural networks are now relying on enormous amounts of training data to train very large models to go from raw inputs to the final decision.

This high level knowledge does however provide significant information that could complement the internal knowledge the network extracts from raw data. Why train on a vast corpus of noisy code when documentation is ready and available? Why force the model to learn species characteristic from scratch when we already have detailed taxonomies available? Is it truly necessary for an autonomous car to guess the surrounding world when HDMaps are on hand?

As such, I am currently invested in exploring how standard “perception based” neural networks can be made to accept and benefit from additional high level knowledge.

Work on the matter:

• Preprint paper “Mind the map! Accounting for existing map information when estimating online HDMaps from sensor data”. In this work, we propose a way to incorporate information on existing maps in online HDMap estimation and show this leads drastic improvements of performance.
• Workshop paper “Exploring the Road Graph in trajectory forecasting” in the SG2RL workshop at ICCV 2023. This is a preliminary study on the effect of various interventions on the road graph representation with regards to forecasting performance.