Institute of Cognitive Science, Osnabrück University, Wachsbleiche 27, DE-49090 Osnabrück, Germany.

---

Abstract—Multiple View Geometry (MVG) with its underlying mathematical principle is mainly used for 3D reconstruction. The most common approaches based on MVG are the Structure from Motion (SfM) methods which create 3D point clouds from a collection of images or video frames. The emerging use of artificial neural networks (ANNs) in almost every domain leads to the question, if ANNs can learn the underlying mathematical geometric mappings of SfM pipelines? To answer this question, three different ANN architectures based on the three different key point matching strategies of SfM were benchmarked. Since we want to learn the mathematical, geometrical mapping of SfM approaches and not the categories or shapes of natural 3D objects, we trained and tested our ANNs on 2D projections of random 3D shapes built from small random cubes. For 3D shapes with a grid size of 〈3,3,3〉 voxels, all architectures show a high prediction accuracy of the reconstructed shape. When scaling up the grid size of the 3D cubes, we recognize a significant decrease in accuracy. These initial results show that all of the different ANN architectures we considered can learn to reconstruct unknown 3D shapes from images. In a more detailed analysis of our results, we investigate how the choice of architecture influences the prediction accuracy of the 3D shape on voxel and overall shape level and if non-occluded voxels can be predicted independently of scale. Finally, we discuss, if a voxel-based representation of the 3D shape can be scaled to a useful technical resolution due to its high impact on the size of the ANN as well as the required training data.

Index Terms—3D reconstruction, artificial neural networks, multiple view geometry, structure from motion.

[PDF]

Cite: Julius Schöning, Gunther Heidemann, Ulf Krumnack, "Structure from Neuronal Networks (SfN²)," Journal of Computers vol. 13, no. 8, pp. 988-999, 2018.