ReSyRIS
The Real-Synthetic Rock Instance Segmentation dataset (ReSyRIS) is created for training and evaluation of rock segmentation, detection and instance segmentation in (quasi-)extra-terrestrial environments. It consists of a set of annotated, real images of rocks on a lunar-like surface, a precisely mimicked synthetic version thereof, and respective synthetic assets for training data generation.
Details
The Real-Synthetic Rock Instance Segmentation dataset (ReSyRIS) is created for training and evaluation of rock segmentation, detection and instance segmentation in (quasi-)extra-terrestrial environments. It consists of a set of real world images of rocks on lunar-like surfaces, and a precisely mimicked synthetic version thereof. Both versions include pixel-wise annotations of all rock instances. The assets necessary for creation of the synthetic test set (3D stone meshes and gravel texture) allow for generation of training data in varying settings.
Download
The dataset can be downloaded via Zenodo:
Real test set
The real test set was recorded during the ARCHES Demo-Mission in the valley between Cratere del Laghetto and La Cisternazza on Mt. Etna in Sicily, Italy. The environment there is very similar to the lunar surface. The test set consists of a collection of in total 36 rocks collected on-site, from which six to nine rocks are repeatedly placed in close and spread out configurations on the floor. For every configuration around 20 images are recorded. This process is repeated twelve times to collect in total 222 images.
The scenes were captured with a rc_visard 65 and two Manta G-201 monochrome cameras with a 8mm F1.8 lens. Data necessary for intrinsic and extrinsic calibration was recorded on-site. Every left RGB image of the rc_visard has been manually annotated to yield pixel-wise masks for each stone instance.
Synthetic test set
To create a perfect copy of the real data, textured 3D rock meshes as well as gravel texture are required.
The 3D rock reconstruction is done with photogrammetry in Meshroom. A Sony ILCE-7M4 with a 35mm F1.8 lens is used to capture images of the object slowly rotating on a white table in a well-lit environment. The camera height is adjusted several times during recording to capture various viewing angles. Around 200 images are collected for each sample. All images are first processed in Darktable, where exposure and white balance are corrected according to a gray card placed aside the rock sample. Every image is then cropped to the region of interest, and masked such that all background pixels are set to a uniform (white) value. This facilitates the matching process in Meshroom, since feature extraction should primarily focus on the rock. Finally, Meshroom's photogrammetry pipeline is employed to generate a high-quality textured mesh.
To extract the surface's gravel texture, the same Sony camera is used to record an approximately 25m2 gravel area in small patches from above (ca. 50cm distance). Given enough overlap between the images, the same photogrammetry pipeline can be reused, with one minor difference: Prior to reconstruction, a second set of images is exported in Darktable, with particular emphasis on contrast and sharpness in order to boost fine detail and aid the reconstruction. The first set is solely used at the very end for texturing.
For a realistic texture it is desirable to not only have RGB information available, but also capture the geometric structure of the surface. To this end, a displacement and normal map is extracted in Blender. This is done by using a decimate and shrinkwrap modifier: First, the high-polynomial mesh is decimated to an almost flat surface, followed by iterative subdivision alongside the shrinkwrap modifier, which constrains the surface to the original, undecimated mesh. This displacement from a low-face-count mesh to the high-polynomial mesh can then be captured into a displacement, normal and colour map.
To obtain the original camera poses as well as the initial poses of the stones for the synthetic test set, the photogrammetry pipeline is also applied on the real world test scenes. While the resulting meshes are of comparably low resolution, they are well suited to approximately align the scanned rocks with the real world test set positions. Then, a planar surface is created in OAISYS, onto which the generated gravel texture is applied. All rocks from a scene are then placed at the positions which were estimated before. Finally, the scene is rendered from the camera poses retrieved during photogrammetric reconstruction.
Synthetic training data
Aside mimicking real world test scenes in a synthetic environment, the previously created assets (3D rock meshes and gravel texture) are well suited to create synthetic training data in almost any variations. The dataset consists of exemplary config files to generate such data with OAISYS.
Related publications
Wout Boerdijk, Marcus Gerhard Müller, Maximilian Durner, und Rudolph Triebel, "ReSyRIS: A Real-Synthetic Rock Instance Segmentation Dataset for Training and Benchmarking", in: Proc. of 2023 IEEE Aerospace Conference, AERO 2023. IEEE. 2023 IEEE Aerospace Conference, Big Sky, USA, March 2023, Big Sky, USA, [elib]
Marcus Gerhard Müller, Maximilian Durner, Abel Gawel, Wolfgang Stürzl, Rudolph Triebel, und Roland Siegwart, "A Photorealistic Terrain Simulation Pipeline for Unstructured Outdoor Environments", in: Proc. of 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems, Prague, 27 Sep - 1 Oct 2021, [elib]
