Introduction

As more people move to online shopping, it becomes increasingly important for clothing brands to allow for a better online shopping experience. Especially challenging is the fact that it is not possible to try on clothes and understand how they look on oneself before making the buying decision. To solve that issue, an increasing number of brands are trying to create virtual try-on experiences. This requires both a 3D model of the person trying on the clothes, as well as accurate 3D models of the items of clothing.

Our client – a start-up in it’s pre-IPO phase – contacted OctoShrew to develop such a 3D reconstruction system to go along with their already developed try-on system for scanned 3D models to bring their technology to a wider audience.

Example image from which a 3D model of the jacket should be constructed.

Previous research has gone into accurate reconstruction of 3D clothes, with one of the most promising open source models being PIFuHD – created by facebook. It’s performance was not sufficient for the client largely because of three main factors:

PIFuHD often produces outliers (blobs that do not belong to the person) It is unable to generate colour - especially relevant since the back of the jacket also needs to be rendered The jacket still needs to be extracted from the rest of the 3D model Results The results achieved by applying PIFuHD are shown below. Despite vastly increased performance over the default performance on real world data in our model, it is not possible to get rid of some of the outlier artifacts and reconstruction inaccuracies.

PIFuHD reconstruction (before improvement)

PIFuHD reconstruction (after improvement)

With slight modifications to the open source model and the addition of a new texturing method as well as new post-processing steps to ensure more consistent performance, the project concluded with us being able to consistently achieve the 3D reconstruction quality seen below, which to our knowledge goes beyond the current state of the art methods developed by other research teams. Most notably, no similar accuracy in texture reconstruction can be found in comparable technologies.

Final Result of Reconstruction