Fungal growth in damp or water-damaged buildings worldwide is an increasing problem, which has adverse effects on both the occupants and the buildings. Air sampling alone in moldy buildings does not reveal the full diversity of fungal species growing on building materials. One aim of this study was to estimate the qualitative and quantitative diversity of fungi growing on damp or water-damaged building materials. Another was to determine if associations exist between the most commonly found fungal species and different types of materials. More than 5,300 surface samples were taken by means of V8 contact plates from materials with visible fungal growth. Fungal identifications and information on building material components were analyzed using multivariate statistic methods to determine associations between fungi and material components. The results confirmed that Penicillium chrysogenum and Aspergillus versicolor are the most common fungal species in water-damaged buildings. The results also showed Chaetomium spp., Acremonium spp., and Ulocladium spp. to be very common on damp building materials. Analyses show that associated mycobiotas exist on different building materials. Associations were found between (i) Acremonium spp., Penicillium chrysogenum, Stachybotrys spp., Ulocladium spp., and gypsum and wallpaper, (ii) Arthrinium phaeospermum, Aureobasidium pullulans, Cladosporium herbarum, Trichoderma spp., yeasts, and different types of wood and plywood, and (iii) Aspergillus fumigatus, Aspergillus melleus, Aspergillus niger, Aspergillus ochraceus, Chaetomium spp., Mucor racemosus, Mucor spinosus, and concrete and other floor-related materials. These results can be used to develop new and resistant building materials and relevant allergen extracts and to help focus research on relevant mycotoxins, microbial volatile organic compounds (MVOCs), and microparticles released into the indoor environment.