Cartographic generalization aims at decreasing map or database detail. On one hand, its goal is taking into account map purpose, user constraints and needs, on the other hand maintaining and highlighting characteristic spatial patterns. One of the main challenges in the research concerning cartographic generalization is the evaluation of its results. While previous studies have exclusively concentrated on quantitative evaluation of cartographic generalization results, we complement these studies by considering both quantitative and qualitative evaluation with the map designers and map users. In this pilot study, six participants were asked to analyze both maps manually designed by experienced cartographers and mapsautomatically generalized with the use of selected machine learning and deep learning models, namely random forest (RF), deep learning (DL), decision trees (DT) and decision trees optimized with genetic algorithms (DTGA). Based on four tasks and two datasets containing: source settlements, manually (human) and automatically generalized ones to smaller scales the users had to identify important settlement patterns and judge if the result was machine or human design. The experiment was conducted with the use of a dedicated web application. Additionally, eye-tracking data were recorded using a Tobii X2-30 eye-tracker. The preliminary results, as shown in Figure 1, suggest that the generalization results that successfully keep the specific settlement patterns are: 1) the automated results (AI generalization) with the use of random forest (RF) and deep learning (DL), and 2) the reference atlas map, designed by experienced cartographers (human generalization). In this preliminary study, participants found the decision tree (DT) results the least successful for maintaining the specific settlement patterns.