Improving the effectiveness of fishways requires a better understanding of fish behavior near hydraulic structures, especially of upstream orientation. One of the most promising approaches to this problem is the use of model behavioral rules. We developed a three-dimensional individual-based model based on observed brown trout (Salmo trutta fario) movement in a laboratory flume and tested it against two hydraulically different flume setups. We used the model to examine which of five behavioral rule versions would best explain upstream trout orientation. The versions differed in the stimulus for swim angle selection. The baseline stimulus was positive rheotaxis with a random component. It was supplemented by attraction towards either lower velocity magnitude, constant turbulence kinetic energy, increased flow acceleration, or shorter wall distance. We found that the baseline stimulus version already explained large parts of the observed behavior. Mixed results for velocity magnitude, turbulence kinetic energy, and flow acceleration indicated that the brown trout did not orient primarily by means of these flow features. The wall distance version produced significantly improved results, suggesting that wall distance was the dominant orientation stimulus for brown trout in our hydraulic conditions. The absolute root mean square error (RMSE) was small for the best parameter set (RMSE = 9 for setup 1, RMSE = 6 for setup 2). Our best explanation for these results is dominance of the visual sense favored by absence of challenging hydraulic stimuli. We conclude that under similar conditions (moderate flow and visible walls), wall distance could be a relevant stimulus in confined space, particularly for fishway studies and design in IBMs, laboratory, and the field.