Generally to create behaviors for your AI actors (not necessarily realistic ones), you have two main choices:
Edit all the behaviors manually, micromanaging the AI with a set of scripts for example.
Use an AI algorithm to create the behaviors automatically for you, either with machine learning or planning.
To make those behaviors realistic you need to find the best combination of technology and expert design in this spectrum.
Problems with Extremes
Theoretically, you could simply have a large database created by designers, which would store all the necessary solutions to problems and reactions to events. However, this isn’t feasible in practice, as you’d need a huge amount of behaviors for different environments and conditions.
Also, you could get your programmers to create some basic behaviors as building blocks, and let an automated AI algorithm find the correct combination for each situation. Sadly, this also causes problems:
Machine learning struggles with creating consistent behaviors; certain chosen behaviors just don’t make sense.
Planners on the other hand are more logical, but almost too much so. The resulting behaviors are very mechanical.
So, neither of the fully automated or fully manual solutions are acceptable for creating realistic behaviors.
Combining Design and Technology
The answer to the problem of realism is to combine AI with designer input. The designers edit certain aspects of the behavior that need to be realistic, while the AI planner plugs everything together mechanically. (Machine learning may be used to make things more efficient/effective if necessary.)
In pratice, the best way to achieve this is by using a hierarchical planner.