There are three ways you can edit behaviors, depending on the underlying representation of your logic. Sadly, most of these solutions are not production ready, and if you want to use these things in a commercial game, you may well have to write your own editor.
Full Programming Languages
It’s possible to build a visual editor for standard programming languages based on lambda calculus. Alice is a research project doing just that; it removes the syntax from the typical scripting process.
The advantage is that programmers will have little trouble getting to terms with the process. On the down side, designers who have no idea about lambda calculus will struggle with the same level of complexity than programmers face every day.
Finite State Machines
Alternatively, you can use finite state machines to simplify your logic — and make the editor easier too. There are some simple open-source tools available for editing state machines:
However, it’s not very easy to use these tools to create large behaviors for games. You’ll no doubt need something more AI specific.
Behavior trees are becoming a standard for the games industry, used in many studios like Microsoft Game Studios (Bungie) and Rockstar Games. A good description of the technology is Damian Isla’s paper on Handling Complexity in the Halo 2 AI.
These are simple to create with tree editors, however, there is no such tool available publically. Many studios, however, roll their own rather quickly.