I like to use the vim editor in an xterm for editing .m files. If you don't launch Matlab via an xterm in X11, then Matlab doesn't know to open an xterm.
One workaround is to create a launchvim.sh script as a text file:
launchvim.sh
#!/bin/bash
export DYLD_LIBRARY_PATH="/opt/X11/lib:$DYDL_LIBRARY_PATH"
/opt/X11/bin/xterm -e vim $1
Simply save this in /Applications and give everyone permission to run by running chmod 755 /Applications/launchvim.sh. Then, in Preferences in Matlab, go to Editor, and enter /Applications/launchvim.sh as your text editor.