One has to be careful during a git add, else you may end up adding files that you didn’t want to commit. However, git rm will remove it from both your staging area (index), as well as your file system (working tree), which may not be what you want.
Instead, use git reset:
git reset filename # or
echo filename >> .gitingore # add it to .gitignore to avoid re-adding it
This means that git reset <paths> is exactly the opposite of git add <paths>.