Open source tools are free to use frameworks and applications. Engineers build the tool and have the source code available for free on the internet for other engineers to use.
Vendor tools are developed by companies that come with licenses to use, and often cost money. Since they are developed by an outside source, technical support is often available for use. Example vendor tools include WinRunner, SilkTest, Rational Robot, QA Director, QTP, LR, QC, RFT, and RPT.
An In-house tool is a tool that a company builds for its own use, rather than purchasing vendor tools or using open source tools.