Just because software is open source does not mean someone is actually looking at the code. But depending on the software there are incentives to do so. Some people might be technologically interested on the way a software does something and look at the source code for that. Some people might want to check the benignity for themselves and actively check the source code for malicious features. With community maintained software there are often many different independent people working on the software. Also many open source software projects allow code commits to the software. Many eyes on the software due to many people working on it increases the chance of malicious features or vulnerabilities being discovered. A great thing about FOSS is the possibility to fork it or to use the FOS software of someone else in your software. FOSS allows and even encourages everyone to work with the software of others for ones own purpose and to modify, adapt or embed it. This leads to more people having an eye on the source code just for purely practical purposes. Open source just means publishing the source code, but FOSS is about actively reusing, improving and adapting other people's work in your own work. Security researchers might also have a look on open source software purely for their own research. Another great important aspect are bug bounties. Many developers pay bounties to people who report vulnerabilities to them. That creates an incentive to audit the code. But obviously not every project, especially smaller ones, have bug bounty programs. But you could probably sponsor one for some software you like.
Lastly there are independent third party audits. Those can be done for a number of reasons. There can be community paid audits through donations. VeraCrypt had one for example. Then there might also be other organizations who want to use the software and have an interest in its security. VeraCrypt is also an example for that. The German government paid the Frauenhofer Institute for an audit of VeraCrypt.
In the end it comes down to the specific software. If someone implements a malicious feature in their software it is not necessarily going to be found just because the source code is open. If you find some random unknown software it is not secure just for being open source, but the chance of malicious features or vulnerabilities being discovered is definitely higher if it is possible to look for them in the first place.
Security critical software should be open source and audited.
This work is licensed under CC BY-SA 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0/