When talking about storing any piece of data, our first response is to throw it in a Key Vault. But like any aspect of system design, throwing everything into the same box, or into a smattering of boxes that lie here, there, and everywhere, eventually leads you to having a mess where there is no organization to how and where information is stored.
Think of your key vault like this? Does this seem organized?

Note: I don’t know whose shed this is, if it’s yours, sorry, by the end of the summer, mine are much worse.
With anything related to the structure of code, architecture, or solution, there is no one way to implement the structure. The problem becomes when everyone starts to implement their own structure, and then we fall into chaos.
If you’re trying to think of how to organize your key vaults, here are some ideas to get you started that might help. However, like any well-organized shed, the importance of implementation comes from the maintenance of the shed.
In picking a structure for your key vaults, the process is pretty straightforward;
- Identify a structure.
- Implement the structure.
- Follow the structure.
Number 3 is the killer for those that don’t like following structure, and this doesn’t mean you will never change this structure, of course you will, but what it does mean is that everyone has to agree on the change (and generally that’s where people fall off).
In the following options, I have not taken into account how you have designed your Azure Network Infrastructure; each of these models should lend itself to the security of how your systems are structured, so you are not getting into cross-pollination of code and work.
Environment
There is no One-Size-Fits-All model for Key Vaults. Every environment should have its own dedicated Key Vault; if someone gets access to one, the rest are fine. While your development team might need access to your Dev Key Vault (great), they shouldn’t have access to any other environment going up the line.
This protection in security is for the company as much as it is for you.
Breaking out Key Vaults by Environment, apply to every option.
Function and Environment
Depending on the setup of your organization, if you have external services you offer customers and internal services for back-end systems, it might make sense to break up your Key Vaults by Function and User.
I.e., What systems are connecting to this data, and what do they need to see that I don’t want to worry about?
If you have a set of partner APIs that users are connecting through, this might be a further separation of keys.
The Function Key Vault strategy should be closely tied to how your Azure network is set up. If you have an external access network with its own set of security rules, you don’t want to be adding the Azure Key Vault into this group for internal services into this group and vice versa.
Platform and Environment
Depending on what systems or architecture you have in place, organizing your Key Vaults by Platform might allow you a greater separation and organization of who does what. If you are running many Power Apps App Registrations, Flows and Connections between systems, organizing your vaults by platform is an easy separation that gives your teams access to what they are working on and nothing else.
I.e., if I’m on the PowerApps team, I get access to the Key Vaults that source Power Apps and Dynamics. I don’t need access to internal apps or API key vaults.
Technology and Environment
If I’m running an Azure Data Lake, Azure SQL database, and other “data” related products, do I want to store all their keys in a vault that is combined with web access? Possibly not, so another way you can structure your Key Vaults is simply by the technology using them. Does this mean I create a separate Vault for APIM vs ASP Web Apps? I would think not, but it depends on your use case and scale for what you are working on and with.
I generally see the use of Technology and the Environment as a further delineation for large sites that are organizing key vaults by Platform to break things down further, if they need to go to that level.
When looking at the security of Key Vaults, everything is virtual, but that doesn’t mean we should apply the same structure and control we previously had in place when it came to servers and other systems, when people had to walk into a room to get access.
You can start by throwing your world into one key vault, with some proper naming and tagging strategies that everyone follows, and you will get there. If you want to go a step further in breaking things down, you can do that well add some additional layers of security into the fold to further secure your valuable secrets.