Like many of us over the summer, we had a surprise after July 2nd, where our Dataverse search consumption skyrocketed. When I say sky and rocket, I mean growth not of 1 GB, but of 10, 20, and 30 GB, depending on how your environment is configured.

(Btw this was real)

The primary culprit is changes to the Dataverse Search to enable Generative AI capabilities that left you with only 3 options to reduce your mass growth in storage capacity.

On – keep running along and having to pay extra for data (this maintains your global search bar).

Default – your global search bar is hidden, but you can still do some AI stuff.

Off – no search bar, limited AI.

I didn’t test the bounds of the AI capabilities changes, but like many, having a global search for many years and then having to take it away wasn’t something we wanted to offer our users.

Taming the Index

I can’t remember if this was there before, but at the same time, I started noticing the following screens on my solutions that outlined the data consumption of indexes

When you click “Manage Search Index”, you get a more detailed view of what is happening in each entity and particularly how many columns are used in the index.

From here, if I wanted to keep global search, I would have to reduce my index consumption on my larger tables (i.e., contacts).

Indexes are not Search Enabled Fields

I made the first mistake when I looked at my entity fields and noticed (on contact for instance) that I had over 50+ search-enabled fields, which did not line up with the above 25. Digging deeper, I found that the search indexes are not tied to these fields but instead are tied to your Quick Find Views.

Staring at this view, I could only surmise that each field was being created as its own index; apply that across tables with 500K rows, and each one gets pretty big.

The Experiment

I don’t like making a ton of changes and “seeing” what happens, so I disabled a few columns, waited for the evening cycle to pass, and then monitored the results of these changes, which are displayed below. In my one contact table that I had 25 indexes, I removed 9 fields, bringing it down to 14.

You can see the drop in consumption, which gets this back to a more normal level.

Interesting to note, I had to wait 3 – 4 days to start seeing changes take effect and not the standard overnight fair.

I have a few entities to go through that are our higher runners, and hopefully, this will give us a cleaner view across all our environments, saving us some $$$ on storage overages.

Our users get to keep the search bar, and we clean up an invariably bloated view that might not need all those costly indexes on it.