We want to monitor index bloat every hour, but the data set is getting to large to handle in Excel. This is because the table is adding 100,000+ rows of data an hour. We have created a view on this data that reduces the data to one row per day, but that means that after 10 days worth of data we have hit Excel’s row limit of 1,048,576 rows and can no longer use the pivot table and graphing in Excel.
This left me with a problem of how to look at this data when I did not have an application that could deal with this much data.
In reading some presentations on Data Warehousing and Postgres, I saw people adding R into the Data Warehouse.