In my previous post, Why truncating a target table is a bad practice in ETL?, I strongly suggest that we do not truncate our target tables.
In this blog, I’ll tell you the worst possible way I’ve seen to truncate a target table.
Before I get into that, I’ll share with you the experience a business user has shared with me:
….Sherry, can you check the report again? It has failed with the new data load today again.
….What caused it to do that? Has been happening sporadically lately where the whole report just goes down.
….I thought the issue has been resolved since last change.
Before the “last change”, the report has no staging data. It’s pulling large amount of data directly from our data source via OPENQUERY (you can read my other posts to know my opinions about OPENQUERY). If you are lucky, the report will send back data within 5 minutes.
So the last change was made is to pull data every morning and store it in a table.
Here is how this table (in my opinion, this is your target table, not a staging table) is populated every morning:
It is truncated first, then with an attempt to populate it. Unfortunately, almost 5 out of 10 days, the OPENQUERY failed, leaving the target table empty 5 days out of 10. In those 5 lucky days, the target table will remain empty for about 10 minutes, which is how long it takes for the table to be fully populated.
That explains the business users’ experience with the report.
Do you still need more convincing not to truncate your target tables?
