An application ran slower than before and the application only invoked a procedure.
After checking I found:
- One section about inserting data into temp table(activity_ID 2) ran more than 50 times and each time of running took more than 120s, even, the largest run time would be about 350s. The reason of more running times is the procedure code is changed and the insert statement is generated by while loop. The updated while condition generated more insert statements than before. I got all of the information by checking the procedure create time and the code management tool P4V.
- There are large number of latch waits during the procedure running.
Running time information:
The latch wait information:
Latch name is SQLO_LT_SQLB_DIRTY_LIST_SET__appendLatch. I guess it is about operating the dirty page list in a buffer pool. I cannot get any information useful about this latch in google. Latch holder EDU name is db2pfchr, which is about the prefetcher. So the whole logic would be:
- Large number of inserting operation increased the dirty pages in the buffer pool.
- Prefetcher read pages into buffer pool.
- Buffer pool page was not enough.
- Dirty page cleaning up was triggered by the prefecher and the latch is hold by this EDU.
- Other applications inserted or read data into the buffer pool but found no page left either, dirty page cleaning up was also triggered but the dirty list latch was already hold by prefetcher. The latch wait occurred.
Based on all of above and the statement is about inserting date into temp table, I guessed there may be I/O issue about the temp table space and its buffer pool.
Buffer pool information for user temp table space:
- Async write occupied only 19.87%.
- Async write time is 27.27ms.
User temp table space information:
- Average write time is 43.62ms whereas the average read time is only 0.13ms.
So, the final reason includes:
- too many inserting operation makes the buffer pool full
- buffer pool is too small to hold all of the pages .
For the first reason, the procedure code should be changed and for the second reason, the buffer pool should be extended.
Questions:
- There may be any I/O error for the disk? No. because other table spaces share the same paths as the temp table spaces and they did not meet any I/O issue.
- Why did I decide to check the procedure creating time? I forget but when I was handling this issue some logic made me check that. Maybe I thought there is no problem for this procedure before so I guess if there is any change about the procedure.