I ran a latency monitor for a while and it reported some problems. Did some digging around with Google and found posts about CPU Throttling causing latency problems. I was using the Windows 8.1 balanced power option, so I checked Processor Power Manager and found the minimum processor state was 5%. I changed the power option to High Performance and that solved the problem. No more audio drop outs.