How do you tell?
I think you are confusing:
a . The command and control channels used by malware and criminals, which are not directly harmful to anyone other than to the real world owner (and users) of the system taken under virtual control by this means, with
b. The use to which the malware infected system is put, including fake money transfers, sending out spam email and creating high traffic levels on websites or using other protocols as part of a DDOS. Because criminals are greedy, usage b. will likely result in higher volume traffic, ( unless the infected machine is part of a very, very large and stealthy botnet, individual hosts of which are only used very lightly). Note that it is the unusual volume of activity (e.g. in relation to persistent requests for the same webpages or abnormally high outgoing email count) that makes an infected machine detectable to the ISP.
ISPs are only justified in taking down protocols (e.g. BT, IRC etc. ) typically used for C&C (relevant to activity a.) if they detect activity b. Activity b. while difficult to detect reliably, is relatively easier for an ISP to detect compared to activity a. Detecting activity b. does need some fairly advanced packet monitoring and firewall rules likely to need updating routinely, but occasional and intermittent encrypted C&C activity could be made almost indistinguishable from other traffic.
The ISPs could certainly get better at detecting malicious patterns of activity by forming better collaboration channels for data sharing with other ISPs, e.g. so information about current DDOS endpoints and attack characteristics are more widely publicised, enabling more accurate idenfication of zombies participating in DDOSs. But better data sharing requires better trust metrics amongst ISPs to enable criminal ISPs to be disconnected sooner rather than later and this hasn't proved achievable in such a rapidly expanding 'wild west' business environment.
As to transparent reporting of problems with individual hosts, spamhaus.org do a good job here (they keep between 600 and 1400 spams out of my network per week), but you only discover a problem when something gets blacklisted by them. ISPs are not transparent because they don't like to give too much information away about the details of the monitoring they are carrying out. The reasons for this reticence seem obvious; they don't want to give attackers the information needed in real time to get around their defences, e.g. by enabling attackers to slow down a DDOS to prevent this getting blocked one infected machine at a time.