How many messages are you reading per second?
I had a few problems with my spout originally but it was either because
1) was not acking the messages and because of max pending they weren't been thrown away from the "queue"
2) buffer size and fetch size was too small: have you tried to figure out how many bytes you write from Kafka and increase the sizes to that size? this helped in my case.
3) was trying to read too far from the past when I restarted the topology so ended up consuming only latest offset.
With the above tweaks I was able to increase my throughput to 9 times more..it obviously depends on size of messages but this helped me..
as Haralds suggested, have a look at the dashboard and try to understand where the problem is..