Someone is spamming using my email address

[#1]

I am getting very regular "address not known" type emails into my catch all email folder. The sender address changes but it is always under my @username.plus.com email address. Emails are being sent to random names that are unfamiliar to me (so can't be hacking into my address book - I've no virus!)
Ive double checked other aspects of my email and I'm pretty sure I've not been hacked or anything, I'm just a bit concerned about the amount of folk who are getting spam seemingly from me!
I'm guessing there isn't much we do about folk using our email addresses to spam from - other than just delete returned mail and hope it all goes away soon.
Any advice?

[#2]

What you are seeing is known as backscatter which some mail servers unfortunately regard as being helpful. The article explains some of the sources from which spammers harvest their 'sender' addresses. Since in your case the spammers are using random local parts (before the @) you could turn off catchall if you don't otherwise make use of it.
Hopefully it will stop of its own accord before long when the spammers move to a different (forged) email address.

[#3]

To keep your email INBOX tidy, as an alternative to switching off your 'catch all' folder -
Depending on which email client you use,  you could set up an incoming email filtering rule which looks for -
    "Subject:"  contains  "address not known"
and any messages that match that rule, you set to delete when received, or move to a special email folder.
Again, depending on your email client, you could refine the filter rule to only delete messages that DON'T contain the email addresses that you actually use, and therefore will only tidy away the randomly generated addresses.

[#4]

I got hit by major backscatter once and ended up with almost 1000 emails an hour being bounced back to me.
I had to disable my catchall account for a few days until it all calmed down.

[#5]

Thanks for your replies.
I can see there is not a lot I can do about it other than filter the server bounced email. flippin spammers