I'm guessing they're basing this dismissal of their own drawing abilities upon an assumption that drawing is an observational skill only - if their drawing doesn't resemble the still life, or life model, or a picture copied from a book - that it has no worth.
Recently I was contributing to the running of the social media of a small publishing firm. It was so inspiring to be with a group including a successful newspaper editor, and a publisher - all with a huge amount of publications under their belts. Their technical chat meant I was unclear at times. Typically for me, I asked them to visualise the discussion. I was met with horror, and the 'Oh we can't draw' statement.
When I clarified I was looking for a sketch, a schematic, a doodle, a working drawing - not something that looked like the end product newspaper or book, the following drawn layouts were sketched out in seconds. I was immediately impressed - and said so. I thought these drawings communicated exactly the information that I needed to know - therefore they were good drawings.
My colleagues remain bemused that I think so.