Alcohol abuse is more common, and abusers usually do not suffer from physical withdrawal symptoms if they abstain. Alcohol abuse can take the form of binge drinking, i.e. consuming large quantities of alcohol occasionally, or more moderate drinking that is associated with unhealthy habits or repercussions. An example would be a person who drinks to “self-medicate” for depression, insomnia, etc. Another example would be someone who alienates him or herself from family and friends by behaviors that are offensive, hurtful, or obnoxious. Serious examples would include dangerous behaviors like drunk driving, or losing a job or loved one due to inability to control one's drinking. Even without the obvious physical symptoms of alcohol dependence, alcohol does have cumulative affects on the body, mostly the liver, which can lead to serious health problems.
How much is too much is hard to quantify. That depends on cultural norms; it is considered acceptable and normal in some cultures to have one or more drinks daily. From a physiologic standpoint, how one tolerates alcohol depends on gender, size, frequency of use, and other factors such as how one's body processes alcohol. My advice to this questioner is to take a step back and try to evaluate the situation without being defensive. If you, or someone who cares about you, has concerns about your drinking, you should take it seriously. It may be worth cutting it out to prevent progression of a problem and perhaps improve your relationships.