Evidence doesn't have to be physical evidence. Witness testimony as to the witness' observations and perceptions is evidence as well. If someone says you did something illegal, that person's word alone can be sufficient for the police to arrest you. The law even says that testimony of just a single witness, if believed beyond a reasonable doubt by the judge or jury, is sufficient to prove a fact. So if someone is accusing you of a crime, you should take that accusation seriously and talk to an attorney about your options.