This research introduces a conceptual framework for enforcing privacy-related restrictions in smart video surveillance systems based on danger levels and incident types to be handled. It increases the selectivity of surveillance by restricting data processing to individuals associated to incidents under investigation. Constraints are enforced by usage control, which is instantiated for video surveillance for the first time and enables tailoring such systems to comply with data protection law.