Machine Learning for Classifying Malware in Closed-set and Open-set Scenarios