Image processing-based fire/flame detection has become popular in recent years. In this paper, a novel fire/flame detection system based on a new conversion matrix and artificial bee colony algorithm was presented. Flame and non-flame image pixel values were combined to have a new feature matrix. A conversion matrix was generated randomly. The conversion matrix was multiplied by the feature matrix. The error of this multiplication result was calculated using theK-means clustering algorithm. The conversion matrix was updated until getting desired performance using artificial bee colony algorithm. At the end of the updating process, updated conversion matrix was multiplied with all images in the dataset to move all images to new color space. The final images were converted into binary images. Otsu method was used to get binary images. These binary images were compared with the corresponding ground truth images in the dataset. The aim of this comparison is to calculate the similarity ratio of the two images. This ratio shows the extent to which the original image features are preserved. A forest fire dataset was used which has 500 forest fire images. It is publicly available and called as Corsican Fire Database. Jaccard and Dice similarity measure parameters were used to evaluate the proposed system performance and compared with other similar study such as particle swarm optimization. Evaluated mean Jaccard index value was 0.76, and mean Dice index value was 0.85. This evaluation was made for 500 images. These results provide that this system can be used in fire/flame detection systems.