Epilepsy is a life-threatening and challenging neurological disorder, which is affecting a large number of people all over the world. For its detection, encephalography (EEG) is a commonly used clinical approach, but manual inspection of EEG brain signals is a time-consuming and laborious process, which puts a heavy burden on neurologists and affects their performance. Several automatic systems have been proposed using traditional approaches to assist neurologists, which perform well in detecting binary epilepsy scenarios e.g. normal vs. ictal, but their performance degrades in classifying ternary case e.g. ictal vs. normal vs. inter-ictal. To overcome this problem, we propose a system that is an ensemble of pyramidal one-dimensional convolutional neural network (P-1D-CNN) models. Though a CNN model learns the internal structure of data and outperforms hand-engineered techniques, the main issue is the large number of learnable parameters, whose learning requires a huge volume of data. To overcome this issue, P-1D-CNN works on the concept of refinement approach and it involves 61% fewer parameters compared to standard CNN models and as such it has better generalization. Further to overcome the limitations of the small amount of data, we propose two augmentation schemes. We tested the system on the University of Bonn dataset, a benchmark dataset; in almost all the cases concerning epilepsy detection, it gives an accuracy of 99.1±0.9% and outperforms the state-of-the-art systems. In addition, while enjoying the strength of a CNN model, P-1D-CNN model requires 61% less memory space and its detection time is very short (< 0.000481 sec), as such it is suitable for real-time clinical setting. It will ease the burden of neurologists and will assist the patients in alerting them before the seizure occurs. The proposed P-1D-CNN model is not only suitable for epilepsy detection, but it can be adopted in developing robust expert systems for other similar disorders.