Excitable cells and cell membranes are often modeled by the simple yet elegant parallel resistor-capacitor circuit. However, studies have shown that the passive properties of membranes may be more appropriately modeled with a non-ideal capacitor, in which the current-voltage relationship is given by a fractional-order derivative. Fractional-order membrane potential dynamics introduce capacitive memory effects, i.e., dynamics are influenced by a weighted sum of the membrane potential prior history. However, it is not clear to what extent fractional-order dynamics may alter the properties of active excitable cells. In this study, we investigate the spiking properties of the neuronal membrane patch, nerve axon, and neural networks described by the fractional-order Hodgkin-Huxley neuron model. We find that in the membrane patch model, as fractional-order decreases, i.e., a greater influence of membrane potential memory, peak sodium and potassium currents are altered, and spike frequency and amplitude are generally reduced. In the nerve axon, the velocity of spike propagation increases as fractional-order decreases, while in a neural network, electrical activity is more likely to cease for smaller fractional-order. Importantly, we demonstrate that the modulation of the peak ionic currents that occurs for reduced fractional-order alone fails to reproduce many of the key alterations in spiking properties, suggesting that membrane capacitive memory and fractional-order membrane potential dynamics are important and necessary to reproduce neuronal electrical activity.