We examine the effects of paired delayed excitatory and inhibitory feedback on a single integrate-and-fire neuron with reversal potentials embedded within a feedback network. These effects are studied using bifurcation theory and numerical analysis. The feedback occurs through modulation of the excitatory and inhibitory conductances by the previous firing history of the neuron; as a consequence, the feedback also modifies the membrane time constant. Such paired feedback is ubiquitous in the nervous system. We assume that the feedback dynamics are slower than the membrane time constant, which leads to a rate model formulation. Our article provides an extensive analysis of the possible dynamical behaviors of such simple yet realistic neural loops as a function of the balance between positive and negative feedback, with and without noise, and offers insight into the potential behaviors such loops can exhibit in response to time-varying external inputs. With excitatory feedback, the system can be quiescent, can be periodically firing, or can exhibit bistability between these two states. With inhibitory feedback, quiescence, oscillatory firing rates, and bistability between constant and oscillatory firing-rate solutions are possible. The general case of paired feedback exhibits a blend of the behaviors seen in the extreme cases and can produce chaotic firing. We further derive a condition for a dynamically balanced paired feedback in which there is neither bistability nor oscillations. We also show how a biophysically plausible smoothing of the firing function by noise can modify the existence and stability of fixed points and oscillations of the system. We take advantage in our mathematical analysis of the existence of an invariant manifold, which reduces the dimensionality of the dynamics, and prove the stability of this manifold. The novel computational challenges involved in analyzing such dynamics with and without noise are also described. Our results demonstrate that a paired delayed feedback loop can act as a sophisticated computational unit, capable of switching between a variety of behaviors depending on the input current, the relative strengths and asymmetry of the two parallel feedback pathways, and the delay distributions and noise level.