The liver is a complex immunological organ. It has both immunogenic and tolerogenic capacity. Tolerogenic potential of human liver with its protective firewalls is required to guard the body against the continuous influx of microbial product from the gut via the sinusoids and biliary tree. Immunotolerance and anergic state is maintained by a combined effort of both immune cells, parenchyma cells, epithelial and endothelial cells. Despite this, an unknown trigger can ignite the pathway towards breakdown in hepatic tolerance leading to autoimmune liver diseases. Understanding the initial stimulus which causes the hepatic immune system to switch from the regulatory arm towards self-reactive effector arm remains challenging. Dissecting this pathology using the current technological advances is crucial to develop curative immune based therapy in autoimmune liver diseases. We discuss the hepatic immune cells and non-immune cells which maintain liver tolerance and the evidence of immune system barrier breach which leads to autoimmune hepatitis, primary biliary cholangitis and primary sclerosing cholangitis.