The primary responsibility is to the shareholders as they are the principal in the principal agent relationship. The shareholders agree the standards set and the auditor must report to them.
However there are two other groups that must be considered. Firstly the firm itself. For example an auditing team from PWC may be hired by Shell to audit a company in a region. The shareholders may set standards but PWC, but the actual team of auditors will have to report back to the firm of auditors, in this case PWC. They could have different instructions between them and the shareholders leading to conflicts such as the reporting of discrepancies. For example the team may find some issues with bribery of local Government officials, which they should report back to the shareholders, but the firm will tell them not to do this as it could endanger them with losing this lucrative account.
The other group who they have responsibility to is the Governments of the jurisdictions where the companies are based and the relevant audit and reporting laws present their. There could be differences between how shareholders will want actions interpreted and how the Government will, again in this case of local bribery, which could lead to conflicts of interest.