Article 4 of the U.N. Charter:
Membership in the United Nations is open to all other [emphasis added] peace-loving states which accept the obligations contained in the present Charter and, in the judgement of the Organization, are able and willing to carry out these obligations.
In Article 4, notice the word other, which is meant to distinguish from the original founding members (1945).
1) be a state (peace-loving)
2) accept the obligations in the Charter
3) The organization has to judge that the applicant state is able and willing to carry out these obligations
The first ten years of the United Nations produced a lot of disputes about criteria and vetoes were caste. This stopped in 1955 when a package deal was reached: each side of the Cold War had their applicants (allies) admitted.
In 1948, a question arose:
When a members state voted to admit an applicant, could the consider other factors (other than Article 4) in reaching a decision? This question was a result of states voting no on admittance and claiming their vote was contingent on whether or not their allies would be admitted as members. The question was sent to the International Court of Justice (ICJ). The ICJ said no-- other factors cannot be considered.
Members states must base their decision on the criteria contained in Article 4 (whether in the General Assembly or the Security Council).
The Rules of Procedure
1) The Security Council must recommend the admission of the applicant state
2) Following recommendation from the SC, the application goes the General Assembly
This is contained in Article 4(2)
A positive recommendation is needed from the SC. This is the same in other areas: picking a Secretary General, attempts to alter the number of permanent SC seats. The process is documented in the Security Council Provisional Rules of Procedure.
Chapter 10 Rules 58-60
The Secretary General receives the application and sends it to the Security Council. The rules provide that the Council with refer the matter to a committee, which is normally the case.
This is referred to a "Committee of the Whole" -- like most SC committees. A Committee of the Whole is all fifteen members. This committee on new members recommend or does not recommend. The process then goes to the parent organ -- the Security Council. A resolution is usually put into the Committee report and the Council will adopt that resolution and send it the the GA for deliberation. In practice, the GA always votes positively when the SC recommends admission. If the SC doesn't recommend the applicant, the Council (once the info is received from the committee) must make a special report to the GA. It will document that they have reviewed the application but that there is no recommendation for admission. If the SC does not get the required 9 votes with no veto, a report still goes to the GA.
General Assembly Rules of Procedure
Chapter 14 Admission of New Members Rules 134-138
Rule states if the Council does not recommend or if the consideration is postponed, the GA may send, after full consideration of the report, the application back to the SC with a full record of the General Assembly discussion-- for the Security Council's further consideration
In the early years of the U.N., these rules resulted in a back and forth between the the SC and GA. Over the last three decades, this has not been the practice-- it has not been contentious. It is important to note that the U.N. Charter does not have a mandate to recognize states. The recognition of states is up to governments. However, rival governments have attempted use the U.N. process to further a particular agenda.
One notable example of recognition within the UN framework was General Assembly Resolution 2758, where the People's Republic of China was recognized as "the only legitimate representative of China to the United Nations." The PRC assumed its position as one of the permanent five members of the Security Council.
How did it start?
Non-member states asked the Secretary General for facilities to observe the work of the General Assembly.
For example, the Swiss in the late 1940's. Observer states have a nameplate and an office. The Secretary General is the guardian / custodian of treaties. The SG keeps a list of which states are parties and which agreements have been ratified, signed, and /or acceded). Most treaties are open to all states. This is to encourage participation universally. The next question to ask is the following: How does the SG know if an entity is a state or not? Neither the UN or the SG has a mandate to recognize states.