Check the official list of countries which citizens (and stateless persons permanently residing in the countries) are able to enter and/or stay in Georgia, or travel through the territory of Georgia withouta visa for the period of one year. If your country is on the list, then your passport will be stamped on the border. No other formalities are needed.

If your country is not listed, go to the the official Georgian e-visa site and follow instructions. The process is also described in a short and fun movie:

If allowed into the country, by Georgian law you can only visit regions within Georgian territory. Meaning that you can only visit Abkhazia, as the border to South Ossetia is closed. For visiting Abkhazia you will need an Abkhazian visa, but it should not be glued into your passport.
Please read these featured articles: How to get Abkhazia from Georgia, Abkhazian Visa and South Ossetia Visa.

Get more details about the Georgian visa at the official Ministry of Foreign Affairs of Georgia website.