The ancient Hellenes viewed the Earth as a round disk divided into equal parts by the Mediterranean Sea and the (Black) Sea (first called the Inhospitable Sea by Pindar, then the Hospitable Sea once the shores became inhabited). Okeanos, a mystical river, flowed around the entire disk, and mysterious peoples—the Hyperboreans in the north, the Ethiopians in the far south and the Kimmerians in parts unknown—lived outside Okeanos' perimeter. 

The ancient Hellenes also believed that the heaven was made up of celestial spheres. The celestial spheres, or celestial orbs, were the fundamental entities of the cosmological models developed by great philosophers and astronomers like Plato, Aristotle, and Ptolemy. It was believed that the stars were fixed and did not change their positions relative to one another. As such, it was argued that they must be on the surface of a single starry sphere. The apparent motions of the fixed stars and the planets are accounted for by treating them as embedded in rotating spheres made of an aetherial, transparent fifth element (quintessence).

According to the Homeric hymns, the ancient Hellenes also believed that there are columns that keep apart earth and heaven. It's most likely these columns or pillars were mountains in the centre of the disk, or at the edge of the disk. It were these pillars that Atlas guarded, or he was the actual pillar who held up the spheres--and thus the actual mountain. As Hesiod writes in the Theogony

"[At the ends of the earth, where lie the roots of earth, sea, Tartaros :] There stands the awful home of murky Nyx wrapped in dark clouds. In front of it [Atlas] the son of Iapetos stands immovably upholding the wide heaven upon his head and unwearying hands." [744]

Now we are on the topic, it also depends upon tradition if Atlas held up the celestial orbs as punishment, or if he was chosen to do it as an honour to him. The punishment part is perhaps best described by Hyginus in his Fabulae:

"After Juno [Hera] saw that Epaphus, born of a concubine, ruled such a great kingdom, she saw to it that he should be killed while hunting, and encouraged the Titanes to drive Jove [Zeus] from the kingdom and restore it to Saturn [Kronos]. When they tried to mount to heaven, Jove with the help of Minerva [Athene], Apollo, and Diana [Artemis], cast them headlong into Tartarus. On Atlas, who had been their leader, he put the vault of the sky; even now he is said to hold up the sky on his shoulders." [150]

Hómēros, in his Odysseia, takes an opposite--or at least milder--view, implying the honour in Atlas' position:

"Atlas the baleful; he knows the depths of all the seas, and he, no other, guards the tall pillars that keep the sky and earth apart." [1.52]

It makes sense, in fact, to have holding up the heavens be an honour: if Atlas slacks even a moment, the celestial spheres would crash down onto Earth and destroy it, killing everyone. It's a huge responsibility, and to give that to someone pissed off for losing a war seems foolish. 

Like many of the myths concerning the ancient Titans, there are versions of mythology where Atlas no longer bears the heaven. In this case: Pindar describes it as such in the Pythian Odes:

"Does not even now great [Titan] Atlas struggle to bear up the weight of heaven, far from his fathers’ land and his possessions? But almighty Zeus set free the Titanes, for as time passes and the breeze abates, the sails are set anew." [4]

The Farnese Atlas is a 2nd-century Roman marble copy of a Hellenistic sculpture of Atlas kneeling with the celestial spheres, not a globe, weighing heavily on his shoulders. It is the oldest extant statue of Atlas, as well as the oldest known representation of the celestial sphere. The globe shows a depiction of the night sky as seen from outside the outermost celestial sphere, with low reliefs depicting 41 (some sources say 42) of the 48 classical Hellenic constellations distinguished by Ptolemy. The sphere is made up of solid marble, and contains no actual stars. This may have aided the formation of the common misconception that Atlas was forced to hold the Earth on his shoulders. I hope this post clears up a bit of the confusion.