Bacchylides 5.160-162 (said by Meleager in Hades, tr. David A. Campbell):

Best for mortals never to be born, never to set eyes on the sun's light.

θνατοῖσι μὴ φῦναι φέριστον,
μηδ᾽ ἀελίου προσιδεῖν
φέγγος.

Euripides, fragment 285, lines 1-2 (from the play Bellerophon, tr. Christopher Collard and Martin Cropp):

I myself affirm what is of course a common word everywhere, that it is best for a man not to be born.

ἐγὼ τὸ μὲν δὴ πανταχοῦ θρυλούμενον
κράτιστον εἶναι φημὶ μὴ φῦναι βροτῷ.

Euripides, fragment 908, line 1 (from an unknown play, tr. Christopher Collard and Martin Cropp):

Not to be born is better than life for mortals.

τὸ μὴ γενέσθαι κρεῖσσον ἢ φῦναι βροτοῖς.

Aristotle, fragment 44 Rose (from the dialogue Eudemus, or On the Soul, preserved by [Plutarch], Consolation to Apollonius 27 = Moralia 115 C-E, tr. Frank Cole Babbitt):

"And in addition to this you observe how the saying, which is on the lips of all men, has been passed from mouth to mouth for many years." "What is this?" said he. And the other, again taking up the discourse, said: "That not to be born is the best of all, and that to be dead is better than to live. And the proof that this is so has been given to many men by the deity. So, for example, they say that Silenus, after the hunt in which Midas of yore had captured him, when Midas questioned and inquired of him what is the best thing for mankind and what is the most preferable of all things, was at first unwilling to tell, but maintained a stubborn silence. But when at last, by employing every device, Midas induced him to say something to him, Silenus, forced to speak, said: 'Ephemeral offspring of a travailing genius and of harsh fortune, why do you force me to speak what it were better for you men not to know? For a life spent in ignorance of one's own woes is most free from grief. But for men it is utterly impossible that they should obtain the best thing of all, or even have any share in its nature (for the best thing for all men and women is not to be born); however, the next best thing to this, and the first of those to which man can attain, but nevertheless only the second best, is, after being born, to die as quickly as possible.' It is evident, therefore, that he made this declaration with the conviction that existence after death is better than that in life."

"καὶ ταῦθ᾽ οὕτως ἀρχαῖα καὶ παλαιὰ παρ᾽ ἡμῖν, ὥστε τὸ παράπαν οὐδεὶς οἶδεν οὔτε τοῦ χρόνου τὴν ἀρχὴν οὔτε τὸν θέντα πρῶτον, ἀλλὰ τὸν ἄπειρον αἰῶνα διατελεῖ νενομισμένα. πρὸς δὲ δὴ τούτοις διὰ στὸματος ὂν τοῖς ἀνθρώποις ὁρᾷς καὶ ἐκ πολλῶν ἐτῶν περιφέρεται θρυλούμενον." "τί τοῦτ᾽;" ἔφη. κἀκεῖνος ὑπολαβών "ὡς ἄρα μὴ γενέσθαι μέν" ἔφη "ἄριστον πάντων, τὸ δὲ τεθνάναι τοῦ ζῆν ἐστι κρεῖττον. καὶ πολλοῖς: οὕτω παρὰ τοῦ δαιμονίου μεμαρτύρηται. τοῦτο μὲν ἐκείνῳ τῷ Μίδᾳ λέγουσι δήπου μετὰ τὴν θήραν ὡς ἔλαβε τὸν Σειληνὸν διερωτῶντι καὶ πυνθανομένῳ τί ποτ᾽ ἐστὶ τὸ βέλτιστον τοῖς ἀνθρώποις καὶ τί τὸ πάντων αἱρετώτατον, τὸ μὲν πρῶτον οὐδὲν ἐθέλειν εἰπεῖν ἀλλὰ σιωπᾶν ἀρρήκτως: ἐπειδὴ δέ ποτε μόγις πᾶσαν μηχανὴν μηχανώμενος προσηγάγετο φθέγξασθαί τι πρὸς αὐτόν, οὕτως ἀναγκαζόμενον εἰπεῖν 'δαίμονος ἐπιπόνου καὶ τύχης χαλεπῆς ἐφήμερον σπέρμα, τί με βιάζεσθε λέγειν ἅὑμῖν ἄρειον μὴ γνῶναι; μετ᾽ ἀγνοίας γὰρ τῶν οἰκείων κακῶν ἀλυπότατος ὁ βίος. ἀνθρώποις δὲ πάμπαν οὐκ ἔστι γενέσθαι τὸ πάντων ἄριστον οὐδὲ μετασχεῖν τῆς τοῦ βελτίστου φύσεως ἄριστον ὰρα πᾶσι καὶ πάσαις τὸ μὴ γενέσθαι: τὸ μέντοι μετὰ τοῦτο καὶ πρῶτον τῶν ἀνθρώπῳ ἀνυστῶν, δεύτερον δέ, τὸ γενομένους ἀποθανεῖν ὡς τάχιστα.' δῆλον οὖν ὡς οὔσης κρείττονος τῆς ἐν τῷ τεθνάναι διαγωγῆς ἢ τῆς ἐν τῷ ζῆν, οὕτως ἀπεφήνατο."

---

Analogous is the idea that one should mourn when a child is born and rejoice when a man dies, e.g. Herodotus 5.4 (tr. A.D. Godley):

The Trausi, who in all else conform to the customs of other Thracians, do as I will show at the times of birth and death. When a child is born, the kinsmen sit around it and lament all the ills that it must endure from its birth onward, recounting all the sorrows of men. The dead, however, they bury with celebration and gladness, asserting that he is rid of so many ills and has achieved a state of complete blessedness.

τούτων δὲ τὰ μὲν Γέται οἱ ἀθανατίζοντες ποιεῦσι, εἴρηταί μοι: Τραυσοὶ δὲ τὰ μὲν ἄλλα πάντα κατὰ ταὐτὰ τοῖσι ἄλλοισι Θρήιξι ἐπιτελέουσι, κατὰ δὲ τὸν γινόμενόν σφι καὶ ἀπογινόμενον ποιεῦσι τοιάδε: τὸν μὲν γενόμενον περιιζόμενοι οἱ προσήκοντες ὀλοφύρονται, ὅσα μιν δεῖ ἐπείτε ἐγένετο ἀναπλῆσαι κακά, ἀνηγεόμενοι τὰ ἀνθρωπήια πάντα πάθεα: τὸν δ᾽ ἀπογενόμενον παίζοντές τε καὶ ἡδόμενοι γῇ κρύπτουσι, ἐπιλέγοντες ὅσων κακῶν ἐξαπαλλαχθεὶς ἐστὶ ἐν πάσῃ εὐδαιμονίῃ.

and Euripides, fragment 449 (from the play Cresphontes, tr. Christopher Collard and Martin Cropp):

We would do better to assemble and bewail a newborn child for all the troubles he is entering, and when a man dies and has his rest from hardships to see him from his home with joy and cries of gladness.

ἐχρῆν γὰρ ἡμᾶς σύλλογον ποιουμένους
τὸν φύντα θρηνεῖν εἰς ὅσ᾽ ἔρχεται κακά,
τὸν δ᾽ αὖ θανόντα καὶ πόνων πεπαυμένον
χαίροντας εὐφημοῦντας ἐκπέμπειν δόμων.

Latin translation of Euripides' lines by Cicero, Tusculan Disputations 1.48.115:

nam nos decebat coetus celebrantis domum
lugere, ubi esset aliquis in lucem editus,
humanae vitae varia reputantis mala;
at, qui labores morte finisset gravis,
hunc omni amicos laude et laetitia exsequi.

---

On the persistence of this idea, Robert J. O'Hara writes in an email:

A verse that appears on a few 18th-century gravestones of children is one that derives ultimately, through several poetical intermediates, from those lines of Theognis. It's striking because it's a very non-Puritan sentiment to find in these late-Puritan New England burying grounds.

I attach two photos here of the gravestone of Nathaniel Thurston of Fitchburg, Massachusetts, who died 24 January 1772, aged one day. The verse reads:

Happy the babe Who privileg'd by fate
To shorter labor & a ligter Weight
Recev'd but yesterday ye gift of breath
Ordored tomarrow to return to death

So there's a direct line from Archaic Greece to early New England through these brief lines.