The early Fathers taught that baptism is a key means God uses to convey his grace to us. They recognized the Bible's teaching that In the ark a few (eight persons) were saved through water. Baptism, which corresponds to this, now saves you, not as a removal of dirt from the body but as an appeal to God for a clear conscience, through the resurrection of Jesus Christ.

Letter of Barnabas
Regarding baptism, we have the evidence of Scripture that Israel would refuse to accept the washing which confers the remission of sins and would set up a substitution of their own instead.

Hermas
'I have heard, sir,' said I, 'from some teacher, that there is no other repentance except that which took place when we went down into the water and obtained the remission of our former sins.' He said to me, 'You have heard rightly, for so it is'.

Justin Martyr
Whoever is convinced and believe that what they are taught and told by us is the truth, and professes to be able to live accordingly, are instructed to pray and to beseech God in fasting for the remission of their former sins, while we pray and fast with them. Then they are led by us to a place where there is water, and they are reborn in the same kind of rebirth in which we ourselves were reborn: 'In the name of God, the Lord and Father of all, and of our Savior Jesus Christ, and of the Holy Spirit,' they receive the washing of water. For Christ said, 'Unless you be reborn, you shall not enter the kingdom of heaven'.

Theophilus of Antioch
Moreover, those things which were created from the waters were blessed by God, so that this might also be a sign that men would at a future time receive repentance and remission of sins through water and the bath of regeneration all who proceed to the truth and are born again and receive a blessing from God.

Clement of Alexandria
When we are baptized, we are enlightened. Being enlightened, we are adopted as sons. Adopted as sons, we are made perfect. Made perfect, we become immortal 'and sons of the Most High' - Ps. 81:6. This work is variously called grace, illumination, perfection, and washing. It is a washing by which we are cleansed of sins, a gift of grace by which the punishments due our sins are remitted, an illumination by which we behold that holy light of salvation.

Tertullian
Happy is our sacrament of water, in that, by washing away the sins of our early blindness, we are set free and admitted into eternal life. But a viper of the Gnostic Cainite heresy, lately conversant in this quarter, has carried away a great number with her most venomous doctrine, making it her first aim to destroy baptism which is quite in accordance with nature, for vipers and asps generally do live in arid and waterless places. But we, little fishes after the example of our Great Fish, Jesus Christ, are born in water, nor have we safety in any other way than by permanently abiding in water. So that most monstrous creature, who had no right to teach even sound doctrine, knew full well how to kill the little fishes by taking them away from the water! Baptism itself is a corporal act by which we are plunged into the water, while its effect is spiritual, in that we are freed from our sins - ibid., 7:2.

Augustine
It is an excellent thing that the Punic North African Christians call baptism salvation and the sacrament of Christ's Body nothing else than life. Whence does this derive, except from an ancient and, as I suppose, apostolic tradition, by which the Churches of Christ hold inherently that without baptism and participation at the table of the Lord it is impossible for any man to attain either to the kingdom of God or to salvation and life eternal? The sacrament of baptism is most assuredly the sacrament of regeneration. Baptism washes away all, absolutely all, our sins, whether of deed, word, or thought, whether sins original or added, whether knowingly or unknowingly contracted. This is the meaning of the great sacrament of baptism, which is celebrated among us: All who attain to this grace die thereby to sin as he himself, Jesus is said to have died to sin because he died in the flesh that is, 'in the likeness of sin' and they are thereby alive by being reborn in the baptismal font, just as he rose again from the sepulcher. This is the case no matter what the age of the body. For whether it be a newborn infant or a decrepit old man--since no one should be barred from baptism--just so, there is no one who does not die to sin in baptism. Infants die to original sin only; adults, to all those sins which they have added, through their evil living, to the burden they brought with them at birth.

Jerome
This much you must know, that baptism forgives past sins, but it does not safeguard future righteousness, which is preserved by labor and industry and diligence and depends always and above all on the mercy of God.

INFANT BAPTISM
Peter explained what happens at baptism when he said, Repent, and be baptized every one of you in the name of Jesus Christ for the forgiveness of your sins; and you shall receive the gift of the Holy Spirit - Acts 2:38. But he didn't restrict this teaching only to adults. He added, For the promise is to you and to your children and to all that are far off, every one whom the Lord our God calls to him. We also read: Rise and be baptized, and wash away your sins, calling on his name - Acts 22:16. These commands are universal, not restricted to adults. The connection between baptism and salvation is clear, not only because of these passages, but because of other explicit passages such as 1 Peter 3:21: Baptism now saves you, not as a removal of dirt from the body but as an appeal to God for a clear conscience, through the resurrection of Jesus Christ.

Other Benefits of Baptism
Along with this forgiveness of sins comes an infusion of grace. It is this grace that makes the soul spiritually alive and capable of enjoying heaven. There are other benefits, too, such as the elimination of punishment due for sins and the right to special graces necessary to enable the baptized to fulfill his baptismal promises. But a consideration of these aspects of the sacrament would take us away from the narrow topic this tract is about:, infant baptism.
Jesus said that no one can enter heaven unless he has been born again of water and the Holy Spirit - John 3:5. His words can be taken to apply to anyone capable of having a right to his kingdom. He asserted such a right even for children: Let the children come to me, and do not hinder them; for to such belongs the kingdom of heaven - Matt. 19:14.
More detail is given in Luke's account of this event, which reads: Now they were bringing even infants to him that he might touch them; and when the disciples saw it, they rebuked them. But Jesus called them to him, saying, Let the children come to me, and do not hinder them; for to such belongs the kingdom of God - Luke 18:15-16.

In Place of Circumcision
Furthermore, Paul notes that baptism has replaced circumcision - Col. 2:11-12. In that passage, he refers to baptism as the circumcision of Christ and the circumcision made without hands. Of course, it was mainly infants who were circumcised under the Old Law; circumcision of adults was rare, there being few converts to Judaism. If Paul, in making this parallel, meant to exclude infants from baptism, he would have said so.

Biblical References
But, one might ask, does the Bible ever say that infants or young children can be baptized? The indications are fairly clear. Lydia was converted by Paul's preaching. She was baptized, with her household - Acts 16:15. The Philippian jailer whom Paul and Silas had converted to the faith, who had been about to commit suicide when they were miraculously freed from their imprisonment, was baptized that night along with his household. We are told that the same hour of the night . . . he was baptized, with all his family - Acts 16:33. And in his greetings to the Corinthians, Paul recalled that, I did baptize also the household of Stephanas - 1 Cor. 1:16.