Quick answer: Baptism in the Spirit is the giving of the Spirit, and it happens at conversion. Water baptism is a meaningful ritual that symbolizes the giving of the Spirit and our conversion, and we should do it right away after conversion.
Long answer: Baptism is derived from washing images and rituals in the Old Testament. The OT prophets often promised that a new age was coming, called the new covenant or the messianic age. They eagerly awaited not only the coming of the Messiah, but also the giving of God’s Spirit on all of God’s people. The prophets often used water as a picture of the cleansing work of the Spirit. It would be a cleansing of the inner person, not just the ritual cleansing practiced under the Law (see esp. Ezek 36:25-27; also Isa 32:15-17, 44:3, Ezek 39:28-29, Joel 2:28-29, Zech 12:10, 13:1).
When John the Baptist began his ministry, he baptized with water. He described this as “a baptism of repentance for the forgiveness of sins.” His baptism also prepared the people for the “coming one,” who would baptize with the Spirit (Mk 1:4-8). That is, John’s water baptism was a picture of the giving of the Spirit from the Messiah. So “baptism with the Spirit” is a metaphor for receiving the Spirit. (By the way, there is no difference in meaning between the phrases by/in/with the Holy Spirit.)
But Jesus did not immediately give the Spirit. John tells us that the giving of the Spirit had to wait until after Jesus’ “glorification” – that is, his death, resurrection and ascension (John 7:37-39). This giving of the Spirit (= the baptism of the Spirit) happened first at the day of Pentecost (or maybe in the upper room, but I’ll save that for another post). Even the disciples had not yet received the Spirit, because they had to wait until Jesus’ atoning death.
Although most Christians no longer use the term this way, in the New Testament, “baptism with the Holy Spirit” is another picture of conversion. When we come to Christ, we receive the Holy Spirit, and are therefore baptized with the Holy Spirit (1 Cor 12:13; see also Rom 8:9, 14, 2 Cor 1:21-22, Gal 3:2-6, 4:6). Later in life, we may have many experiences where we sense the work of the Spirit through us. Acts describes these as “being filled with the Spirit” (Ac 4:8, 4:31, 13:9-10, and others).
After the death and resurrection of Jesus Christ, the church began to practice water baptism immediately upon conversion (Ac 2:41). The water ritual served as a picture of the Spirit baptism that was taking place at conversion. Immersing in water also became a picture of the believer’s union with Christ, dying and rising with him (Col 2:12).
Do we need to get baptized with water? For a full answer, see this post. A quick answer: we don’t need baptism for salvation, but the NT commands it, and it should be a ritual of celebrating our new life in Christ. You can go simple and just sign a marriage contract to get married – but why not have a wedding to celebrate?
Another common view: Pentecostals sometimes use the phrase “baptism in the Spirit” to refer to a significant experience of the Spirit later in one’s Christian life, accompanied by speaking in tongues. They use the term this way because they think all believers should have the same experience as the first disciples: belief in Jesus, and then some time later, baptism of the Spirit with speaking in tongues (as at Pentecost). However, if you read through Acts, you can see that the pattern of Pentecost is never exactly repeated, and is probably a unique historical event like the resurrection of Jesus. Further, the rest of the New Testament never indicates that baptism of the Holy Spirit is a later event, or that it is accompanied by speaking in tongues, or that all believers should speak in tongues.