how to correctly limit number of goroutines

Question

I'm getting in 'stdin' lines of URL's like: $ echo -e 'https://golang.org\nhttps://godoc.org\nhttps://golang.org' | go run 1.go . The task is to get from each WEB-page number of word "Go". But I'm not allowed to start more than 5 goroutines and can use only standard library Here is my code:

    package main

    import (
      "fmt"
      "net/http"
      "bufio"
      "os"
      "regexp"
      "io/ioutil"
      "time"
    )

func worker(id int, jobs<-chan string, results chan<-int) {
  t0 := time.Now()
  for url := range jobs {
    resp, err := http.Get(url)
    if err != nil {
      fmt.Println("problem while opening url", url)
      results<-0
      //continue
    }
    defer resp.Body.Close()
    html, err := ioutil.ReadAll(resp.Body)
    if err != nil {
      continue
    }
    regExp:= regexp.MustCompile("Go")
    matches := regExp.FindAllStringIndex(string(html), -1)
    t1 := time.Now()
    fmt.Println("Count for", url, ":", len(matches), "Elapsed time:", 
t1.Sub(t0),  "works id", id)
    results<-len(matches)
  }
}

func main(){
  scanner := bufio.NewScanner(os.Stdin)
  jobs := make(chan string, 100)
  results := make(chan int, 100)
  t0 := time.Now()
  for w:= 0; w<5; w++{
    go worker(w, jobs, results)
  }
  var tasks int = 0
  res := 0
  for scanner.Scan() {
      jobs <- scanner.Text()
      tasks ++
  }
  close(jobs)
  for a := 1; a <= tasks; a++ {
    res+=<-results
  }
  close(results)
  t2 := time.Now()
  fmt.Println("Total:",res, "Elapsed total time:", t2.Sub(t0) );
}

I thought it works until I passed more than 5 URL (one of them was incorrect) to stdin. The output was:

 goroutine 9 [running]:
 panic ...

Obviously, extra goroutnes have been started. How to fix it? May be there are more convenient way to limit number of goroutines?

Answer 1

goroutine 9 [running]:

Some goroutines are started by the runtime, and by web fetches.

Looking at your code, you only started 5 goroutines.

If you really want to know how many go routines you are running use runtime.Numgoroutine