TPU VM access Cloud Storage 403 forbidden when writing files

Question

When I run my python command to train my model on my tpu-vm, it failed on writing files to Cloud Storage.

Traceback (most recent call last):
File "device_train.py", line 302, in <module>
  save(network, step, bucket, model_dir,
File "device_train.py", line 62, in save
  with open(f"gs://{bucket}/{path}/meta.json", "w") as f:
File "/usr/local/lib/python3.8/dist-packages/smart_open/smart_open_lib.py", line 235, in open
  binary = _open_binary_stream(uri, binary_mode, transport_params)
File "/usr/local/lib/python3.8/dist-packages/smart_open/smart_open_lib.py", line 398, in _open_binary_stream
  fobj = submodule.open_uri(uri, mode, transport_params)
File "/usr/local/lib/python3.8/dist-packages/smart_open/gcs.py", line 105, in open_uri
  return open(parsed_uri['bucket_id'], parsed_uri['blob_id'], mode, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/smart_open/gcs.py", line 146, in open
  fileobj = Writer(
File "/usr/local/lib/python3.8/dist-packages/smart_open/gcs.py", line 427, in __init__
  self._resumable_upload_url = self._blob.create_resumable_upload_session()
File "/usr/local/lib/python3.8/dist-packages/google/cloud/storage/blob.py", line 2728, in create_resumable_upload_session
  _raise_from_invalid_response(exc)
File "/usr/local/lib/python3.8/dist-packages/google/cloud/storage/blob.py", line 3936, in _raise_from_invalid_response
  raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.Forbidden: 403 POST https://storage.googleapis.com/upload/storage/v1/b/my-bucket/o?uploadType=resumable: {
"error": {
  "code": 403,
  "message": "Access denied.",
  "errors": [
    {
      "message": "Access denied.",
      "domain": "global",
      "reason": "forbidden"
    }
  ]
}
}
: ('Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.CREATED: 201>)

I have two service accounts, one is like [email protected], and the other is like [email protected].
I try to add the storage admin to both of them. But it doesn't work. Looking for your help!