Event Grid-triggered Function Keeps Triggering After Function Successfully Runs?

Question

I have an Azure Function triggered by Event Grid Events. Event Grid Events are created only when a blob is uploaded to a Storage Account.

This is now deployed and working well although for some reason, the Function keeps getting triggered by the same event even though it successfully processed?

Example:

• Yesterday, 8 successful tests; all good:

• Today I review the logs, and the function has continued to execute!

• Error: "Blob does not exist"

• I deleted the blob after last test yesterday. Why is the Event Grid still firing?

Code snippet:

def main(event: func.EventGridEvent):

        result = json.dumps({
            'id' : event.id,
            'data' : event.get_json(),
            'topic' : event.topic,
            'subject' : event.subject,
            'event_type' : event.event_type
        })

        logging.info('EventGrid trigger processing an event: %s', result)

        credential = DefaultAzureCredential()

        download_start_time = datetime.datetime.now()
        logging.info(f'######## Starting downloading blob from storage at ' + str(download_start_time) + ' ########')

        # =============================
        # Download blob from storage container:
        # =============================

        blob_client = BlobClient.from_blob_url(event.get_json()["url"], credential)
        blob_data = blob_client.download_blob().readall()
        blob_byte_stream = io.BytesIO(blob_data)

EDIT 1: This is still happening, this time a bit different.

• Now, the EventGrid keeps firing after SUCCESSFULLY passing messages and the Function running

How do I debug this?

Answer 1

I finally figured out what the issue was...using the BlobClient.from_blob_url method worked fine when testing blob uploads using Azure Storage Explorer. But when using Azure Data Factory, a different API is used and the data.url property in the EventGrid message is not the actual blob url (contained dfs instead of blob).

Oddly enough, soon after I brought this issue up to support team, a new blobUrl property was added to EventGrid data object.

In my code, I simply changed "url" to "blobUrl" and the method succeeded. (I also improved the error handling of the Python code to accommodate such errors in the future.)

Documented EventGrid message (as of 12/10/2020):

• No blobUrl property

[{
  "topic": "/subscriptions/{subscription-id}/resourceGroups/Storage/providers/Microsoft.Storage/storageAccounts/my-storage-account",
  "subject": "/blobServices/default/containers/my-file-system/blobs/new-file.txt",
  "eventType": "Microsoft.Storage.BlobCreated",
  "eventTime": "2017-06-26T18:41:00.9584103Z",
  "id": "831e1650-001e-001b-66ab-eeb76e069631",
  "data": {
    "api": "CreateFile",
    "clientRequestId": "6d79dbfb-0e37-4fc4-981f-442c9ca65760",
    "requestId": "831e1650-001e-001b-66ab-eeb76e000000",
    "eTag": "\"0x8D4BCC2E4835CD0\"",
    "contentType": "text/plain",
    "contentLength": 0,
    "contentOffset": 0,
    "blobType": "BlockBlob",
    "url": "https://my-storage-account.dfs.core.windows.net/my-file-system/new-file.txt",
    "sequencer": "00000000000004420000000000028963",  
    "storageDiagnostics": {
    "batchId": "b68529f3-68cd-4744-baa4-3c0498ec19f0"
    }
  },
  "dataVersion": "2",
  "metadataVersion": "1"
}]

Actual EventGrid message coming through now:

• Notice blobUrl has been added to the schema

{
    "id": "long-string",
    "data": {
        "api": "CreateFile",
        "requestId": "long-string",
        "eTag": "0x8D89B4FF7150079",
        "contentType": "application/octet-stream",
        "contentLength": 0,
        "contentOffset": 0,
        "blobType": "BlockBlob",
        "blobProperties": [{
            "acl": [{
                "access": "u::rw,u:long-string:rwx,u:long-string:rwx,g::rx,g:long-string:rx,m::rw,o::",
                "permission": "0660",
                "owner": "long-string",
                "group": "$superuser"
            }]
        }],
        "blobUrl": "https://myfunction.blob.core.windows.net/container/20201208/730420201208080239.csv",
        "url": "https://myfunction.dfs.core.windows.net/container/20201208/730420201208080239.csv",
        "sequencer": "0000000000000000000000000000692c00000000000e1a99",
        "identity": "long-string",
        "storageDiagnostics": {
            "batchId": "long-string"
        }
    },
    "topic": "/subscriptions/long-string/resourceGroups/myResourceGroup/providers/Microsoft.Storage/storageAccounts/storageAccount",
    "subject": "/blobServices/default/containers/container/blobs/20201208/730420201208080239.csv",
    "event_type": "Microsoft.Storage.BlobCreated"
}

Another caveat here...notice the Content-Length in the message above

• The CreateFile API is not the actual message which indicates the blob has been created.
• The FlushWithClose API is

There is a note in the docs about this. So I also had to setup an EventGrid Advanced Filter which only triggers when FlushWithClose events are generated (!)