AWS Codedeploy failed during Install: UnknownError "\xCC" from ASCII-8BIT to UTF-8

Question

I have tried to troubleshoot an UnknownError returned by the CodeDeploy Agent on an Amazon Linux EC2 instance. I push files from my local git repo on my Mac to a S3 bucket, and deploy from there to the EC2 instance.

I have edited the /etc/codedeploy-agent/conf/codedeployagent.yml and set :verbose: to true on a EC2 instance. I tried to deploy again and open codedeploy-agent.log at /var/log/aws/codedeploy-agent.

The last lines in log before failing shows that the agent installs ELB scripts provided by AWS.

2015-06-19 08:09:13 DEBUG [codedeploy-agent(7637)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandBuilder: Copying /opt/codedeploy-agent/deployment-root/c6b25857-8911-4f61-8b30-b39f0c34c395/d-6UZ0VFID9/deployment-archive/scripts/stop_server.sh to /var/www/html/scripts/stop_server.sh
    2015-06-19 08:09:13 DEBUG [codedeploy-agent(7637)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandBuilder: Copying /opt/codedeploy-agent/deployment-root/c6b25857-8911-4f61-8b30-b39f0c34c395/d-6UZ0VFID9/deployment-archive/scripts/register_with_elb.sh to /var/www/html/scripts/register_with_elb.sh
    2015-06-19 08:09:13 DEBUG [codedeploy-agent(7637)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PutHostCommandComplete: "Code Error" 
    2015-06-19 08:09:13 INFO  [codedeploy-agent(7637)]: [Aws::CodeDeployCommand::Client 200 0.084236 0 retries] put_host_command_complete(command_status:"Failed",diagnostics:{format:"JSON",payload:"{\"error_code\":5,\"script_name\":\"\",\"message\":\"\\\"\\\\xCC\\\" from ASCII-8BIT to UTF-8\",\"log\":\"\"}"},host_command_identifier:"WyJjb20uYW1hem9uLmFwb2xsby5kZXBsb3ljb250cm9sLmRvbWFpbi5Ib3N0Q29tbWFuZElkZW50aWZpZXIiLHsiZGVwbG95bWVudElkIjoiQ29kZURlcGxveS9ldS13ZXN0LTEvUHJvZC9hcm46YXdzOnNkczpldS13ZXN0LTE6NzMzNTE3MzY5MDAyOmRlcGxveW1lbnQvZC02VVowVkZJRDkiLCJob3N0SWQiOiJhcm46YXdzOmVjMjpldS13ZXN0LTE6NzMzNTE3MzY5MDAyOmluc3RhbmNlL2ktZWRjYTkzNDciLCJjb21tYW5kTmFtZSI6Ikluc3RhbGwiLCJjb21tYW5kUG9zaXRpb24iOjQsImNvbW1hbmRBdHRlbXB0IjoxfV0=")

The last files installed are not related. Since I also have tried to remove the scripts I mention above. The error seems to be upstream and Ruby related.

2015-06-19 08:09:13 ERROR [codedeploy-agent(7637)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: Encoding::UndefinedConversionError - "\xCC" from ASCII-8BIT to UTF-8 - /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/install_instruction.rb:165:in `encode'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/install_instruction.rb:165:in `to_json'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/install_instruction.rb:165:in `to_json'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:38:in `block in install'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:37:in `open'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:37:in `install'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:103:in `block in <class:CommandExecutor>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:51:in `execute_command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:123:in `process_command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:56:in `perform'
/opt/codedeploy-agent/lib/instance_agent/agent/base.rb:28:in `run'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:38:in `block in run'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:55:in `with_error_handling'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:37:in `run'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:70:in `block in run_with_error_handling'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:55:in `with_error_handling'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:69:in `run_with_error_handling'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:33:in `block in start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:22:in `loop'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:22:in `start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:198:in `block in spawn_child'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:196:in `fork'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:196:in `spawn_child'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:188:in `block in spawn_children'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:187:in `times'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:187:in `spawn_children'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:133:in `start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:36:in `block in start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:35:in `fork'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:35:in `start'
/opt/codedeploy-agent/bin/codedeploy-agent:37:in `block (2 levels) in <main>'
/opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/command_support.rb:130:in `call'
/opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/command_support.rb:130:in `execute'
/opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/app_support.rb:262:in `block in call_command'
/opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/app_support.rb:275:in `call'
/opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/app_support.rb:275:in `call_command'
/opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/app_support.rb:69:in `run'
/opt/codedeploy-agent/bin/codedeploy-agent:84:in `<main>'

I have tried $ grep --color -in -r '\xCC' /my/folder/ which returns something like

Binary file /my/folder/a.png matches

But also returned /my/folder/wp-content/plugins/wordpress-seo/admin/class-metabox.php:348:'Ì'

So I removed the plugin and tried to convert the filenames from ASCII to UTF-8 with convmv (it didn't convert anything). But without success.

I have also updated several gems according to a couple of StackOverflow threads.

How could I mitigate this error? I find the error message from Amazon CodeDeploy vaugue and hard to troubleshoot and I've run out of ideas.

Answer 1

The LANG environment variable must be defined when the deployment starts or else the CodeDeploy install event fails.

You can ensure that CodeDeploy has it by editing the CodeDeploy agent service configuration:

sudo vim /etc/init.d/codedeploy-agent

You can export the variable when the service first starts with this line: export LANG="en_US.UTF-8".

So the start block will look like this:

start() {
  echo -n $"Starting $prog:"
  export LANG="en_US.UTF-8"
  cd $AGENT_ROOT
  nohup $BIN start >/dev/null </dev/null 2>&1  # Try to start the server
  exit $?
}

Some additional notes

For me the problem was only occurring during an auto scale event. In other words I could manually launch the instance and then deploy just fine, but not right when a new instance was launched. This indicates a race condition in the sense that the environment is not ready as soon as the service starts. I tried changing the upstart timing by running:

update-rc.d -f codedeploy-agent remove
update-rc.d codedeploy-agent start 99 2 3 4 5 . stop 20 0 1 6 . 

but this did not work. Setting the variable in the start block guaranteed that it was available in time.

To ensure you have all required environment variables available to CodeDeploy as soon as it starts it's probably wise to follow Amazon's advice:

we recommend that you keep the agent service in a stopped state and use the launch scripts to start the agent service

Answer 2

Try updating codedeploy-agent. It worked for me. SSH into your instance and run this command: sudo /opt/codedeploy-agent/bin/install auto

Answer 3

Issue looks to be LANG env has not been set when Installing / Starting CodeDeploy.

I found that if I were to restart CodeDeploy via the CLI then the app would deploy without issue.

You can check the LANG env of the codedploy process by running

ps auxe |grep codede

We deploy CodeDeploy via puppet and have to set (environment => 'LANG=en_US.UTF-8') when running the install script.