Docker

Network Issues

Experienced a problem where docker containers are not allowed to access stuff on the 192.168.x.x subnet.

If this happens, you can empty iptables and restart docker:

  sudo service docker stop
  
  sudo iptables -P INPUT ACCEPT
  sudo iptables -P FORWARD ACCEPT
  sudo iptables -P OUTPUT ACCEPT
  sudo iptables -t nat -F
  sudo iptables -t mangle -F
  sudo iptables -F
  sudo iptables -X
  
  sudo service docker start

GCloud Auth

	  Building imagename
	  ERROR: (gcloud.auth.docker-helper) There was a problem refreshing your current auth tokens: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})
	  Please run:
	  
	    $ gcloud auth login
	  
	  to obtain new credentials.
	  
	  If you have already logged in with a different account, run:
	  
	    $ gcloud config set account ACCOUNT
	  
	  to select an already authenticated account to use.
	  ^CERROR: Aborting.

Annoying hook that takes over when you run docker build.

Solution is: https://stackoverflow.com/questions/59148598/how-to-get-gcloud-auth-docker-helper-out-of-the-habit-to-hook-into-non-related-d

edit .docker/config.json and remove credshelpers hooks:

Before

{
  	  "auths": {},
  	  "credHelpers": {
  			  "asia.gcr.io": "gcr",
  			  "eu.gcr.io": "gcr",
  			  "gcr.io": "gcr",
  			  "marketplace.gcr.io": "gcloud",
  			  "staging-k8s.gcr.io": "gcr",
  			  "us.gcr.io": "gcr"
  	  }
}

After

  {
		  "auths": {},
		  "credHelpers": {
		  }
  }

Nvidia

Instructions for installing Nvidia runtime in docker can be found here and then tested by running:

 docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Which should output the normal nvidia-smi info from the host system.

Docker Compose GPU Options

Add a deploy clause to the service that should have a GPU attached to it:

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]
 

You can pass device ID as recorded in nvidia-smi or you can pass a device count and have docker allocate available devices:

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]