ApplicationsLabelbox (Deprecated)

THIS VERSION OF THE APP IS NO LONGER USED AND THIS DOCUMENT IS ONLY KEPT FOR HISTORICAL PURPOSES.

Access Instructions

Labelbox User Interface

Labelbox Admin Interface

See LastPass:

Database CLI

  1. SSH to Labelbox server

ACE Infra Team user accounts are on Labelbox server so you can login as yourself.

# example
ssh bushnet1@labelbox-prod.ecd-ai.com
  1. Identify MySQL container
kubectl get pods -n replicated-<TAB AUTOFILL> | grep mysql
 
For example:
$ kubectl get pods -n replicated-baefce429a9e43bf449a974bd9fb8d4a | grep mysql
labelbox-mysql-master-0                          1/1     Running     0          157d
labelbox-mysql-slave-0                           1/1     Running     0          157d
  1. Connect to MySQL container
kubectl exec -it labelbox-mysql-master-0 -n replicated-<TAB AUTOFILL> bash
 
For example:
$ kubectl exec -it labelbox-mysql-master-0 -n replicated-baefce429a9e43bf449a974bd9fb8d4a bash
I have no name!@labelbox-mysql-master-0:/$
  1. Login to MySQL

The MySQL root password can be found in the Replicated settings page, under Root password for MySQL.

I have no name!@labelbox-mysql-master-0:/$ mysql -u root -p
Enter password: < Get this from the Labelbox Admin UI >
...
mysql>
  1. Set Database to default
mysql> use default@default;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
 
Database changed
mysql>

LDAP Configuration

Currently the configuration cannot be enabled for encryption due to certificate errors. This needs to be addressed before we can move to encrypted channel for LDAP.

Temp Configuration

This config is valid as of 3/1/2021

We are using the temp ECD-AI LDAP service in AWS.

  • Hostname: ldap-temp1.ecd-ai.com
  • Search User: cn=search,CN=Users,DC=ldap-temp,DC=ecd-ai,DC=com
  • Base DN: DC=ecd-ai,DC=com
  • User Search DN: DC=ldap-temp
  • Username: samaccountname
  • Email Field: mail
  • Organization Name: ECD-AI
  • LDAP Sync Filter: (objectClass=*)

Regular Configuration

Pending Labelbox bugfix

  • Hostname: ldap.ecd-ai.com this points to a target group in AWS which points to 10.112.254.61
  • Port: 3268 Port for encrypted will be changed to 3269 once encryption is setup
  • Encryption Type: Plain This will be changed to StartTLS to enable encryption
  • Search User: CN=gcr01prd,OU=UsersGeneric,OU=IT,OU=SanFrancisco,OU=AdminUnits,DC=nala,DC=roche,DC=com
  • Search password: See Secrets Manager —> cloud_admin ldap_admin key
  • Base DN: DC=com
  • User Search DN: DC=roche
  • Restricted User Group: GLOFCT_Labelbox-gCORE
  • Username field: samaccountname
  • Email field: mail
  • Organization name: ECD-AI
  • LDAP Sync Filter: (objectClass=*)

Disaster Recovery

See dedicated section in our Disaster Recovery document.

Log Diving

Summary

Troubleshooting Labelbox (and most apps) requires access to logs. As of this writing (September 2020) we don’t have a proper production log management solution (coming soon) so we’ll have to SSH to the server and use the command line.

Labelbox is a Kubernetes application so familiarity with kubectl is required here. I’m not going to teach you kubectl, but I’ll navigate you to where you can use this utility to do a little spelunking.

First you’ll need to SSH to the Labelbox server

ssh username@labelbox-prod.ecd-ai.com

From there you can use kubectl to interact with the cluster. Log tailing with kubectl can be done:

  • per container
  • per pod
  • per deployment

NOTE: There are other tools that provide a greater scope (all logs), but that’s beyond the scope of this section… at least for now.

The primary application deployment for Labelbox is called replicated

kubectl get deployment
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
replicated                              1/1     1            1           152d
replicated-premkit                      1/1     1            1           56d
replicated-sidecar-controller-default   1/1     1            1           152d
replicated-statsd                       1/1     1            1           56d
retraced-api                            1/1     1            1           152d
retraced-cron                           1/1     1            1           152d
retraced-nsqd                           1/1     1            1           152d
retraced-postgres                       1/1     1            1           152d
retraced-processor                      1/1     1            1           152d

To view the logs for this deployment just run the command below. Note that you can remove the -f if you just want to see the logs and then exit. Adding -f creates a “live tail” so you can watch new logs as they come in.

kubectl logs deployment/replicated --all-containers=true -f

LDAP Troubleshooting

Requires tailing two logs simultaneously

# labelbox api log
kubectl logs deployment/api -n replicated-baefce429a9e43bf449a974bd9fb8d4a -f

# replicated log
kubectl logs replicated-6956bbffb8-sr9ww replicated -f

Certificate Management

Application Certificate (*.labelbox.ecd-ai.com)

Kubernetes Cluster Certificates

Kubernetes cluster certificates are configured for annual expiration, as we surprisingly found out on April 26, 2021.

Check Expiration

Expiration can be checked by SSH’ing to the Labelbox server and running the following command

bushnet1@labelbox.ecd-ai.com 19:39:47   /etc/kubernetes
$ sudo kubeadm alpha certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Apr 27, 2022 01:00 UTC   364d            no
apiserver                  Apr 27, 2022 01:00 UTC   364d            no
apiserver-etcd-client      Apr 27, 2022 01:00 UTC   364d            no
apiserver-kubelet-client   Apr 27, 2022 01:00 UTC   364d            no
controller-manager.conf    Apr 27, 2022 01:00 UTC   364d            no
etcd-healthcheck-client    Apr 27, 2022 01:00 UTC   364d            no
etcd-peer                  Apr 27, 2022 01:00 UTC   364d            no
etcd-server                Apr 27, 2022 01:00 UTC   364d            no
front-proxy-client         Apr 27, 2022 01:00 UTC   364d            no
scheduler.conf             Apr 27, 2022 01:00 UTC   364d            no

Renew Certificates

Certificate renewal is quite straightforward. The key is that it be done BEFORE the certificates expire and the cluster goes down. At time of writing, there is no process, monitoring or automation in place to address this, but this is coming soon.

The following command renews the cluster certificates.

sudo kubeadm alpha certs renew all

NOTE: After updating the certificates, I expected everything to work, but the same errors persisted in logs

Apr 27 16:31:35 labelbox kubelet[10187]: E0427 16:31:35.595276   10187 bootstrap.go:263] Part of the existing bootstrap client certificate is expired: 2021-04-24 15:27:29 +0000 UTC

I think this is because the renewal process DOES update the client configuration (/etc/kubernetes/admin.conf (KUBECONFIG)), but DOES NOT update the kubelet configuration (/etc/kubernetes/kubelet.conf).

After stumbling around in the dark for a bit I was able to get the cluster running again like so:

# write new kubelet file
sudo kubeadm alpha kubeconfig user --client-name system:node:labelbox | sudo tee /etc/kubernetes/kubelet.conf
 
# restart kubelet
sudo service kubelet restart

The kubelet restarted, the errors disappeared and within a minute, the admin UI was restored, enabling me to restart the application.

Support / Howto

How to generate a Support Bundle

Occasionally, Labelbox Support will ask us to generate and send a “Support Bundle” so they can troubleshoot an application issue. There are two methods for achieving this, but note that there is an issue with the GUI approach so recommend CLI until further notice.

GUI

  • Login to the Labelbox Admin Interface (link)
  • Click Support
  • Click Download Support Bundle

Bundles are too large to send via Slack so you’ll need to email them. Just ask the requestor for an email address to which to send the bundle.

NOTE: If you get the following error:

Error while creating support bundle: Error: Unsuccessful HTTP response

You’ll have to use the CLI method.

CLI

  • SSH to labelbox-prod.ecd-ai.com with your UNIX ID.
  • Run replicatedctl support-bundle command to create support bundle.
  • SCP the resulting file back to your workstation.
  • Email it to requesting Labelbox engineer(s).

NOTES

  • replicatedctl make take 10 minutes or so to run.
  • KUBECONFIG is automatically set at login so manually setting should not be required.

Troubleshooting

Error: “Successful” User Authentication Returns User to Login Page

  • User logs in with correct credentials
  • User returned to login page as if the login failed
  • No user facing error messages
  • Logs show Could not find org membership for userId error

Perusing the logs shows the following:

kubectl logs deployment/api -n replicated-<TAB> -f
...
...
{"exception":{"message":"Could not find org membership for userId: ckp2zykm90igy106zbgxd2w2r, organizationId: ck9g2vm820ffs0996ouensbsk","stack":"Error: Could not find org membership for userId: ckp2zykm90igy106zbgxd2w2r, organizationId: ck9g2vm820ffs0996ouensbsk\n    at Object.exports.getOrgMembership (/app/apps/api/dist/resolvers/query/permissions/organization-permissions.js:53:15)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:94:5)\n    at async exports.organizationRole (/app/apps/api/dist/resolvers/query/permissions/organization-permissions.js:96:27)"},"level":"error","message":"[captureExceptionSentry] called with error but sentry is disabled","dd":{"trace_id":"2100051288466001889","span_id":"2214705110631839571","service":"api","version":"1.0.0"},"timestamp":"2021-05-27T19:24:10.497Z"}
...
...

Solution is to delete the user from the database. The user can then login fresh and the provisioning process will run again, hopefully correctly this time.

After connecting to the database per instructions above, you just:

  1. select the user where name=...
  2. delete the row where id=...

Below is an example that worked for me.


# select the user

mysql> select * from User where name="chenl195";
+---------------------------+-------------------------+-------------------------+---------+----------+----------+----------+---------+----------+---------+-------------+---------------------------+
| id                        | createdAt               | updatedAt               | role    | email    | nickname | name     | picture | fcmToken | deleted | auth0UserId | organizationId            |
+---------------------------+-------------------------+-------------------------+---------+----------+----------+----------+---------+----------+---------+-------------+---------------------------+
| ckp2zykm90igy106zbgxd2w2r | 2021-05-24 19:22:52.264 | 2021-05-24 19:22:52.264 | LABELER | chenl195 | NULL     | chenl195 | NULL    | NULL     |       0 | chenl195    | ck9g2vm820ffs0996ouensbsk |
+---------------------------+-------------------------+-------------------------+---------+----------+----------+----------+---------+----------+---------+-------------+---------------------------+
1 row in set (0.00 sec)

# delete the user

mysql> delete from User where id="ckp2zykm90igy106zbgxd2w2r";
Query OK, 1 row affected (0.00 sec)

”Your Account Has Been Suspended” Error

Logging in yields the following response <!— image: image (from original wiki uploads) —>

The corresponding application logs (accessible to administrators) will yield the following

WARN 2020-09-24T18:40:26+00:00 sessions/session.go:70 Failed to get apps: unexpected end of JSON input
WARN 2020-09-24T18:40:26+00:00 sessions/session.go:70 Failed to get apps: unexpected end of JSON input
ERRO 2020-09-24T18:40:26+00:00 daemon/renderer.go:348 unexpected status code 401
WARN 2020-09-24T18:40:26+00:00 sessions/session.go:70 Failed to get apps: unexpected end of JSON input
ERRO 2020-09-24T18:40:26+00:00 daemon/renderer.go:348 unexpected status code 401

In my experience, this occurs when you try to login after someone has removed you from the organization via the Labelbox UI.

Resolution requires connecting to the database and flipping the deleted field from 1 to 0. For example, this is the state of my user after being removed from the organization

mysql> select * from User;
+---------------------------+-------------------------+-------------------------+---------+------------------------------+----------+------------------------------+---------+----------+---------+------------------------------+
| id                        | createdAt               | updatedAt               | role    | email                        | nickname | name                         | picture | fcmToken | deleted | auth0UserId                  |
+---------------------------+-------------------------+-------------------------+---------+------------------------------+----------+------------------------------+---------+----------+---------+------------------------------+
| ck9g2vmgc0fg70996poe14szg | 2020-04-25 20:30:16.427 | 2020-09-23 16:35:40.281 | ADMIN   | bushnet1                     | NULL     | Todd Michael Bushnell        | NULL    | NULL     |       1 | bushnet1                     |

Here’s the fix

mysql> update User set deleted = false where email = 'bushnet1';
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0
 
mysql> select * from User;
+---------------------------+-------------------------+-------------------------+---------+------------------------------+----------+------------------------------+---------+----------+---------+------------------------------+
| id                        | createdAt               | updatedAt               | role    | email                        | nickname | name                         | picture | fcmToken | deleted | auth0UserId                  |
+---------------------------+-------------------------+-------------------------+---------+------------------------------+----------+------------------------------+---------+----------+---------+------------------------------+
| ck9g2vmgc0fg70996poe14szg | 2020-04-25 20:30:16.427 | 2020-09-23 16:35:40.281 | ADMIN   | bushnet1                     | NULL     | Todd Michael Bushnell        | NULL    | NULL     |       0 | bushnet1                     |
```1
Weights & Biases (WandB)Labelbox S3 Integration