Add and remove users from the diary