Get users where something where.not something

I'm trying to get group's users with specific ids that are not admin.

For the moment I have:

group.users
     .joins(:roles)
     .where(id: user_ids)
     .where.not(roles: { role_type: Role::Type::ADMIN })
     .pluck(:id)

In my log I have:

SQL to load the group:

  (0.3ms)  SELECT "users"."id" FROM "users" INNER JOIN "groups_users" 
  ON "users"."id" = "groups_users"."user_id" 
  WHERE "groups_users"."group_id" = $1  [["group_id", 137375]]

SQL for the query above:

  (0.6ms)  SELECT "users"."id" FROM "users" INNER JOIN "roles" 
  ON "roles"."user_id" = "users"."id" AND "roles"."is_destroyed" = $1 
  INNER JOIN "groups_users" ON "users"."id" = "groups_users"."user_id"
  WHERE "groups_users"."group_id" = $2 AND "users"."id" IN (82884, 82885)
  AND "roles"."role_type" != $3  [["is_destroyed", "f"],
  ["group_id", 137375], ["role_type", 1]]

The problem is I always get all the users of the group with matching user_ids. The where.not is not effective.

I had to do something like

users_in_group  = group.users.where(id: user_ids).pluck(:id) 

users_in_group -= group.users.joins(:roles).where
                 (roles: { role_type: Role::Type::ADMIN}).pluck(:id)

I don't understand why.