The man claimed the move would interfere with the constitutional rights of his three children who lawfully reside here