Some problems with user sync'ing and org organic group creation

[https://id.mayfirst.org/jamie]

Not sure if they are related, but i suspect they are.

For reference, see: #379 and #434.

[https://id.mayfirst.org/jamie]

And #438.

[https://id.mayfirst.org/jamie]

Sha discovered the following groups have registered but do not show up on the organize site (see #434):

• Amnesty International, Group 78, Detroit
• Detroit Greens
• League of Revolutionaries for a New America
• JASECon
• Skills for The New Millennium Topur (i think it's supposed to be "Tour")
• Small Schools Workshop
• WOEIP

The good news is that, on close examination, I've found that:

• All groups are successfully registered on community.
• Have user accounts successfully created on the community site
• Have user accounts successfully created on the organize site

The only missing part is the creation of the organization organic group.

I've also found a SQL statement that lists the problem cases - and fortunately the list is limited to the ones identified by Sha:

mysql> SELECT uid,name, FROM_UNIXTIME(users.created) FROM users LEFT JOIN og_uid USING(uid) WHERE data LIKE '%profile_is_primary_registrant%' AND og_uid.nid IS NULL;
+-----+----------------------+------------------------------+
| uid | name                 | FROM_UNIXTIME(users.created) |
+-----+----------------------+------------------------------+
| 193 | freddetroit          | 2010-01-31 13:58:08          | 
| 216 | gkgrunow             | 2010-02-02 15:22:07          | 
| 264 | margaretgordon       | 2010-02-05 14:31:55          | 
| 268 | jerome               | 2010-02-05 15:33:02          | 
| 367 | skillstour           | 2010-02-12 16:28:17          | 
| 368 | tripledawn           | 2010-02-12 16:28:17          | 
| 412 | smallschoolsworkshop | 2010-02-15 09:11:13          | 
+-----+----------------------+------------------------------+
7 rows in set (0.02 sec)

mysql>

I'm still working on why the organization groups were not created.

jamie

[https://id.mayfirst.org/jamie]

It looks like the problem is not caused by accounts that already exist on the organize site - all of these accounts were created post-launch. However, I'm still not sure why they failed.

jamie

[https://id.mayfirst.org/jamie]

I'm going through these one at a time to try to understand the pattern.

JASEcon was created properly ( http://organize.ussf2010.org/org/jasecon).

League of Revolutionaries... Walda registered and is the registering contact. Her account already existed on the organize site, which I think caused the breakdown in this case (leaving Jermoe with an account, but not a member of any group, nor coded as primary registrant).
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9286

Skills for the New Millennium - skillstour was the registering user and triplesdawn was an additional user, both with emails. Both accounts were created on organize, however, neither user is coded on organize as being the primary_registrant (this record produced two results in the table below).
 https://community.ussf2010.org/en/civicrm/contact/view?action=view&reset=1&cid=9509

Detroit Greens - two registrants, two identical names, only one with email (freddetroit), user on the organize site with profile_is_primary_registrant as false.
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9119

Amnesty International - two registrants, two names, only one with email (gkgrunow), shows up as profile_is_primary_registrant false on organize
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9178

Small Schools - same as above (user is smallschoolsworkshop)
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9589

WOEIP - two registrants, two different names, only one with an email (margaretgordon). One account on organize, not coded as primary_registrant.
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9276

Seems like the main problem is that, for some reason, some contacts are not coming over to organize coded as the primary registrant.

jamie

[https://id.mayfirst.org/jamie]

See #445.

[https://id.mayfirst.org/jamie]

I just manually fixed the League of Revolutionaries (which takes jerome out of the mix).

I'm still seeing this problem happen with select groups though - here are the latest results of the query:

mysql> SELECT uid,name, FROM_UNIXTIME(users.created) FROM users LEFT JOIN og_uid USING(uid) WHERE data LIKE '%profile_is_primary_registrant%' AND og_uid.nid IS NULL;
+-----+----------------------+------------------------------+
| uid | name                 | FROM_UNIXTIME(users.created) |
+-----+----------------------+------------------------------+
| 193 | freddetroit          | 2010-01-31 13:58:08          | 
| 216 | gkgrunow             | 2010-02-02 15:22:07          | 
| 264 | margaretgordon       | 2010-02-05 14:31:55          | 
| 367 | skillstour           | 2010-02-12 16:28:17          | 
| 368 | tripledawn           | 2010-02-12 16:28:17          | 
| 412 | smallschoolsworkshop | 2010-02-15 09:11:13          | 
| 459 | montyneill           | 2010-02-18 12:45:49          | 
| 460 | mchaneyt             | 2010-02-18 12:45:49          | 
| 473 | harlemtenants        | 2010-02-19 12:34:50          | 
| 488 | abdelhakeem2003      | 2010-02-20 11:46:12          | 
| 511 | breastsnotbombsnyc   | 2010-02-22 16:58:21          | 
+-----+----------------------+------------------------------+
11 rows in set (0.04 sec)

mysql>

[sgroganb@…]

wow, really great work on this jamie. thanks so much for slugging through this. just wanted to give thanks.

[https://id.mayfirst.org/jamie]

I just did some very substantial re-factoring of the code that is called on the organize site after a user or organization registers. I think the code logic is a little simpler, is now heavily commented, and includes a lot of logging so if the problem persists we should have an easier time figuring out why.

We still need to fix the previously broken accounts, which I'm working on now.

jamie

[http://josephlacey.myopenid.com/]

Most of these users have used their email (and maybe name) as one or more of the additional participants. This results in one contact record with multiple registrations because of the deduping process. This is going to be a problem when we let them back in to edit a record. We will need to separate these registrations into multiple contact records.

Amnesty International, Group 78, Detroit
Mary Geraldine Grunow (uid 216, gkgrunow)
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9176

WOEIP
margaret gordon (uid 264, margaret gordon)
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=8254

Skills for The New Millennium Topur
Megan Wilson (uid 367, skillstour)
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9507
Lorca Blanco, a third additional participant (uid 368, tripledawn)

Small Schools Workshop
Susan Klonsky (uid 412, smallschoolsworkshop)
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9587

Harlem Tenants Organization
Lisa Jones (uid 473, harlem tenants)
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9758

puplic serviese union
lyla sobaih (uid 488, abdelhakeem2003)
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9790

Breasts Not Bombs!
Ingrid Romero (uid 511, breastsnotbombsnyc)
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9845

Detroit Greens
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9119
The registering attendee took the contact name of the default additional participant, so the same but the names were reversed.
Additional 2 Detroit Greens (uid 193, freddetroit)

I'm thinking the failure happens because civicrm first dedupes the contacts, so the 'is_primary_registrant' designation no longer exists. The user is now associated with an additional participant record (but also a primary registrant user), and the code fails because it can't find a primary registrant. If this is true, the solution could be to interrupt the deduping process by changing the rules so it takes more than an email address match. But we should look more closely and try to duplicate this to make sure this is correct, and explore all the solutions here.

The other two users don't fit the model above.

North Dakota Study Group
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9723
Registering Attendee is fine - Ken Jones
But these two additional participants didn't get associated with the org og at all. I didn't see any reason why this would have happened.
Monty Neill (uid 459, montyneill)
Greta McHaney-Trice (uid 460, mchaneyt)

[sgroganb@…]

great work, jamie and joseph.

i'm not sure if this is related, but there are a number of organizations that are now listed twice, or in some cases multiple times, on the organize site.

These are the ones it appears to be happening with:
* Bike It
* Building Powerful Networks
* Friends of the MST
* La Raza Centro Legal (this org is listed like 5 or 6 times)
* Philly Stands Up
* SOLIDARITY
* The Birth Attendants: Prison Doula Project

[https://id.mayfirst.org/jamie]

Ug. This is getting really bad.

Joseph: I just confirmed your theory in my test environment. When your additional participant is a duplicate of the registering contact, then the organization organic group fails to be created on the organize site. Nice sleuthing!!

I think that rather than changing the de-duping logic - is it possible for us to insert some kind of custom validation on the submission of the additional registrant and issue an error if we see that their using the same email address as the original registrant? I think that would have the added benefit of letting them now they can truly register three individuals (if they are repeating a registrant, they only get two).

Sha - the dupes are getting bad. I'm trying to find the source of the problem. I though it was hitting the submit button more than once, but that doesn't seem to be causing dupes to be created. I'm still working on it now.

jamie

[http://josephlacey.myopenid.com/]

Email validation:
This logic is already built in, that you can't register two participants with the same email address. There was some reason Mark and I discussed that we thought it was better to allow this. I'd need to look again to remember why. If we did decide to introduce this, that would soften the issue of clicking the button multiple times (though I think a js hide or lock of the button once it's clicked is worth adding), it would prevent additional participants deduping, and it would help with an issue that was brought up earlier, allowing someone to register at some future time. An issue with this is if someone registers multiple orgs or under an org and as an individual. See:

Kali Akuno
Two registered organizations
US Human Rights network
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9640
Malcolm X Grassroots Movement
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9645

Catherine Wilkerson
Registered as an individual
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9123
and then registered an organization:
 https://community.ussf2010.org/en/civicrm/contact/view?reset=1&cid=9874

Multiple orgs:
I have a list of multiple registrations that is partially related to this issue. It includes BikeIt (sent an email), Building Powerful Networks, Philly Stand Up (sent an email) and The Birth Attendants. I'll take a look at Friends of the MST, La Raza Centro Legal and SOLIDARITY. These probably need to be dealt with on a another ticket. I'll create that and mention it here.

[http://josephlacey.myopenid.com/]

See related ticket #476.

[https://id.mayfirst.org/jamie]

Thanks Joseph for the detailed follow up on #476 and your explanation on this ticket.

Based on what we know now - do you think we'll solve these problems by:

• Reverse the decision to allow two participant with the same email to register (assuming we determine that the original reason for doing that is no longer valid given these problems we are having) AND
• Used some kind of javascript to prevent the final submit button from being pressed twice

That's my understanding from what you've researched - and I think it makes a lot of sense. Even if there are other issues lurking, getting these two fixes in place seems prudent.

Unfortunately, I'm travelling this weekend (i'm actually writing this from a virgin america flight - in fligh wifi!) so my timing is bad. However, I know libkuman is ready to get down again so I'm hoping we can rope him in.

The problem is pretty severe - I think people are getting confused by having the multiple records (and the data clean up job is mounting), so I'd say this probably the number one ICT priority at the moment.

jamie

[http://josephlacey.myopenid.com/]

These two solutions, as conceived, seem to address all the issues related to multiple registrations, if these diagnoses are correct. I'd like libkuman to weigh in though.

Adding the js is relatively simple. Adding the email validation will be more work, the ripples we'll need to work through (like form workflow, translation of error messages, etc). But I agree this should probably be bumped to the top of the priority list.

[http://josephlacey.myopenid.com/]

I've added the javascript hide once a user clicks the register button. It hides both the go back and register buttons and displays a wait message. See ticket #476 for a few more details on that.

I also tested the email validation on my local instance, and it works at what its supposed to do but too well. If you register an additional participant with the same email address, it asks for a unique email address. This may need to be tested with a few more limit cases to be sure it will work and not conflict with anything else.

But there is a small bug. If you add both First Name and Last Name profile fields, it is supposed to allow a non-required Email field. It is configured as non-required and the first additional participant can register without an email address, but all subsequent additional participants have to enter an email. I assume we can interrupt this validation in the code, but it will take a little more work.

Another bug, but maybe intentionally. You can still register with the same email address on the first page of the registration form. As noted in comment 12, there are cases where this may be desired, but that would open the door for some possible mistaken duplicate registrations.

[https://id.mayfirst.org/jamie]

Hi Joseph,

Is the reason subsequent registrations need to enter an email address because the system detects two empty email addresses and thus the validation error of not allowing two identical email addresses kicks in? If so - that makes sense and sounds like we could interrupt it. However, not sure if there's a different reason for the failure.

As for registering with an already used email address when you are the primary reigstrar for an organization... I think the drupal sync code will prevent duplicate drupal users from being created and should generally do that right thing (provided the sync code properly passes along the flag that this is the primary registrant for the org). However, I'm not sure about the Civi code - will that properly use the existing contact record and add the right additional relationships?

If we're at all in doubt - or even as a first phase - I think it would be better to go backwards and remove this functionality in the short term until we can ensure we have all the basis covered.

jamie

[http://josephlacey.myopenid.com/]

Mark and I went through this and we think it's fine to go forward with it, but we'd like another opinion. I've fixed the null email address uniqueness problem. Jamie, you were right it was just doing a simple comparison without actually checking whether the email address had a value other than null. I've added that additional validation hurdle, and it works fine now. Here are the scenarios Mark and I walked though.

Email address isn't in the CRM at all, so the user is allowed to submit the registration form and a new contact record is created.

Contact with an email address exists in the CRM but isn't registered, so the user is allowed to submit the registration form and it is attached to the existing record.

Additional participant is attempted with a registered email address, but a validation error is thrown to make email address unique. This error deals with most of the registrations on this ticket, which is case 1 on ticket #476.

Future attempt to register with a registered email address, but a validation error is thrown, and user is redirected to the default CiviCRM event page. This redirect needs to be changed. (Any thoughts about where to redirect them? A message will need to be displayed explaining the problem. Returning the user to the registration form has a small problem. If you're logged in, it checks your registration status automatically and will redirect you, so if you're redirected back to the form, you'll get caught in a loop. I see this only being a problem for admins, but a problem nonetheless.) This error deals with multiple registrations at different times, which fall under case 3 of ticket #476.

There is one idiosyncrasy. If someone registers and selects to pay later (which includes scholarship requests) and they return to register either intentionally under another org or mistakenly, they will be allowed to because they technically aren't registered yet since they haven't paid. We can find this code and make the restrictions tighter to include pay later registrants, but Mark and I thought this shouldn't prevent the current solution from implementation. (In comment 16, I misdiagnosed this behavior.)

After the redirect is decided, the steps to implement would be to pull the new code live, and flip the unique email address checkbox under the event configuration.

[https://id.mayfirst.org/jamie]

Thanks Mark and Joseph for working through this!

On the redirect question - I'm not sure I understand the question. I think the scenario is that a person is on the initial organization registration page and they have just filled in the first page of info. The primary contact (which is on the first page) has an email address of a contact that is already registered.

Why do they need to be redirected anywhere? Is it possible for them to simply get the same page they just filled in, but with a validation error at the top?

Also - when you say that currently you are redirecting them to the default civicrm Event page - do you mean the first page you go to when registering as an organization?

I also agree on moving forward without the tighter restrictions on pay later (but that these tighter restrictions should be implemented as soon as we can.

I'm ready to test and push stuff live when you feel that you are ready. My schedule is a little weird given my travel - I should be "on" tonight very late and then again tomorrow afternoon.

jamie

[alfredo@…]

I think we should ask OC/NPC to get volunteers for a "tester team" and those folks have to do tests for a 24 hour period before we push anything other major functionality into production. I think what has "failed" us isn't our coding work; all code is buggy at first. Hell, what we've produced is miraculousy functional given the dearth of testing support we've gotten. It's our testing that has failed us and that's not something we as a WG can handle alone.

Our current approach is to do the work, open ad hoc testing, activate for production use and then react to glitches people find. Sometimes it's perfectly cool; sometimes we can get hit with whoppers. When the latter, the world doesn't end but it's disruptive of *our* work and our schedule of work and that's what I want to avoid if possible.

Mallory and I, in Detroit, encountered enormous respect for this WG's work and tremendous willingness to accept our explanations of problems and flexibility about deadlines, etc. You guys have earned us that respect; let's use it productively by asking some folks to volunteer to work with us as testers.

What do youse think?

Alfredo

[https://id.mayfirst.org/jamie]

I think the tester approach is a good one. It does mean, however, that we'll need to complete $394 first (fully functional sandbox/testing sites).

jamie

[http://josephlacey.myopenid.com/]

I'm also not sure why the user is redirected in this case. But this "already registered" validation is treated differently than the normal form validation. We can certainly just take out the redirect, which I tried last night. I set the page to display the error message but somehow my local instance got hosed. I think it's a php problem, but that means I can't test this change, and I 'm a little wary of pushing this without testing it even a single time. I'll try to find someone on IRC today for the fix; it's fairly straight forward.

This is the default event page that CiviCRM provides. We're not using it because of the complexity of language and registration types in our process.

 https://community.ussf2010.org/en/civicrm/event/info?id=4&reset=1

The tester scenario would be great. I think we'll need it after we finalize the solution for this problem and move onto the next priority, ticket #447. We'd want to make sure we don't run into any similar difficulties when adding and editing existing registrations.

[https://id.mayfirst.org/jamie]

Hi Joseph - I'm on IRC now and can do local testing. I (should) have a setup testing environment.

jamie

[http://josephlacey.myopenid.com/]

I've sorted out the remaining problems with the unique email implementation. As discussed with Jamie, I removed the preProcess registration check, I changed the redirect to return the user to a refreshed version of the form, and I set the status to display a message stating the email address has already been registered.

You are already registered for this event. If you want to change your registration, please login to your account here http://organize.ussf2010.org/user, or if you feel that you've gotten this message in error, please contact online-support@ussf2010.org.

I've edited the additional participant message as well; it states the last clause of the last sentence. The exact wording should be reviewed but in this case I think we should definitely include this direct contact information. Once there's consensus on this wording, it needs to be submitted for translation.

Without the preProcess check, if you are logged in and enter an email address that isn't associated with your account and is already in the CRM, you will get a UFMatch conflict error. A very limit case that I don't think needs to be addressed.

Someone should test these changes locally before pulling live.

Once this fix is in place, we'll need to go through and start unraveling the data problems.

[https://id.mayfirst.org/jamie]

Nice Joseph!!!

Thanks for pushing this through, it's really important.

I tested locally with duplicate email address and non-duplicate email address for:

• individual registrant
• primary registrant of organization
• additional registrant of organization

I also tested with a unique email address as the primary registrant for an organization. And then entered the same address as an additional registrant.

And it worked in all scenarios!!

I made one small change to the error message. I removed the link to organize and the direction to login there if you want to change your registration because it's not clear how or what aspects of the registration could be changed.

I also noted one usability problem: if you add a duplicate on an additional registrant (which is the most common scenario) - the page is refreshed with the error message and all fields filled out as you original filled them out so you can easily see your error and make a correction.

However, if you add a duplicate email on the initial page of an individual or organizational registration, then the page refreshes with the error message, but does not retain any of your original information in the form.

I don't think this usability problem is a major concern since it will be triggered infrequently and is only mildly annoying.

I've pulled these changes onto the live site.

I can help with the data fixing this weekend. Great work Joseph!!!

jamie

[https://id.mayfirst.org/jamie]

I just went through the last of the registered organizations that didn't get related organizations on the organize site. I think these have all been taken care of (and the code should now prevent this from happening in the future).

We still have a hideous amount of data clean up on #476, but this ticket can now be closed.

jamie